DevOps in the AI World

· 5 min read · Tim Thomas

I run a media company from my phone. The agents do the devops. Here is how a change goes from a voice note to live, tested properly, with a rollback faster than a kick drum.

Picture this. I am walking the dog in the park. Somewhere between the duck pond and the car I think: the reels we post to TikTok every day should be on the site too, on the reels page, so people can swipe them there. I say that into my phone. Not type. Say. The agent asks two questions, I answer them, and I put the phone back in my pocket.

Ten minutes later it is on the dev site. I look at it on the same phone, still in the park. I say "ship it". Five minutes after that it is live for everyone, in every country, and nobody watching noticed a thing.

That is the whole devops story, and I want to tell it properly because people assume "one person and an AI" means cowboy deploys onto a live site at midnight. It is the opposite. This is the most disciplined release process I have ever run, and I do not run it. The agents do.

The voice note

Everything starts as a conversation. I describe what I want in the way you would to a mate who happens to be a very good engineer: what it should do, what it should feel like, what is wrong with the current one. The agent reads the codebase, reads the brand guide (which is written as code, so it can), and builds it.

Then it tests it. Not "I ran it once and it looked fine" testing. Unit tests, the boring kind, hundreds of them across the API and the social machinery, and every change runs the lot. If the change is to something that posts to the outside world, there is a dry-run mode that prints what would have gone out. If it is to the site, it builds the whole site and type-checks it. Only when all of that is green does it tell me it is ready to look at.

Dev is a full copy of live

Here is the bit the enterprise crowd will recognise. There are two of everything: two sites, two APIs, two databases. Dev runs on a fresh copy of the live database, refreshed from last night's backup with one command, so what I am looking at is not test data. It is yesterday's real radio, real plays, real library. If a chart looks wrong on dev, it would look wrong on live, and I find out first.

I test on dev myself. On my phone, on the TV, on the laptop. I am the QA department and I am fussy. Nothing goes further until I say so, and "I say so" is a literal instruction, not a vibe. Dev cannot touch production. That is wired into the machine, not into my memory.

Blue and green

Production is two identical slots. Call them blue and green. One is live. The other is asleep.

When I say ship it, the new build goes into the sleeping slot. It builds there, starts there, gets health-checked there, all while the live slot keeps playing music to the world. Only when the new slot answers correctly does the front door swap over, gracefully, letting anyone mid-request finish. Then it checks again from the outside, through the real address, and if anything is off it swaps straight back before I have finished reading the message.

The old slot stays warm. So a rollback is not a rebuild. It is a switch flip.

We timed it. Zero dropped requests on the first real deploy. Rollback: 0.074 seconds. Seventy-four milliseconds. That is faster than the gap between two kick drums at 128 BPM.

When going back costs nothing, going forward costs nothing. I ship on a Tuesday afternoon. I ship on a Sunday night. I shipped three times yesterday and thought about it about as much as I think about saving a file.

The bit where it bit me

Two days after this went in, production went weird. Every page rendered. No button worked. Looked perfect, did nothing. The site was a beautiful corpse for an hour.

The cache. Every build names its scripts with a fingerprint of their contents. The edge was still serving the old pages, pointing at scripts the new slot did not have. So now the deploy purges the edge every time, and the runbook for that step opens with "this is not housekeeping, read this before you remove it". Written by the agent, for the next agent, because the next one has not seen the failure and I will have forgotten it.

Receipts

Every deploy and every rollback writes one line to a ledger: when, which slot, which commit, how many seconds to switch. "How often do you actually ship?" gets a number, not a memory. Since the ledger started: most days, sometimes three times.

Who is actually doing this

Not me. I want to be clear about that, because it is the interesting part.

I did not build the slots, the health checks, the graceful swap, the purge, the ledger, the status command or the runbook. I described the outcome: never break the live site, never lose a change, always be able to go back, and let me do it from my phone. The agent designed the mechanism and built it in an afternoon. It also runs it. When I say ship, it promotes, watches, reads the logs and tells me what happened. When something misbehaves at 2am, the same agent reads the same runbook and follows it.

There is even a rule for when several agents are working at once, because two of them once overwrote each other in the same folder. Now every piece of work gets its own checkout and merges when it is done. The rule is in the repo with the date and the incident that caused it. The agents read it before they start. I never think about it.

Why this matters if you are building the same way

The boring infrastructure is the whole game. An agent that can only touch dev, needs a literal "ship it" to go live, and can roll back in seventy milliseconds, is an agent you can let run while you sleep. The same agent with one environment and no undo is a liability with good taste.

Build the slots first. Then go and talk to your phone.

Earlier in this log: the admin area. Next: the day a cable stayed plugged in and the site went down anyway.

More from the build log