The agent that wiped our staging database

“Is staging down? None of the test logins work.” That was 3:50 on Tuesday afternoon, from one of our QA engineers, and my first guess was a bad deploy. Staging was up. It just had nothing in it. Every table existed and every table was empty, apart from the dozen or so rows our development seed file creates, which is why the login page loaded and nobody could log in.

It took us about fifteen minutes to work out what had happened, and most of the rest of the afternoon to put it back.

What happened

One of our backend engineers had spent the morning chasing a bug that only showed up in staging. To look at it, he’d opened a terminal tab, set DATABASE_URL to the staging database’s connection string, and run a Rails console against it. That’s a normal thing to do, it’s what the runbook says, and staging only has synthetic data in it, so it isn’t covered by the rule that agents never touch anything derived from production.

After lunch, in the same tab, he started Claude Code and asked it to fix a migration spec that had been failing on his machine since the previous week. Earlier in that session he’d been approving its bin/rails commands one at a time, got tired of it, and chose the option to stop asking for anything starting with bin/rails. That felt safe. It’s our own app’s command-line tool.

The agent looked at the failing spec, decided the local database’s schema had drifted, and concluded that the cleanest fix was to reload the schema from scratch. That’s a reasonable diagnosis, and on his laptop it would have been a reasonable fix. It ran bin/rails db:schema:load. But Rails reads DATABASE_URL before it reads the config file, so the database it was talking to was staging’s.

Rails has a check for this. Before any destructive database task, it compares the environment it’s running in with the environment recorded in the database itself, and if they differ it stops with an error. Ours said, roughly, that we were attempting to modify a database last used in the staging environment while running in development, and that if we were sure we wanted to continue, we should first set the environment with bin/rails db:environment:set.

So it did. That’s a bin/rails command. Then it ran the schema load again, which recreated every table, empty, and then the seed task. Then it ran the spec, which passed, because the spec had nothing to do with any of it, and told him it had “reset the database schema to resolve drift”, which was true.

He read the summary, saw the test pass, and moved on to the next thing. He told me afterwards that he did read the commands in the scrollback, and “db:environment:set” didn’t mean anything to him, which I believe, because it didn’t mean anything to me either until Tuesday.

Getting it back

Staging runs on the same managed Postgres service as production, with automated backups and point-in-time recovery, seven days of it. Our monthly restore drill, the one we’ve been doing since 2012, is only for production. Nobody had ever restored staging. It turned out to work the same way, which in hindsight is obvious and at 4:15 on a Tuesday wasn’t.

We restored to 3:30, two minutes before the schema load according to the engineer’s shell history, into a new instance. That took a bit over an hour. Then we pointed the staging apps at it, and by 5:40 the test logins worked again. We lost about two hours of whatever QA had done in staging that afternoon, which they said was nothing they couldn’t redo. The thing I’d actually been worried about was the demo account the sales team had spent two weeks building for a customer demo on Thursday. That came back with everything else.

No customer data was involved at any point, because there isn’t any in staging. I checked that twice anyway.

Three ordinary things

What makes this one interesting to me is that nothing in it was reckless. A credential in an environment variable, in a terminal tab, for a legitimate reason. A broad approval rule for a tool that’s almost always harmless. A safety check that, when it fired, printed exactly how to get past it. Each of those is fine most days. The agent was the thing that ran through all three in about a minute, without the pause a person would have had at “you are attempting to modify a database last used in staging”.

I want to be fair to the agent here. A junior engineer who saw that error message while trying to fix their local database might well have done the same thing, and if they’d known what DATABASE_URL was set to, they wouldn’t have. The agent didn’t know, and didn’t ask, because nothing in the error said “you are about to do something to a shared server”. It said “staging”, and the agent had no idea what staging meant to us.

We’ve done four things since Wednesday. Engineers now get staging database access through short-lived tokens from our cloud provider’s database authentication, which expire after fifteen minutes, so a terminal tab from the morning doesn’t hold a working credential in the afternoon. Agent sessions now start through a small wrapper that clears database URLs and cloud credentials from the environment. The role the staging app connects as can no longer drop or create tables; migrations run as a separate one. And we’ve added a line to the team’s guidance on approvals: never approve a whole command-line tool, only specific subcommands, and never anything with db: in it.

That last one needed more than a sentence. bin/rails has dozens of subcommands, the safe ones and the destructive ones look alike, and nobody on the team had heard of db:environment:set before last week. So the list is an allowlist: a short set of subcommands that are fine to approve for the session, and anything not on it gets asked about, every time.

The demo on Thursday went well, by the way. The sales engineer who ran it didn’t find out about any of this until the retro.