The cron job that ran every minute

schedule_expression = "cron(* 4 * * ? *)"

That’s the line. If you read it quickly it says “every day at 4 o’clock”, and that’s what the engineer who reviewed it read, and what I read when I looked at the PR afterwards before I knew there was a problem. What it actually says, in EventBridge’s six-field cron format, is “every minute of the 4 o’clock hour, every day”. The minutes field is *. It should have been 0.

What the job does

Our customer success team asked in August for a short written summary of each account’s last 30 days, waiting for them each morning: usage trends, open tickets, anything that changed in their plan. It’s internal, it’s read by people, and it goes through the same model provider as our support drafts, which has been on our subprocessor list since 2023, so the data side was already covered. About 2,900 accounts, one model call each, written into a table keyed on account and date.

It’s exactly the kind of internal tool our autonomy table lets an agent build unattended, and that’s what happened. The spec said “run once a day at 04:00 UTC, before the CS team starts”. The agent wrote the handler, the tests, and the Terraform for the schedule and the ECS task. The reviewer agent we run on every PR passed it, an engineer reviewed it and ran it once by hand in staging, and it was merged on Thursday 3 September. The first scheduled run was at 04:00 UTC on the Friday.

So was the second, at 04:01.

Why nobody noticed

Each run took about four and a half minutes, so for an hour every night there were four or five copies of the job running at once, all summarizing the same accounts. The model provider rate-limited some of the calls, and the job retried them with backoff, as the agent had very sensibly written it to do. The table write was an upsert on account and date, so the 60th run of the night overwrote the 59th with a nearly identical paragraph. Every morning the CS team found a fresh summary for every account, which is what they’d asked for.

From the inside, it was working perfectly. The logs said so, sixty times a night, in a log group nobody had a reason to open. The rate-limit errors were there too, but they were retried and succeeded, so nothing alerted.

What did alert was money. Each run cost about $17 in model calls. Sixty a night is about $1,020 a day, against $17 a day planned. We have a spend alert on the provider account, set as a percentage of a monthly budget, and it fired on Thursday the 10th. It went to a shared inbox, and it was the Monday, 14 September, before someone read it, went to the provider’s usage page and saw a sawtooth with a tooth every night at the same hour.

By then it had run 660 times instead of 11. The extra 649 runs came to about $11,000.

Finding the line

It took about fifteen minutes from “why is usage spiking at 4am” to the schedule expression, and most of that was finding which job ran at 4am. The fix was one character. I asked the agent that had written it, in a fresh session, to explain the expression, and it said straight away that it fires every minute between 04:00 and 04:59 and that it was probably meant to be cron(0 4 * * ? *). It knows. It just didn’t apply what it knows at the moment it was writing the line, the same way the reviewer has written a few hundred crontab entries over twenty years and still read past it.

The tests didn’t catch it because nothing tested the schedule. The agent had written good tests for the handler, and the schedule was Terraform, which we don’t unit test. The staging run didn’t catch it because it was triggered by hand.

Afterwards

The job’s calls now go through their own API key, in its own project on the provider side with its own monthly limit, so the worst case for any one job is that job’s budget, not everyone’s. The spend alert goes to the on-call channel instead of an inbox. And there’s a CI check now that finds any schedule expression a PR adds or changes and posts the next five times it will fire as a comment on the PR. For this one it would have said 04:00, 04:01, 04:02, 04:03, 04:04, which nobody would have read past.

We didn’t ask the provider for a refund. The calls were real and we made them.

The part I can’t make a rule for is the review itself. The engineer who approved it is careful, and he did read the Terraform. He read the line, recognized the shape of a cron expression with a 4 in it, and moved on, which is how everyone reads things they’ve seen a thousand times. The reviewer agent hadn’t flagged it either. The agent had written about 600 lines in that PR and 599 of them were fine. Afterwards I showed cron(* 4 * * ? *) to three other engineers, cold, and asked what it meant. Two of them said 4am every day.