The runaway cloud bill weekend

$0.045 a gigabyte. That’s what AWS charges to push data through a NAT gateway in us-east-1, on top of the hourly charge, and I’d have told you a month ago that I knew that. I did know it, in the sense that it’s on a pricing page I’ve read. I didn’t know it in the sense of looking at an architecture diagram and seeing a meter.

Troy Hunt wrote up how he got pwned by his cloud costs yesterday, more than AU$11k of Azure egress after Cloudflare quietly stopped caching his Pwned Passwords downloads once they grew past 15GB, and reading it I felt a lot of sympathy and a bit of relief that his was bigger than ours. Ours happened two weekends ago. I’d been meaning to write it up for the team anyway, so here’s the public version.

Friday

We have a job that builds the data exports customers can schedule, a weekly or monthly dump of their account’s activity as CSV files. For years it ran from cron on one long-lived instance in a public subnet, which was one of the last pets we had. The plan was to move it onto AWS Batch so it could scale with the number of customers instead of with how long that one box took, and an engineer on my team did the move over a couple of weeks before the holidays. It was reviewed, it ran fine in staging, and it went to production on a Friday afternoon, which I’ll come back to.

Two things were different in production. The new Batch compute environment lived in our private subnets, like everything else we’ve built in the last few years, and its only route to S3 was through the NAT gateway, because that VPC had never had an S3 endpoint. Nobody noticed, because nothing breaks when you do that. It just costs money per gigabyte.

The second thing was a bug. Each export run reads the customer’s raw event files out of S3, builds the CSVs, writes them out, and then writes a small marker object saying “this period is done”. The new job’s role couldn’t write to the prefix where the markers lived. The old box’s role could. The write failed, the code caught the exception and logged a warning (a warning!), and the scheduler, seeing no marker, decided every period for every customer still needed doing and queued them all again. And again. Each pass read about 3.6TB of event files through the NAT gateway. The compute environment’s maximum was 512 vCPUs, because that’s the number somebody typed into the form, so the scheduler happily ran as many passes in parallel as that allowed.

Because the failure was the very last step, nothing customer-visible went wrong. No duplicate emails (the notification is sent after the marker write), no errors in the app, nothing on our dashboards except a log line at warning level that nobody watches on a weekend.

Sunday

The engineer who’d shipped it checked on the job on Sunday evening, more or less out of nerves, and found the Batch console showing close to a hundred completed runs of a job that should have run once. He disabled the job queue at about 9:30 PM, messaged me, and we spent an hour working out what it had cost.

From roughly 6:30 Friday evening to 9:30 Sunday night, about 51 hours, the job had pushed around 347TB through the NAT gateway. At $0.045 a gigabyte that’s about $15,600. The compute, on-demand because spot was on the list for “phase two”, came to around $1,250, and the S3 requests were a rounding error. Just under $17,000 in a weekend, for a company whose whole AWS bill is normally about $37,000 a month.

We had a billing alarm. We’d set it up in 2015, during the move out of the colo, at a monthly total that seemed high then. We’ve long since grown past it, so it fired every month, and at some point someone had made an email filter for it. I’m fairly sure that someone was me.

Monday and after

The fix for the bug was a one-line IAM change plus making the marker write fail the job loudly instead of logging a warning. The fix for the money was even smaller: an S3 gateway endpoint on that VPC, which costs nothing and sends S3 traffic around the NAT gateway instead of through it. It’s the kind of thing every AWS best-practices talk tells you to do, and we’d done it in our main production VPC years ago. This VPC was newer, and nobody had thought to.

We also opened a support case and explained what had happened. AWS credited back part of it, and was clear that this was a one-time thing, which seems fair, and I wouldn’t plan on it.

We’ve since set up AWS Budgets per account with forecast alerts that go to the ops channel instead of email, and turned on Cost Anomaly Detection, which I’d assumed we already had. The Batch environment has a maximum that matches what the job actually needs, and the job itself has a cap on how many times it’ll retry a period before it pages someone.

The part I haven’t solved is the review. The pull request was good. It had tests, it had a staging run, two people approved it, and nobody, me included, looked at the Terraform for the compute environment and asked how its traffic would reach S3. Code review is set up to catch bugs, and this wasn’t really a bug until it was multiplied by a bill. I don’t have a good answer yet for how you make the cost of a design visible in the place where the design gets reviewed. Some of the people who sell tools for this would say I should buy a tool for it, and they might be right.

(And yes, the Friday afternoon deploy. We’ve stopped doing those for batch jobs, though I don’t think a Tuesday would have saved us much. Somebody would have noticed sooner, and the number would have been smaller, and we’d still have been paying a meter nobody knew was there.)