Finding money in the cloud bill

Ten minutes. That’s how long the compute in Ludic’s story stayed running after every query, in a post from the weekend called I accidentally saved my company half a million dollars. Their Snowflake warehouses were set to suspend after ten minutes idle, which is the default, the queries took a couple of seconds, and so they were paying for something like ten minutes of compute for every two seconds of work. The projected bill was heading for a million dollars. He changed the setting, it came down to about half that, and the rest of the post is about how the organization made him regret it, which is funnier and sadder than the technical part.

I read it with some recognition, because we spent September and part of October doing the same exercise on our AWS bill, without the drama. Nobody asked us to. Money’s more expensive than it was two years ago, everyone we talk to is trimming, and our bill had grown from about $37,000 a month at the time of the runaway weekend to about $58,000, while the business had grown by less than that. I wanted to know where the gap was.

The budgets and anomaly alerts we set up after that weekend do their job, which is catching sudden spikes. They don’t catch slow waste, things that cost a bit too much every single month and have done for a year, because to an anomaly detector that’s just normal. You have to go and look.

Where it was hiding

Things that are on because nobody turned them off

This was the biggest one, and it’s the Snowflake story in a different costume. We have a staging environment and two shared dev environments, each a scaled-down copy of production: a few app servers, a database, a cache, a queue. They ran 24 hours a day, every day, for about 730 hours a month. People use them from roughly 8 to 8 on weekdays, which is about 260 hours. We now scale the app tier to zero and stop the databases outside those hours (with a Slack command to bring an environment up if someone’s working late), and it took about $4,000 a month off. The other one in this category was a database read replica that had been added for a reporting migration in the spring of last year and never removed. Nobody had queried it in five months.

Logs

CloudWatch Logs charges $0.50 a gigabyte to ingest in us-east-1, before you store anything. During an incident in March someone had turned one of our busiest services up to debug logging to chase a problem, and it was still at debug in October, writing a bit under 4TB a month. That was close to $2,000 a month for logs that nobody read, about a service that was fine. Putting it back to info, and setting retention on the log groups that had been on “never expire” since they were created, saved about $1,700.

Storage nobody deletes

EBS snapshots, mostly. Our image builds take a snapshot every time they produce an AMI, the old AMIs got deregistered by a cleanup job, and the snapshots behind them didn’t, because the cleanup job didn’t know they existed. Add unattached volumes from instances that were terminated with “delete on termination” off, and the last few volumes still on gp2 (gp3 is about 20% cheaper per gigabyte and at least as fast for us), and that was about $1,300 a month.

Commitments we hadn’t bought

We’d bought Savings Plans in 2021 sized for the fleet we had then. The fleet grew and the commitment didn’t, so only about 40% of our compute was covered, and the rest was paying on-demand rates. Strictly that’s a discount we weren’t taking rather than waste, but it’s still money. Topping up the commitment to cover the floor of our usage, the part that never goes away even at 3 AM on a Sunday, saves about $2,900 a month. I’d rather under-commit than over-commit, since a commitment you don’t use is the one kind of cloud waste you can’t fix later.

Data moving between places

We learned about the NAT gateway the hard way last year. This time it was cross-AZ traffic: AWS charges a cent a gigabyte in each direction between availability zones in the same region, and one chatty internal service had its callers spread across all three zones while its own instances were mostly in one. It’s a small number per gigabyte with a lot of gigabytes behind it. Keeping traffic in-zone where we could, and accepting it where we couldn’t, saved about $700.

That’s about $10,600 a month, a bit over 18% of the bill. None of it needed new tools. It was Cost Explorer grouped by usage type, the CUR in Athena for the questions Cost Explorer can’t answer, and a lot of asking “what is this and who owns it?” in Slack.

The part that wasn’t about AWS

What Ludic’s story is really about is that he saw the problem a month after joining and wasn’t allowed to fix it, because it wasn’t anyone’s job, and the process for making it someone’s job took longer than the money took to burn.

We had milder versions of that. The read replica was “the data team’s”, except that the person who’d asked for it had moved teams. The debug logging was turned on by someone who was entirely right to turn it on during the incident, and there was no step in our incident process to turn it off afterwards. Now there is, in the postmortem template, under “temporary changes made during the incident”. Every one of the five had a person who could have fixed it in an afternoon and didn’t know it was there.

The thing I’ve changed for the long run is tagging. Every resource now needs an owner tag that’s a team, not a person, and a monthly report goes to each team with their slice of the bill and how it moved. Nobody’s budget gets docked, so it’s not a chargeback, but it turns out people are quite motivated to fix a number with their team’s name on it. The first report went out last week. One team found a forgotten load test cluster within an hour of reading it, which I’m counting as a sign it works and not as a sixth category.