Somebody else's six-figure bandwidth bill, and our smaller one

190TB at $55 per 100GB is exactly $104,500. I checked, because the first thing I wanted to know when I read the Reddit post that was all over Hacker News on Tuesday was whether the bill even added up. It does. A developer with a small static site on Netlify’s free plan, a few hundred visitors a day, woke up to an overdue invoice for $104,500 after something hammered one audio file on the site for four days. Support called it a DDoS and offered to knock it down to 5%, about $5,000, which is still a lot of money for a hobby site. Then the post went viral, and Netlify’s CEO said on HN that they wouldn’t be charging for it, and that their policy is to forgive bills like this after the fact rather than take free sites offline during a spike.

I’m glad it ended that way for them. I’m less glad that “post on Reddit and hope” is the escalation path, and I say that as someone who paid our own version of this bill three weeks ago without anyone forgiving anything.

Ours

Our docs site is static. It’s built from Markdown in a repo, deployed to one of the popular static hosting platforms (not Netlify, though I don’t think the details would be different on any of them), and it’s been there for years without anyone thinking about it. The plan includes a terabyte of bandwidth a month. Past that, you buy more in 100GB blocks, at $45 a block. Nobody here had ever needed to know the second number.

The docs site also hosts the downloads for our command-line tool, which customers use for bulk imports and for scripting against the API. It’s a C# app published as a self-contained single file, so it carries its own copy of the .NET runtime and each build is about 41MB. There’s one for Windows, one for Mac and one for Linux, sitting at a stable “latest” URL, because that’s what makes the install instructions one line long.

You can probably see where this goes. One of our customers set up a CI pipeline that installs the tool at the start of every job, from the “latest” URL, with no caching. That alone would have been fine, a few hundred downloads a day. Then a step in their pipeline started failing, and their CI system was configured to retry failed jobs, and the retries were configured, as far as we can tell, to do that indefinitely. For eleven days, one of their runners downloaded our 41MB binary, failed, and started again, around the clock, sometimes several jobs in parallel. 386,000 downloads. About 15.8TB.

How we found out

The CFO found it, on the company card statement, when the hosting charge came through at $6,705 instead of the usual small number. 15.8TB minus the included terabyte is just over 14.8TB, which rounds up to 149 blocks, at $45 each. She asked in Slack what it was, and for about an hour nobody knew.

The host had warned us. It sends an email when you pass 50%, 75% and 100% of your included bandwidth, and it sent all three. They went to the address the account was registered with, which was a shared inbox for the website that was set up years ago, before the docs site was really ours, and which nobody reads.

What stung a bit is that we’d done a lot of work on exactly this kind of surprise. After the runaway weekend two years ago we set up AWS Budgets with forecast alerts to the ops channel and turned on Cost Anomaly Detection, and they work. They’d have caught this in a day or two if the traffic had been in AWS. But the docs site isn’t in AWS. It’s a separate vendor with its own dashboard, its own billing and its own emails, paid on a card. It’s on our vendor list, and since last March that list even has a column for which bank and card each vendor is paid from, but nobody had connected “paid on a card” to “how much it could cost if something went wrong”.

The customer

The logs made it obvious, since every request came from a small range of IPs with the same CI user agent. We found the account they belonged to, and our support lead emailed their admin, politely, with a summary of what we were seeing. They were mortified, fixed the retry policy the same afternoon, and started caching the binary. Nobody had done anything malicious, and I’m not interested in sending them the bill. We’d made the “latest” URL easy to use in exactly the way they used it.

We asked the host for a credit, explaining what had happened. They were nice about it and said no, which I think is fair. The traffic was real requests for a real file that we’d chosen to put there.

The caps

The binaries have moved off the docs site. They’re now in S3, behind CloudFront, in our own AWS account, where the Budgets alerts and the anomaly detection can see them. CloudFront gives you a free terabyte a month and then charges $0.085 a gigabyte in North America for the first 10TB, a bit less after that, so the same 15.8TB would have cost us about $1,240 there, which is a lot less, but still not nothing, and the alert would have fired on day two. There’s also a rate limit now, through AWS WAF, on how many requests one IP can make for the downloads in a five-minute window, set high enough that a busy office behind one NAT won’t hit it.

The install instructions now tell CI users to pin a version and cache it, with a two-line example for the CI systems most of our customers use.

The hosting account’s notifications go to the ops channel, not to an inbox, and it’s registered to a team address that actually exists. And we went through the vendor list looking for every other usage-billed service paid on a card, which turned up four more, all of them small, and none of them with a cap set.

Two of those four offered a hard spending cap, and we turned it on for both. Our docs host doesn’t offer one on our plan (I asked), so the rule we settled on is that anything usage-billed gets a cap if the vendor has one, and an alert in the ops channel at twice its normal monthly spend if it doesn’t. I’d rather the docs go down for a day than pay for a stranger’s retry loop, and with the binaries gone, the docs site’s normal month is about 300GB, a long way from the terabyte.