us-east-1 again

The first page came at 3:14 on Monday morning, Eastern time, and it was for webhooks. Our outgoing webhook deliveries were failing, the retry queue was growing, and the on-call engineer, who’d been asleep for about four hours, opened the AWS status page and found it already had an entry: increased error rates and latencies for multiple services in us-east-1.

That’s where we run. We’ve run there since before the 2012 outage, we were there for the S3 afternoon in 2017, and I’ve argued against going multi-region since at least 2017. So I’ve been asked “are we going to move?” five times since yesterday, and I’ll get to it.

I’m writing this the day after. AWS hasn’t published its full write-up yet, so what follows is what their status page said during the day, and what it looked like from our side.

What AWS said

The problem started shortly before midnight Pacific on Sunday. A little after 2 a.m. Pacific, AWS said the likely cause was DNS resolution for the DynamoDB API endpoint in the region: the name that every client, including a lot of AWS’s own services, uses to reach DynamoDB wasn’t resolving. That was fixed within the next hour or so. But by then a lot of things that depend on DynamoDB internally had fallen over and were recovering in a pile, and the problem moved. For most of the rest of the day the trouble was launching new EC2 instances, which AWS throttled while it recovered, and then a problem with the health checks for network load balancers. They said everything was back to normal at about 3 in the afternoon Pacific, with some services still working through backlogs after that.

That’s roughly fifteen hours, start to finish.

What it looked like from here

The webhook failures were the DynamoDB part. We use DynamoDB for one small thing, the idempotency records that stop us sending the same webhook twice, and when the delivery workers couldn’t reach it they did the right thing and stopped. The queue backed up for about two and a half hours and drained by itself once DNS came back. A few customers got webhooks late. Nobody got them twice, which I was pleased about.

Then it got quieter, and for a couple of hours it looked like we’d got away with it.

Our traffic has a shape. It’s low overnight, and it climbs steeply from about 8 Eastern on a weekday as the East Coast starts work, and Monday is the steepest day of the week. On a normal Monday the web tier’s auto scaling group roughly doubles between 7:30 and 10. Yesterday it tried to, and the launches failed. The instances we already had kept running perfectly well. They were also all we had, and by 9:30 they were at about 90% CPU, and the app was slow, p99 response times a bit over four seconds on the main pages, for the next several hours.

The other casualty was deploys. Our CI runners are a pool of EC2 instances that keep their caches, and the pool shrinks overnight and grows during the working day. On Monday it had shrunk to one runner, and it couldn’t grow. Builds queued behind each other for hours. That turned out to be fine, because we’d have frozen deploys anyway, but it does mean that if we’d needed to ship a fix yesterday, it would have waited in line behind a dependency bump.

Our Postgres databases didn’t notice anything. Neither did S3, the load balancers in front of the web tier, or the instances that were already up. And the status page, which moved off our stack in 2017 for exactly this reason, worked all day, which is the first big AWS outage since 2017 where we really needed it and it was there.

The pattern

When I drew that out on a whiteboard on Monday evening, the line down the middle was very clear. Everything that was already running kept running. Everything we needed to create during the incident, a new instance, a new runner, a scaled-out web tier, failed.

AWS has a name for the design idea, static stability, and a paper about it from their Builders’ Library a few years ago. The argument is that a system should keep working through a failure without needing to change anything, because the machinery for making changes is exactly what’s most likely to be broken during a big event. Running instances are the data plane, and the data plane is built to keep going. Launching instances goes through the control plane, which is a large, complicated distributed system with its own dependencies, apparently including DynamoDB.

We’d built most of our system to recover by creating things. Auto scaling is creating things. Our CI is creating things. Our runbook for a bad web server is “terminate it and let the group replace it”. That’s a fine design on 364 days a year, and it’s the wrong one on the day the region is having trouble, which is the day you need it.

So, are we moving?

No, or at least not yet, and I’ll try to be honest about why.

A warm standby in a second region, kept close enough to take traffic in an hour, would cost us something like 35 to 40% more on the AWS bill by our rough estimate, plus a database replication setup we don’t have today, plus probably two engineers for most of a quarter to build and test it, plus a failover drill we’d have to keep running. And it wouldn’t free us from us-east-1 entirely, because some of AWS’s global services have their control planes there. Yesterday cost us about six hours of a slow app and two and a half hours of late webhooks. I keep doing that sum and getting the same answer I got in 2017.

What we’re doing instead is aimed at the line on the whiteboard:

• Run the web tier with more headroom. We’ve been scaling to about 60% CPU at peak. If we’d started Monday at the size we usually reach by 10 a.m., yesterday would have been slow for nobody. That’s the most expensive change on this list, and it’s a lot cheaper than a second region.
• Keep at least three CI runners up overnight, so a deploy doesn’t depend on launching anything. If we can’t ship a fix during an outage, the outage gets to decide how long our own bugs last.
• Change the bad-web-server runbook so the first step is taking the instance out of the load balancer, not terminating it.
• Look at every place where recovery means creating something, and ask whether it could mean “already exists” instead.

The one we argued about longest was the webhook idempotency store. It stopped deliveries, which is the safe failure, and I’d wanted it to degrade instead, keeping a short local record so deliveries could continue with a small risk of duplicates. One engineer pointed out that our docs promise customers “exactly once”, and that a duplicate payment webhook is a much worse day for a customer than a late one. He won. The store keeps failing closed, and the docs now say what actually happens, that during an outage deliveries may be delayed but won’t be duplicated, which is a promise we kept on Monday.