The afternoon our cloud went dark

Last month’s EC2 outage in US-East took a decent chunk of the internet down for an afternoon (Reddit, Heroku, Foursquare and a bunch of others) and it took us down with it. The post-mortem is done and things have calmed down, so I want to write up what happened on our side. We didn’t handle it especially well, and I suspect a lot of small teams are set up exactly the way we were.

We’re a small team running a web app that people use during business hours. Like a lot of people we’d moved off our own boxes onto EC2 over the last couple of years, and the architecture was what you’d draw on a whiteboard in your first week of AWS: two web servers behind an Elastic Load Balancer, a MySQL primary and read replica both on EBS volumes, a worker box for background jobs, nightly EBS snapshots and a mysqldump shipped to S3. All of it in one availability zone. We knew that was a risk, it was on the list. It was on the list for about eight months.

What happened

It started in the early afternoon our time. The first sign wasn’t an alert, it was a customer emailing to say pages were “hanging”. Our monitoring said the web servers were fine and in a sense they were, they were up and waiting on the database. On the database box MySQL was alive but any query that touched disk just sat there. iostat showed the EBS volume pegged at 100% utilization with basically zero throughput, which is what a volume that’s stopped responding looks like. The replica was in the same zone on the same kind of volume, doing the same thing.

About twenty minutes later the AWS status page went yellow with the usual “we are investigating increased error rates”. There’s a moment of relief when you find out it isn’t you, and it lasts maybe thirty seconds, until you realize it also means there’s nothing for you to fix.

The mistake we made in the first hour was going quiet. We were heads-down debugging and nobody told customers anything for about 45 minutes. When we did, we put a banner on the site, which was served by the same app that was hanging, so hardly anyone saw it. The email went out a bit after that.

Getting back up

Then we had to decide whether to wait for AWS or rebuild somewhere else. We gave it an hour. During last year’s big EBS event some volumes took a very long time to come back, and we didn’t want to bet the rest of the day on it.

We launched a test instance with a fresh EBS volume in each of the other zones and wrote some data to disk. Two of them were fine. Every API call was slow, presumably because everyone in the region was doing exactly what we were doing. Creating a volume from last night’s snapshot was going to be slow if it worked at all, since EBS was the thing that was broken, but the mysqldump in S3 didn’t depend on EBS. So we restored the dump onto a new instance in a healthy zone. That left us missing about twelve hours of data, but we had the app’s own logs and could replay the most important writes by hand.

Our database host was hardcoded in a config file on each web server, so repointing the app meant SSH-ing in and editing files. ELB was also having problems in our zone, so we launched two new web servers in the good zone and registered those with the load balancer instead.

We were back up, read-write, a little under four hours after that first customer email. The original volumes came back that evening, and we spent most of the next day reconciling the twelve hours of data between the old database and the new one, which was nobody’s idea of fun.

What AWS said

Amazon’s post-mortem came out later that week and is worth reading. A data collection server had been replaced, the DNS update for it didn’t fully propagate, and the agent on the EBS storage servers that reports to it kept retrying the old address. There was a memory leak in that retry path, and it slowly ate memory on the storage servers until they couldn’t keep up with requests. A DNS change and a slow leak in a monitoring agent nobody was looking at. In my experience that’s what most big outages look like up close.

What we changed

The database replica now lives in a different availability zone from the primary and the web servers are split across two zones. Yes, there’s a little more cross-zone traffic cost, it’s nothing compared to an afternoon of downtime.

We restore from backup once a month now, into a scratch instance, with a timer running. We had literally never restored the dump before that day. The first drill took 40 minutes, versus the couple of hours it took us on the day when we were working it out as we went. We also ship the MySQL binary logs to S3 every five minutes, so the next restore loses five minutes of data instead of twelve hours.

The database host is now a DNS name we control with a short TTL, so failing over is one change instead of SSH-ing into every web server. The status page is hosted somewhere completely separate from our own infrastructure. And the first line of the runbook is now “post on the status page and email customers”, before anyone opens a terminal. The thing customers complained about most wasn’t the outage, it was those first 45 minutes when they didn’t know whether we knew.

The embarrassing part is that every single one of these was already on a list somewhere. Apart from last year’s EBS mess and the storm that took out part of US-East in June, the cloud had just been up, every day, for two years, and it felt like it would keep being up. It took a bad afternoon to actually do them.