Object storage went down and took our status page with it

I’m writing this a few hours after S3 in us-east-1 came back, so some of it may turn out to be wrong once Amazon publishes whatever they publish. This is just what it looked like from our side.

A bit before 10 AM Pacific, our error rate on file uploads went from basically zero to basically everything. Then thumbnails stopped loading in the app. Then our CI builds started failing, which is when the engineer on call messaged me, because a failing build is the alert people actually read. Within ten minutes it was obvious it wasn’t just us. Half the internet was reporting broken images and failed uploads, a bunch of sites were simply down, and somebody on our team pointed out that the AWS Service Health Dashboard was showing all green.

It turned out the dashboard was green because the dashboard itself depends on S3, and couldn’t load the icons that would have turned it red. AWS put a banner at the top and tweeted that the colors not changing was related to the S3 issue. It took them until about 11:35 Pacific to get the dashboard updating again. I’d laugh harder if we hadn’t done exactly the same thing.

Our status page is a small static site, and it’s been moved twice already. After the 2012 outage we took it off our own servers, and after the Dyn DNS attack last October we gave it its own domain on a different DNS provider. Where it actually lived through all of that was an S3 bucket behind CloudFront, in a separate AWS account, because that’s the cheapest, most reliable way to host a static site that anyone knows about, and because “what if S3 is down” didn’t feel like a serious question. A separate account felt like “separate” at the time. The bucket was in us-east-1, like everything else we run. So at 10:15, when we wanted to post “we know, uploads are failing, it’s an AWS problem”, we couldn’t. The deploy script for the status page does an aws s3 sync, and it hung. CloudFront kept serving the cached copy, which said “All systems operational”, in large friendly green letters, until S3 came back and someone could finally push an update.

We ended up doing what everyone else did. We posted on Twitter, put a banner in the app (the banner text lives in our database, not in S3, by luck rather than design), and answered the support queue by hand. Most customers were fine with it once they knew. The ones who were annoyed were annoyed about the green page, not the outage, which matches what we learned the last time a region went dark on us: people forgive downtime much more readily than they forgive silence.

Object retrievals started coming back around 1 PM, and adding new objects was working again by about 1:50. We were mostly back to normal by 2:30 after clearing a backlog of failed thumbnail jobs.

The part I hadn’t appreciated is how many things depended on S3 that I wouldn’t have listed if you’d asked me yesterday. I made a list this afternoon while it was fresh, and I’m still adding to it:

• user uploads and thumbnails, obviously
• the status page
• CI, because our build caches and artifacts live in a bucket
• our private Docker registry, whose storage backend is S3, so no new containers could be pulled and autoscaling would have failed if we’d needed it
• the log shipper, which buffers to local disk when it can’t write to S3, and two of the boxes came within about 15% of filling their disks
• a nightly database backup job, which wasn’t due to run yet but would have failed silently because its alerting was, you guessed it, a file written to S3

None of these are wrong choices on their own. S3 really is very reliable, and if I had to bet on S3 versus anything we could run ourselves, I’d still bet on S3. The problem is that we’d made the same bet six times without noticing, and all six were in the same region.

For next sprint, the status page is moving for the third and hopefully last time, to a hosted status page service that runs entirely on someone else’s stack. It costs a few dollars a month, and I’d rather pay that than build something clever. The deploy for it won’t depend on any of our tooling, so someone can update it from a phone. Our registry and build caches are getting a replica in a second region. That one’s mostly a config change, and I’m annoyed with myself that we hadn’t done it. And the backup job is getting a heartbeat check that alerts when it doesn’t hear from the job, instead of relying on the job to report its own failure.

I’m not going to try to make the whole product multi-region over this. A four-hour outage of the most reliable service AWS has, once in a few years, isn’t worth a project that size for a company our size. But the stuff we use to talk about outages has to survive the outage, and I should have seen that one coming a long time ago.

I’m curious what the root cause turns out to be. AWS usually writes a decent post-mortem, and I’ll be reading it closely.