The day DNS went away

The worst part of Friday was that every one of our servers was fine. CPU was flat, error rates were zero, the database was bored. Our monitoring, which runs inside the same network as the app, was green across the board. Meanwhile our support inbox was filling up with people saying the site wouldn’t load, and when I tried it from my laptop at home, it wouldn’t.

dig told the story in about ten seconds. Our domain’s name servers were all Dyn’s, and none of them were answering.

What happened

Starting at about 7:10am Eastern on Friday, Dyn, one of the big managed DNS providers, was hit with a huge DDoS against its DNS infrastructure, starting with its East Coast data centers. If your domain’s DNS was at Dyn, resolvers on the East Coast couldn’t look it up, and so as far as a lot of the internet was concerned, you didn’t exist. Twitter, GitHub, Reddit, Spotify and a long list of others were down or flaky for hours, and so were we. Dyn got the first wave under control by mid-morning, a second wave hit around lunchtime and was wider, and a third in the afternoon was mostly mitigated.

Flashpoint, a security firm, said on Friday that at least part of the traffic came from Mirai botnets, the malware that infects internet-connected cameras and DVRs with default passwords. That’s the same malware that was behind the 620 Gbps attack on Brian Krebs’s site last month, and its source code was posted publicly at the end of September. So it isn’t a surprise, exactly, that someone pointed it at something bigger.

Our morning

We had all our eggs in Dyn’s basket. Our main domain, the API domain, and, worst of all, the domain our status page is on, because when we moved the status page off our own infrastructure after the 2012 outage, we moved the hosting and never thought about where its DNS lived. So the first thing customers on the East Coast saw when they went looking for an update was the status page failing to load too. We ended up posting updates from our Twitter account, which was itself up and down, because Twitter was on Dyn as well.

The obvious fix is a second DNS provider, and the obvious reason we didn’t already have one is that it’s annoying to keep two in sync. We decided around 9:30 to do it anyway, live, with Route 53 as the second, and it took most of the day.

The first problem was getting our records out of Dyn. Their web console was struggling, which is understandable, and Route 53 can’t pull a zone by zone transfer, it only takes records through its API or a zone file import. We had a few hundred records in the main zone, a lot of them CNAMEs for customer custom domains. One of our engineers ended up writing a script against Dyn’s API, with lots of retries, to dump everything into a BIND-format zone file, which we then hand-checked against what we thought should be there and imported. That was about two hours, most of it waiting on API calls that timed out.

The second problem was the registrar. Adding Route 53’s four name servers alongside Dyn’s is a change at the registrar, which needs a login, which since the 2FA rollout in August needs a second factor. I’d like to report that this went smoothly, and it did, but only because one of the two people with the registrar token was already in the office.

The third problem was the one I didn’t think about until it was happening. The registry for .com updated our delegation within minutes, but resolvers that had already cached our old NS records kept using them until those records expired. The NS records at the .com level have a two-day TTL. In practice the big public resolvers seemed to pick up the change much faster than that, and some ISP resolvers clearly didn’t, so for the rest of Friday we were up for some people and down for others, which is the most confusing kind of outage to explain to a support team.

And one more thing, which the 2012 version of me would have found funny. After that outage we put the database behind a DNS name with a short TTL, so we could fail over by changing one record, and over the years we’d done the same thing to most of our public records too, 60 seconds each. That’s great when the DNS provider is up. When it’s down, a 60-second TTL means resolvers forget your address after a minute and have to ask again, and there’s nobody there to answer. A domain with an hour-long TTL would have survived a short DNS outage for anyone who’d looked it up recently. Ours didn’t.

What we’re changing

We’re keeping both providers. Every record change now goes through a script that writes to both APIs and then checks that they agree. It’s about 150 lines and I’d rather have a tool for this, but I haven’t found one I trust yet. Netflix’s Denominator is the closest thing I’ve seen.

The status page is getting its own domain on a different DNS provider from everything else, so it shares nothing with the product.

TTLs are going up, to an hour for most records, and we’ll lower them deliberately ahead of a planned change. The emergency case that the short TTLs were for, failing over the database, now goes through an internal name that doesn’t depend on public DNS at all.

And we’re adding an external check that resolves our domains from a few places outside our network, because “every server is fine” and “nobody can reach us” were both true for about twenty minutes before anyone on our side knew.

What stays with me is how little of this was about us. Somebody’s DVR, with a password nobody ever changed, was part of what took our product offline for a lot of our customers on Friday. There are reportedly hundreds of thousands of these devices on Mirai already, and nobody is going to patch them. On Monday one of the Chinese camera makers said it would recall some of its devices in the US, which is a start, but we’re doing the second-provider work on the assumption that this happens again.