When our CDN went down

6:27 a.m., Tuesday. Our external checks, the ones that run from outside our network since the Dyn outage, started failing on the main app, and the on-call engineer’s phone went off. She opened the app and got a Cloudflare error page, the one that says the problem is between Cloudflare and the site, with our name in it. Our origin servers were fine. CPU flat, no errors, the load balancer healthy and receiving almost no traffic, because almost nothing was getting through to it.

We’ve had Cloudflare in front of the app for a few years now, for caching, the web application firewall, and bot protection on the login page. And for the first ten minutes, the on-call engineer did what anyone would do, which was assume it was us. She checked the last deploy (Monday afternoon, nothing interesting), checked our Cloudflare configuration, or tried to, and couldn’t log in to the Cloudflare dashboard. That was the first real clue. By about 6:50 Cloudflare’s own status page had an incident up, and the news sites had started on it soon after, with the usual list of big names that were down.

The runbook we’d tested

I got on the call a few minutes after that. We have a runbook for going around the CDN, written not long after we moved behind Cloudflare, when I’d remembered their regex outage in 2019 and realized we had no way to serve the app without them at all. We drill it twice a year on the staging environment. It’s four steps. Open the origin load balancer to the internet, since it normally accepts traffic only from Cloudflare’s address ranges. Switch our AWS firewall rules on the load balancer from counting to blocking, so we’re not entirely unprotected. Point the app’s DNS record at the load balancer instead of Cloudflare, through the script that writes to both our DNS providers. Then wait.

The wait is the reason the app’s record is one of the few we kept at a five-minute TTL, when we raised everything else to an hour in 2016. It’s in the runbook, with a note saying why, which is the only reason nobody has “fixed” it since.

We decided to run it at 7:20, after half an hour of Cloudflare’s updates saying they were investigating, with no time estimate. The last step was done by 7:38, and by about 7:45 most customers could reach the app again, going straight to our load balancer. That part went the way the drills had, within a few minutes. I was quite pleased with it, for about ten minutes.

The two things we hadn’t tested

The first was custom domains. About forty of our customers use their own domain for the app, something like portal.theircompany.com, pointed at us with a CNAME. Those certificates are issued and served by Cloudflare. Our load balancer has certificates for our own domains and nothing else. So when the shared record moved, those forty customers went from a Cloudflare error page to a browser certificate warning, which is worse, because some of their users clicked through it and then emailed their IT departments asking whether they’d been hacked.

We couldn’t fix that quickly. Issuing forty certificates for domains we don’t control means a validation step in each customer’s DNS, and we weren’t going to do that at 8 in the morning. So we split the record: our own domain stayed on the load balancer, and the custom domains went back through Cloudflare, which meant those forty customers stayed down until Cloudflare recovered. Our drill on staging had never included a custom domain, because staging doesn’t have any.

The second was the login page. Our login form has a Cloudflare Turnstile widget on it, the bot check that replaced captchas, and our server won’t accept a login without a valid token from it. Turnstile was one of the things that was broken on Tuesday, and the widget’s script comes from Cloudflare, not from us. So with the app reachable again, anyone who was already logged in was fine, and nobody else could log in at all. That’s the same reason we couldn’t get into Cloudflare’s dashboard earlier, as it turned out. They had the same problem on their own login page.

The engineer who owns auth found it at about 7:55 when our support lead asked why customers were saying the app “loads but won’t let me in”. We have a feature flag that skips the Turnstile check, there for local development, and it took twenty minutes of discussion and two minutes of work to turn it on in production with rate limiting tightened on the login endpoint. Logins came back at about 8:20.

Cloudflare’s side

Cloudflare said core traffic was mostly flowing again by about 9:30 our time, and everything was back by a little after noon. We moved our own domain back through Cloudflare at 7 that evening, which is the part of the runbook we drill least and which went fine, and turned Turnstile back on at the same time.

Their write-up went up the same day, which I’ll give them a lot of credit for. A permissions change in one of their databases made a query that builds a configuration file for their bot management system return duplicate rows, the file came out about twice its usual size, and it went over a size limit in the software that loads it on every machine in their network, which then failed. The file was regenerated every few minutes, and for a while the good and bad versions alternated, so the network kept recovering and failing again, and they spent the first part of the morning thinking it might be an attack. The bot-management connection made me laugh a bit grimly, given that bot management is what took our login page down.

It’s the third big provider outage in a month, after AWS in October and Microsoft’s Azure Front Door at the end of October, which didn’t affect us directly.

Failing open

The custom domains are a project now. We’ll get certificates for them in AWS ahead of time, with the customers’ help, so the bypass covers them too. It’s a few weeks of emails, and staging is getting two fake custom domains so the drill can test it.

The login page is a harder question. Turnstile is there because we get credential-stuffing attempts against our login every week, and it works. What we have to decide is what happens when the bot check itself is unavailable. Failing closed, which is what we had, means a Cloudflare outage locks every customer out of our product. Failing open means anyone who can take down or block the widget gets an unprotected login form, and that’s not a hypothetical. On Tuesday we picked failing open, by hand, with tighter rate limits, for about eleven hours. I think that was right for Tuesday. I’m less sure it’s right as the automatic behavior, and the two engineers I’ve asked have given me opposite answers.