The day our routes disappeared

“Your tunnels are up, but you’re not advertising anything to us.”

That was a network engineer at one of our customers, on a call with our support team at about five past three on a Thursday afternoon back in March, and the support person relayed it to the engineering channel word for word because she had no idea what it meant. I knew what each word meant. It took us another half hour to believe it.

I’ve been meaning to write this up since, and last Monday gave me the push. When Facebook, Instagram and WhatsApp all disappeared for about six hours, the explanation that came out over the next day was, at its core, the same thing that happened to us: some routers politely told the rest of the world they no longer knew how to reach a set of addresses, and the rest of the world believed them.

A little BGP

The internet is tens of thousands of separately run networks, called autonomous systems, each with a number (an ASN). Your ISP is one, AWS is a few, Facebook is one. BGP, the Border Gateway Protocol, is how they tell each other what they can reach. A network announces to its neighbors, “I can deliver traffic to these address ranges” (prefixes, like 157.240.0.0/16), its neighbors pass that on to their neighbors with themselves added to the path, and so on, until every router on the internet that matters has an opinion about how to get to that prefix. Routers pick the best path they’ve heard of, usually the shortest, subject to a lot of policy.

The other half of BGP is the withdrawal. If a network can no longer reach a prefix it announced, it sends a message saying so, and that message spreads the same way the announcement did. Once every path to a prefix has been withdrawn, routers simply have no route to it. Packets addressed there get dropped at the first router that doesn’t know where to send them. There’s no error that reaches the application, no “server down”, just silence and timeouts.

BGP mostly works on trust. A router believes what its neighbors tell it, which is why the occasional route leak or hijack makes the news, but it also believes withdrawals, and it should. If the network that owns a prefix says it can’t reach it anymore, the honest thing for everyone else to do is stop sending traffic there.

Our very small internet

We’re on AWS and we don’t run BGP on the public internet. But six of our larger customers don’t send us their data over the internet at all. Their security teams want it over a private connection, so each of them has an IPsec VPN from their data center into a transit gateway in our AWS account, and our private address range, call it 10.20.0.0/16, is only reachable through those tunnels.

Each tunnel runs BGP. It’s a tiny version of the internet: two autonomous systems, AWS’s side with a private ASN and the customer’s router with theirs. Our side announces 10.20.0.0/16 to them, their side announces whichever ranges their systems send from, and both routers install what they hear. The transit gateway decides what to announce based on its route tables, and a VPC’s range ends up in a route table through something AWS calls a propagation. Remove the propagation and the transit gateway stops announcing the range, which, in BGP, means sending a withdrawal.

The Thursday

The change was a Terraform refactor. We’d been adding a staging environment that needed its own route table on the transit gateway, and an engineer reorganized the networking module so production and staging were defined the same way. The plan had about thirty changes in it, almost all of them renames inside Terraform’s state, and one of them was a destroy-and-recreate of the propagation that put our production VPC’s range into the VPN route table. Destroy and recreate is normally harmless, a gap of a second or two. But the new one was pointed at the staging route table by a copy-paste slip, so the destroy happened and the recreate went somewhere else. The engineer who wrote it and the one who reviewed it both read the plan. Neither caught it, and I don’t think I would have either, because the line looked exactly like the twenty-nine renames around it.

It was applied at 2:10. Within seconds, the transit gateway sent a withdrawal for 10.20.0.0/16 on all six tunnels, and six customer routers removed the route. The tunnels stayed up. The BGP sessions stayed up. The customers’ systems kept trying to send us data every fifteen minutes, and their packets for 10.20.x.x went to their default route, which points at the internet, where a private address goes nowhere.

On our side, nothing looked wrong. Our alarms watched the tunnel state, which was up, and our ingest lag alert has a two-hour threshold because a couple of these customers send in big irregular batches. So the first we heard of it was the customer’s call at 3:05, and even then we spent the first part of the half hour looking at tunnel status and saying “but they’re up”. Someone eventually opened the transit gateway’s route table and saw that it no longer had our VPC’s range in it.

We put the propagation back by hand in the console at 3:50. The routes were re-announced within a minute, and data started flowing. All told it was 100 minutes of six customers’ feeds going nowhere. Five of them retry, and caught up by the evening. The sixth doesn’t, and they had to re-run the afternoon’s export the next morning, which was a slightly awkward phone call.

Facebook’s version

Facebook’s own write-up from last Tuesday says that during routine maintenance, “a command was issued with the intention to assess the availability of global backbone capacity”, and it took down all the connections in their backbone, the links between their data centers. They have a tool that audits commands like that, and a bug in it let this one through. Then the part that looks like ours: their DNS servers are designed to stop announcing their own prefixes over BGP when they can’t reach the data centers, which is a sensible safety feature, since a DNS server cut off from the rest of the network is one you don’t want answering. With the backbone down, every one of them concluded it was cut off, and withdrew. Cloudflare saw the burst of BGP withdrawals at about 15:40 UTC. Within minutes, resolvers all over the internet couldn’t find Facebook’s name servers at all, and facebook.com stopped resolving.

And then the part where their scale makes it so much worse. Their internal tools ran on the same network that had vanished, and getting engineers physically into the data centers took a while, because, as they put it, “it took extra time to activate the secure access protocols needed to get people onsite”. It was about six hours before things came back.

I’m not comparing our 100 minutes to their six hours. But the shape is the same. A routine change by competent people, a withdrawal that was BGP doing exactly what it was told, and a system whose health checks all said fine, because they measured whether the sessions were up and not whether anything was being said over them.

The part I haven’t fixed

We now have a check that runs every minute and compares what the transit gateway is announcing on each tunnel with what it should be announcing, and alarms if any expected prefix is missing. That would have caught March in a minute. We added a policy that any Terraform plan touching the networking module that destroys a propagation or a route needs a second reviewer who’s specifically looking at the destroys, which is clumsy but cheap.

The gap I haven’t closed is that the check is ours. It tells us what we’re announcing. It doesn’t tell us what our customers’ routers have accepted, and that’s the thing that actually matters. A customer could filter our prefix, or their own change could drop the route, and every number on our side would look perfect while their data went nowhere. The only party who can see that is the customer. One of the six has agreed to run a probe from their side to an endpoint in our VPC and send us an alert if it fails. The other five haven’t answered the email yet.