Healthcare.gov and the myth of the big-bang launch

Healthcare.gov launched on October 1st and three weeks later most people still can’t get through it. The account-creation step, where you have to register before you can even look at plans, fell over almost immediately. Pages time out, and insurers are reportedly getting garbled enrollment records on the other end. Yesterday the President stood in the Rose Garden and said there’s “no excuse” for it, and HHS has announced a “tech surge” of outside experts to help fix it.

It’s easy to pile on, and there’s a lot of schadenfreude in tech right now, mostly from people who’ve never shipped anything to millions of people on a deadline written into law. I don’t want to do that. I’ve been on launches that went badly. But I do think this is a giant version of a mistake I see all the time at much smaller scale, the big-bang launch.

You know the pattern. A large system with lots of parts, built by lots of teams, often from different organizations. Everything gets integrated late, and the whole thing gets tested end to end just before launch day, if at all. Then on launch day all of it goes live at once, for everyone. From what’s been reported so far, Healthcare.gov fits this pretty well. A long list of contractors built different pieces. The New York Times reported earlier this month that a test a few days before launch had trouble with just a few hundred simultaneous users. The date couldn’t move, and on day one the whole country was invited in at once. Each of those decisions makes sense on its own. Put together they mean the first real test of whether the system works is the launch.

Integration is where the bugs live. Every piece can pass its own tests and the whole can still fall over, because the bugs are in the assumptions between the pieces: timeouts, data formats, what “required” means, what happens on an error. Systems built by separate teams almost always disagree about something, and you only find out when they’re talking to each other under real load. Load isn’t something you bolt on at the end either. Making every visitor create an account before they could browse took one of the hardest steps to scale and made it the very first thing everyone did.

I’ve never built anything at this scale, so take this with appropriate salt. But I think the principles are the same ones that work for a ten-person team. Get a skeleton of the whole flow working end to end on day one, with fake data and stub services, and then fill in the real pieces behind it while keeping that path green every single day. Separate the launch from the release. The law said enrollment had to be open on October 1st, it didn’t say every feature had to work for every visitor that day. Launch in one state first, or let people browse plans without an account, or open the doors in waves, so each slice is a real test with a limited blast radius. And send real traffic at the thing before it’s live. Facebook famously did this with Chat, running the code and generating fake load from users’ browsers for weeks before anyone could see the feature.

The other thing I’d want is one owner, a single person with the authority and the technical depth to make calls across every contractor and every component. When nobody owns the whole system, problems that cross team boundaries have nobody responsible for them.

Paul Graham said in Do Things that Don’t Scale this summer that big launches almost never work for startups (I wrote about that essay in July). I didn’t expect to be pointing at that essay about a federal website three months later, but here we are. Shipping small pieces early feels slower, and in my experience it almost never is.