Twenty-seven laptops in a boot loop

The first message came in at 6:52 a.m. Eastern on Friday, from someone on our support team: “my laptop is doing a blue screen thing and then restarting, is this a me problem?” By 7:30 there were nine more, and by then the reason was all over the news. It wasn’t a her problem.

You’ve probably read the outline already. At 04:09 UTC on Friday, CrowdStrike pushed a content update for its Falcon sensor on Windows, a “channel file” (number 291) that tells the sensor how to evaluate named pipe activity. A logic error in it crashed the sensor, and since the sensor runs in the kernel, it crashed Windows, on boot, every boot. CrowdStrike pulled the file at 05:27 UTC. Microsoft’s estimate on Saturday was about 8.5 million Windows machines, under one percent of the total, which is a small percentage that happens to include a lot of airlines, hospitals and banks. Macs and Linux machines don’t use that file and were fine.

We run Falcon on our laptops; we moved them to it from Defender in 2022. Defender was the “endpoint security product” on the list of fourteen agents we made after SolarWinds, and Falcon is the same kind of agent I said at the time I didn’t want to stage updates for. More on that at the end.

Who got hit

The 78 minutes the bad file was live are what decided it. In US time that’s 9:09 to 10:27 p.m. Pacific on Thursday, or just after midnight Eastern. A laptop only got the file if it was awake and online in that window. Everyone who’d closed their lid at the end of the day opened it on Friday morning to a perfectly normal machine, because the sensor had picked up the fixed file before anything could go wrong.

So it was a lottery, weighted by habit. We have 68 Windows laptops, mostly in support, sales, finance and ops; nearly all the engineers are on Macs. Twenty-seven were hit. They were the ones docked at home with sleep turned off, plus a couple of people on the West Coast who’d been working late Thursday and watched their screen go blue in front of them. One of those posted in Slack at 9:40 p.m. Pacific, and nobody saw it until the morning, which I’m not going to pretend I’d have handled better if I had.

Our servers are Linux. The Windows build agents don’t run Falcon. Production never noticed.

The fix, and the catch

The fix CrowdStrike published early Friday is simple to describe: boot into Safe Mode or the Windows Recovery Environment, go to the CrowdStrike folder under the system drivers directory, delete the bad channel file, reboot. On a server in a rack that’s a few minutes of someone’s time. On a VM you can detach the disk and fix it from another machine.

On a laptop with BitLocker, which is every one of ours, there’s a catch. Safe Mode and the recovery environment need the drive unlocked, and the drive is unlocked by the TPM during a normal boot, which is exactly the thing that’s crashing. So the laptop asks for the BitLocker recovery key, a 48-digit number in eight groups of six, and nobody has that memorized, which is the point of it.

Our recovery keys are escrowed in the device management system, which our managed IT provider runs. That part worked. Every laptop they’d enrolled had its key there. What didn’t scale was the people. The provider has a lot of clients, a lot of them were having the same morning, and their help desk queue was, in the words of the account manager, “biblical”. Our IT lead had access to view keys in the console. Nobody else did. At 7:15 she gave me and our support lead the same role, which took about four minutes, and that was the most useful thing anyone did all day.

Reading numbers to people

We’re fully remote, so all 27 laptops were in people’s homes. The procedure we settled on was: the person calls, or we call them, they get to the BitLocker screen, one of us reads the key in groups of six, they type each group and read it back, then we talk them through the recovery menus to a command prompt, the delete, and the reboot. It took between fifteen and forty minutes per laptop, depending on how comfortable the person was with a command prompt, and how many times the recovery menu came up in an order that didn’t match the screenshot we were working from.

There’s a security problem hiding in that procedure, and we’d talked about it before, in the help desk post from 2020. A recovery key is a secret. “Someone on Slack says they’re in finance and need their key” is the kind of request the rule we wrote for customer 2FA resets four years ago says not to act on without a second channel. On Friday the second channel was easy, because we called people back on the number we have on file, and most of them were on video anyway with the laptop on the kitchen table behind them. But I did notice how quickly the rule started to feel like an obstacle at 8 a.m. with eleven people waiting. By Saturday CrowdStrike was warning about fake “hotfix” downloads aimed at its customers, and CISA was warning about phishing using the outage. A real outage is the best cover a social engineer will ever get.

By about 2 p.m. Friday we had done 22 that way. Three more had come back on their own: people who’d rebooted over and over had the sensor pick up the fixed file before the crash, which other people were reporting too and Microsoft’s guidance for Azure VMs mentioned, though on our machines it took between six and a dozen reboots. One belonged to someone on vacation, and we did it Monday morning.

The last one had no key in escrow. It was one of the oldest laptops in the fleet, set up before the IT provider took over, and its enrollment had been migrated without the recovery key ever being backed up to the new system. There was no way into that drive, so it was wiped and reimaged, and the owner lost a few days of local files that weren’t synced anywhere. The provider ran a report on Monday and found two more machines in the same state, neither of them affected this time. Those have new keys escrowed now.

Staging content updates

Back to the 2020 post. I wrote then that for the endpoint security agent in particular, holding back updates means holding back detections, which is the opposite of what you want from a security product, so we let it update itself. What I didn’t spell out was that we did set a policy for the sensor software itself: laptops run one version behind the latest, which is a standard Falcon option.

It made no difference. What went out on Thursday night wasn’t a sensor version. It was content, the kind of file the sensor pulls several times a day, and the version policy doesn’t govern it. I think a lot of administrators found that out on Friday. I’d assumed, without ever checking, that “N-1” covered more than it did.

So the question is whether I’d want a staging knob for the content updates if CrowdStrike offered one. Every argument I made in 2020 still holds: new detections are the whole value of the thing, and a three-day delay on them is three days that a new piece of malware has a free run. On the other hand, 27 laptops and most of a day for three people, and we got off lightly compared to Delta. I’d probably take a delay of an hour or two, enough for a vendor’s own canaries to fall over first. Whether that’s something a customer should have to configure, or something the vendor should have been doing anyway, is a question I’m going to put to our account team when they call, and they will call.