WannaCry Friday

The text came in a little before 11 on Friday morning, Eastern time: “all the files on the shared drive have .WNCRY on the end, is that bad?”

It was from the office manager at a small business I do some consulting for on the side, about 30 people, one server, a mix of Windows 7 desktops and a couple of laptops. I’d been reading about a ransomware outbreak hitting British hospitals since early that morning, and the honest answer was yes, that’s bad, unplug everything.

I’m writing it up because I think a lot of small offices had exactly this Friday, and the details of how it got in are more useful than the headlines.

What WannaCry is

By now most people have read the outline. The ransomware itself is ordinary: it encrypts documents, shows a ransom note asking for $300 in bitcoin, doubles it to $600 after three days, and threatens to delete the files after seven. What made it a worm is that it spreads over SMB, using EternalBlue, one of the NSA exploits the Shadow Brokers dumped on April 14th. Microsoft had already patched the bug in MS17-010 on March 14th, a month before the leak. Once a machine is infected it scans its local network on port 445, and random addresses on the internet as well, and infects anything that hasn’t got that patch.

Around 11 our time a researcher who goes by MalwareTech noticed the malware checked a long nonsense domain before doing anything, registered it, and that turned out to stop new infections from detonating. Microsoft took the unusual step of putting out patches for XP, Windows 8 and Server 2003 on Friday night. Europol said on Sunday that it had hit more than 200,000 machines in 150 countries.

How it got into the office

The server was a Windows Server 2008 R2 box that runs their line-of-business application and also, because it was there, the file shares. It hadn’t been patched since last year. The application vendor’s support line had told them, at some point, not to install Windows updates because one had broken their software once, and that advice had been followed faithfully ever since.

That alone would have been survivable. The problem was the router. Some time ago the same vendor had needed to pull files off the server for support, and the easiest way to do that was to forward port 445 from the internet straight to it. Nobody ever closed it. So the server was an unpatched Windows box with SMB open to the whole internet, on the one day in years when that mattered most. My best guess, from the timestamps on the first encrypted files, is that it was hit around 9:40 in the morning by one of the random internet scans, before the kill switch domain was registered.

From the server it went sideways. The office network is completely flat, one subnet, every desktop can talk to every other desktop and to the server on every port. Seven of the desktops hadn’t had updates in months, because automatic updates had been switched off after a forced reboot ate someone’s spreadsheet. All seven got encrypted. The rest had been patched in April and were fine, which is the most convincing argument for Windows Update I’ve ever been able to show a client.

The weekend

We didn’t pay. The backups were the part I was most worried about, and they were half good. There was a nightly backup to a USB drive plugged into the server, which was mounted as a drive letter, which meant WannaCry encrypted it along with everything else. But about a year ago I’d talked them into a cheap cloud backup for the shared drive and the application’s database, with versioning, and Thursday night’s copy was intact. So they lost Friday morning’s work and nothing else.

Rebuilding took most of Saturday and part of Sunday. The server got a clean install, every patch, and the application reinstalled from the vendor’s media (I called the vendor’s support line on Saturday and very politely asked whether MS17-010 would break their software. They said no). The seven desktops got wiped and reimaged. The port forward came off the router, and I put the server and the desktops on separate VLANs with only the ports the application actually needs open between them. Updates are back on everywhere, set to install overnight so nobody loses a spreadsheet.

Our own fleet

Saturday night, once the client’s restore was running, I went through our own stuff, which I’ll admit I should have done Friday morning instead of answering texts.

We’re a SaaS company and nearly all of production is Linux, so the exposure there was small. The things I checked were the Windows machines we do have: a few build agents for the Windows version of a client we ship, a couple of utility servers, and everyone’s laptops. I checked every security group for 445 open to anything outside our VPC (none, good), ran Get-HotFix across the Windows servers to confirm the March or later updates were installed (all of them were, because they’re patched automatically), and asked people to check their laptops. Two weren’t up to date. One belonged to an engineer who’d paused updates during a deadline and forgot to unpause them. The other was a test machine under a desk that nobody had logged into since February. Neither had been exposed to anything, but both got patched on Saturday night. On Monday we pushed a group policy to turn off SMBv1 everywhere, since nothing we run needs it.

That was all fine, and mostly luck. It’s the client’s server I can’t stop thinking about. A vendor told a small business not to patch, and they didn’t, because why would they second-guess the people who wrote their software? I’ve heard the same story from people who run IT for hospitals and factories, and on Friday we all saw what it does at scale. Microsoft’s Brad Smith wrote on Sunday that governments stockpiling exploits is the root of this, and he’s got a point about the NSA. But the exploit was public for a month and the patch had been out for two, and that box was still never going to get it.

If you look after a small office, go and check your router for port forwards today. It takes ten minutes. (And if you want a related story, the card data scare at a friend’s stores a few years ago had the same flat network at the bottom of it.)