The root certificate on our sales laptops

Last Thursday morning one of our engineers dropped a link in chat with the message “uh, didn’t we buy Lenovos for sales?” We had. Four of them, back in November, consumer models, because we’d hired faster than planned and they were cheap and in stock at the store down the road. The link was to the Lenovo forum thread and the early write-ups about Superfish, which by then had been confirmed as adware that Lenovo preloaded on consumer laptops sold in the last few months of 2014.

If you missed it, Superfish is a “visual search” thing that injects shopping ads into web pages. To inject ads into HTTPS pages it has to be able to read them, so it uses a library from a company called Komodia that sits in the Windows network stack, installs its own root certificate, and then mints a certificate on the fly for every site you visit. Your browser sees a perfectly valid certificate for your bank, signed by a CA it trusts, because Superfish told Windows to trust it. That alone would be bad. What made Thursday bad was that the same root certificate, with the same private key, was on every affected machine, and by mid-morning Robert Graham at Errata Security had pulled the key out of the software and cracked its password. The password was komodia.

So as of Thursday, anyone who wanted to could sit on the same coffee shop Wi-Fi as one of our salespeople and impersonate any HTTPS site to them, our own admin console included, with a certificate their laptop would accept without a warning. Our sales team spends a lot of time on coffee shop Wi-Fi.

Thursday

The first thing was to find out which machines were affected, and the quickest way turned out to be Filippo Valsorda’s test page, which tries to load a resource with a certificate signed by the Superfish key and tells you whether your browser accepted it. We had the four salespeople open it, which was its own small adventure because two of them were out with customers. Three of the four laptops failed. The fourth had been reimaged in December after a hard drive problem, from a plain Windows install, which nobody had planned.

On the three bad ones you could see it plainly once you knew to look. Run certmgr.msc, open Trusted Root Certification Authorities, and there’s an entry issued to “Superfish, Inc.” sitting between the real CAs. There was also “Superfish Inc. VisualDiscovery” in the installed programs list, which I’m almost sure I scrolled past when I set those machines up.

We pulled all three machines in that afternoon. Uninstalling the program doesn’t remove the certificate, and at that point nobody was sure what else the installer had touched, so we didn’t try to clean them in place. We backed up their documents, wiped them, and reinstalled from a clean Windows image we already used for engineering laptops. That took most of the evening. Microsoft pushed a Windows Defender update on Friday that removes Superfish and its certificate, and Lenovo put out a removal tool, so if you have a lot of these the in-place route is more reasonable now than it was on Thursday afternoon.

Then the uncomfortable part. These laptops had been intercepting their owners’ TLS traffic since November, and since Thursday the key was public. We had no evidence anyone had used it against our people. We also had no way to prove that nobody had. So we treated every credential those three people had typed into a browser as exposed: our own product, email, the CRM, the expense system, the shared sales logins we had been meaning to get rid of anyway. We reset the passwords, killed all their sessions, and made them set up two-factor on everything that offered it. They were nicer about this than I would have been.

One thing I learned while checking our own site. We’d added HSTS last year (see PRISM and the systems we build), and I’d half assumed that, plus certificate pinning in Chrome for the big sites, would help here. Neither does. HSTS only makes sure you use HTTPS, and Superfish’s fake certificates are HTTPS. Chrome deliberately doesn’t enforce pins against roots that were installed locally, so that corporate proxies and antivirus products keep working. Which is exactly the category Superfish lives in.

What we changed

It’s a short list. New laptops get wiped and reinstalled from our own image before anyone uses them, whoever they’re for and whatever brand they are. We buy the business lines rather than whatever’s on the shelf (ThinkPads weren’t affected, as far as anyone has said). And we added a check to our laptop setup script that dumps the trusted root store and diffs it against a known-good list, which flagged one more thing on an engineer’s machine, a root left behind by an old debugging proxy he’d forgotten he installed. That one was harmless, but it’s the same shape of problem.

The other thing I’d say is that Superfish isn’t unique. Over the weekend Hanno Böck found that PrivDog, which ships with some Comodo products, of all things, does something similar and arguably worse, and the Komodia library has apparently turned up in other products too. Filippo also showed that Komodia’s own certificate checking is broken, so in some cases it will turn a certificate your browser would have rejected into one it accepts. Anything that does TLS interception on the client is in a position to get this wrong, and a fair number of “security” products do it. If you run a team, go look at your own root stores this week. It took us ten minutes per machine to check, and much longer to clean up the three we’d missed.