GDPR week: what we actually changed

I got 61 “we’ve updated our privacy policy” emails between Wednesday and Friday. I know because I started counting on Wednesday, out of spite, and then felt bad on Thursday when I realized we’d sent one too.

GDPR came into force on May 25th. If you haven’t been near it, it’s the EU regulation on personal data, it applies to anyone handling data about people in the EU no matter where the company is, and the maximum fine is 4% of global annual turnover or €20 million, whichever is higher. That last number is the reason every company you’ve ever given your email address to wrote to you last week.

We’re a SaaS company with a decent number of European customers, so this has been on my plate since the start of the year. Not since 2016, when the regulation was actually adopted, which would have been the sensible time. Since January, like most people I know. This is a write-up of what we actually did, mostly for other people running small engineering teams who are still in the middle of it. The usual caveat applies: I’m not a lawyer, we paid one, and you should too.

The thing that shaped everything was figuring out which hat we were wearing. For the data our customers put into the product, we’re a processor: we handle it on their behalf and their instructions, and they’re the controller who decides what happens to it. For our own stuff (account signups, billing contacts, marketing lists, the analytics on our website) we’re the controller. The obligations are different, and once we’d sorted every piece of data into one bucket or the other, about half of the panicked list we’d started with went away, and the other half got a lot more concrete.

The first real piece of work was the data map, which is a grand name for a spreadsheet. Every system that holds personal data, what’s in it, where it physically lives, who can get to it, how long we keep it, and which hat applies. Before we started I’d have told you we had nine such systems. We found 21. The production database and its replicas, sure, and the backups. But also the application logs, which had email addresses in them all over the place; the error tracker, which captured request bodies; the support desk; the CRM; the email provider; the data warehouse; a feedback tool someone in product had signed up for with a credit card; and, my favorite, a handful of engineers’ laptops with copies of production data on them from debugging sessions months ago. Each of the external services also became a line on our list of subprocessors, which our customers now get to see, and each one needed a data processing agreement. Chasing those took longer than any of the engineering.

Deletion was the hard engineering part. GDPR gives people a right to erasure, and for us it mostly arrives as a customer telling us to delete one of their users, or all of their data when they leave. Deleting rows from the primary database is easy. The question is everything downstream. We ended up with a single deletion job that knows about every system on the map, and for the ones it can’t reach automatically, a ticket that goes to a person with a deadline. Logs were simple once we decided to cut retention to 30 days and scrub email addresses at write time, which we should have been doing anyway. Backups were the tricky one. You can’t sensibly edit a backup to remove one person, so what we did (and what our lawyer was comfortable with) is document that backups expire on a 35-day cycle, keep a list of deleted IDs, and make our restore procedure re-apply those deletions before a restored database ever serves traffic. I tested that restore procedure myself, and it found two bugs in the new step that re-applies deletions. Our monthly restore drill had never exercised that step, because until now there was nothing to re-apply.

Then there’s the consent banner trap, and I’ll admit this is the bit I care about most. Somewhere in March, marketing wanted a big “by using this site you consent to everything” banner on the website and in the app. That’s what a lot of sites have done this week. It’s also mostly wrong, as far as I can tell. Under GDPR, consent has to be freely given, specific, informed and an actual affirmative action: pre-ticked boxes and “by continuing to browse” don’t count. And consent is only one of six lawful bases for processing. Most of what we do with customer data is necessary to perform the contract, and some is legitimate interest, and neither one needs a banner. Asking for consent you don’t need is worse than not asking, because consent can be withdrawn, and then you’ve built a system where a user can click a button and switch off something the product needs to work. (The cookie part of the banner question is actually a different EU law, the ePrivacy directive, which made it all more confusing.) So instead of a banner, we took most of the marketing trackers off the app entirely, kept the ones on the website behind an opt-in, and wrote the privacy policy in sentences a person could read.

I didn’t expect to feel vindicated quite so quickly, but on Friday itself Max Schrems’ new group, noyb, filed complaints against Google, Facebook, Instagram and WhatsApp over exactly this, “agree or leave” consent. Meanwhile some American news sites, including the LA Times and the Chicago Tribune, simply stopped serving pages to European visitors. I understand why. It’s a lot of work. But it’s a strange look, telling 500-odd million people that you’d rather not have them as readers than write down what you do with their data.

The last thing on the list was the incident runbook. GDPR requires notifying the supervisory authority of a personal data breach within 72 hours of becoming aware of it, and as a processor we have to tell our customers without undue delay, so they can meet their own clocks. We added that step, with names next to it, and ran a tabletop exercise on a fake breach. We burned 20 of the 72 hours just working out who decides whether it counts as a breach, so that’s now written down too.

All in, it took about 14 engineer-weeks spread over five months, most of it in the deletion work, plus a lot of my own time chasing vendors for agreements. I think that’s cheap for what it bought. I now know where our data is, which I could not have honestly told you in December.

The part I can’t close off is the laptops. We asked everyone to delete their local copies of production data, and they did, and we’ve got a policy now and a sanitized dataset for debugging. But I have no way of knowing whether there’s a copy on a laptop that left the company two years ago. I suspect most companies don’t, and I suspect very few of them put it on their data map.