The cloud account we forgot we had got suspended

The Tell HN post about Google Cloud suspending someone’s production projects at 1 a.m. on a Saturday has been on the front page since yesterday, with well over a thousand points. Their story is that a monthly payment failed, possibly because of a billing outage on Google’s side, they paid the balance by hand, billing support told them the projects would not be suspended, and then the projects were suspended anyway, in the middle of the night, on a weekend.

The comments are full of people saying the same thing happened to them, and of people saying “this is why you get an account manager and pay by invoice”. Both are right. What I want to add is our version, from June, which was smaller and entirely our own fault, and which I’ve been meaning to write up since.

Saturday

Our support lead messaged me a bit after 7 a.m. on a Saturday in June: a customer had written in to say they couldn’t finish signing up. The address field on the signup form just spun.

That address field uses Google’s Places autocomplete, and when you submit the form we geocode the address so we can set the account’s time zone and region. We don’t run anything else on Google Cloud. Everything we own is in AWS. But in 2017 one of our engineers set up a Google Cloud project to get a Maps API key, and when Google started requiring a billing account for the Maps APIs in 2018, he put his company card on it. The monthly charge is small, usually well under a hundred dollars, and it has never once come up in a planning meeting.

He left in March. His offboarding went by our checklist, the one we’ve had since the VPN account that outlived its owner: directory account disabled and later deleted, GitHub, AWS, the vault, and so on. Finance cancelled his company card on his last day, which is also correct. Google tried that card at the start of May and again at the start of June, and sent its warnings to the project’s billing admins, which meant to him, at an address that no longer existed. Early on that Saturday the billing account was suspended, the Maps APIs started refusing our requests, and the signup form’s validation, which required a geocoded address, rejected every signup.

Getting in

The first problem was that nobody could log in to the project. Its only owner was the deleted account. If you’ve never had to do this, that’s the moment your stomach drops. As it turned out, the project sat under our company’s Google Cloud organization, because it was created by someone signed in with a company account, and our IT lead is a super admin on the Google Workspace side, which lets her grant herself admin roles on the organization and from there on the project. That took about forty minutes, most of which was reading documentation on a phone, because the only person who’d ever used that console had left five months earlier.

The second problem was the billing. Adding a new card and paying the overdue balance was quick. Getting the billing account out of suspension took longer than that, and the APIs didn’t come back for a while after it said it was active again, long enough that we started wondering whether it had worked. Meanwhile one of our engineers shipped a change so that a failed geocode no longer blocks signup. We default the time zone from the browser and fix the region later. That went out at about 10:30 and the Maps APIs came back about half an hour after that.

From the first failed signup to the fix, it was a little over five hours on a Saturday morning. We lost roughly 40 signups that we know of, and our support lead emailed every one of them on Monday. Most came back.

The diagram

We have an architecture diagram. It’s pretty good, actually. Every AWS account, every service, the databases, the queues, the CDN, the third-party APIs we call on the request path. Google Maps is on it as a small box with an arrow labeled “geocode”. What isn’t on it, and what I don’t think is on anyone’s architecture diagram, is the thing that box depends on, which was one person’s credit card and one person’s email address.

It wasn’t missing from everything. It was in our GDPR subprocessor list, which says “Google LLC, geocoding”, because we wrote that list in 2018 and keep it current for legal reasons. And it was on our quarterly access-review sheet, sort of, as part of a row that said “Google” and that everybody read as meaning Workspace. Nobody was wrong, exactly. Each list was about something else, and none of them was about who pays and who gets the warnings.

Since June we’ve gone through every vendor we pay, which finance could pull from the card statements in an afternoon, and added two columns to the access-review sheet: which card or invoice pays for it, and which address gets its billing and security notices. Thirteen services were paid on a card in one person’s name. Four of those people have left, and their cards were already cancelled or would have been, which means four more versions of that Saturday were already on their way. Every one of them now bills to a shared finance card, and every one sends its notices to a group address that four of us are on. Anything with a billing admin role has at least two of them.

The bigger version of this is the one I haven’t solved. Our AWS bill is paid by invoice now, with an account team we can phone, and AWS has always been reasonable with us. But everything we have runs on one cloud account structure, paid by one company, under terms that let the provider suspend it. The Tell HN story is about a company that paid its bill and got suspended anyway. If the same thing happened to our AWS organization at 1 a.m. on a Saturday, we would have backups, in another account and another region, of all our data, and nowhere to run any of it.

I don’t think running on two clouds is the answer for a company our size, the cost and the complexity would be bigger than the risk. But I’d like to at least have the conversation with an account manager before we need one, and know what our phone number is. Right now I’m not sure I could tell you.