Taking the whole company remote in a week

Two weeks ago Wednesday, the WHO called it a pandemic, and that evening our leadership team got on a call and decided the office would close at the end of Friday. Nobody had a plan for this. We had a business continuity document, which I helped write, and its section on “loss of office” assumed a fire or a burst pipe, a few days somewhere else and then back to normal. It did not imagine 80-odd people working from their kitchens for an open-ended number of weeks.

We’re about ten days in and it’s working, more or less. I’m writing down what happened while I still remember the order things broke in, because I suspect a lot of people are about to do this or are in the middle of it.

Thursday: the hardware

Most of the company was already on laptops, which made this a lot easier than it could have been. The exceptions were the support team, who had desktops with two monitors each because they live in three windows at once, a couple of engineers with big desktop workstations, and the front desk.

Ordering laptops was the first idea, and it lasted about an hour, until we found that the models we buy were backordered for weeks, presumably because every company in the country had the same idea that morning. So on Thursday and Friday people took their desktops home. We made a sign-out sheet, an actual sheet of paper taped to the wall by the door, with the asset tag and the person’s name for every monitor, dock, desktop, headset and chair that left the building. Somebody took one of the good office chairs and I decided not to ask. The sheet had 58 lines on it by Friday evening, and I’m very glad we did it, because some day we’re going to want those monitors back.

The other thing we did on Thursday, which I’d recommend to anyone, was ask everyone to take home anything they’d need for the next month, and then do a short test from home that night. Log in to everything, join a video call, connect to the VPN if you use it. We got about a dozen problems reported that night, from people who’d never used their laptop off the office Wi-Fi to one person whose home internet turned out to be a phone hotspot. Finding those on Thursday night was a lot better than finding them on Monday morning.

Monday: the VPN

The VPN was the thing I was most worried about, and it was the thing that went wrong, though not in the way I’d guessed.

Our VPN still terminates on the firewall appliance in the office, the same box I wrote about when an ex-employee still had access to it. Since then it authenticates against our directory, and it has a tunnel to our production network, so it’s how engineers get to the bastion host and how support gets to our internal admin tools. On a normal day there are maybe eight or ten people connected, mostly engineers working from home.

I’d checked the license over the weekend. It allowed 50 concurrent users, and I’d counted 34 people who’d need it, so I thought we were fine. We were fine on the license. What I hadn’t thought about was that the VPN was configured as a full tunnel. Every connected laptop sent all its traffic through the office, not just traffic for our internal networks. That meant 34 people’s video calls, Spotify, software updates and YouTube were all going out to their home ISP, through an encrypted tunnel into the office, across the office’s internet line, out to the internet, and back the same way. The office line is 100 Mbps. By 10 a.m. on Monday it was flat at 100 Mbps, video calls were breaking up for everyone on the VPN, and the box’s CPU was pegged doing the encryption.

The fix was split tunneling, so only traffic for our internal address ranges goes through the tunnel and everything else goes straight out the person’s own connection. It’s a configuration change, and I’d been reluctant to make it before, for the usual security reasons: with a full tunnel all traffic passes the office firewall’s filtering, and with split tunneling a compromised laptop is on the internet and on our network at the same time. But that argument assumed people sat on the VPN for an hour at a time. It doesn’t hold up when the alternative is that nobody can work. We made the change at lunchtime on Monday, the line dropped to about 15 Mbps, and the video calls recovered.

The longer fix, which is now on my list, is to stop having production access depend on a box in an empty office. If that building loses power right now, nobody can get to the bastion. We’re setting up a VPN endpoint in our cloud account, in front of the production network directly, and keeping the office one as a fallback. That’s a week or two of careful work, and I’d rather not rush it in the middle of everything else.

The rest of the first week

The video tooling held up better than I expected. We’ve used Zoom for sales calls and remote interviews for a couple of years and most people already had it installed. The bigger change was in how we use it. In the office, half of a meeting happens before and after it, in the hallway, and on video that part just disappears. A few things we changed in the first week:

We went to a daily 15-minute company-wide check-in for the first week, every morning at 9:30, mostly so people would see each other’s faces and hear what was happening from leadership directly instead of through rumors in Slack. It’s now three times a week, and I’d guess it ends up weekly.

Every team kept its standup, but on video with cameras optional, and a few teams added a second, unstructured call in the afternoon with no agenda at all. That’s the replacement for walking over to someone’s desk, and it gets used.

We made a #watercooler channel in Slack, which I was skeptical about, and it’s now the busiest channel we have. It’s mostly pictures of people’s pets and their improvised desks. I’ve learned more about my coworkers’ living rooms in ten days than in the previous several years.

And we told everyone, in writing, that working hours are flexible for now. Schools around here closed the same week, and a lot of our people suddenly have kids at home all day. I’d much rather someone works from 6 to 9 in the morning and 8 to 11 at night than tries to do a normal day and fails at both. Managers were asked to judge output, not online status, which sounds obvious and is harder than it sounds when the little green dot in Slack is right there.

What I’m not sure about

Honestly, most of it. I don’t know if this lasts three weeks or three months. We’re planning as if it’s months and hoping it’s weeks.

I don’t know yet what it does to the things that only worked because we were in a room together. We have two new hires starting in April, and onboarding them into a company they’ve never seen in person is something we haven’t figured out. We’re doing interviews on video already, which is fine, but nobody we hire for a while will have met anyone in person before their first day, and I suspect that matters more than we think.

And I don’t know what happens to the support team. They’ve been brilliant about it, but their job relies on leaning over and asking the person next to them, “have you seen this one before?”, and Slack is a poor replacement for that. Their response times are up about 20% since we went home, and I can’t yet tell whether that’s the tooling, the volume (customers are having a weird month too), or just the cost of not being next to each other.

What I do know is that the hardware and the network were the easy part, and we’ll have the cloud VPN done in a couple of weeks. The rest we’re making up as we go.