Leaving the colo cage

A few weeks ago I handed back the keycard to our colo cage. We’d had it for about six years, and for the last three of them it had mostly been a monument to things nobody wanted to move.

When we moved our main product onto EC2 a few years ago (I wrote about one of the less fun days that came with it), we moved the customer-facing app and its database and left everything else where it was. “Everything else” was two racks with four VMware hosts running about forty VMs, plus a handful of physical boxes. Build servers, the internal wiki, the old reporting database, a file server, an FTP server for the two customers who still sent us data that way, monitoring, and some things we’d find out about later. The plan in January was to have it all out by the end of March. It took until the middle of September.

The inventory is the project

The first thing I’d tell anyone doing this is that the move itself was the easy part, and finding out what we actually had took about half of the total time.

We had a head start, or thought we did. After the Shellshock weekend last fall we’d started a list of every machine we own, and it had all forty VMs on it, with “ops” as the owner of half of them, which turned out to mean nobody. It was about 80% right on what existed and a lot less right on what anything did, since half the VMs had names like util02. So we did it the slow way. For every VM we looked at what was listening on which ports, what connected to it (we ran netstat on a schedule for two weeks and diffed it), and what was in its crontab, and then we found a human who’d admit to owning it. Seven VMs had no owner at all. We shut them down one at a time and waited a week for anyone to notice. Five of them, nobody did. The other two turned out to be generating a weekly revenue report that went to our finance person, who noticed on the Monday, and a license server for a piece of desktop software the support team used.

The physical boxes were worse. One of them, a 1U server with a sticky note on it that said “DO NOT REBOOT”, was the NTP server for everything in the cage, and also, for reasons nobody could reconstruct, the SMTP relay for the build server’s email. I like to think every colo cage has one of those.

Moving the data

The data was bigger than we thought, because nobody had looked at the file server’s disk usage in years. About 4TB in total. Our uplink from the cage was 100Mbps, so that’s 32 terabits at 100 megabits a second, about 3.7 days of transfer if nothing else is using the link and nothing goes wrong. Both of those assumptions were false. We ended up copying the file server to disks and shipping them to Amazon with their Import/Export service, which worked, and was more paperwork than I expected. Amazon announced a rugged 50TB box they’ll ship to you, Snowball, at re:Invent a couple of weeks ago, which would have been exactly what we wanted in April.

For the VMs we mostly did not try to convert the VMware images. AWS has a VM import tool and we tried it on a few, and it worked on some of them, but every one we moved that way came with ten years of whatever had been installed on it. For anything we cared about we built a fresh instance with our configuration management and moved the data and the app. For things we didn’t care about, we asked whether they needed to exist, which is how the FTP server finally died (we gave those two customers an SFTP drop that lands files in S3, and one of them thanked us).

The bill

I’d done a cost model before we started, and the steady-state AWS number came out a bit lower than the colo plus the VMware support contract plus a hardware refresh we’d otherwise have needed next year. That’s roughly what we’ve ended up at, but the first three months were about 40% over the model. Some of it was running both sides at once, which I’d budgeted for, though not for six months. The rest was the stuff everyone warns you about and you underestimate anyway: EBS snapshots that nobody was expiring, test instances left running over a weekend, and data transfer charges for the replication between the cage and AWS while both were live. Buying reserved instances once we knew what we needed got the bill back to the model. I’d wait until you’ve been running in AWS for a month or two before buying them, because our instance sizes changed a lot in the first month.

The colo contract was its own small lesson. It auto-renewed annually with 90 days’ written notice, and our renewal was at the start of October, so we had to give notice in early July without being sure we’d be out in time. We were, with about two weeks to spare. If I’d read the contract in January instead of June, that date would have been at the top of the project plan.

The people part

This is the part I didn’t plan for well. We had one ops engineer who had built most of what was in that cage, racked every server, and was the only person who knew which cable went where. Moving to AWS made a lot of what he knew less useful, and he knew it before I did. For the first couple of months he was the slowest person on the project, and I read that wrong. I thought he didn’t care about it. It turned out he was worried that the job he was good at was going away.

What helped was being explicit. We sat down and talked about what his job would look like after the cage was gone, more automation and more AWS, less hardware, and I paid for him to do the AWS certification training. He ended up running the hardest part of the move, the reporting database, and he’s now the person on the team who knows our AWS setup best. But I lost about two months by not having that conversation in January, and if you’re leading one of these moves, have it before you announce the project, not after.

Last day

We wiped every drive before it left the cage, which took two days by itself. We sold the newer servers to a reseller, recycled the older ones, and kept one as a test box in the office, mostly out of sentiment. The last thing to come out was the “DO NOT REBOOT” server, which we rebooted, and nothing happened. I took the sticky note home.