The vendor API that vanished in 30 days

The email arrived on a Tuesday at the end of May, from a no-reply address, with the subject line “Important changes to your plan”. It went to a shared inbox that three people read and nobody owns, and it sat there for six days.

It was from the vendor whose API fills in company details during our sign-up flow. You type in your work email, and by the time you’ve reached the next screen we’ve looked up the domain and pre-filled the company name, rough size, industry and logo, which saves people some typing and, more importantly for us, routes the sign-up to the right onboarding path. We’d been on their “startup” plan for three years, a flat monthly fee that was small enough that it went on a company card and nobody thought about it. The email said that plan was being retired on June 30th, and that we’d move to usage-based pricing. A link went to the new price list.

The same week, Christian Selig, who makes the Apollo app for Reddit, posted that under Reddit’s new API pricing, $12,000 per 50 million requests, his app would cost him about $20 million a year to run. I read that with a lot of sympathy and some smugness, because we don’t build on anyone’s platform like that. Then on the following Monday someone in finance forwarded me our email with “is this a problem?” and I worked out what the new price list meant for us. By the time we’d finished dealing with it, Selig had announced that Apollo would shut down on June 30th, along with several other third-party Reddit apps, and thousands of subreddits had gone dark in protest.

What we were actually using

The first question was how many calls we made, and we didn’t know. The vendor’s dashboard said 1.4 million in May. That seemed impossible. We get nowhere near that many sign-ups.

It took an afternoon to find out why. The enrichment call wasn’t only in the sign-up flow. Two years ago someone had added it to the internal admin page for an account, so our support team could see a customer’s company details, and that page called the API every time it loaded, with no caching. Then the sales team’s CRM sync started using the same helper function to fill in missing fields, once an hour, for every account that had an empty field, which was most of them, because the vendor simply doesn’t know about a lot of small companies and returns nothing. Every hour we asked about the same few thousand domains and every hour got the same empty answers. The sign-up flow, the thing we actually paid for, was about 3% of the calls.

At the new per-call price, May would have cost us a bit over 30 times what we’d been paying. Sign-ups alone would have cost a little less than the old plan.

Thirty days

We had about 24 days left by the time anyone looked at the email properly. The fixes for the waste were easy, if tedious to do in a hurry. We put a cache in front of the API with a 30-day expiry, which is about how often a company’s size or logo changes, and cached the empty answers too, for a week, so we’d stop asking about companies the vendor didn’t know. The admin page now reads from our own database, which the sign-up flow had been writing the results into all along. The CRM sync now only asks about accounts created in the last day. By the middle of June we were down to about 40,000 calls a month, nearly all of them real sign-ups.

That made the new price tolerable. The new pricing has a monthly minimum, so we’re still paying about half as much again as we used to, for the same thing, and we’ve signed up for it, because the alternative was replacing the vendor in three weeks, and I’ve done enough of those rushed migrations to know that’s how you end up with a different set of problems.

We looked at alternatives anyway, in parallel. There are at least four other companies selling roughly the same data. Two of them had better coverage of small companies in our test of 200 domains, and one was cheaper. None of them could be switched to in less than a sprint, because the response formats differ and our onboarding routing depends on how each one categorizes industries. That’s the part I’d been underestimating: the API call is ten lines of code, but the categories it returns had leaked into our routing rules, our analytics and a couple of dashboards the sales team lives in.

The question I can’t answer

After Log4Shell we built a spreadsheet of every vendor whose software touches our systems. After our bank failed in March it got a column for which bank is behind each card or invoice, and in April one for whether it’s quietly sending our data to an AI model. This vendor was on it, row 31, with an owner, who’d left the company last year, and a renewal date, which was blank because the plan was month-to-month. Nothing on it would have told anyone that the vendor could change the terms with thirty days’ notice, or that the usage had grown thirty-fold without anyone choosing to grow it.

What I’d like is a rule for which dependencies are worth the effort of being ready to replace. Reddit’s API was, for Apollo, the whole business, so it’s easy to say Selig should have had a plan B, and I’m not sure what a plan B for “the platform your app is a client for changes its mind” would even be. Ours was a convenience on a sign-up form. Somewhere between those two is a line where it’s worth paying for a second vendor, or an abstraction, or a cache, or at least a call-count alert, before you need one. We’ve got about 50 rows on that spreadsheet that are some kind of API we call. I’ve asked each owner to write down, in one sentence, what happens to the product if that API disappears in thirty days. Eleven have answered so far, and three of the answers were “I’m not sure”, which is more honest than I’d have been a month ago.