Building a data team at a startup

Somewhere around the fiftieth employee, someone senior comes back from a conference and says we need a data scientist. It happened to us this year. The CEO had seen a talk about a churn prediction model, and he wanted one, and he wanted to know why we didn’t have one already.

The honest answer was that we couldn’t have built one if we’d hired the best data scientist in the world that week, because we couldn’t answer much simpler questions. How many active customers did we have last month? It depended on who you asked. Sales had a number from the CRM, finance had one from billing, and product had one from the app database that counted anyone who’d logged in, including our own support staff impersonating customers. They were within 10% of each other, which is close enough to sound right and far enough apart to start an argument in every board meeting.

So we spent most of this year building a data team, small, and it went better than I expected, mostly because we did things in the unglamorous order. This is what I learned, with the caveat that I’m an engineering leader whose own data science credentials consist of Andrew Ng’s online class in 2011 and a lot of SQL.

The first hire is not a data scientist

The first person we hired was what people are starting to call a data engineer, although the job description we posted was a page of guesses. Their job was to get data from where it lives into one place where you can query it, and to keep it there reliably. That meant nightly loads from our production Postgres, the billing system’s API, the CRM’s export, and our event tracking, all into a Redshift cluster. Six months in, that pipeline is still most of their job.

I’d hire this person first every time, ahead of a data scientist, for two reasons. Every analysis anyone will ever do depends on the pipeline being right, and when it’s wrong, it’s wrong quietly. A broken web server pages someone. A broken nightly load just means the dashboard shows that revenue went down 30% on Tuesday, and someone in a meeting makes a decision about it. The other reason is that a data scientist hired into a company without that plumbing will spend their first year building it, badly, and resenting it, and then leave.

The second hire was an analyst, someone good at SQL and good at talking to non-technical people, and better at the second than the first. Most of the value in the first year has come from this person sitting with sales, support and finance and turning vague questions into precise ones. “Which customers are at risk?” turned into “which accounts on an annual plan have had fewer than three active users in each of the last four weeks?”, and that question you can answer with a query, and it turned out to be about 80% as useful as the churn model the CEO wanted.

We haven’t hired a data scientist yet. We will next year, probably, and now they’ll have something to work with.

The thing that matters most is a definitions doc

If I had to pick one artifact from the year, it’s a shared document called “Metric definitions” that says, in plain English and then in SQL, exactly what we mean by an active customer, a churned customer, monthly recurring revenue, a trial conversion, and about twenty other things. Every dashboard links to it. When someone disagrees with a number, the argument is about the definition, which is an argument you can have once and settle, instead of about whose spreadsheet is right, which you have every week.

Getting the definitions agreed took longer than building the pipeline. The definition of “active customer” alone took three meetings and a short, slightly tense email thread between the heads of sales and finance. I’d do it again in exactly the same way, except earlier.

Things I’d tell the CEO not to buy

Don’t build a Hadoop cluster. We’re nowhere near big enough to need one, and almost nobody at our stage is. All our data, every event since we started tracking, fits comfortably in a small Redshift cluster, and could probably fit in a big Postgres box. When a vendor tells you that you need “big data” infrastructure, ask them to tell you how big your data is. (We’ve been here before: last year we turned off an EMR job that a shell pipeline on one box could replace.)

Don’t buy the expensive all-in-one analytics platform yet. We looked at three. They were all impressive in the demo, because the demo data was clean. Ours isn’t, and a tool can’t fix that for you. We use a plain SQL-based dashboard tool and a lot of saved queries, and I expect we’ll outgrow it, which is fine.

Be careful with real-time. Everybody asks for real-time dashboards at first. Almost nobody makes a decision that changes because a number is an hour old instead of a second old. Nightly was fine for the first six months, and hourly is fine now. Real-time is a lot more engineering for very little benefit, and every bit of complexity in a pipeline is another way for it to be quietly wrong.

Where the data team lives

We put the data team in engineering, reporting to me, and I’m not sure that was right. It means the pipeline is built well, with code review, tests on the transformations and monitoring on the loads. But it also means the analyst’s best customers, sales and finance, sometimes feel like they’re filing tickets with IT. A few companies I’ve talked to put the analysts in the business teams and the data engineers in engineering, and I can see the appeal. I don’t think there’s a right answer at our size, and I’d rather the analyst spend half their week sitting with sales than move the reporting line.

The churn model, by the way, is on the plan for next year. The CEO is still asking.