GraphQL, six years later

I was the one who argued for GraphQL, in a design review in the spring of 2018, so I read Matt Bessey’s Why, after 6 years, I’m over GraphQL with the slightly hunched posture of someone reading a review of their own work. It was on the front page of Hacker News yesterday. His examples are in Ruby with graphql-ruby, which is exactly what we use, and his six years are our six years almost to the month.

His argument, if you haven’t read it: he started using GraphQL in production in 2018, championed it, and now says “I would not recommend GraphQL to most people”. The reasons are the attack surface (every field needs authorization, and query cost is unbounded, so rate limiting is hard), performance (N+1 queries that a client can create without any backend change, so you end up wrapping everything in dataloaders defensively), and coupling, where authorization and batching logic end up living in the transport layer. For most teams he’d use a typed REST API described with OpenAPI instead, with generated clients. He’s careful to say REST has some of the same problems, just with simpler solutions.

I agree with most of it. Our six years were a bit more mixed than his.

Why we did it

In 2018 we were rebuilding the main web app’s front end in React, and the existing API was a pile of REST endpoints that had grown one screen at a time. The account settings page made eleven requests to render. Half the endpoints returned far more than the page needed and the other half returned too little, so the front end engineers spent a lot of time asking the back end engineers for “just one more field” on an endpoint, and waiting. GraphQL promised to fix that: one typed schema, the front end asks for exactly what it needs, and the type definitions for the client get generated from the schema.

That part worked, and still works. Our front end team will tell you, without me asking, that the generated TypeScript types are the best thing in the codebase. When someone removes a field, the front end build fails in CI, not in a customer’s browser. Building a new screen out of data that already exists in the graph needs no back end change at all, which in 2018 was a real bottleneck and today mostly isn’t.

What it cost

Authorization, first and worst. Bessey’s example is a user type where object-level checks are fine until some field on it (an email, a list of blocked users) needs its own rule depending on who’s asking. We had exactly that. In 2020 we found that the email field on our User type was readable by anyone who could see the user at all, which on a shared project meant any collaborator, including guest accounts. The REST endpoints had never returned email to guests, because nobody had written an endpoint that did. In GraphQL, nobody had to. It was a field on a type, and the type was reachable. Our front end never asked guests’ sessions for it, so the question was whether anyone had asked by hand. We went through the GraphQL query logs back as far as we kept them, about 60 days, and found no query from a guest session that selected email on another user. Since the GDPR work two years earlier, though, “we found no evidence” isn’t where you stop, so we told the affected customers’ admins what had been exposed, to whom, and for how long, and that we’d found no sign it had been used. We fixed it, and then spent two weeks going through every field on every type with a spreadsheet (of course), and we now have a rule that every field declares its authorization explicitly, with a CI check that fails if one doesn’t.

Then N+1. We wrapped most resolvers in batch loaders early, after a list page brought the database to its knees in the first month, but every so often someone adds a field to a list query in the front end that resolves through a path nobody batched, and that page gets slow in a way that never shows up in a back end PR. It’s the thing he describes, a performance problem created entirely by a client change, and it’s still the most common cause of our slow-page tickets.

And observability, which he doesn’t dwell on much. Every request is a POST to /graphql. Our APM tool, out of the box, shows one endpoint with a latency distribution that’s the sum of everything the app does. It took a surprising amount of custom instrumentation (operation names on every query, a span per resolver above a threshold) before the p99 charts meant anything again.

What we’re doing

Not ripping it out. The graph behind the web app is big, it’s well understood by the people who work on it, and the cost of replacing it is a year of work with no customer-visible result. The problems are known now and mostly fenced in.

But we’re not extending it either. Our public API has always been REST and will stay that way, because customers integrating with us want endpoints they can call with curl and cache, and they don’t want to learn our schema. New internal APIs between services are REST with an OpenAPI spec, and the clients get generated from that, which gets most of what I liked about GraphQL’s typed schema without making every field a public decision. Bessey mentions TypeSpec for writing those specs. One of our engineers tried it on a new service last week, and liked it more than writing the YAML by hand, which isn’t a high bar.

If I were making the 2018 call again with what I know now, I think it’d still be GraphQL, honestly, for that one case: one big product UI, many screens, a front end team that was waiting on the back end every week. I’d just go in knowing that every field is an endpoint, and budget for that from the first day instead of the third year.