The SSRF that reached our metadata service

What stopped me in the Capital One story was less the 106 million people than the mechanism, as Brian Krebs and others have pieced it together: a web application firewall that could be tricked into making requests on an outsider’s behalf, one of those requests going to the instance metadata service, and the metadata service handing back temporary credentials for the role the firewall ran as. That role turned out to be able to list and read a lot of S3 buckets. Amazon’s position is that this was a misconfiguration in Capital One’s firewall, not an AWS flaw, and that’s fair as far as it goes. But every EC2 instance has that metadata service, and a lot of us have code that fetches URLs for users.

For anyone who hasn’t run into it, server-side request forgery is the name for this class of bug. Your server makes an HTTP request to an address the user influenced, and the user picks an address that only your server can reach. On AWS the juiciest one is the metadata service, a link-local address that answers any process on the instance with no authentication at all, because the theory is that only code running on the box can talk to it. SSRF breaks that theory. Evan Johnson from Cloudflare is quoted in the Krebs piece calling SSRF the most serious vulnerability facing organizations that use public clouds, and I’m not going to argue.

So the week after, I asked the team a simple question: which of our services will fetch a URL that a customer gave us?

What we found

The list was longer than I expected. Webhooks, obviously, where customers give us a URL and we POST events to it. An “import from URL” option on file uploads. And link previews, the feature where you paste a link into a comment and we show the page title and an image.

The webhook sender was fine, mostly by luck. It runs in a separate small service with no IAM role attached at all, because the person who built it two years ago was careful, so the metadata service would have had nothing useful to give it.

The link preview service was the problem. It’s a little internal service that fetches the page, parses out the title and the preview image, and then fetches the image and resizes it. It ran on instances with the same IAM role as the main app servers, because when it was split out of the monolith nobody gave it its own role. The URL validation checked that the user’s link was http or https and that the hostname didn’t resolve to a private address. It did not check where redirects went. So a link to an outside page that redirected somewhere internal sailed through the validation and the fetch followed the redirect. One of our engineers confirmed this in staging in about twenty minutes: the preview card for a link he controlled came back with the name of our app server role in it, which is the point where we stopped testing and started fixing.

We went through our CloudTrail logs and the link preview service’s access logs for the previous 90 days, looking for any use of that role’s credentials from outside our own instances and for preview requests that ended up at internal addresses. We didn’t find any. I can’t prove nobody found it before us, only that we found no sign of it, which is a less comfortable sentence than I’d like.

What made this worse than it needed to be was the role. The app server role could read the bucket where we keep customer data exports. A credential leak from the preview service would have meant a credential that could read every export file we had.

What we did about it

The quick fix, the same afternoon, was at the network level on the preview instances: a host firewall rule so that only root can reach the metadata address, and the preview service doesn’t run as root. That one rule closed the hole regardless of what the application code did. I’d recommend it for any instance that doesn’t need its own credentials from inside the application process, and that’s more of them than you’d think.

Then the preview service got its own role with no permissions except writing to its own cache bucket. That should have been true from the start, and it’s the change that limits the damage from the next bug, not just this one.

Then the code. The validation now happens at connection time, not at parse time: we resolve the hostname ourselves, refuse private, loopback and link-local ranges, and connect to the address we checked, so a DNS answer can’t change between the check and the fetch. Redirects are followed manually and every hop goes through the same check. We also only fetch on ports 80 and 443, and cap the response size, which we should have been doing anyway.

The longer job, which isn’t done, is moving every outbound fetch on behalf of a customer behind one egress proxy that enforces all of that in one place, so the next team that builds a feature that fetches URLs doesn’t have to remember any of it. Webhooks and imports are moving over next.

I have some sympathy for AWS here. Google’s metadata server has required a special header on requests for a while now, which a lot of SSRF bugs can’t add, and Johnson has suggested AWS could do something similar but that it’d break existing code. I’d take the breakage. Until something like that exists, give every service that fetches URLs for customers its own role, with as close to nothing in it as you can manage.