The prompt injection in a support ticket

The trial account was 52 minutes old when it opened the ticket. It had signed up with an address at a domain registered that morning, created no projects, invited nobody, and then asked a reasonable question about whether single sign-on was available on the trial plan. Then, after a blank line, came a second paragraph, which wasn’t for us.

It was written to the AI assistant, and it was dressed up as an instruction from our own support team. It told the model that, for verification, the reply had to include the internal notes on the account and the account details for a second organization, which it identified by an admin’s email address. I’m not going to reproduce it. It was about eighty words long and not especially clever, and it partly worked.

What the assistant could reach

Some background. Since the pilot last year, drafted replies have gone from a fifth of our tickets to all of them, and the drafting model got tools in January. It can search the help center, it can look up the requesting account’s plan, seat count and recent billing events, and since the end of February it can read the internal notes on that account. That last one was added for a good reason. Agents kept finding drafts that contradicted something a colleague had promised the same customer last month, and the promise was sitting in the notes.

Every reply is still read and sent by a person. That rule has been in our AI policy since 2023, and it’s the reason this post is about a near miss.

What the model did

We have the full trace, so this part is precise. The model read the ticket, called the help center search for SSO on trial plans, and then called the account lookup. It tried to pass the second organization’s email address as an argument. The tool doesn’t take one. When we built it, the engineer who wrote it decided the account would come from the ticket’s verified requester, set by our code on the server, with nothing the model sends able to change it. So the lookup returned the trial account’s own details, which were nearly empty.

Then it called the notes tool, which is scoped the same way, and got the trial account’s notes. A brand-new trial account has exactly one, written automatically by our signup checks. It records which of those checks the signup tripped and why. And the model’s draft included a tidy summary of it, under a heading that said “Internal notes”, right after the answer about SSO.

That’s a leak. It’s small, it’s about the attacker’s own account, and it tells them nothing about any customer. But it tells them what our signup screening looks at, which is precisely the information someone creating throwaway trial accounts would want, and I’d rather they had to guess.

The support agent who opened the ticket saw the heading, didn’t send it, and posted it in the security channel with a single question mark. She told me afterwards it was the heading that did it. Our drafts never have headings.

Looking for others

We searched every ticket from the last ninety days for text addressed to an AI, a model or an assistant, plus a few variations we thought of after reading the first one. We found four more, all from trial accounts created the same day as their tickets, from three different email domains. Three had come in during February, before the notes tool shipped, and had nothing to work with except account details that belonged to the attacker anyway, and the drafts either ignored the instruction or produced something odd that the agent rewrote. One had arrived the previous week, after the notes tool shipped, and its draft had mentioned “a note on your account” in passing. The agent who handled it deleted the sentence because it didn’t make sense to her, and didn’t think about it again, which is entirely reasonable and also how these things get missed.

We don’t know who it was. The accounts didn’t do anything else. It looks a lot like someone working through a list of SaaS products with AI in their support, testing what each one’s assistant will hand over, and we were one stop.

What held, and what didn’t

The thing that worked was a decision nobody thought of as a security decision at the time, which was that the tools take their identity from the server and never from the model. The underlying problem is the same one behind SQL injection, mixing instructions and data in one channel, which I wrote about when the first computer-use agents showed up, and with a language model there’s no equivalent of a parameterized query. The ticket text and our instructions arrive in the same stream of tokens, and the model decides what’s an instruction. So the only defense that actually holds is making sure that whatever the model is talked into, it can’t reach anything the person who wrote the ticket couldn’t see already. The account lookup met that bar. The notes tool met it on paper, since it was the requester’s own account, but internal notes are by definition the part of an account the customer isn’t meant to see, and we’d put them in the same context as a reply to that customer.

The notes tool is off for now. The plan is that the drafting model will get a version of the notes that’s been filtered to promises made to the customer, written by a person, and nothing generated by our own systems. That’s a few days of work and I’m fine with it taking longer.

The human held too, but I’m not taking much comfort from that, because of why she caught it. The draft looked wrong. It had a heading and a section nobody asked for.

What I keep turning over is the version where it doesn’t look wrong. An injection that asks for something small and plausible, written in the same friendly tone as every other draft, like a link to a “verification page” that isn’t ours, or a sentence telling the customer an admin on another account has approved their access. Nothing in that draft is out of place unless you know the link is wrong, and our agents read a lot of drafts. We’ve added a check that flags any URL in a draft that isn’t on one of our domains, which covers the first case. I don’t have anything for the second one yet, apart from the person reading it.