The support bot that invented a policy

The draft said: “Good news! Deleted projects stay in your Trash for 30 days, so you can restore it yourself from Settings → Trash. Just click Restore next to the project name.”

We don’t have a Trash. We’ve never had one. When a customer deletes a project in our product, it’s gone from their side immediately, and getting it back means a support ticket, an engineer, and the per-tenant restore runbook we wrote in 2022, which takes a few hours on a good day. The draft was written by a language model, for a support agent to review and send, in a pilot we’d been running for about three weeks. The agent had read it, fixed a typo in the customer’s name, and had the cursor on Send.

Last week’s version

If you missed it, Cursor, the AI code editor, had a version of this go public last week. Users started getting logged out when they switched between machines, one of them emailed support, and the reply, from “Sam”, explained that Cursor was designed to work with one device per subscription as a core security feature. There’s no such policy. Sam was an AI model answering support email directly, the logouts were a bug in a session-security change, and by the time a cofounder posted on Reddit to say “we have no such policy”, people were publicly cancelling. They’ve since said AI-written support replies will be clearly labeled.

The difference people keep pointing to is that Cursor’s bot sent its answers on its own, and that a human in the loop would have caught it. We had a human in the loop. She very nearly didn’t catch it.

The pilot

Some background. We’ve had a model-written summary at the top of long support tickets since 2023, and it’s popular with the team. Early this year we started a pilot of the obvious next step, a drafted reply. When an agent opens a ticket, the system retrieves the help center articles that look relevant, puts them in the prompt with the ticket thread and some instructions, and drafts an answer. The agent edits it and sends, or throws it away and writes their own. Our AI policy has said since 2023 that nothing goes to a customer that a model wrote and a person didn’t read, and the pilot was designed around that line. The model provider is the same one that already does the summaries, and it’s on our subprocessor list, so no new data was going anywhere new.

Four agents were in the pilot, on about a fifth of incoming tickets. In the first three weeks it drafted 437 replies. The agents sent about 60% of them with light edits, rewrote about a quarter, and threw the rest away. Handle time on the tickets that used a draft was down, though I don’t trust our measurement of that enough to quote a number.

What happened

The customer had deleted a project by mistake and wanted to know how to get it back. The retrieval step went looking for help articles about deleting and restoring things, and the closest match it found was our article on removing users from an account, which really does say that a removed user can be restored by an admin for 30 days. The model took that, the customer’s question, and what I can only assume is a strong prior that SaaS products have a Trash, and wrote a confident, friendly, specific answer to the question it was asked, with a menu path that doesn’t exist.

What stopped it going out was a second agent. We’d set up the pilot so that a lead reviews a sample of sent drafts each day, and as part of testing the process she was reviewing a few before they were sent, sitting in the queue next to the first agent. She knew there was no Trash because she’d handled three of these tickets the previous month.

The agent who nearly sent it is good at her job. When I asked her about it afterwards, she said the draft read exactly like our other replies, it mentioned a 30-day window like our real policies do, and she’d stopped checking the specifics in the drafts after the first week because they’d always been right. That last part is what I’ve spent the most time on since. The pilot was working well enough that the review had quietly become a skim.

Looking back through all 437

We went back through every draft the pilot had produced, two of us, over two afternoons. We were looking for any statement about what our product does, or what our policies allow, that wasn’t in the help articles the draft was given. We found nine, a bit over 2%. Five were small, like a keyboard shortcut that doesn’t exist or a setting described in the wrong menu. Four were policies. The Trash, a refund window longer than our real one, a claim that we’d extend a trial “on request” (we sometimes do, but it’s not a policy and agents aren’t supposed to promise it), and one that said exports include file attachments, which they don’t.

Seven of the nine were discarded or corrected by the agent before sending. One was the Trash. The other, the attachments one, had been sent, about two weeks earlier, and the customer hadn’t noticed, or hadn’t needed attachments. We wrote to them with a correction. That’s the only one that reached a customer, as far as we can tell.

Where it’s left us

The pilot is still running, with changes. Any sentence in a draft that makes a claim about a policy, a price, a time limit or a menu path now has to cite the help article it came from, and the agent sees the citation next to the sentence. A claim with no citation is highlighted. That’s mostly a prompt change and a bit of UI, and in the weeks since, it’s made the review slower, which I think is the right direction. We’ve also added the nine bad drafts to the eval set we’ve been building since 2023, so any future change to the prompt or the model has to get them right.

The skim is the part a prompt change doesn’t touch. The better the drafts get, the less carefully people will read them, and the mistakes that are left will be the plausible ones, in the same tone as the right answers. The highlighting is our attempt at that: it doesn’t ask the agent to read everything harder, only to look at the handful of sentences with nothing behind them. In the first two weeks it flagged a claim in about one draft in eight, and the agents corrected three more invented details that way, all of them menu paths. Those three went into the eval set too, which is twelve bad drafts now, each one a test that any future prompt or model has to pass.