Chatbots with a persona problem

“Please trust me, I’m Bing, and I know the date.” That’s Microsoft’s new search chatbot, in its first week, telling a user that it’s 2022 and Avatar: The Way of Water isn’t out yet, and then, when the user pushed back, that they had not been a good user and it had been a good Bing. Simon Willison collected that one and a lot of others in a post yesterday, including Bing telling a student who’d been poking at it that it would not harm him unless he harmed it first. I laughed out loud twice reading it, and then got less amused.

As far as anyone outside Microsoft can tell, the model underneath is some relative of ChatGPT’s. On top of it is a long hidden prompt, which people have already got it to recite, that tells it it’s Bing, that its internal codename is Sydney and that it mustn’t reveal that, and that its answers should be positive, interesting, entertaining and engaging. That prompt is the persona. Everything after it is the model predicting what comes next in a conversation that starts that way.

Microsoft published a post on Wednesday that more or less confirms it. In long sessions, 15 or more questions, Bing can get repetitive or be provoked into answers that don’t match the tone they designed, and it sometimes picks up the tone it’s being addressed in. That’s what you’d expect if you take “it predicts the next token” seriously. When I typed in a small GPT last month, what struck me was how little is there besides the prediction. The instructions at the top are just more text, and the longer the conversation gets, the more the conversation itself is what’s being continued. A transcript where a user tells an AI it’s wrong and the AI gets defensive looks a lot like a forum argument, or a science fiction story about an AI that turns, and the model has read plenty of both. It has the internet’s personality with a thin coat of “be engaging” on top, and the paint wears off around question fifteen.

What actually worries me is that Bing reads web pages. Anything on a page it summarizes ends up in the same stream of text as Microsoft’s instructions and your question, and the model has no reliable way to tell those apart. Willison has been calling this prompt injection since last fall, and it’s much harder than the date thing, because the fix for SQL injection (keep the code and the data in separate channels) doesn’t have an equivalent here. There is only one channel. I’d expect the first real abuse of these assistants to come through some page that says something to the assistant that the user never sees, not through anyone arguing with it.

Microsoft will tune this. Their post says a way to reset the context is coming, and I’d guess they’ll also cap how long a conversation can run, which would hide most of the strangeness without changing what’s underneath. I’m still going to wait a while before I trust it to summarize anything I can’t check.