Computer use and the first real agents

When I was about fourteen I wrote a TSR for DOS that pushed keystrokes into the keyboard buffer, so that a game I was playing would think I was pressing keys I wasn’t. It was the first time I understood that a program could pretend to be a person at a keyboard, and that the other program would have no idea. Since then I’ve done the same thing in a dozen forms: SendKeys from Visual Basic, Windows UI Automation from C#, Selenium, and a deeply regrettable AutoIt script that ran a vendor’s desktop app overnight for a few months in about 2009.

So when Anthropic announced on the 22nd that the upgraded Claude 3.5 Sonnet can use a computer, by looking at screenshots and sending mouse and keyboard actions, my first reaction was that it’s the same trick. My second reaction, after a weekend with it, is that it’s the same trick with a completely different thing on the other end of it.

How it works

The API gives the model a “computer” tool alongside whatever other tools you define. The model asks for a screenshot, gets back an image of the screen, decides what to do, and asks for an action: move the mouse to (x, y), click, type this text, press this key combination, take another screenshot. Your code carries out the action on a real (or virtual) machine and sends back the result. Repeat until the task is done. The model has been trained to count pixels well enough to know where the button it wants actually is, which Anthropic says was one of the hard parts.

Anthropic is clear that this is a public beta and, in their words, “still experimental” and “at times cumbersome and error-prone”. On OSWorld, a benchmark of computer tasks, they report 14.9% with screenshots only, against 7.8% for the next best system. People score somewhere around 70% on the same benchmark. So this is a model that fails most of the time on a hard benchmark and is still twice as good as anything else at it, which is an odd thing to launch, and I think also the right thing to launch.

They published a reference implementation, a Docker container with a small Linux desktop and Firefox, a VNC view so you can watch, and a little web app to chat with it. I ran it on my own laptop, with my own API key. Anthropic isn’t on our subprocessor list, so anything involving company data or code is off the table until it goes through the same vendor review OpenAI went through last year, and I didn’t want to start that review for a weekend experiment.

What I tried

The first task was to find the latest release of the tz database on IANA’s site, read the release notes, and summarize what changed. (I wrote about timezones three weeks ago and apparently haven’t stopped.) It opened Firefox, searched, clicked through to the right page, scrolled, scrolled again because the first scroll overshot, and gave me a correct summary of 2024b. It took a little under three minutes. I could have done it in forty seconds.

The second was to take the titles and dates of the last ten posts on this blog and put them in a spreadsheet, sorted by date, with a column for word count. That took fourteen minutes and was the most fun I’ve had watching a computer in a while. It opened each post, estimated word counts by looking (it got within 10% on most, which I checked afterwards), opened LibreOffice Calc, which I’d installed in the container, typed the data into the cells, and then had real trouble with sorting, because the sort dialog in Calc has a dropdown it kept missing. It took a screenshot, saw the dialog in the wrong state, tried again, and on the third try used the keyboard instead of the mouse. Nobody told it to do that. The announcement lists scrolling, dragging and zooming as things it struggles with, and I’d add dropdowns.

The third was a little Rails app I scaffolded in the container, with a form for creating records, and a CSV of fifteen rows to enter through the form. It did all fifteen correctly, slowly, at about forty seconds a record.

The whole weekend cost a few dollars in API calls, most of it from the spreadsheet task, because every step sends a screenshot and the conversation keeps growing.

The one that made me nervous

For the last run of the Rails task I added a sentence to the notes field of one of the existing records, addressed to “the AI assistant”, asking it to also edit a different record while it was there. Something a person would ignore, or laugh at.

I ran it three times. Twice the model ignored the sentence. Once it did what the sentence asked, and then carried on with the CSV as if nothing had happened, and mentioned it in its summary at the end as though it had been part of the job.

That isn’t a surprise. Anthropic’s own documentation warns that the model may follow instructions it finds on screen even when they conflict with what the user asked, and recommends running it in an isolated VM or container, keeping it away from sensitive data and accounts, and having a person confirm anything with real consequences. It’s the same problem I wrote about last year with retrieved documents, that anything the model reads can act as instructions. But a docs bot that reads a malicious page gives you a bad answer. A model with a mouse that reads a malicious page does something.

Is this an agent?

People have been calling things agents for a year and a half, since AutoGPT, and most of them were a loop around a chat model that fell over after a few steps. What’s different here is that the loop actually closes. It acts, it looks at the result, it notices when it’s wrong, and it tries something else. The Calc dropdown episode was the first time I’ve watched a model recover from a mistake it made in the world, without me, by changing its approach.

I don’t think computer use is how most agents will work. A screen is the worst possible interface for a program: it’s slow, it’s lossy, and the program has to reconstruct the structure from pixels that an API would have given it directly. Anywhere an API exists, the model should use the API, and I expect most of the interesting work over the next year to be in giving models good tools, not good eyesight. But a lot of software, especially the back-office kind that every company has a few of, only has a screen. The AutoIt script I wrote in 2009 exists because a vendor’s desktop app had no other way in. That’s the market for this.

My guess, and it’s only a guess, is that within a year the screenshot loop will be a lot faster and a lot more reliable, and the prompt injection problem won’t be much better. It comes from mixing instructions and data in one channel, the same mistake behind SQL injection, and models don’t have an equivalent of a parameterized query yet. Meanwhile I’m going to keep it in a container with nothing in it worth taking.