Vercel Labs' agent-desktop Lets Coding Agents Drive Native Apps Through the Accessibility Tree. Here Is the Test I Would Run Before Trusting It.
Vercel Labs now has a tool called agent-desktop, an Apache-2.0 command-line program and C library that lets an AI agent read and operate native apps on macOS, Windows and Linux. Instead of taking screenshots and asking a vision model what is on screen, it reads the operating system's accessibility tree and hands the agent a list of elements it can act on. It was announced on October 31, and one early write-up calls it "agent-browser, for your whole computer." For a solo operator, the interesting part is the apps with no API.
What it is, and what it is not
The tool reads each app's accessibility tree: the AX API on macOS, UI Automation on Windows and AT-SPI on Linux. A snapshot comes back as JSON, with a reference for every element the agent can act on. The agent runs a command against a reference, then takes a new snapshot to check what happened. The documentation describes a four-step loop: map the app with a skeleton snapshot, drill into a region, act on a reference, then read the interface to verify.
It contains no language model. It is a tool an agent calls, so anything that can run shell commands, including Claude Code and Codex, can use it. It does not automate web pages, and Vercel points to its separate agent-browser tool for those. It cannot see apps that publish no accessibility tree. Requirements are macOS 13 or later with Accessibility permission, Windows 10 version 1809 or later, or Ubuntu 24.04 or later with an AT-SPI bus running. Install is npm install -g agent-desktop.
One detail I like: the docs warn that a successful action does not prove the interface changed, which is why the verify step exists. Anyone who has watched an agent confidently click a button that did nothing will appreciate that.
The claim I could not verify
You may see a number floating around that a 52-step Apple Maps task dropped from 198,000 tokens to 31,000 with this tool. I could not find it in the repository, in the documentation or in the announcement coverage I read. The docs make no token claims and publish no comparison figures, and the early coverage says independent tests against screenshot-based tools have not been published. The design argument is reasonable. Text and element references are usually smaller than images. But reasonable is not measured, so I would not plan around any specific saving.
Why I care as a solo operator
The software that eats my week rarely has an API. Desktop accounting tools, a bank's export wizard, a vendor portal packaged as an app, a design tool with one export I need every month. Today I either do these by hand or avoid them. If an agent can operate them reliably, a monthly chore becomes a script. That is where I would aim this, not at anything exciting.
The price of that is access. The tool needs Accessibility permission on a Mac, which is broad control over what the machine shows and does. The docs also say snapshots, clipboard reads and screenshots may contain private data and should be kept out of public logs. An accessibility tree can include whatever text is on screen, and on my machine that sometimes means account balances or customer names.
The ten-run test
Before I rely on it, I would do this on a spare setup. Pick one boring task in one app, something like exporting last month's transactions to a folder. Run the same agent against it ten times. Write down how many runs finished correctly, how many failed loudly, and how many looked fine but produced the wrong file. The third number is the one that decides everything. If even one run in ten silently does the wrong thing, I would not let it near anything with money in it.
I would run the test under a separate user account with no password manager open, no mail client and no browser sessions. Then I would read one full set of logs, to see what a snapshot actually captured.
Where I could be wrong
The project is young, and I am working from the repository, its documentation and early coverage, not from months of use. Reports differ on its popularity, so I will not cite a star count. Apps that expose poor accessibility trees will behave badly, and many cross-platform apps are in that group. Screenshot-based tools may turn out to be more forgiving in the real world even if they cost more. I would also not be surprised if a model vendor's built-in computer use catches up fast.
What I'd actually do
I would not install it on my daily machine today. I would pick the one desktop app I resent most, set up a throwaway user account, run the ten-run test this weekend, and decide from the count. If it clears nine out of ten with no silent failures, it goes on the shortlist for monthly chores. If it does not, I have lost an afternoon and learned where the edges are.
Author
Lukas
@lukcombinator