Drive WhatsApp via accessibility APIs is mostly string matching, not tree walking

The shape of the problem

A web-automation engineer reading about driving WhatsApp through accessibility APIs imports a mental model from CSS selectors: find the right node, click it, type. That model is wrong here, in a way that matters.

The accessibility tree of a Catalyst app exposes almost no typed structure. There is no data-message-id, no role="listitem" with a clean child structure. What you get is a flat list of (role, description, x, y, w, h) tuples where description is a localized human-readable VoiceOver string. That string is the whole API surface.

WhatsApp Catalyst's description strings happen to be templated, so they parse with about thirty lines of regex. The work of writing an MCP server for WhatsApp on macOS is mostly figuring out those templates and writing the parsers; the AX plumbing is identical to what you would do for any other Catalyst app and takes a single afternoon.

Step 1: attach by bundle id

The PID lookup uses NSRunningApplication.runningApplications(withBundleIdentifier:). If WhatsApp is not running, launch it via /usr/bin/open -a WhatsApp.app and wait two seconds for the AX tree to settle. The bundle id is stable; window titles change with the active chat and should not be used as an attach key.

Set an explicit messaging timeout right after attach. AX calls are XPC under the hood; without a timeout, a stutter in the target app hangs your caller indefinitely. The MCP server uses 2.0 seconds for the permission probe (where you want to fail fast) and 5.0 seconds for the real traversal.

Step 2: traverse once, keep everything you might need

Walk kAXChildrenAttribute recursively to a depth of 15. On every node, read role, description, value, title, position, and size in a single pass. Keep the node if it has any text or if its role is in the meaningful set: AXButton, AXTextField, AXTextArea, AXStaticText, AXHeading, AXGenericElement, AXLink.

Doing it in one pass matters. Going back to the AX system per-node per-attribute roughly doubles the time spent talking to the kernel. One walk, every attribute, flat array of structs out the other end. Every operation downstream filters that array; the array itself is regenerated on every tool call because WhatsApp's tree mutates on every keystroke.

Step 3: the actual vocabulary

This is the table no other accessibility-API guide will give you, because writing it requires reading WhatsApp Catalyst's AX tree on a real Mac. Each row is one of the description-string shapes the app emits, the role you find it under, and what to do with it. The patterns are taken verbatim from main.swift.

Step 3a: the regex that turns description strings into MessageInfo

With the vocabulary known, the parser is straightforward. Take the AXGenericElement description, peel off the prefix (Your message, or message, ), then peel off the routing suffix (, Sent to {name} or , Received from {name}), then peel off the timestamp tail. Whatever remains is the message body.

Step 3b: spatial heuristics for search results

Description matching gets you about 80% of the way there. The other 20% is coordinate filtering, because WhatsApp's sidebar contains a lot of buttons that match description filters but are not what you want: tab pills (All, Unread, Favorites), media filter chips, navigation buttons, and so on.

Three thresholds, all encoded as constants in main.swift, do the filtering:

The thresholds are encoded as named constants, which means a WhatsApp layout change is a one-line edit, not a rewrite. The point is to keep the policy explicit and editable, not to derive it from first principles every time.

Step 4: act, then verify

Find the compose AXTextArea by description. Click it via a CGEvent mouse-down/mouse-up pair at the element center (after saving the user's cursor position so the cursor does not visibly jump). Paste through the system clipboard rather than typing each character, because per-character CGEvent typing fails on emoji and fights with WhatsApp's IME. Press Return.

Verification is the part where most accessibility-driven automations leave the LLM hanging. After hitting Return, re-walk the tree and look for an AXGenericElement whose description starts with Your message, containing the body you just pasted. If you find it, return verified: true. If you do not, return verified: false plus a snippet of whatever did appear, so the calling agent can self-correct instead of pretending success.

What this approach looks like next to alternatives

The competitor here is not another macOS accessibility library, but the other transports people pick to reach WhatsApp from an agent: headless Chromium driving web.whatsapp.com, the Signal Protocol fork (whatsmeow), or the Meta WhatsApp Business Cloud API. The comparison is between driving the native app via AX and those three. Below is what each row of the AX-driven approach actually buys you.

Honest limitations

This approach is macOS-only. It depends on a real WhatsApp Desktop install staying logged in, on the host process having Accessibility permission, and on the user's screen being unlocked when the agent is acting (the AX tree is empty on a locked screen). It is single-user; the agent shares whatever pairing the user has, with all the consent implications that follow.

The description strings are localized. The matchers in main.swift assume an English locale. A French or Japanese WhatsApp Desktop ships translated VoiceOver labels and the parsers will need locale-aware variants. The cleanest fix is to keep WhatsApp Desktop in English regardless of system locale; the more invasive fix is to extend each regex with localized alternates.

Latency is real: one full tree walk on a chat with a few hundred rendered rows takes ~120ms on an M-series Mac. For send + verify that is two walks plus the paste sleep, around 700ms end to end. That is fine for an interactive agent and bad for a batch sender. If you need volume, the WhatsApp Business Cloud API is the right tool and this is the wrong one.

Frequently asked questions