agent-device

What It Does

agent-device (by Callstack, the React Native consultancy behind React Native CLI and much of the RN ecosystem tooling) gives AI coding agents a live feedback loop in running apps. It ships as one runtime with three entry points: a 68-command CLI (agent-device open, snapshot, press, fill, screenshot, close), a stdio MCP server (agent-device mcp, 57 tools) that exposes the same commands to clients like Claude Code, Cursor, and Codex, and a typed Node.js client (createAgentDeviceClient()) for building your own agents. It drives iOS, tvOS, Android, HarmonyOS, Amazon Vega OS TV, web, macOS, and Linux targets; agents read token-efficient accessibility snapshots with @ref handles instead of reasoning over screenshots, act through refs or selectors, and save screenshots, logs, traces, and replayable .ad scripts as review evidence. The repo also bundles four Agent Skills Open Standard files (agent-device, ios-simulator, android-emulator, dogfood) that teach agents the workflow without probing commands first. The user it targets is a mobile or cross-platform developer whose coding agent needs to verify its own changes in the app rather than guessing from unit tests.

The Good

The test discipline is the best GearScope has measured in this category. The repo's own unit lane (vitest projects unit-core + subprocess-stub) passed 7,426 of 7,426 tests across 988 test files in 246 seconds on our host, out of 1,136 .test.ts files total. The suite then failed its own pipeline with exit code 1 because two Swift-typecheck tests exceeded 2x their wall-clock budget: the repo enforces per-test duration budgets in the unit lane, and the gate fired exactly as designed on a slow machine. Around that sit 23 GitHub Actions workflows, including weekly mutation testing (Stryker), nightly concurrency-torture and perf lanes, Maestro conformance differential runs, fuzzing for parsers, dependency-graph checks, layering checks, dead-code baselines (fallow), and a CI workflow named agent-guidance.yml that validates the agent-facing guidance text. This for a tool with only 6 runtime dependencies (undici, yaml, tar-stream, yauzl, ipaddr.js, @limrun/api).

The MCP server is implemented to current spec and verified deterministic. The initialize handshake echoes the exact protocol revision the client requests (asked for 2025-06-18, got 2025-06-18; the default is 2025-11-25), carries a 1,378-byte instructions workflow card deliberately sized under the 2KB Claude Code truncation limit, and answers tools/list with 57 tools (264,862 bytes) where every tool carries an inputSchema, 56 command tools are sorted, and the MCP-only help tool rides at the end. Two separate server sessions returned byte-identical tool arrays (verified with a field-by-field diff, zero differences), which is what makes the tool list cacheable. The help tool serves the same version-stamped guides as the CLI (help(workflow) returned 9,356 bytes starting with agent-device 0.20.10: workflow), so MCP-only clients need no separately installed skill.

Every failure we produced came back structured, with a hint and a diagnostic ID. Calling a nonexistent tool returns isError: true with Error (INVALID_ARGS): Unknown command tool: does_not_exist plus a hint. Requesting an iOS snapshot without Xcode returns Error (COMMAND_FAILED): unable to find utility "simctl" with Hint: Retry with --debug, a Diagnostic ID, and the on-disk path of the request's ndjson log. Android without adb returns Error (TOOL_MISSING): adb not found in PATH with an install hint, exit code 1. The daemon even detected that our fresh-prefix binary differed from the running one and replaced it with a logged code-signature mismatch notice. Errors this consistent are rare; most MCP servers we test crash or hang on at least one of these probes.

The security posture extends into the skill text itself. The bundled ios-simulator skill instructs agents to treat npm install -g agent-device as a user-owned step: "Do not run that command autonomously or substitute a mutable npx -y agent-device@latest invocation." The agent-setup docs repeat the rule (avoid agents fetching mutable npm packages unprompted), SECURITY.md documents private vulnerability reporting with response timelines, release assets ship sha256 checksums plus manifests, and the unreleased changelog hardens the repository-level ./agent-device.json config against operator-controlled values loading before local modules.

Version and metadata hygiene is scripted, not aspirational. Five version surfaces agree on 0.20.10: git tag v0.20.10, npm latest, package.json, server.json (the MCP registry manifest), and the MCP serverInfo handshake. The version npm script runs sync:mcp-metadata and re-stages server.json, and check:mcp-metadata gates releases on it. The server.json name io.github.callstack/agent-device matches the package's mcpName field, and both server.json and glama.json parse clean. Registry demand is real: 112,093 npm downloads last week, and 11,126 and 6,008 skills.sh installs for the agent-device and dogfood skills.

The Bad

The managed web backend installer fails on a reproducible user-agent 403. agent-device web setup downloads Chrome for Testing from storage.googleapis.com and failed twice in a row with Error (COMMAND_FAILED): Download failed: server returned HTTP 403 Forbidden. The URL itself is fine: curl -I on the exact same URL returns HTTP 200 from the same host within the same minute, and curl -A undici on it reproduces the 403. The tool's downloader (undici 7.29.0) sends a user-agent Google's CDN rejects, so the documented --platform web path cannot be set up on affected machines at all until the UA is fixed or a system browser is used.

The Apple capture path silently requires full Xcode, and doctor under-reports it. On our Command-Line-Tools-only Mac, agent-device open Calculator --platform macos worked (the app launched), but the first snapshot failed with xcodebuild build-for-testing failed because the Apple runner must be compiled first. That is a 15GB+ dependency with no warning at install time. doctor reports the missing adb and hdc but never names the missing Xcode or simctl, even though capabilities --platform ios fails on exactly that. A CLT-only Mac is a common CI configuration and the doctor output does not surface this blocker.

doctor exits 0 while printing "Doctor: fail". We verified the exit code directly: the command prints Doctor: fail / Blockers found before the run and returns 0, while real command failures like snapshot --platform ios correctly return 1. Nothing in help doctor documents this, so any CI script or agent gating on doctor's exit code gets a false green.

Session cleanup has a rough edge after a failed runner build. Following the failed macOS snapshot, agent-device close timed out (Error (COMMAND_FAILED): Daemon request timed out, with the daemon reset as compensation), and a second close then reported SESSION_NOT_FOUND. Recoverable, but the first close should have torn the session down.

The changelog stops at 0.15.0. CHANGELOG.md contains two sections: ## Unreleased (enormous, excellent per-issue entries) and ## 0.15.0. Versions 0.16.0 through 0.20.10 shipped as git tags and npm publishes with no versioned changelog entries, so consumers tracking breaking changes (and there are breaking changes, like the ADR 0014 ref-frame expiry) must read git log or the Unreleased blob. Relatedly, the MCP 2026-07-28 stateless revision support and ttlMs/cacheScope cache hints documented in the Unreleased changelog are not in the published 0.20.10: server/discover answered an empty result in our stateless probe. Documentation ahead of the shipped package is normal, but worth knowing before relying on those features.

No Hermes Agent or OpenClaw install docs, and no tool annotations. The agent-setup page covers Cursor, Codex, Claude Code, Windsurf, Cline, and Goose; Hermes works fine through the universal installer (we verified npx skills add callstack/agent-device --agent hermes-agent lands a byte-identical SKILL.md in ./.hermes/skills/) but is not mentioned anywhere in the repo. None of the 57 MCP tools carry annotations (readOnlyHint and friends), which newer MCP clients use for cheap safety heuristics.

Smoke Test Results

Host-based runs on macOS 26.5.2 aarch64, Node 24.13.1, Python 3.9.6 (the sbx microVM daemon could not authenticate to Docker this session, so the standard three variants ran directly on the host with logs captured to the sandbox paths). The repo was shallow-cloned to /tmp/agent-device-review (54 MB).

Run A - Fresh sandbox, no deps preinstalled

$ npm install --prefix /tmp/agent-device-fresh-run agent-device
✅ added 29 packages in 20s (6 runtime deps; unpacked size 3.1 MB)

$ agent-device --version
✅ 0.20.10 (matches npm latest, package.json, server.json, git tag)

$ agent-device help workflow | head -1
✅ first line prints the version then the topic: agent-device 0.20.10 workflow
 (help is version-stamped as documented)

$ agent-device doctor
✅ reports honest blockers: adb not found, hdc not found, web backend not installed

$ agent-device capabilities --platform macos
✅ host Mac identified as booted macos target, supports 32 commands

$ agent-device apps --platform macos | head -3
✅ real inventory with bundle ids (Brave Browser com.brave.Browser, ...)

Pass rate: 6 of 6. A clean npm-only install reaches a working CLI with live host inventory; what it cannot reach without Xcode or adb is capture, which doctor discloses.

Full sandbox log →

Run B. Sandbox with deps preinstalled

$ pnpm install --frozen-lockfile
✅ dev tree installs clean (pnpm 11.x via isolated prefix, lockfileVersion 9.0)

$ pnpm test:unit
✅ Test Files 988 passed (988); Tests 7426 passed (7426) in 246s; 0 failed

$ pnpm test:unit (exit code check)
✅ pipeline exits 1 via the repo's own "Slow-test gate" after 2 Swift-typecheck
 tests exceed 2x wall-clock budget (29.91s and 12.03s vs 2.5s) - the duration
 budget is enforced, not decorative

$ find skills -name "SKILL.md" | wc -l
✅ 4 (agent-device, ios-simulator, android-emulator, dogfood)

$ frontmatter check across skills/*/
✅ 4 of 4 have YAML frontmatter with name + description

$ version consistency (package.json vs server.json vs npm view)
✅ 0.20.10 = 0.20.10 = 0.20.10

$ grep mcpName package.json
✅ "io.github.callstack/agent-device" matches server.json name

$ node JSON.parse server.json glama.json
✅ both registry manifests parse clean

$ docs / CI / test-file surface
✅ 23 website doc pages, 23 CI workflows, 1136 .test.ts files

Pass rate: 9 of 9. With dev dependencies present, the project's own quality gates all pass on a host it has never seen; the only nonzero exit belongs to the repo's own wall-clock discipline firing on two Swift compiles.

Full sandbox log →

Run C. Functional verification (does it do what it claims?)

$ python3 sandbox/skills/agent-device/mcp_probe.py
✅ 14 of 14: initialize echoes requested revision 2025-06-18; serverInfo
 agent-device 0.20.10; instructions card 1378 bytes (<2KB); tools/list 57
 tools / 264862 bytes, every tool has inputSchema; 56 sorted + MCP-only help
 appended last; unknown tool → isError INVALID_ARGS with hint; help() returns
 CLI→tool decision card; help(workflow) embeds version-stamped guide;
 capabilities(macos) returns live 32-command inventory over MCP; ios snapshot
 (no Xcode) fails as structured isError naming simctl; tools/list
 byte-deterministic across two sessions (field diff: 0)

$ npx skills add callstack/agent-device --agent hermes-agent --skill agent-device --copy --yes
✅ lands SKILL.md in ./.hermes/skills/agent-device/ (cwd-level, per skills CLI
 convention)

$ diff -q .hermes/skills/agent-device/SKILL.md repo/skills/agent-device/SKILL.md
✅ byte-identical to the repo copy (registry blob is current; the skill is a
 single 1643-byte file so there is no references/ tree to go stale)

$ agent-device snapshot --platform android
✅ exit 1 with Error (TOOL_MISSING): adb not found in PATH + Hint +
 Diagnostic ID + ndjson log path (negative path verified)

Functional pass rate: 4 of 4. What could not be verified: driving a real app's UI (needs full Xcode for the Apple runner, adb for Android, or the web backend, whose setup 403s; see Functional-Reason). The web-setup 403 was reproduced deterministically and root-caused to the downloader's user-agent via curl -A undici returning 403 where the default curl user-agent returns 200.

Full functional log →

What the runs tell you

The install path is clean and the MCP surface is verifiable end to end without any device, which is good design for a device tool: everything except capture works on a bare host, and every missing-toolchain failure names itself. The two gaps that remain are environmental (Xcode, adb) except the web-setup 403, which is a real defect in the shipped package.

Setup Walkthrough

1. Check Node first: the CLI requires Node 22.12+, and web automation requires Node 24+. Our host ran Node 24.13.1 with no issues.
2. Install globally: npm install -g agent-device@latest (29 packages, about 20 seconds). The npm package bundles the macOS helper and Android helper APK sources, so no post-install builds for basic use.
3. Run agent-device doctor and read the text, not the exit code (it exits 0 even on failure). On our CLT-only Mac it correctly flagged adb and hdc but missed the Xcode requirement that blocks all Apple capture.
4. Skim agent-device help workflow: it is an agent-oriented 8KB card describing the open, act with --settle, verify, close loop, and it is version-matched to the installed binary.
5. For MCP clients, add {"mcpServers":{"agent-device":{"command":"agent-device","args":["mcp"]}}}. The server carries its own instructions, so skill installation is optional for MCP-only clients.
6. For skill-aware agents, run npx skills add callstack/agent-device. In non-interactive scripts, pass the full flag set (--agent --skill --copy --yes); the bare command opens an interactive scope picker that stalls and installs nothing under a closed stdin, which is a skills CLI behavior, not an agent-device defect.
7. For iOS work, budget for full Xcode; Command Line Tools are not enough. For Android, install platform-tools (adb). For web, run agent-device web setup and hope your network's CDN tolerance extends to the undici user-agent; on ours it did not.

Alternatives

1. mobile-next/mobile-mcp - a 5,965-star MCP server for iOS and Android automation over WebDriverAgent and uiautomator2. Prefer it when you want a thinner, device-cloud-focused MCP without agent-device's replay, evidence, and profiling layers.
2. Maestro - declarative YAML flow testing with a large ecosystem. Prefer it when the flows are known ahead of time and humans, not agents, maintain them; agent-device can export its recorded runs to Maestro YAML.
3. vercel-labs/agent-browser - the web-only sibling whose inspect-act-verify process agent-device adopted. Prefer it when web is the only target.