pentest-ai (ptai)

What It Does

ptai is an offensive-security MCP server and CLI for AI-driven penetration testing. It exposes 52 MCP tools to clients like Claude Code, Cursor, VS Code Copilot, Codex, and Claude Desktop, covering recon, authenticated web testing, API security, Active Directory, cloud, mobile, wireless, browser automation, credential testing, privilege escalation, exploit chaining, and PoC validation. The headline architectural choice is the "Trust Spine": a finding stays a candidate until a named machine oracle re-runs the exploit and reproduces it N out of N times, and only then earns a VERIFIED badge. Third-party scanner output from nuclei, nikto, and ZAP is held back until an oracle re-proves it. The target users are AppSec teams wiring it into CI, consultants, bug bounty hunters, red teamers, and Claude Code or Cursor users who want to drive a pentest in plain English using their existing subscription (no separate API key on the MCP path).

The Good

The oracle-verification gate is real and the bundled demo proves it. ptai demo (no API key, no external tools) spins up a bundled vulnerable app, finds three flaws, and oracle-VERIFIES all three: Reflected XSS on /search, Open Redirect on /go, and SQLi login bypass on /auth/login. It then replays one proof capsule live ("replay 3/3"), writes a tamper-evident capsule JSON (1,159 bytes, contains integrity_sha256, receipt, finding, recipe), and re-runs the same routes on a hardened twin that returns zero findings. The README's "no proof, no finding" pitch is not a slogan. It is what the binary does.

The MCP server implements the full protocol surface, cleanly. A stdio handshake against ptai mcp returns serverInfo, protocolVersion: 2024-11-05, and capabilities experimental, logging, prompts, resources, tools, extensions. tools/list returns 52 tools including start_engagement, get_findings, get_attack_chains, list_tools, run_tool, run_probe, http_request, test_web_app, test_active_directory, test_cloud, test_api_security. Resources and prompts are wired (return empty lists) and the newer extensions capability is advertised. This is a complete MCP server, not a thin wrapper.

Test suite depth exceeds the README's own claim. The README says "1,000+ tests." The repo ships 236 test files containing 2,383 test functions, on a CI matrix of Python 3.10, 3.11, 3.12, and 3.13. The honeypot harness (tests/honeypot/, including test_mcp_honeypot_e2e.py and test_e2e_probes.py) and the clean-app zero-false-positive gate (tests/cleanapp/test_clean_fp.py) ship in the repo, so the precision claims are reproducible rather than screenshots. Benchmark raw artifacts live under benchmarks/results/2026-05-12/ through 2026-05-15-juiceshop/ with a README. Few security tools publish both their high-water mark (OWASP Juice Shop) and their honest lower-bound (private honeypot) side by side.

Scope enforcement and AUP gating are baked in, not bolted on. First run requires AUP acceptance (persisted to ~/.pentest-ai/aup-consent.txt, bypassable in CI via PENTEST_AI_AUP_ACCEPTED=1). A scope file is loaded at startup and out-of-scope hosts are refused at tool-invocation time. PoCs are non-destructive by default. Auth profiles store references (env vars, op:// 1Password paths, Vault paths, AWS Secrets Manager ARNs), never credential values. The OAST (out-of-band callback) section documents exactly what metadata is server-visible versus payload-encrypted, and gives a one-flag --no-oast kill switch. SECURITY.md ships a real coordinated-disclosure policy with a 90-day window.

The Bad

Version metadata is inconsistent across every surface that reports one. The CLI ptai --version returns 1.0.0. The pyproject.toml version is 1.0.0. The VERSION file says 0.14.0. The README references 0.13.0 and 0.14.0 ("49 of them as of 0.14.0"). The CHANGELOG's latest entry is [1.0.0] - 2026-06-25. The MCP initialize handshake returns serverInfo.version: 3.4.2, which matches none of the above. For an MCP server, serverInfo.version is the field compatibility-checking clients read. A client that gates on version will see 3.4.2 and make wrong decisions. This is the most concrete defect in the package.

Tool and probe counts drift between the CLI banner, the README, and the runtime. The ptai --help banner advertises "197 wrapped tools." The README "What's inside" section says "200+ tool wrappers" and the pyproject description says "200+ tools." The pipeline scan description said "205 wrapped tools." On the MCP side, the README says "49 MCP tools," the source has 47 @mcp.tool decorators, and the runtime tools/list returns 52. Web probes: the README says "60 curated web probes," the engine/probes/web/ directory has 50 probe files (excluding __init__.py and the payloads library). None of these deltas are large, but for a tool whose entire value proposition is precision and proof, the counts should not disagree with the binary.

The git history is a single squashed commit. git log --oneline returns one entry (the merge of PR #19, "2.0-trust-spine"). The CHANGELOG documents a real release history (0.17.0 through 1.0.0 over May to June 2026), but none of that history is visible in the commit log. Contributors who want to understand why a decision was made, bisect a regression, or credit prior work have only the CHANGELOG narrative to go on. This is a maintainer-style choice (squash-on-merge), not a defect, but it lowers the bar for outside contribution and makes the single-commit-plus-1,102-stars profile read as unusually top-heavy for a third-party reviewer.

The "17 specialist agents" framing oversells what the agents are. The agents/ tree contains 18 directories (17 phase agents plus a selection router), each holding Python orchestrator classes that subclass BaseAgent. They are internal coordination code that calls the LLM and dispatches to tool wrappers and probes. They are not SKILL.md agent-skill files and cannot be loaded into Claude Code, Cursor, or Codex as standalone skills. The repo is an MCP server with internal orchestrator classes, not a pack of 17 portable agent skills. Readers who interpret "17 specialist agents" the way the agent-skill community uses "agent" will be mildly misled.

Smoke Test Results

Tested on macOS 26.5.1 (aarch64) host, Python 3.11.14 via uv, isolated venv. The PyPI package was installed fresh (no repo clone) for the install and server runs. The repo was shallow-cloned separately for source inspection and test counting.

Run A. Fresh sandbox, no deps preinstalled

$ uv venv --python 3.11 /tmp/ptai-smoke-fresh && source /tmp/ptai-smoke-fresh/bin/activate
$ uv pip install ptai
Using CPython 3.11.14 ... Successfully installed ptai-1.0.0 (and 21 deps)
✅ install exit 0

$ ptai --version
ptai 1.0.0
✅ version exit 0

$ ptai --help
 Usage: ptai [OPTIONS] COMMAND [ARGS]...
 Offensive-security MCP server with 197 wrapped tools, 17 specialist agents,
 and SPA-aware probes ...
 Commands: start, mcp, doctor, menu, serve, list, reap, resume, chain,
 status, live, demo, replay, findings, tool-results, tools, route, ps ...
✅ help renders full command tree

$ ptai doctor
 Resolved provider: openai Model: gpt-4o
 OPENAI_API_KEY: not set Findings DB: missing
 Reachability: FAIL (OpenAI endpoint unreachable: Illegal header value b'Bearer ')
✅ doctor exits 1 with a concrete reason (no API key)

Pass rate: 4 of 4. Clean install, version, help, and doctor all behave. Doctor's FAIL on reachability is correct behavior, not a bug, since no key was provided.

Full fresh-install log →

Run B. MCP server starts and the handshake completes

$ printf '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{...}}\n' | ptai mcp
=== INITIALIZE ===
serverInfo: {"name":"pentest-ai","version":"3.4.2"}
protocolVersion: 2024-11-05
capabilities: ['experimental','logging','prompts','resources','tools','extensions']
✅ initialize responded

$ tools/list
=== TOOLS/LIST ===
tool count: 52
first 12 names: ['start_engagement','get_engagement_status','get_findings','health',
 'get_attack_chains','list_tools','run_tool','run_recon','test_web_app',
 'authenticated_scan','test_active_directory','test_cloud']
resources/list: 0 resources prompts/list: 0 prompts
✅ tools respond

Pass rate: 3 of 3. Server starts, initializes, and enumerates 52 tools. This is the canonical "does the MCP server actually work" run. The serverInfo.version: 3.4.2 mismatch is noted in The Bad.

Full MCP handshake log →

Run C. Functional verification (does it do what it claims?)

$ PENTEST_AI_AUP_ACCEPTED=1 PTAI_NON_INTERACTIVE=1 ptai demo --capsule /tmp/ptai-demo-capsule.json

 Vulnerable build
 ┏━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━┓
 ┃ Severity ┃ Finding ┃ Endpoint ┃ Verdict ┃
 ┡━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━┩
 │ MEDIUM │ Reflected XSS │ /search │ VERIFIED │
 │ MEDIUM │ Open Redirect │ /go │ VERIFIED │
 │ CRITICAL │ SQLi login bypass │ /auth/login │ VERIFIED │
 └──────────┴──────────────────────────┴─────────────┴──────────┘
 3 vulnerability finding(s), 3 oracle-verified, 0 informational
 Re-proven live: replayed the proof against the running target, oracle
 re-confirmed (replay 3/3).

 Hardened build: same routes, vulnerabilities fixed
 0 vulnerability finding(s), 0 informational
✅ demo exit 0, capsule written (1,159 bytes, integrity_sha256 present)

Functional pass rate: 3 of 3. The core "Trust Spine" claim is verified: findings appear on the vulnerable build with VERIFIED badges, the proof capsule replays 3-of-3, and the hardened twin returns zero findings. The capsule JSON contains capsule_schema_version, ptai_version, created_at, finding, receipt, integrity_sha256 (tamper-evident receipt confirmed).

Full functional log →

What the runs tell you

The install path is clean (pip install, no API key, no external security tools needed for the core MCP plus demo surface). The MCP server is complete and responds correctly to the full protocol handshake. The headline oracle-verification thesis works as documented on the bundled target. What remains untested is the live-engagement path: a real scan needs written authorization, an LLM (Claude Code subscription, an Anthropic or OpenAI key, or local Ollama), and the external tools ptai wraps (nmap, sqlmap, nuclei, ffuf, and the rest). The demo proves the verification machinery works. It does not prove the 200-plus tool wrappers all integrate cleanly against a live target.

Setup Walkthrough

1. Install Python 3.10 or newer. macOS ships 3.9.6, so install via Homebrew (brew install python@3.11) or use uv, which handles the interpreter for you.
2. pip install ptai from PyPI. The package is 1.0.0, MIT licensed, pulls 21 dependencies including fastmcp, httpx, pydantic, typer, rich, scapy, paramiko, impacket, and bloodhound.
3. For the MCP path (recommended for Claude Code, Cursor, Codex users): claude mcp add pentest-ai -- ptai mcp. No API key needed. Your subscription is the LLM. For other MCP clients, ptai setup --mcp auto-detects installed clients and writes their config files.
4. For the standalone CLI path (CI, air-gapped, no MCP client): set one of ANTHROPIC_API_KEY, OPENAI_API_KEY, PENTEST_AI_LLM_PROVIDER=ollama, or use LiteLLM for 300-plus models. Then ptai start https://your-target.com. A default $10 per-engagement spend cap applies on this path only.
5. Install external security tools on demand. ptai setup --tier core installs roughly 6 essentials in 30 seconds. --tier recommended adds fuzzers, crawlers, and password tools in about 5 minutes. --tier full installs everything in about 30 minutes. Or let the planner prompt you at engagement start.
6. Verify the install with ptai demo (zero API key, two minutes, proves the oracle pipeline) and ptai doctor (diagnoses provider config and reachability).

Gotcha: first run in interactive mode prompts for AUP acceptance. In CI, set PENTEST_AI_AUP_ACCEPTED=1 and PTAI_NON_INTERACTIVE=1 to bypass. The findings database and evidence live under ~/.pentest-ai/.

Alternatives

1. HexStrike - the closest competitor (also MIT, also MCP, 150-plus tool wrappers). Broader cloud, binary, and CTF coverage. ptai is sharper on SPA web pentests with curated probes and is the only one with oracle-verified findings. The README's own comparison table is honest about where HexStrike wins.
2. ZAP (OWASP ZAP) - the incumbent free scanner. Massive template and rule library, mature proxy. No LLM coordination, no MCP, no oracle verification. The published benchmark shows ZAP returning 593 Juice Shop findings with a 47 percent false-positive rate versus ptai's 88 at 0 percent. Different philosophy: ZAP is broad and noisy, ptai is narrow and proven.
3. PentestGPT - the LLM-first approach. Reasoning about pentest steps, no tool wrappers, no MCP, no PoC validation. Useful as a brainstorm partner. ptai is the inverse: curated deterministic probes do the detection, the LLM coordinates.