JetBrains benjamin-plus

What It Does

benjamin-plus is JetBrains' first official agent skill: a five-rule, 2,996-byte text payload that gets injected into a coding agent's context (via a Claude Code SessionStart hook, an AGENTS.md append, or any system prompt) to cut session cost. The rules change how the agent looks things up, not what it builds: gather facts in one pass, read the smallest slice that answers the question, probe the environment once, treat the task's own check as the definition of done, and poll long-running commands in 30-second slices instead of every second. The repo (236 stars, 6 days old at test) contains the ruleset, the injection payload, a hook script, a checksum manifest, and an unusually detailed A/B results document claiming 17.9% lower median cost with quality unchanged. It targets developers running metered coding agents at any volume.

The Good

The eval write-up is the most honest one GearScope has reviewed in this category. EXPECTED-RESULTS.md keeps the v5 archive next to the v6 headline, explains the medians-versus-totals disagreement ("the skill wins the middle of the distribution and gives some of it back on trap-task tails"), discloses that the control arm drifted +10.5% between run days, refuses to overclaim quality ("not powered as an equivalence test; 80 pairs rules out large effects only"), lists iteration shortcuts that were fine for gating but not quotable, and states total program spend ($153 across 254 billed trials). Every number in the README traces back to this file, and the internal consistency holds: 225 instances times 3 replicas equals the "675 paired replicas" the README cites, and the token claim (745) matches our chars-divided-by-four heuristic (2,996 bytes yields 749) almost exactly.

The install-method finding is load-bearing research, not packaging opinion. The team ran hook-injected delivery against skill-folder delivery head to head on Java SWE-bench with Codex CLI: injection saved 4.4% (p=0.003) while the skill folder saved nothing (−0.5%, not significant) because fetching SKILL.md cost a median of 3 steps with 73% path misses. So the repo ships injection only, and the README says so in plain terms. This is the first vendor skill we have reviewed that opts out of the SKILL.md ecosystem on measured evidence, and it corroborates what GearScope found in the orca review (agents burn steps finding skill files) from the opposite direction.

The payload itself is disciplined, and the single-source-of-truth claim holds. Five rules, each behaviorally specific (chain probes with ; and label sections, | head -50 on inspection only, one wider look if a peek was too narrow), each carrying its own correctness guard: keyhole rules "apply to inspection, never to ingestion," and "efficiency never outranks correctness." We verified that the RULESET.md body (post-frontmatter) is byte-identical to the injected-instruction.md payload, that SHA256SUMS.txt verifies all 8 files against the working tree, and that the 8-line hook script emits the full 55-line payload with the banner from any working directory.

It comes from a lab that stress-tests its own claims. This is the same JetBrains research program that externally verified caveman's prose-compression numbers at 8.5% on real coding runs (a smaller figure than caveman's own marketing, published anyway). The v6 polling rule was added because Java traces showed 45.8% of treated-arm steps were polls, 3,982 of them at sub-5-second intervals. The repo is CODEOWNERS-gated ("every change needs an approval from the repo owner," internal ticket PEN-3817), and the one merged PR was a plain-language README rewrite.

The Bad

The "Reproducing" section reproduces nothing. EXPECTED-RESULTS.md ends with four commands (gen_instruction.py, compare.py, adoption.py, mediators.py) and points at LAB-NOTES.md and winning-strats.md. None of those files exist in the repo, and no eval code ships at all. The headline 17.9% figure is therefore trust in a well-written document, not something a reader can rerun. The same file also points at dist/benjamin-plus/SHA256SUMS.txt while the repo ships the manifest at root, which is harmless drift from the internal research repo but confirms the public repo was carved out of a larger private one.

The universal installer rejects the repo, and the frontmatter invites the attempt. npx skills add JetBrains/benjamin-plus-skill exits 1 with "No valid skills found. Skills require a SKILL.md with name and description." That is deliberate (the README's whole thesis is inject-don't-install), but RULESET.md still carries full SKILL.md-format frontmatter (name, a 461-character description under the 1,024 limit, license), so anyone following muscle memory hits a clean but confusing dead end. There is no skills.sh registry presence at all.

It is six days old with a three-commit history. One contributor, no tags, no releases, no CI, no tests, one open community issue. Nothing here is load-bearing infrastructure yet; the entire value proposition rests on one eval document from one research run plus one external cross-check.

Every agent that is not Claude Code or Codex gets one sentence. "Any other agent: append injected-instruction.md to the system prompt." True, and the payload is plain text so it works, but Hermes, OpenClaw, Cursor, and Copilot users get no file paths, no hook equivalents, and no guidance for the re-injection-after-compaction problem that the hook matcher explicitly solves for Claude Code. The hook script's own header documents the registration better than the README does for non-Claude hosts.

Smoke Test Results

Hands-on host test on macOS 26.5.2 (aarch64), bash 3.2, Node 24.13.1, npm 11.8.0, Claude Code CLI 2.1.152. The repo is a text payload plus an 8-line hook script, so runs exercise clone integrity, emission, packaging, and the live-session path. Full logs: fresh, with-deps, functional.

Run A. Fresh clone, no deps preinstalled

$ git clone --depth 1 https://github.com/JetBrains/benjamin-plus-skill /tmp/benjamin-plus-review
✅ 2.1 MB, 11 files outside .git
$ bash hooks/sessionstart.sh | head -1
✅ BENJAMIN-PLUS MODE ACTIVE
$ bash hooks/sessionstart.sh | wc -l
✅ 55 (full payload, exit 0, executable bit set as cloned)
$ shasum -a 256 -c SHA256SUMS.txt
✅ 8 of 8 checksums OK (README, RULESET, payload, hook, 3 assets, results doc)
$ cmp <(sed -n '14,65p' RULESET.md) <(sed -n '4,55p' injected-instruction.md)
✅ byte-identical: ruleset body matches injected payload

Pass rate: 5 of 5. Zero dependencies beyond bash and coreutils; the entire skill is verifiable with the tools already on any Unix host.

Run B. Host with deps preinstalled (Node 24.13.1, Claude Code 2.1.152)

$ npx -y skills add JetBrains/benjamin-plus-skill --agent claude-code --yes
❌ exit 1: "No valid skills found. Skills require a SKILL.md with name and description." (nothing landed; deliberate per README, see The Bad)
$ python3 -m json.tool hook-settings.json
✅ VALID (the README's SessionStart JSON block parses and matches the documented hooks schema)
$ claude --version
✅ 2.1.152 (Claude Code) present on host
$ bash hooks/sessionstart.sh # invoked from / (unrelated cwd)
✅ banner emitted, exit 0 (script resolves its own path via dirname)

Pass rate: 3 of 4. The single failure is the universal-installer path, which is the repo's documented intent and fails fast with a clear error, but it is still the path most agent users will try first.

Run C. Functional verification (does it do what it claims?)

$ claude -p "quote the first line of the BENJAMIN-PLUS block in your session context"
❌ "Failed to authenticate. API Error: 401 OAuth access token has expired." (host CLI credential expired; policy prohibits entering credentials)
$ echo $(( $(wc -c < injected-instruction.md) / 4 ))
✅ 749, within 0.5% of the claimed 745-token payload (2,996 bytes, 505 words)
$ awk 'NR>=4 && NR<=9' RULESET.md | wc -c
✅ 461 chars: description fits the 1,024-char agent-skill limit

Functional pass rate: 2 of 3. The delivery mechanics are fully verified (emission, byte-identity, checksums, valid hook config). The behavioral claim, that agents receiving the payload run 10 to 18% cheaper, was not exercised end to end: the host Claude Code CLI could not open an authenticated session, and the A/B harness that produced the claim is not published.

What the runs tell you

The artifact is exactly what it claims to be: a small, checksummed, dependency-free text payload with a working delivery hook. What cannot be verified from the repo is the effect size; you either trust the A/B document or run your own paired comparison, and the README explicitly invites the latter with a template issue format for regressions.

Setup Walkthrough

1. Clone the payload repo: git clone https://github.com/JetBrains/benjamin-plus-skill ~/.benjamin-plus (2.1 MB, no dependencies).
2. Claude Code: add the SessionStart hook to ~/.claude/settings.json with matcher startup|resume|clear|compact and command cat ~/.benjamin-plus/injected-instruction.md. The matcher re-injects after /clear and compaction, the two events that silently wipe injected context. Verify with /hooks. We confirmed the JSON block is valid and the hook script variant works from any cwd.
3. Zero-config alternative for any repo: cat ~/.benjamin-plus/injected-instruction.md >> CLAUDE.md.
4. Codex CLI: append the same file to ~/.codex/AGENTS.md or a repo's AGENTS.md.
5. Any other agent (Hermes, OpenClaw, Cursor, Copilot, custom): append injected-instruction.md to whatever the agent loads as persistent context. There is no SKILL.md, so skill registries and npx skills add will not work, by design.
6. Removal is the inverse of whatever you did: delete the hook block or the appended lines. Nothing else is installed anywhere.

Alternatives

1. JuliusBrussee/caveman (reviewed 4.5/5) cuts output tokens by making the agent talk in compressed prose, verified at 8.5% on real runs by this same JetBrains program. Different mechanism (output style versus lookup discipline), so the two stack; benjamin-plus changes process, caveman changes phrasing.
2. DietrichGebert/ponytail (reviewed 5/5) is the YAGNI-discipline skill that ships the laziest correct solution (54% less code on a benchmark). It attacks code volume, benjamin-plus attacks conversation steps; ponytail also ships as a real multi-platform skill with an MCP server if registry install matters to you.
3. ayghri/i-have-adhd (reviewed 4.5/5) spends tokens on clarity, shaping output so readers get the answer first. It is the philosophical opposite on token spending and a useful counterweight if you tried compression skills and lost too much signal.