Microsoft Waza

What It Does

Waza is a Go CLI for evaluating AI agent skills. It scaffolds eval suites, runs benchmarks against skills defined in SKILL.md format, grades results using deterministic and LLM-based graders, and produces comparison reports across models. The tool targets skill authors who want to measure and improve the quality of their agent skills before distribution. It includes 14 grader types (code, text, file, behavior, diff, trigger, prompt, tool calls, action sequence, JSON schema, program, inline script, skill invocation, and tool constraint), an HTTP dashboard for visual exploration of results, MCP server mode for IDE integration, and CI workflow templates.

The Good

Serious engineering depth. 371 Go source files totaling 90,182 lines of code. 163 test files containing 1,716 test functions. That is not a typo: the test-to-code ratio is high, with 49,408 lines of test code against the implementation. This is one of the most thoroughly tested skill tools reviewed by GearScope. The internal package structure (43 packages under internal/) shows clean separation: graders, execution, orchestration, scoring, reporting, tokens, triggers, validation, and more.

14 grader types covering the evaluation surface. Deterministic graders (code assertions, regex, file checks, diff comparison) work without any LLM. LLM-based graders (prompt rubrics, behavior scoring, tool call accuracy) use a model-as-judge pattern with configurable rubrics. The Azure ML-derived rubric library (9 rubrics in examples/rubrics/) provides battle-tested evaluation criteria for task completion, intent resolution, response completeness, and tool usage dimensions. Each rubric includes chain-of-thought instructions, rating levels, and JSON output schemas.

Scaffolding generates real structure. Running waza init creates a project with skills/ and evals/ directories, a CI workflow, .gitignore, and .waza.yaml config. Running waza new skill test-skill scaffolds a SKILL.md, eval.yaml with metrics and graders, three task files (basic usage, edge case, negative test), and a fixture file. The generated eval.yaml includes meaningful defaults: code grader checking output length, text grader checking relevance patterns, task completion metric with 0.8 threshold. This is not stub code.

Spec compliance checking is thorough. waza check test-skill runs 9 spec compliance checks, advisory checks (module count, complexity, negative-delta-risk, procedural content, over-specificity, cross-model density, body structure, progressive disclosure, scope reduction), token budget analysis, schema validation of eval.yaml, and task file validation. It outputs a structured report with actionable next steps.

CI integration out of the box. The init command generates a GitHub Actions workflow (eval.yml) for running evals on pull requests. There is a dedicated skills-ci-example.yml for reference. The eval results can be compared across runs with waza compare, enabling regression detection.

Multiple distribution channels. Pre-built binaries for 6 platforms (darwin/linux/windows x amd64/arm64). Install script with checksum verification. Azure Developer CLI extension registry. Source build with Go 1.26+. The 110MB arm64 binary is large but includes embedded Copilot SDK binaries.

MCP server mode. waza serve exposes eval.run, eval.list, eval.get, eval.validate, task.list, task.get, run.status, and run.cancel as JSON-RPC methods. IDE integrations can trigger evals programmatically. The .copilot/mcp.json config file is included in the repo.

The Bad

Copilot SDK dependency gates the best features. The default executor is copilot-sdk, which requires GitHub Copilot authentication. Running waza models returns a 403 error without Copilot access. The waza quality command (LLM-as-judge for skill quality scoring) also requires Copilot. The waza run command with real skill execution needs Copilot. You can use the mock executor for testing the framework, but actual skill evaluation against a running agent requires Copilot credentials. This limits the addressable audience to Copilot subscribers for the core use case.

110MB binary size is heavy. The darwin-arm64 binary is 110MB because it embeds the Copilot CLI binary (version 1.0.49) for all supported platforms. This is unavoidable given the architecture but matters for bandwidth-constrained environments.

The suggest command errored in testing. Running waza suggest skills/test-skill --dry-run returned exit code 2 with "parsing suggest response: response is not valid suggestion YAML." This is likely because the suggest command requires Copilot access to generate suggestions, but the error message does not make this clear. A better error would say "Copilot authentication required for suggestion generation."

Skill runner SKILL.md is thin. The bundled waza-runner/SKILL.md is 95 lines and describes the eval runner at a high level but does not include detailed references. It references EVAL-SPEC.md, WRITING-TASKS.md, and GRADERS.md in its References section, but these files do not exist under waza-runner/. The references exist in the docs/ directory at the repo root, creating broken cross-references for anyone following the SKILL.md paths.

Go 1.26 requirement limits source builds. Building from source requires Go 1.26, which is bleeding-edge. Most systems ship Go 1.22 or earlier. The binary install avoids this, but contributors who want to build from source need a very recent Go toolchain.

Smoke Test Results

Tested on macOS (host), aarch64. Downloaded pre-built v0.33.0 binary from GitHub Releases.

Run A. Fresh install, no pre-existing config

$ curl -fSL -o /tmp/waza-darwin-arm64 https://github.com/microsoft/waza/releases/download/v0.33.0/waza-darwin-arm64
✅ Downloaded 110MB binary

$ chmod +x /tmp/waza-darwin-arm64 && /tmp/waza-darwin-arm64 --version
waza version 0.33.0
✅ Version check passed

$ /tmp/waza-darwin-arm64 --help
Waza is a command-line tool for evaluating Agent Skills.
Available Commands: cache check compare completion coverage dev grade help init models new quality results run serve session suggest tokens
✅ All 16 subcommands present

Pass rate: 3 of 3. Binary install works on first try with no dependencies.

Run B. Functional verification (mock engine)

$ /tmp/waza-darwin-arm64 init /tmp/waza-test-project --no-skill
✅ Project created: 6 items set up

$ cd /tmp/waza-test-project && /tmp/waza-darwin-arm64 new skill test-skill
✅ Skill created: 6 file(s) scaffolded

$ /tmp/waza-darwin-arm64 check test-skill
📋 Compliance Score: Low (description too short)
📐 Spec Compliance: 9/9 checks passed
📊 Token Budget: 78 / 500 tokens
🧪 Evaluation Suite: Found (eval.yaml detected)
📐 Schema Validation: Passed (3 task file(s) validated)
✅ Check command ran successfully

$ /tmp/waza-darwin-arm64 tokens count skills/
skills/test-skill/SKILL.md 78 tokens
✅ Token counting works

$ /tmp/waza-darwin-arm64 coverage
Coverage: 100.0% (1/1 fully covered)
✅ Coverage grid works

$ /tmp/waza-darwin-arm64 suggest skills/test-skill --dry-run
parsing suggest response: response is not valid suggestion YAML
❌ Suggest command errored (likely needs Copilot auth)

$ /tmp/waza-darwin-arm64 models
Error: 403 "unauthorized: not authorized to use this Copilot feature"
❌ Models list requires Copilot authentication

$ /tmp/waza-darwin-arm64 quality skills/test-skill
Error: requires Copilot authentication
❌ Quality scoring requires Copilot authentication

Pass rate: 5 of 8. The mock-engine features (init, new, check, tokens, coverage) all work. The Copilot-dependent features (suggest, models, quality) correctly identify the missing auth but could have clearer error messages.

Run C. Structural validation (repo inspection)

$ find . -name "*.go" -not -path './.git/*' | wc -l
371 Go files

$ find . -name "*_test.go" | wc -l
163 test files

$ grep -r "^func Test" --include="*_test.go" | wc -l
1716 test functions

$ find internal/graders -name "*.go" -not -name "*_test.go" | wc -l
14 grader implementations

$ find examples -type f | wc -l
40+ example files
✅ Structural metrics verified

Structural pass rate: 5 of 5. File counts, test counts, grader counts, and example counts all verified against documented claims.

What the runs tell you

The binary installs cleanly with zero dependencies (a single download). The scaffolding, checking, and structural analysis features work without any authentication. The Copilot dependency is real but well-scoped: it affects only the execution, model listing, and LLM-judge features. For skill authors who use Copilot, this is transparent. For others, the mock engine and structural checks still provide significant value.

Setup Walkthrough

1. Download the binary: curl -fsSL https://raw.githubusercontent.com/microsoft/waza/main/install.sh | bash (auto-detects OS and architecture, verifies checksum).
2. Verify: waza --version should output waza version 0.33.0 or later.
3. Initialize a project: waza init my-project && cd my-project.
4. Create a skill: waza new skill my-skill. Edit skills/my-skill/SKILL.md with your skill definition.
5. Check readiness: waza check my-skill. Fix any compliance issues.
6. Run evals: waza run my-skill (requires Copilot auth) or use the mock executor for structural testing.

Post-install gotcha: waza models, waza quality, and waza run with the default copilot-sdk executor all require GitHub Copilot access. Set executor: mock in your .waza.yaml to run structural evals without Copilot.

Alternatives

1. anthropics/skill-creator - Anthropic's meta-skill for creating skills. Teaches agents to write SKILL.md files but does not evaluate them. Complementary to Waza (create with skill-creator, evaluate with Waza).
2. snyk/agent-scan - Snyk's security scanner for agent skills. Focuses on vulnerability detection rather than quality evaluation. Different concern, can be used alongside Waza.
3. tech-leads-club/agent-skills - A validated skill registry that includes manual review. Community-driven quality gate rather than automated tooling. Waza automates what this repo does manually.