SkillOpt

What It Does

SkillOpt is a Python framework from Microsoft that automatically improves natural-language skill documents (like SKILL.md files) through iterative training. The core idea borrows from deep learning: a skill document is treated as "model weights," agent task execution is the "forward pass," and an optimizer model proposes text edits (the "gradients") based on failure analysis. Over multiple epochs and steps, the skill document evolves to produce better agent outcomes. The framework supports six benchmarks (SearchQA, ALFWorld, DocVQA, LiveMathematicianBench, SpreadsheetBench, OfficeQA), four model backends (Azure OpenAI, OpenAI, Anthropic Claude, local Qwen via vLLM), and provides a Gradio-based WebUI for monitoring. The target user is a researcher or engineer who wants to systematically improve how well an LLM agent follows a skill's instructions.

The Good

Deep learning analogy is more than marketing. The mapping from DL concepts to skill optimization is implemented in real code, not just documentation. Learning rate maps to edit budget per step (configurable 2 to 16 edits). LR schedulers (constant, linear, cosine, autonomous) control how aggressively the skill changes over training. Gradient clipping maps to edit ranking and selection. Momentum maps to "slow update," a longitudinal comparison mechanism at epoch boundaries that prevents catastrophic forgetting. Validation gating accepts or rejects candidate skills based on held-out performance. The code for all of these is in skillopt/optimizer/scheduler.py, skillopt/optimizer/clip.py, skillopt/optimizer/slow_update.py, and skillopt/evaluation/gate.py.

Solid codebase architecture. At 19,489 lines across 90 Python files, the project has clear separation of concerns. The environment adapter pattern (skillopt/envs/base.py) lets new benchmarks plug in with four methods: build_train_env, build_eval_env, rollout, and reflect. A template directory (skillopt/envs/_template/) provides starter code for new adapters. The optimizer pipeline has six explicit stages (rollout, reflect, aggregate, select, update, evaluate) orchestrated by a 1,912-line trainer (skillopt/engine/trainer.py). The skill edit engine supports five operations (append, insert_after, replace, delete, skip) with a protected "slow update region" that prevents the optimizer from overwriting longitudinal guidance.

Four edit application modes give flexibility. The skill_update_mode config supports patch (surgical edits), rewrite_from_suggestions (targeted rewrite), and full_rewrite_minibatch (complete rewrite based on batch analysis). The ranking prompt (skillopt/prompts/ranking.md) instructs the optimizer to prioritize edits by systematic impact, complementarity, generality, and actionability. This is not random text mutation: the optimizer is explicitly prompted to rank edits by how many failures they would prevent.

Documentation goes deep. The docs include a complete DL-to-SkillOpt analogy table, per-stage explanations with code analogies, a skill document guide, a configuration reference, and benchmark-specific setup instructions. The docs/guide/dl-analogy.md page is particularly useful: it lists 22 concept mappings and provides transfer rules for what DL intuitions apply (cosine schedule works, moderate LR is best, slow update helps) and what does not (larger batch size does not always help, skills converge faster than neural networks at 2 to 4 epochs).

CLI entry points work out of the box. Both skillopt-train --help and skillopt-eval --help produce clean usage output after a standard pip install -e .. The training script supports auto-resume from the last completed step, which is practical for long-running optimizations with expensive API calls.

The Bad

No public test suite. There are zero test files in the repository. No tests/ directory, no test_*.py files, no CI configuration (no .github/workflows/). For a 19,489-line Python project from Microsoft Research, this is a significant gap. The code quality is high, but without tests, there is no automated verification that the edit application logic, scheduler math, or patch aggregation remains correct as the code evolves. The pyproject.toml lists pytest under dev dependencies but provides no test runner configuration.

Requires paid API access for every operation. The training loop calls LLM APIs at every stage: rollout (target model executes tasks), reflect (optimizer model analyzes failures), aggregate (merging patches), select (ranking edits), and the optional slow update and meta skill stages. A single 4-epoch run with batch size 40, 8 analyst workers, and 3 analyst rounds will make hundreds to thousands of API calls. There is no --dry-run or --mock mode. Users cannot evaluate the framework's behavior without committing API budget. This is a high barrier for exploration.

Single commit on the public repo. The public repository has exactly one commit (4c1b74f). This means the entire development history was squashed before publication. While common for Microsoft research releases, it makes it impossible to trace when specific features were added, whether there were known issues that were fixed, or how the code evolved. The gitignore references internal docs and experiment files that were stripped before release, confirming this was cleaned for publication.

Six benchmark configs but no benchmark data. The configs reference data paths (data_path, split_dir) but no datasets are included or linked. The README says "Benchmark datasets are not included in this repository." The data/ directory is gitignored. Users must prepare their own data in the expected JSON format (train/val/test splits with specific fields). There is no sample dataset or data generation script to get started quickly. This means the "Quick Start" section of the README is not actually quick: you need to find or create a compatible dataset first.

Alpha status with no stability guarantee. The pyproject.toml classifies the project as Development Status :: 3 - Alpha. The version is 0.1.0. The one commit has no tags. There are no release notes, changelog, or semantic versioning history. Users adopting this should expect breaking changes.

Smoke Test Results

All smoke tests ran on macOS (aarch64) using Python 3.11 with a clean venv.

Run A. Fresh install, core modules

$ pip install -e .
 Successfully installed skillopt-0.1.0

$ skillopt-train --help
 usage: train.py [-h] --config CONFIG --split_dir SPLIT_DIR
 [--azure_openai_endpoint ...] [--num_epochs N] ...

$ skillopt-eval --help
 usage: eval_only.py [-h] --config CONFIG --skill SKILL
 [--split {valid_unseen,valid_seen,train,all}] ...

Pass rate: 10 of 10. All core modules import cleanly. Both CLI entry points respond to --help. The scheduler, edit engine, patch application, slow-update protection, and config loading all pass functional assertions.

What the runs tell you

The package installs cleanly and all non-API-dependent functionality works correctly. The edit application logic (append, replace, delete, insert_after) with protected region enforcement is verified. The scheduler math (cosine decay, autonomous mode) produces expected values. What cannot be tested is the actual training loop, which requires LLM API credentials for every stage. The gap between "everything that can be tested locally" (all pass) and "the core value proposition" (untestable without API keys) is where the partial functional verification comes from.

Setup Walkthrough

1. Clone and install: git clone https://github.com/microsoft/SkillOpt.git && cd SkillOpt && pip install -e .
2. Configure API credentials: cp .env.example .env and fill in at least one backend (Azure OpenAI recommended, or OpenAI/Anthropic/local Qwen).
3. Prepare benchmark data: create data/my_split/{train,val,test}/items.json following the format in the README.
4. Run training: python scripts/train.py --config configs/searchqa/default.yaml --split_dir data/my_split --azure_openai_endpoint
5. (Optional) Launch monitoring: pip install -e ".[webui]" && python -m skillopt_webui.app

Post-install gotcha: the AZURE_OPENAI_ENDPOINT environment variable is required even if you use a different backend. Without it, LLM calls fail. The .env.example file does not make this clear enough.

Alternatives

1. microsoft/waza - Microsoft's skill quality testing framework. Where SkillOpt optimizes skills, Waza tests them. They are complementary: use SkillOpt to improve a skill, then Waza to validate it.
2. darkrishabh/agent-skills-eval - A lighter-weight TypeScript test runner for agent skills (522 stars). Focused on CI/CD-style testing rather than optimization. Easier to adopt if you just want automated skill quality checks.
3. Manual iteration - Writing a skill, testing it manually, and editing it by hand. Free but slow. SkillOpt automates the manual loop at the cost of API spend.