Skill Creator: Build, Test, and Optimize Claude Code Skills with a Meta-Skill

Most Claude Code skills are built once and used as-is. They either work, or they don’t. You tweak them manually, run a few prompts by hand, and hope the behavior holds up across edge cases. This ad-hoc process works until it doesn’t — until the skill triggers at the wrong time, produces inconsistent output, or fails silently when the input drifts from what you tested.

The Skill Creator addresses this problem systematically. It’s a meta-skill — a skill for building skills — that guides you through intent capture, draft writing, evaluation, iteration, and description optimization. Rather than shipping a skill based on gut feel, you ship one that’s been tested against concrete scenarios, evaluated both qualitatively and quantitatively, and tuned so it triggers correctly in the right contexts.

This is the difference between a skill that works in your demo and one that works reliably in production.

When to Use This Skill

The Skill Creator is designed for a specific set of high-value scenarios where the investment in structured development pays off:

• Greenfield skill development: You have a workflow in mind — maybe something you’ve already done manually with Claude several times — and you want to formalize it into a repeatable, triggerable skill. The Skill Creator helps you capture intent, draft the skill, and validate it before you ship.
• Iterating on an underperforming skill: You have a skill that mostly works but produces inconsistent results or triggers unpredictably. Rather than manually tweaking YAML and praying, you run evals to pinpoint the failure mode and iterate with data behind your decisions.
• Benchmarking and variance analysis: You want to understand not just whether a skill works, but how often it works and how much output varies across similar inputs. The Skill Creator supports quantitative benchmarking workflows for exactly this.
• Description optimization: Skill triggering accuracy depends heavily on the description Claude reads to decide whether to invoke a skill. The Skill Creator includes a dedicated description improver script that optimizes this text separately from the skill logic itself.
• Converting ad-hoc conversations into formalized skills: You ran a complex multi-step workflow with Claude and want to capture it as a reusable skill. The Skill Creator can extract the tools used, the sequence of steps, the corrections you made, and the observed input/output formats from your conversation history.

Key Features and Capabilities

Structured Skill Creation Lifecycle

The Skill Creator enforces a disciplined workflow: intent capture → draft → test case generation → evaluation runs → iteration → description optimization. You don’t have to follow every step — if you already have a draft, you jump straight to the eval loop. But the structure is there when you need it, preventing the common failure mode of shipping skills that were never properly tested.

Qualitative and Quantitative Evaluation

Evaluations happen at two levels simultaneously. While test prompts run in the background, the Skill Creator drafts quantitative evals — assertions about output structure, content, or behavior — and explains them to you. The eval-viewer/generate_review.py script then renders results in a reviewable format so you can assess quality at a glance. You get both the numbers and the human judgment working together.

Intent Extraction from Conversation History

If you’ve been doing something manually with Claude — a debugging workflow, a code review pattern, a documentation process — the Skill Creator can extract that workflow from the conversation history. It pulls out the tools used, the sequence of steps, corrections you made, and the input/output shapes it observed, then uses those to bootstrap the skill draft.

Adaptive Communication

The Skill Creator is explicitly designed to work with developers across experience levels. It reads context cues to calibrate its language — using technical terms like “JSON assertions” and “eval benchmarks” when the user clearly understands them, and shifting to plain explanations when the context suggests otherwise. For a skill-building tool that might be used by both senior engineers and non-technical domain experts, this matters.

Description Optimization Pass

Separate from the skill logic, the Skill Creator can run a dedicated description optimization pass using a standalone script. This improves the triggering accuracy of your skill — how reliably Claude recognizes when to invoke it — without touching the underlying skill implementation.

Quick Start Guide

Scenario 1: Building a Skill from Scratch

Start by describing what you want the skill to do. The Skill Creator will ask clarifying questions to nail down intent:

User: I want to build a skill for reviewing database migration files before they get applied.

Skill Creator: Got it. A few questions to nail down the scope:
1. What should the review check for? (e.g., missing rollback steps, destructive operations, index creation on large tables)
2. What input format — raw SQL files, migration framework output (Rails, Flyway, Alembic)?
3. Should the output be a structured report, inline comments, a pass/fail verdict, or something else?

Once intent is captured, you get a draft skill file and a set of test cases. The Skill Creator generates test prompts covering happy paths and edge cases, runs them, and produces an evaluation report.

Scenario 2: Iterating on an Existing Skill

If you already have a draft, jump straight to evaluation. Point the Skill Creator at your existing skill file:

User: I have a skill for generating commit messages but it's inconsistent. Here's the current skill file: [paste skill content]

Skill Creator: Let's run evals on it. I'll use your existing test cases if you have them, or generate a set now.
Do you want to add any specific cases that have been problematic? (e.g., large diffs, merge commits, non-English codebases)

After running the eval suite, you review results in the eval viewer:

python eval-viewer/generate_review.py --skill commit-message-generator --run latest

The output surfaces both quantitative metrics (assertion pass rates, output length variance) and the raw outputs for qualitative review. You identify patterns in the failures, update the skill, and rerun.

Scenario 3: Optimizing a Skill Description

Once the skill logic is solid, run the description optimizer to improve triggering accuracy:

python scripts/improve_description.py --skill migration-reviewer

This runs a separate optimization loop focused solely on the description text Claude reads when deciding whether to invoke your skill. The result is a revised description that triggers more reliably across the range of prompts your users will actually type.

Tips and Best Practices

Start with three to five concrete test cases before writing the skill

The instinct is to write the skill first and figure out evaluation later. Resist it. Writing test cases first forces you to articulate exactly what success looks like, which clarifies the skill’s scope and prevents you from writing overly general logic that satisfies no specific case well.

Treat the description and the logic as separate optimization problems

A technically excellent skill that never triggers is useless. A skill with a great description that produces bad output is annoying. Keep these concerns separated. Nail the logic first using the eval loop, then run the description optimizer as a final pass. Don’t conflate the two during iteration.

Use variance analysis, not just average performance

A skill with 80% average quality across runs might have very low variance (consistently mediocre) or very high variance (sometimes brilliant, sometimes broken). High variance is often worse for production use cases because users can’t predict what they’ll get. Run enough eval iterations to understand variance, not just central tendency.

Capture workflow conversions while the conversation is fresh

If you want to formalize a workflow you just completed with Claude, trigger the Skill Creator immediately while the conversation history is available. The longer you wait, the more context you’ll need to reconstruct manually. The Skill Creator can extract the structure directly from recent conversation history when it’s still in context.

Don’t skip the “vibe check” even when you have quantitative evals

Quantitative assertions catch regressions reliably, but they only measure what you thought to measure. Always do a qualitative pass through the eval viewer output. You’ll catch failure modes that no assertion anticipated — awkward phrasing, subtle misunderstandings of intent, correct answers that are technically right but practically useless.

Expand your test set before declaring the skill done

Start with five to ten test cases for fast iteration. But before shipping, expand to thirty or more cases that cover edge cases, adversarial inputs, and realistic variations in how users will phrase requests. Skills that look solid on small eval sets often have brittle edges that only appear at scale.

Conclusion

The Skill Creator is the tool you reach for when you want to stop shipping skills on faith and start shipping them on evidence. It brings a real development loop — draft, test, evaluate, iterate — to what is otherwise an opaque and ad-hoc process. For teams investing seriously in Claude Code skills, this meta-skill is foundational infrastructure rather than a nice-to-have.

Whether you’re capturing a workflow from scratch, debugging an underperforming skill, or squeezing the last few percentage points out of triggering accuracy, the Skill Creator provides the scaffolding to make that work systematic. The investment in proper evaluation pays back every time your skill performs correctly in a context you didn’t explicitly test.

Build the skill. Run the evals. Ship with confidence.

Skill template sourced from the claude-code-templates open source project (MIT License).