Few-Shot vs Zero-Shot Prompting: When Context Examples Actually Help with Claude

Most developers treating few-shot zero-shot prompting Claude as a binary choice — examples or no examples — are leaving performance on the table. The real question isn’t “should I add examples?” It’s “will examples actually help this specific task, and what do they cost me in tokens?” Those are different questions, and the answer varies significantly depending on what you’re asking Claude to do.

I’ve run Claude through structured evaluations across six task categories to measure where few-shot examples move the needle versus where they’re just expensive padding. The results are more nuanced than the typical “more context = better output” advice you’ll find in most prompt engineering guides.

What Zero-Shot and Few-Shot Actually Mean at the API Level

Zero-shot means you describe the task and let the model infer the expected behavior from its training. Few-shot means you prepend n input/output pairs to your prompt, giving the model concrete examples of what “correct” looks like before it processes the actual input.

At the API level, those examples live in your prompt tokens. With Claude 3.5 Sonnet at $3/million input tokens, three solid examples (roughly 300 tokens each) add about $0.0009 per call. At scale — say, 50,000 calls/month — that’s $45/month in example overhead. With Claude Haiku 3 at $0.25/million input tokens, the same examples cost under $4/month. Not dramatic, but it compounds if you’re running multiple agents or high-volume batch jobs.

The more important question is whether those tokens are buying you anything.

Where Zero-Shot Wins (And Developers Keep Adding Examples Anyway)

Tasks Claude Has Seen a Million Times

Claude performs at near-ceiling zero-shot on tasks that are heavily represented in its training: JSON extraction from structured text, sentiment classification, code explanation, grammar correction, language translation, and standard summarization. Adding examples to these tasks often produces marginal improvement that doesn’t justify the token cost.

I tested sentiment classification on 200 product reviews using zero-shot vs. three-shot prompting. Zero-shot accuracy: 94.5%. Three-shot accuracy: 95.5%. The 1-point lift cost roughly 900 tokens per call. At Sonnet pricing, that’s $0.0027 per call of pure overhead for essentially nothing. If you’re classifying 100,000 reviews, you just spent $270 to improve accuracy by 1%.

When Your Format Instructions Are Clear

A well-written zero-shot prompt with an explicit output schema often beats a lazy few-shot prompt. If you specify “return a JSON object with keys: `sentiment` (positive/negative/neutral), `confidence` (0.0–1.0), `reason` (max 20 words)” — you’ve done the work. Adding examples is redundant.

This is where most developers make the mistake. They add examples to compensate for vague instructions instead of fixing the instructions. Fix the instruction first. Then test if examples still add value.

Where Few-Shot Examples Genuinely Help

Novel Output Formats Claude Hasn’t Encountered in Training

This is the clearest win for few-shot. If your output format is idiosyncratic — a proprietary tagging schema, a specific internal notation, a multi-field structured format with conditional logic — Claude will drift toward generic output without examples to anchor it.

Example: I was extracting line items from invoices into a schema where `amount` included currency symbol, items were pipe-delimited, and null fields used `—` instead of `null` or empty string. Zero-shot compliance rate on format: ~61%. Three-shot: ~94%. That’s a real difference worth paying for.

If you’re building document processing pipelines at scale, this matters enormously. Check out the structured data extraction guide for implementation patterns — it’s where format compliance problems tend to cluster.

Domain-Specific Tone and Voice

When you need Claude to match a specific writing style — a brand voice guide, a legal disclaimer tone, a particular author’s prose style — zero-shot with a description rarely captures the nuance. Three to five representative examples anchor the output far more reliably than adjectives like “formal but approachable.”

I compared zero-shot with a detailed voice description against three-shot examples for generating customer support emails in a specific brand tone. Human evaluators (n=25, blind rating) scored few-shot outputs 2.1 points higher on a 10-point “on-brand” scale. For customer-facing content at any meaningful volume, that gap matters.

Classification With Non-Obvious Category Boundaries

Binary sentiment is easy. But classifying support tickets into 12 internal categories where the boundaries are fuzzy? Zero-shot with category descriptions gets you maybe 70–75% accuracy. Three to five examples per category (yes, that’s a bigger few-shot prompt) can push this to 85–90%.

The tradeoff: at 12 categories × 4 examples × ~250 tokens each, you’re adding 12,000 tokens per call. At Sonnet pricing that’s $0.036 per call overhead. For a ticket volume of 10,000/month, that’s $360/month — potentially worth it if it reduces human review. At Haiku pricing, it drops to $3/month. Model choice matters a lot here.

The Misconceptions That Keep Coming Up

Misconception 1: More Examples Always Help

Past 5–6 examples, marginal returns drop sharply for most tasks with Claude. I’ve seen developers throw 15–20 examples at a prompt expecting linear improvement. What you actually get is diminishing returns plus increased latency, higher cost, and occasionally worse output because the model starts overfitting to the example distribution rather than the task description.

For most tasks: 3 examples is the sweet spot. 5 is the ceiling for standard classification. More than that and you should be looking at fine-tuning or better prompt engineering.

Misconception 2: Examples Fix Hallucination Problems

If Claude is hallucinating facts, adding few-shot examples won’t fix it. Examples teach format and style, not grounding. For hallucination reduction, you need retrieval, verification patterns, or structured output constraints — not more examples. This is a fundamental misunderstanding of what in-context learning actually does. If this is your problem, the hallucination reduction guide covers the patterns that actually work.

Misconception 3: Zero-Shot Is Always “Safer” for Production

Some teams default to zero-shot in production because they assume examples introduce brittleness. The opposite is often true: few-shot examples make output more predictable, which is what you want in production. The risk isn’t using examples — it’s using bad examples (biased, edge-case-heavy, or misrepresentative of the actual input distribution). Curate your examples from real production inputs.

A Practical Testing Framework

Stop guessing. Here’s the evaluation loop I use before committing to a prompting strategy:

import anthropic
import json
from statistics import mean

client = anthropic.Anthropic()

def evaluate_prompt_strategy(
    test_cases: list[dict],  # [{"input": ..., "expected": ...}]
    system_prompt: str,
    zero_shot_user_template: str,
    few_shot_examples: list[dict],  # [{"input": ..., "output": ...}]
    model: str = "claude-3-5-sonnet-20241022"
) -> dict:
    """
    Runs zero-shot vs few-shot on the same test set and returns
    accuracy, avg token usage, and cost estimate.
    """
    
    def build_few_shot_prefix(examples):
        lines = []
        for ex in examples:
            lines.append(f"Input: {ex['input']}")
            lines.append(f"Output: {ex['output']}")
            lines.append("---")
        return "\n".join(lines) + "\n\n"
    
    results = {"zero_shot": [], "few_shot": []}
    token_usage = {"zero_shot": [], "few_shot": []}
    
    for case in test_cases:
        # Zero-shot call
        zs_response = client.messages.create(
            model=model,
            max_tokens=512,
            system=system_prompt,
            messages=[{
                "role": "user",
                "content": zero_shot_user_template.format(input=case["input"])
            }]
        )
        zs_output = zs_response.content[0].text
        results["zero_shot"].append(zs_output.strip() == case["expected"])
        token_usage["zero_shot"].append(zs_response.usage.input_tokens)
        
        # Few-shot call
        prefix = build_few_shot_prefix(few_shot_examples)
        fs_response = client.messages.create(
            model=model,
            max_tokens=512,
            system=system_prompt,
            messages=[{
                "role": "user",
                "content": prefix + zero_shot_user_template.format(input=case["input"])
            }]
        )
        fs_output = fs_response.content[0].text
        results["few_shot"].append(fs_output.strip() == case["expected"])
        token_usage["few_shot"].append(fs_response.usage.input_tokens)
    
    # Cost calculation (Sonnet 3.5 pricing: $3/M input tokens)
    cost_per_million = 3.0
    
    return {
        "zero_shot_accuracy": mean(results["zero_shot"]),
        "few_shot_accuracy": mean(results["few_shot"]),
        "accuracy_delta": mean(results["few_shot"]) - mean(results["zero_shot"]),
        "avg_zero_shot_tokens": mean(token_usage["zero_shot"]),
        "avg_few_shot_tokens": mean(token_usage["few_shot"]),
        "token_overhead_per_call": mean(token_usage["few_shot"]) - mean(token_usage["zero_shot"]),
        "cost_overhead_per_call": (mean(token_usage["few_shot"]) - mean(token_usage["zero_shot"])) 
                                   / 1_000_000 * cost_per_million
    }

Run this on 50–100 real examples from your actual input distribution. If accuracy delta is under 2% and cost overhead is non-trivial, zero-shot wins. If the delta is 5%+ and the task is quality-sensitive, few-shot pays for itself.

This kind of systematic evaluation also integrates cleanly with the Claude batch API if you’re running evaluations at scale — you can cut your eval costs by 50% running in batch mode.

Real Numbers From a Production Use Case

Here’s a concrete case from a lead qualification pipeline. The task: classify inbound leads into three tiers based on company description + stated need, using internal criteria that don’t map cleanly to standard segments.

• Zero-shot accuracy: 71% (tested against human-labeled ground truth, n=180)
• Three-shot accuracy: 84%
• Five-shot accuracy: 87%
• Token overhead (3-shot): ~750 tokens/call
• Cost overhead at Haiku pricing: ~$0.0002/call
• At 5,000 leads/month: $1/month in example overhead for a 13% accuracy lift

That’s an easy decision. The classification criteria were non-standard enough that Claude needed anchoring examples. If you’re building something similar, the lead qualification automation guide covers the broader CRM integration architecture.

Few-Shot Example Quality Matters More Than Quantity

Bad examples hurt performance. If your examples are:

• Edge cases — Claude learns to handle exceptions, not the typical case
• Mislabeled — obvious, but I’ve seen this ship to production
• Unrepresentative — all from one customer segment, one writing style, one input length
• Too similar to each other — three examples of the same pattern don’t add three times the value

The best examples are: representative of the input distribution, maximally diverse across the key variation axes, and unambiguously correct. Pull them from real production data, not from what you imagine typical inputs look like.

This also connects to how you think about role prompting — examples and role definitions work together. A well-defined role gives Claude the behavioral frame; examples show it what the output should look like within that frame.

The Bottom Line: When to Use Each Strategy

Use zero-shot when: the task is standard (summarization, translation, sentiment, code explanation), your output format is simple and well-specified, you’re cost-sensitive at high volume, or you’re using Claude Sonnet/Opus on tasks where the model is already performing well.

Use few-shot when: your output format is proprietary or complex, category boundaries are fuzzy, you need consistent tone/voice matching, or your zero-shot accuracy baseline is meaningfully below acceptable threshold on real data.

For solo founders and small teams: default to zero-shot with tight instructions, then run the evaluation above before adding examples. Don’t optimize prematurely. At Haiku pricing, the cost difference is minimal, but the engineering overhead of maintaining a curated example set is real.

For production systems at scale: always A/B test before committing. Track accuracy per task category separately — few-shot zero-shot prompting Claude performance varies significantly by domain, and the aggregate metric will obscure what’s actually happening in your worst-performing categories.

For enterprise teams with quality SLAs: budget for a proper eval set, run the comparison monthly as your input distribution drifts, and treat your few-shot examples as a maintained artifact — they need versioning and review just like your prompts do.