Chain-of-thought debugging for Claude agents: extract reasoning, find failure points, fix behavior

When a Claude agent gives you a wrong answer, you have two choices: guess at what went wrong and iterate blindly, or read the agent’s actual reasoning and fix the exact failure point. Chain of thought debugging agents is the systematic version of the second approach — and it’s what separates agents that get tuned into reliable tools from ones that stay permanently flaky.

By the end of this tutorial, you’ll have a working debug harness that forces Claude to expose its step-by-step reasoning, a parser that flags where reasoning breaks down, and a prompt iteration loop you can actually run in CI.

1. Force verbose reasoning output — instrument your prompts to emit structured thinking
2. Parse and log reasoning traces — extract individual steps into a structured format
3. Classify failure modes — identify which step category caused the wrong output
4. Run a diff-based prompt fix loop — compare reasoning before and after prompt changes
5. Automate regression testing — lock down fixed behaviors against future prompt changes

Why Blind Debugging Doesn’t Scale

Most debugging workflows look like this: agent returns bad output → developer tweaks a word in the system prompt → re-runs the test → repeat. This is vibes-based engineering. It works up to about five test cases, then it completely breaks down when your agent has 40 edge cases and you can’t tell whether a prompt change fixed one thing and broke three others.

The fundamental problem is that LLMs are not black boxes in the way people treat them. Claude, in particular, will tell you exactly why it made a decision if you ask correctly. The reasoning is in there — you just need to extract it systematically.

This is also directly relevant to monitoring AI agents for misalignment: when you can read the chain of thought, you can catch an agent reasoning its way to an incorrect conclusion before it causes downstream damage.

Step 1: Force Verbose Reasoning Output

The first move is to instrument your system prompt so that Claude always emits its reasoning in a structured, parseable format before giving a final answer. Don’t rely on Claude’s default behavior — mandate the format explicitly.

SYSTEM_PROMPT = """
You are a customer support classification agent.

Before giving any final answer, you MUST output your reasoning in the following XML format:

<reasoning>
  <step id="1" category="input_parsing">What exactly is the user asking?</step>
  <step id="2" category="context_retrieval">What relevant context applies here?</step>
  <step id="3" category="rule_application">Which policies or rules are relevant?</step>
  <step id="4" category="decision">What is my decision and why?</step>
  <step id="5" category="confidence">How confident am I? What could go wrong?</step>
</reasoning>

<answer>
  Your final structured answer here.
</answer>

Never skip the reasoning block. Never merge reasoning into the answer block.
"""

The XML format is deliberate. JSON inside a string is fragile when Claude generates multi-line content. XML with clear delimiters survives nearly all edge cases in practice. The category attribute on each step is what lets you classify failure modes later.

You can adapt the step categories to your agent’s domain — a code-generation agent would use categories like requirement_parsing, algorithm_selection, edge_case_check. The point is to make the reasoning structure match the actual decision tree your agent should follow.

Step 2: Parse and Log Reasoning Traces

Once Claude is emitting structured reasoning, you need a parser that extracts individual steps into a format you can analyze across many test runs.

import re
import xml.etree.ElementTree as ET
from dataclasses import dataclass, field
from typing import Optional
import anthropic

@dataclass
class ReasoningStep:
    step_id: int
    category: str
    content: str

@dataclass
class AgentTrace:
    input: str
    steps: list[ReasoningStep] = field(default_factory=list)
    answer: str = ""
    raw_response: str = ""
    parse_error: Optional[str] = None

def parse_trace(raw_response: str, user_input: str) -> AgentTrace:
    trace = AgentTrace(input=user_input, raw_response=raw_response)
    
    # Extract reasoning block
    reasoning_match = re.search(
        r'<reasoning>(.*?)</reasoning>',
        raw_response,
        re.DOTALL
    )
    answer_match = re.search(
        r'<answer>(.*?)</answer>',
        raw_response,
        re.DOTALL
    )
    
    if not reasoning_match:
        trace.parse_error = "no_reasoning_block"
        return trace
    
    if not answer_match:
        trace.parse_error = "no_answer_block"
        return trace
    
    trace.answer = answer_match.group(1).strip()
    
    # Parse individual steps
    try:
        # Wrap in root for valid XML
        xml_str = f"<root>{reasoning_match.group(1)}</root>"
        root = ET.fromstring(xml_str)
        for step_el in root.findall('step'):
            trace.steps.append(ReasoningStep(
                step_id=int(step_el.get('id', 0)),
                category=step_el.get('category', 'unknown'),
                content=step_el.text.strip() if step_el.text else ""
            ))
    except ET.ParseError as e:
        trace.parse_error = f"xml_parse_error: {e}"
    
    return trace


def run_agent(user_input: str, client: anthropic.Anthropic) -> AgentTrace:
    response = client.messages.create(
        model="claude-opus-4-5",  # use Sonnet for cheaper iteration: ~$0.003/call
        max_tokens=2048,
        system=SYSTEM_PROMPT,
        messages=[{"role": "user", "content": user_input}]
    )
    raw = response.content[0].text
    return parse_trace(raw, user_input)

Log every trace to a file or database indexed by test case ID and prompt version. SQLite works fine at this scale — you’re not doing streaming analytics, you’re doing forensics.

Step 3: Classify Failure Modes

Now that you have structured traces, you can classify where the reasoning broke down, not just that it gave a wrong answer. This is the core of chain of thought debugging for agents.

from enum import Enum

class FailureCategory(Enum):
    NO_REASONING = "no_reasoning_block"
    INPUT_PARSING = "input_parsing"
    CONTEXT_RETRIEVAL = "context_retrieval"
    RULE_APPLICATION = "rule_application"
    DECISION = "decision"
    CONFIDENCE = "confidence"
    CORRECT = "correct"

def classify_failure(
    trace: AgentTrace,
    expected_answer: str,
    evaluator_fn  # callable(actual, expected) -> bool
) -> FailureCategory:
    
    if trace.parse_error:
        return FailureCategory.NO_REASONING
    
    is_correct = evaluator_fn(trace.answer, expected_answer)
    if is_correct:
        return FailureCategory.CORRECT
    
    # Heuristic: find the earliest step with thin or suspicious content
    # "thin" = less than 20 chars likely means Claude phoned it in
    for step in trace.steps:
        if len(step.content) < 20:
            return FailureCategory[step.category.upper().replace("-", "_")]
    
    # If all steps look populated but answer is wrong,
    # failure is at the decision/synthesis stage
    return FailureCategory.DECISION


# Run across a test suite
def analyze_test_suite(
    test_cases: list[dict],
    client: anthropic.Anthropic,
    evaluator_fn
) -> dict:
    results = {cat: [] for cat in FailureCategory}
    
    for case in test_cases:
        trace = run_agent(case["input"], client)
        category = classify_failure(trace, case["expected"], evaluator_fn)
        results[category].append({
            "input": case["input"],
            "trace": trace,
            "expected": case["expected"]
        })
    
    return results

After one run over your test suite, you get a breakdown like: 60% of failures happen at rule_application, 30% at input_parsing, 10% are missing reasoning blocks entirely. That tells you exactly where to fix your system prompt rather than guessing.

This structured approach pairs well with reducing LLM hallucinations in production — the same reasoning traces that reveal wrong decisions also expose where the model is confabulating context it doesn’t actually have.

Step 4: Run a Diff-Based Prompt Fix Loop

With failure categories in hand, you can now make targeted prompt changes and measure whether they move the needle. The discipline here is one variable at a time.

import json
from datetime import datetime

def prompt_experiment(
    baseline_prompt: str,
    candidate_prompt: str,
    test_cases: list[dict],
    client: anthropic.Anthropic,
    evaluator_fn,
    experiment_name: str
) -> dict:
    """
    Run the same test suite against two prompts and compare failure distributions.
    Cost at claude-sonnet-4-5 pricing: ~$0.003 per call * n test cases * 2 variants
    """
    global SYSTEM_PROMPT
    
    SYSTEM_PROMPT = baseline_prompt
    baseline_results = analyze_test_suite(test_cases, client, evaluator_fn)
    
    SYSTEM_PROMPT = candidate_prompt
    candidate_results = analyze_test_suite(test_cases, client, evaluator_fn)
    
    baseline_correct = len(baseline_results[FailureCategory.CORRECT])
    candidate_correct = len(candidate_results[FailureCategory.CORRECT])
    
    report = {
        "experiment": experiment_name,
        "timestamp": datetime.utcnow().isoformat(),
        "n_cases": len(test_cases),
        "baseline_accuracy": baseline_correct / len(test_cases),
        "candidate_accuracy": candidate_correct / len(test_cases),
        "delta": (candidate_correct - baseline_correct) / len(test_cases),
        "baseline_failure_distribution": {
            k.value: len(v) for k, v in baseline_results.items()
        },
        "candidate_failure_distribution": {
            k.value: len(v) for k, v in candidate_results.items()
        }
    }
    
    print(json.dumps(report, indent=2))
    return report

Running 50 test cases against two prompt variants with claude-sonnet-4-5 costs roughly $0.30 total. That’s cheap enough to run on every pull request that touches your system prompt. If you’re iterating frequently, check out how system prompt frameworks for consistent agent behavior can help you manage prompt versions at scale.

Step 5: Automate Regression Testing

The final step is turning your debug harness into a regression suite. Every time you fix a failure case, add it to a locked test set with a strict evaluator. If a future prompt change reintroduces that failure, the CI run catches it before deployment.

import pytest

# Locked test cases — these failures were explicitly fixed
REGRESSION_CASES = [
    {
        "input": "I want to cancel my subscription but keep access until month end",
        "expected_category_not_in": [
            FailureCategory.INPUT_PARSING,  # must correctly parse dual intent
        ],
        "expected_answer_contains": "month end"
    },
    # ... add cases as you fix them
]

@pytest.mark.parametrize("case", REGRESSION_CASES)
def test_no_regression(case, anthropic_client):
    trace = run_agent(case["input"], anthropic_client)
    
    # Check reasoning structure is intact
    assert not trace.parse_error, f"Reasoning parse failed: {trace.parse_error}"
    assert len(trace.steps) >= 4, "Missing reasoning steps"
    
    # Check failure categories that were previously fixed stay fixed
    for banned_category in case.get("expected_category_not_in", []):
        for step in trace.steps:
            if step.category == banned_category.value:
                assert len(step.content) > 30, (
                    f"Step '{banned_category.value}' is thin — likely regression"
                )
    
    # Check answer content
    for fragment in case.get("expected_answer_contains", []):
        assert fragment.lower() in trace.answer.lower(), (
            f"Expected '{fragment}' in answer: {trace.answer}"
        )

Common Errors

Claude ignores the reasoning format and answers directly

This happens when the instruction to reason is buried in a long system prompt, or when the user message is phrased as an urgent request. Claude deprioritizes format instructions under conversational pressure. Fix: move the format instruction to the very end of the system prompt with a line like CRITICAL: Always begin your response with the reasoning XML block, regardless of the question type. Also add an assistant prefill: start the assistant turn with <reasoning> to force the format.

# Assistant prefill forces Claude to start in the right format
response = client.messages.create(
    model="claude-opus-4-5",
    max_tokens=2048,
    system=SYSTEM_PROMPT,
    messages=[
        {"role": "user", "content": user_input},
        {"role": "assistant", "content": "<reasoning>"}  # prefill
    ]
)
# Note: prepend "<reasoning>" to the response text before parsing
raw = "<reasoning>" + response.content[0].text

The XML parser breaks on special characters in reasoning content

Claude sometimes puts angle brackets, ampersands, or quotes inside reasoning steps when reasoning about code or HTML. Standard XML parsing chokes on this. Fix: wrap step content in CDATA sections, or use a regex-based fallback parser instead of ElementTree for step content extraction. In practice I use regex as the primary parser and only fall back to ElementTree for validating structure.

Failure classification gives wrong category for multi-step failures

The heuristic of “find the first thin step” breaks when Claude writes verbose but incorrect reasoning. A step can be 200 characters and still be completely wrong. For high-stakes agents, replace the length heuristic with a second Claude call that evaluates each step’s correctness independently. Yes, this doubles cost — roughly $0.006 per evaluation call with Haiku — but for code-generation agents where a wrong step has real consequences, it’s worth it.

What to Build Next

The natural extension of this debugging harness is a live monitoring dashboard: pipe production traces into the same classifier, aggregate failure categories by day, and set an alert when rule_application failures spike above 10%. This catches prompt drift — where a model update changes reasoning behavior without you changing anything. Pair this with the patterns in LLM fallback and retry logic so that when your monitoring detects a spike, your agent automatically routes to a backup model while you investigate.

You could also extend the prompt experiment loop into a Bayesian optimization setup: instead of manually proposing candidate prompts, use a second LLM to generate prompt variants targeting the most common failure category, then auto-select the best performer. That’s roughly three days of engineering work and would eliminate most of the manual prompt-tuning cycle entirely.