Validation, Retry, and Feedback Loops

Official Exam Guide Objectives

Task 4.4: Implement validation, retry, and feedback loops for extraction quality

Knowledge of

• Retry-with-error-feedback: appending specific validation errors to the prompt on retry to guide the model toward correction
• The limits of retry: retries are ineffective when the required information is simply absent from the source document (vs format or structural errors)
• Feedback loop design: tracking which code constructs trigger findings (detected_pattern field) to enable systematic analysis of dismissal patterns
• The difference between semantic validation errors (values don't sum, wrong field placement) and schema syntax errors (eliminated by tool use)

Skills in

• Implementing follow-up requests that include the original document, the failed extraction, and specific validation errors for model self-correction
• Identifying when retries will be ineffective (e.g., information exists only in an external document not provided) versus when they will succeed (format mismatches, structural output errors)
• Adding detected_pattern fields to structured findings to enable analysis of false positive patterns when developers dismiss findings
• Designing self-correction validation flows: extracting "calculated_total" alongside "stated_total" to flag discrepancies, adding "conflict_detected" booleans for inconsistent source data

What You Need to Know

Extraction systems fail in production, and reliably: documents arrive shaped unexpectedly, figures fail to reconcile, values land in the wrong field. What distinguishes a robust pipeline is not the absence of those failures but what happens next. The validation-retry pattern converts them into a workflow that repairs itself.

Retry-with-Error-Feedback

A retry worth making carries three things back to the model:

1. The original document — there is nothing to re-read without it, and a retry is a re-reading
2. The failed extraction — its own previous answer, so the correction is targeted rather than a fresh attempt
3. The specific validation error — the one element that distinguishes this from simply asking again

// Retry with error feedback
const retryMessages = [
  {
    role: "user",
    content: `Original document:\n${originalDocument}\n\n` +
      `Your extraction:\n${JSON.stringify(failedExtraction)}\n\n` +
      `Validation error: Line items sum to £450 but stated_total is £500. ` +
      `Please re-extract, ensuring all line items are captured.`
  }
];

# Retry with error feedback: the message carries all three inputs the model
# needs — the original document, its failed attempt, and the exact error.
retry_messages = [
    {
        "role": "user",
        "content": (
            f"Original document:\n{original_document}\n\n"
            f"Your extraction:\n{json.dumps(failed_extraction)}\n\n"
            "Validation error: Line items sum to £450 but stated_total is £500. "
            "Please re-extract, ensuring all line items are captured."
        ),
    }
]

The gap between this and a bare retry is large. Re-running the same request against the same document invites the same reading of it, so the same mistake comes back. Naming the discrepancy gives the model somewhere to direct its attention — hunting for the line item it missed, checking which field a value landed in, redoing the arithmetic.

The Retry Effectiveness Boundary

This is the concept the exam tests most aggressively in this task statement. Retries have a clear effectiveness boundary:

Retries ARE effective for:

• Dates rendered in the wrong convention, or currency written inconsistently between fields
• Output whose shape is wrong — values nested at the wrong level, or assigned to the wrong property
• A value that is in the document and ended up under the wrong key
• Arithmetic that fails because a line item was skipped on the first pass

Retries are NOT effective for:

• Anything the source simply does not state
• Values that live in a second document nobody supplied
• Fields that would require knowledge from outside the material provided

The line between the two lists is whether the answer is present in what the model was given. Everything above it is a reading failure and re-reading can correct it; everything below it is an absence, and no number of attempts conjures a department name out of a document that never mentioned one. Items will offer a retry for both situations. For the second, the correct response is to route the extraction to human review, or to return null where the schema permits it.

Self-Correction Flow Design

Some validation can live inside the schema rather than in the logic that inspects its output:

calculated_total vs stated_total: Ask for both — the sum the model derives from the line items, and the total the document asserts. A disagreement between two fields is then visible in the extraction itself, with no external arithmetic required to notice it.

{
  "line_items": [
    { "description": "Widget A", "amount": 150.00 },
    { "description": "Widget B", "amount": 300.00 }
  ],
  "calculated_total": 450.00,
  "stated_total": 500.00,
  "total_discrepancy": true
}

conflict_detected booleans: Where a source may contradict itself, give the model a field to say so. A document stating "payment due: 30 days" in one place and "payment terms: net 60" in another should surface both readings with conflict_detected: true, rather than the model quietly selecting whichever it encountered first.

detected_pattern Fields

Review and analysis pipelines benefit from recording what triggered each finding, not only the finding itself:

{
  "finding": "Potential SQL injection vulnerability",
  "severity": "critical",
  "detected_pattern": "string concatenation in SQL query",
  "file": "user_service.py",
  "line": 42
}

That field turns dismissals into data. Group the findings developers reject by detected_pattern and the weak rules identify themselves — if everything triggered by "variable shadowing in nested scope" is being waved through, that pattern's prompt needs work rather than the reviewers needing persuasion. Extract, validate, gather what was dismissed, refine, repeat.

Schema Syntax Errors vs Semantic Validation Errors

The exam distinguishes between these two error categories:

Schema syntax errors — Malformed JSON, missing required fields, wrong data types. Eliminated entirely by tool_use with JSON schemas (covered in Task Statement 4.3).

Semantic validation errors — Structurally valid output carrying wrong values: totals that do not reconcile, dates in an impossible order, data in the wrong field. Nothing about the schema can detect these, because each one satisfies it. They need validation logic written separately, and they are what retry loops exist for.

These two task statements deliberately overlap, and the point being tested is the boundary between them: tool_use closes the first category completely and the second not at all.

Deep Dive

The API already retries malformed tool calls internally

Before you write a single line of retry logic, the Claude API has a baked-in version of this pattern: "If a tool request is invalid or missing parameters, Claude will retry 2-3 times with corrections before apologizing to the user." This is the same retry-with-error-feedback principle operating one layer down — the API surfaces its own validation error to the model and lets it self-correct. Your application-level retry loop for semantic errors (sums, field placement, fabrication) is the same mechanism extended to checks the API cannot perform on its own.

Sourceplatform.claude.com › handle-tool-callsfetched 2026-07-30

is_error is the general-purpose feedback channel

The mechanism for reporting a failure back to the model already exists in the Messages API: "If the tool itself throws an error during execution... you can return the error message in the content along with \"is_error\": true" and "Claude will then incorporate this error into its response to the user." Retry-with-error-feedback for extraction validation is a direct application of this same contract — instead of a tool execution failure, the "error" you report is a semantic validation failure, delivered as specific text in the next user turn rather than a generic is_error tool_result.

Sourceplatform.claude.com › handle-tool-callsfetched 2026-07-30

Rules-based feedback outranks vague or judge-based feedback

Anthropic's Agent SDK guidance ranks feedback quality explicitly: "The best form of feedback is providing clearly defined rules for an output, then explaining which rules failed and why. Code linting is an excellent form of rules-based feedback. The more in-depth in feedback the better." This is precisely the shape of "Line items sum to £450 but stated_total is £500" — a specific rule, a specific violation, not a generic "validation failed." The same guidance places another common alternative near the bottom of the hierarchy: judging output with a second LLM call is "generally not a very robust method, and can have heavy latency tradeoffs" — reinforcing why this task statement favours deterministic, rule-based validation logic over an LLM re-grading its own extraction.

Sourceclaude.com › building-agents-with-the-claude-agent-sdkfetched 2026-07-30

Self-verification is valuable — but only inside the fixable boundary

The Agent SDK blog frames self-checking as a reliability multiplier: "Agents that can check and improve their own output are fundamentally more reliable — they catch mistakes before they compound." Read alongside this task's retry-effectiveness boundary, the claim only holds for errors the model can actually correct by re-examining information it already has access to (the source document). It is not a claim that self-checking can conjure data the source document never contained — that failure mode still routes to human review, not another retry.

Sourceclaude.com › building-agents-with-the-claude-agent-sdkfetched 2026-07-30

Repeated failures on the same pattern call for a formal rule, not another retry

For findings that fail repeatedly in the same recognisable way — the exact scenario detected_pattern tracking is built to surface — the documented fix is architectural, not another retry attempt: "If your agent fails at a task repeatedly, can you add a formal rule in your tool calls to identify and fix the failure?" This is the systematic-improvement half of the loop: detected_pattern data tells you which pattern keeps failing or getting dismissed; the fix is a prompt or schema change for that pattern, not persistence with the retry loop.

Sourceclaude.com › building-agents-with-the-claude-agent-sdkfetched 2026-07-30

Quick Reference

Exam Traps

Practice Scenario

Build Exercise

Build a Validation-Retry Loop for Document Extraction

Difficulty: Advanced (3/4)

60 minutes

1. Define an extraction tool with calculated_total and stated_total fields, a conflict_detected boolean, and detected_pattern fields for tracking which constructs trigger findings

Why: Self-correction fields like calculated_total vs stated_total enable automatic discrepancy detection without external logic. conflict_detected booleans and detected_pattern fields create the data foundation for systematic prompt improvement.

You should see: A JSON schema with separate calculated_total and stated_total number fields, a total_discrepancy boolean, a conflict_detected boolean, and a detected_pattern string field on each finding in the line_items array.

2. Implement validation logic that checks: field completeness, numerical consistency (calculated sum matches stated total), enum validity, and date ordering

Why: Semantic validation catches errors that tool_use cannot. The exam distinguishes schema syntax errors (eliminated by tool_use) from semantic errors (wrong sums, misplaced values) that require validation logic and retry loops.

You should see: A validation function that returns an array of specific, actionable error messages. Each error should state what was expected versus what was found, not just that validation failed.

3. Build the retry loop: on validation failure, construct a follow-up message containing the original document, the failed extraction, and the specific validation error

Why: Retry-with-error-feedback is dramatically more effective than naive retries. Without the specific error, the model has no guidance and typically reproduces the same mistake. With the error, the model can target its self-correction.

You should see: A retry message that includes all three elements: the original document text, the JSON of the failed extraction, and the specific validation error string. The model should produce a corrected extraction on retry.

4. Test with 5 documents: 2 with fixable errors (misplaced values, wrong totals) and 3 with unfixable errors (absent information) — verify the loop retries only fixable cases

Why: The retry effectiveness boundary is the most aggressively tested concept in this task statement. Retries fix format mismatches and structural errors but cannot create information absent from the source. The exam presents both scenarios and expects you to identify which is fixable.

You should see: The 2 fixable documents succeed after 1-2 retries with corrected totals or field placements. The 3 unfixable documents are correctly identified as having absent information and flagged for human review rather than retried.

5. Log detected_pattern data for each finding and analyse which patterns are most frequently dismissed to identify prompt refinement priorities

Why: detected_pattern fields create a systematic improvement loop. When developers consistently dismiss findings triggered by a specific pattern, that pattern likely needs prompt refinement. This turns dismissal data into actionable prompt improvement priorities.

You should see: A log or table showing each detected_pattern, its frequency, its dismissal rate, and a prioritised list of patterns needing prompt refinement. Patterns with high dismissal rates should be at the top.

Sources

• Claude Certified Architect Foundations Exam Guide — Task Statement 4.4 — Anthropic
• Tool Use (Function Calling) — Anthropic
• Building with Claude API (Skilljar) — Anthropic
• Handling Tool Calls — Anthropic
• Building Agents with the Claude Agent SDK — Anthropic

Appendix A — Build Exercise Step Hints

Progressive hints revealed by the "Stuck? Get a nudge" control on each step.

Step 1. Define an extraction tool with calculated_total and stated_total fields, a conflict_detected boolean, and detected_pattern fields for tracking which constructs trigger findings

Why: Self-correction fields like calculated_total vs stated_total enable automatic discrepancy detection without external logic. conflict_detected booleans and detected_pattern fields create the data foundation for systematic prompt improvement.

You should see: A JSON schema with separate calculated_total and stated_total number fields, a total_discrepancy boolean, a conflict_detected boolean, and a detected_pattern string field on each finding in the line_items array.

const extractionTool = {
  name: "extract_invoice",
  input_schema: {
    type: "object",
    properties: {
      line_items: {
        type: "array",
        items: {
          type: "object",
          properties: {
            description: { type: "string" },
            amount: { type: "number" },
            detected_pattern: { type: "string" }
          }
        }
      },
      calculated_total: { type: "number" },
      stated_total: { type: "number" },
      total_discrepancy: { type: "boolean" },
      conflict_detected: { type: "boolean" }
    },
    required: ["line_items", "calculated_total", "stated_total", "total_discrepancy"]
  }
};

extraction_tool = {
    "name": "extract_invoice",
    "input_schema": {
        "type": "object",
        "properties": {
            "line_items": {
                "type": "array",
                "items": {
                    "type": "object",
                    "properties": {
                        "description": {"type": "string"},
                        "amount": {"type": "number"},
                        "detected_pattern": {"type": "string"},
                    },
                },
            },
            # Extracting both totals is what makes the discrepancy detectable
            # without a second model call.
            "calculated_total": {"type": "number"},
            "stated_total": {"type": "number"},
            "total_discrepancy": {"type": "boolean"},
            "conflict_detected": {"type": "boolean"},
        },
        "required": ["line_items", "calculated_total", "stated_total", "total_discrepancy"],
    },
}

Step 2. Implement validation logic that checks: field completeness, numerical consistency (calculated sum matches stated total), enum validity, and date ordering

Why: Semantic validation catches errors that tool_use cannot. The exam distinguishes schema syntax errors (eliminated by tool_use) from semantic errors (wrong sums, misplaced values) that require validation logic and retry loops.

You should see: A validation function that returns an array of specific, actionable error messages. Each error should state what was expected versus what was found, not just that validation failed.

function validateExtraction(result) {
  const errors = [];
  const calculatedSum = result.line_items.reduce((sum, item) => sum + item.amount, 0);
  if (Math.abs(calculatedSum - result.stated_total) > 0.01) {
    errors.push(`Line items sum to ${calculatedSum} but stated_total is ${result.stated_total}`);
  }
  if (result.total_discrepancy !== (calculatedSum !== result.stated_total)) {
    errors.push("total_discrepancy flag does not match actual discrepancy state");
  }
  return errors;
}

def validate_extraction(result: dict) -> list[str]:
    errors = []
    calculated_sum = sum(item["amount"] for item in result["line_items"])
    if abs(calculated_sum - result["stated_total"]) > 0.01:
        errors.append(
            f"Line items sum to {calculated_sum} but stated_total is {result['stated_total']}"
        )
    if result["total_discrepancy"] != (calculated_sum != result["stated_total"]):
        errors.append("total_discrepancy flag does not match actual discrepancy state")
    return errors

Step 3. Build the retry loop: on validation failure, construct a follow-up message containing the original document, the failed extraction, and the specific validation error

Why: Retry-with-error-feedback is dramatically more effective than naive retries. Without the specific error, the model has no guidance and typically reproduces the same mistake. With the error, the model can target its self-correction.

You should see: A retry message that includes all three elements: the original document text, the JSON of the failed extraction, and the specific validation error string. The model should produce a corrected extraction on retry.

async function retryExtraction(originalDoc, failedResult, errors, maxRetries = 3) {
  let attempt = 0;
  let result = failedResult;
  while (errors.length > 0 && attempt < maxRetries) {
    const retryMessage = `Original document:\n${originalDoc}\n\n` +
      `Your extraction:\n${JSON.stringify(result, null, 2)}\n\n` +
      `Validation errors:\n${errors.join("\n")}\n\n` +
      `Please re-extract, fixing the identified errors.`;
    result = await extract(retryMessage);
    errors = validateExtraction(result);
    attempt++;
  }
  return { result, errors, attempts: attempt };
}

def retry_extraction(original_doc, failed_result, errors, max_retries=3) -> dict:
    attempt = 0
    result = failed_result
    while errors and attempt < max_retries:
        retry_message = (
            f"Original document:\n{original_doc}\n\n"
            f"Your extraction:\n{json.dumps(result, indent=2)}\n\n"
            f"Validation errors:\n{chr(10).join(errors)}\n\n"
            "Please re-extract, fixing the identified errors."
        )
        result = extract(retry_message)
        errors = validate_extraction(result)
        attempt += 1
    return {"result": result, "errors": errors, "attempts": attempt}

Step 4. Test with 5 documents: 2 with fixable errors (misplaced values, wrong totals) and 3 with unfixable errors (absent information) — verify the loop retries only fixable cases

Why: The retry effectiveness boundary is the most aggressively tested concept in this task statement. Retries fix format mismatches and structural errors but cannot create information absent from the source. The exam presents both scenarios and expects you to identify which is fixable.

You should see: The 2 fixable documents succeed after 1-2 retries with corrected totals or field placements. The 3 unfixable documents are correctly identified as having absent information and flagged for human review rather than retried.

function isFixableError(error, originalDoc) {
  // Fixable: numerical discrepancies, misplaced values
  if (error.includes("sum to") || error.includes("misplaced")) return true;
  // Unfixable: field references data not in document
  if (error.includes("not found in source")) return false;
  return false; // Default conservative
}

const fixableErrors = errors.filter(e => isFixableError(e, originalDoc));
const unfixableErrors = errors.filter(e => !isFixableError(e, originalDoc));
if (unfixableErrors.length > 0) flagForHumanReview(result, unfixableErrors);

def is_fixable_error(error: str, original_doc: str) -> bool:
    # Fixable: numerical discrepancies, misplaced values
    if "sum to" in error or "misplaced" in error:
        return True
    # Unfixable: field references data not in the document
    if "not found in source" in error:
        return False
    return False  # Default conservative


fixable_errors = [e for e in errors if is_fixable_error(e, original_doc)]
unfixable_errors = [e for e in errors if not is_fixable_error(e, original_doc)]
if unfixable_errors:
    flag_for_human_review(result, unfixable_errors)

Step 5. Log detected_pattern data for each finding and analyse which patterns are most frequently dismissed to identify prompt refinement priorities

Why: detected_pattern fields create a systematic improvement loop. When developers consistently dismiss findings triggered by a specific pattern, that pattern likely needs prompt refinement. This turns dismissal data into actionable prompt improvement priorities.

You should see: A log or table showing each detected_pattern, its frequency, its dismissal rate, and a prioritised list of patterns needing prompt refinement. Patterns with high dismissal rates should be at the top.

const patternStats = {};
for (const finding of allFindings) {
  const p = finding.detected_pattern;
  if (!patternStats[p]) patternStats[p] = { total: 0, dismissed: 0 };
  patternStats[p].total++;
  if (finding.wasDismissed) patternStats[p].dismissed++;
}

// Prioritise by impact: frequency * dismissal rate
const priorities = Object.entries(patternStats)
  .map(([pattern, stats]) => ({
    pattern,
    dismissalRate: stats.dismissed / stats.total,
    impact: stats.total * (stats.dismissed / stats.total)
  }))
  .sort((a, b) => b.impact - a.impact);

pattern_stats: dict[str, dict[str, int]] = {}
for finding in all_findings:
    p = finding["detected_pattern"]
    stats = pattern_stats.setdefault(p, {"total": 0, "dismissed": 0})
    stats["total"] += 1
    if finding["was_dismissed"]:
        stats["dismissed"] += 1

# Prioritise by impact: frequency * dismissal rate
priorities = sorted(
    (
        {
            "pattern": pattern,
            "dismissal_rate": stats["dismissed"] / stats["total"],
            "impact": stats["dismissed"],
        }
        for pattern, stats in pattern_stats.items()
    ),
    key=lambda p: p["impact"],
    reverse=True,
)

Appendix B — Interactive Study Prompts

Two prompts to paste into Claude. B1 drills the judgement the exam actually measures; B3 reviews the code you wrote for the Build Exercise above. The exam simulator between them is the interactive quiz on this page.

B1. Concept Check — Discrimination Drill

B2. Exam Simulator

B3. Build Coach — Code Review

The Build Exercise and its hint ladder are already on this page. This prompt is for the one thing the page cannot do: review the code you actually wrote.