Human Review & Confidence Calibration

Official Exam Guide Objectives

Task 5.5: Design human review workflows and confidence calibration.

Knowledge of

• The risk that aggregate accuracy metrics (e.g., 97% overall) may mask poor performance on specific document types or fields
• Stratified random sampling for measuring error rates in high-confidence extractions and detecting novel error patterns
• Field-level confidence scores calibrated using labeled validation sets for routing review attention
• The importance of validating accuracy by document type and field segment before automating high-confidence extractions

Skills in

• Implementing stratified random sampling of high-confidence extractions for ongoing error rate measurement and novel pattern detection
• Analyzing accuracy by document type and field to verify consistent performance across all segments before reducing human review
• Having models output field-level confidence scores, then calibrating review thresholds using labeled validation sets
• Routing extractions with low model confidence or ambiguous/contradictory source documents to human review, prioritizing limited reviewer capacity

What You Need to Know

Human review is what stands between an automated extraction system and its mistakes. The examinable question is not whether to use reviewers but how to spend a fixed amount of their attention for the most accuracy — which turns on calibrating confidence, distrusting aggregates, and sampling in strata.

The Aggregate Metrics Trap

This is the most dangerous misconception in production extraction systems. A system reports 97% overall accuracy. The team celebrates. Management approves full automation for all high-confidence extractions.

Underneath that number, the picture is not uniform. Dates come off standard invoices at 99.5%. Handwritten receipts run at 60%. Scanned PDFs with poor OCR manage 72%. International documents in unfamiliar formats reach 45%.

An average conceals its worst components by construction, and the segments hidden here are not the harmless ones — receipts handwritten by field staff, invoices from new international suppliers, scanned historical documents pulled for a compliance audit. The places the system is weakest are frequently the places an error costs most.

The rule, then: measure accuracy per document type and per field before anything is automated. An aggregate is never sufficient grounds for that decision on its own.

Standard invoices carry the volume, so they carry the average — and three document types sit at accuracy nobody would sign off if they saw it, weighted into invisibility.

Stratified Random Sampling

Validating once is not enough; accuracy has to be watched. Stratified sampling means drawing a representative sample from each stratum — document type, confidence band, field type — and putting humans on it.

The part that matters, and that the exam tests, is that the high-confidence extractions must be sampled too. Low-confidence items already go to a reviewer, so nothing new is learned by checking them. High-confidence items are the ones running unattended, which makes them the only place a new error pattern can establish itself unobserved.

Stratified sampling serves two purposes:

1. Ongoing accuracy measurement — check each segment is still performing at the rate it was validated at.
2. Novel error pattern detection — surface failure modes that were not present when the validation set was assembled.

Without it the automated portion is unmonitored by definition. A new document format could introduce a systematic error and the first indication would be a downstream business process failing on data nobody checked.

Field-Level Confidence Calibration

Confidence can be reported per field. An invoice extraction might come back as:

{
  "vendorName":  {"value": "Acme Corp",   "confidence": 0.98},
  "invoiceDate": {"value": "2024-03-15",  "confidence": 0.95},
  "totalAmount": {"value": "$1,247.83",   "confidence": 0.72},
  "lineItems":   {"value": [],            "confidence": 0.61}
}

Those numbers are not accuracy rates. A model reporting 0.95 might be right 88% of the time on one field type and 99% on another — the score orders the model's own certainty and says nothing about how often that certainty is warranted.

Calibration requires labelled validation sets (ground truth data). Run documents whose correct extraction you already know, compare reported confidence against what actually happened, and build the curve. What emerges is usable: at 0.90 on date fields the model is right 94% of the time; at 0.90 on amount fields, 82%. Same score, different meaning, which is exactly why an uncalibrated threshold cannot be applied across fields.

Calibrated thresholds then drive routing:

• Above the calibrated threshold → run automatically, with stratified sampling watching it
• Below it → send to a reviewer
• Sitting near the line → send to a reviewer, and put it near the front of the queue

Reviewer Capacity Prioritisation

Reviewer time is finite and expensive, and how it is allocated is examinable.

Route the highest-uncertainty items to reviewers first. This means:

• Fields the model scored low
• Anything pulled from a source that was ambiguous or contradicted itself
• Document types whose measured accuracy has always been weak
• Fields where more than one reading was available and the model said so

Do NOT spread reviewer capacity evenly across all extractions. Even distribution spends attention confirming work the model already does well, and the shortfall lands precisely where judgement was needed — so the same budget produces materially less accuracy.

Order the queue dynamically rather than fixing it in advance. As documents arrive, the pending queue re-sorts by uncertainty, so the next item a reviewer picks up is the most uncertain one outstanding rather than whichever happened to arrive first.

Validation Before Automation

The sequence matters:

1. Measure accuracy by document type and field segment — not aggregate.
2. Calibrate confidence scores using labelled validation sets.
3. Set calibrated thresholds for automation versus human review.
4. Implement stratified random sampling for ongoing verification of automated extractions.
5. Only now scale back review, and only on the segments whose accuracy has held up under all of the above.

Jumping to step 5 on the strength of an aggregate is the trap this whole task statement circles. Each earlier step exists because a specific failure occurs without it — and each one is skippable, which is what makes the shortcut tempting.

Deep Dive

The LLM-as-judge scoring pattern: a template for field-level confidence

Anthropic's own evaluation of their multi-agent research system used "a single LLM call with a single prompt outputting scores 0.0-1.0 and pass-fail grade," which they found "was most consistent," and which "allowed us to scalably evaluate hundreds of outputs." This is a directly transferable pattern for field-level confidence in extraction systems: a bounded numeric score plus a discrete pass/fail-style grade, generated by one call rather than an elaborate multi-step judgement process, is what stays consistent enough to calibrate against ground truth at scale.

Sourceanthropic.com › multi-agent-research-systemfetched 2026-07-30

The evaluation rubric as a model for validation-set design

When Anthropic scored their research agent's outputs, the LLM judge evaluated against a specific rubric: "factual accuracy, citation accuracy, completeness, source quality, and tool efficiency." This decomposition — multiple named dimensions rather than one opaque score — is the same principle behind validating extraction accuracy "by document type AND field segment" instead of a single aggregate number. A validation set with only one axis of measurement hides exactly the kind of per-dimension failure this lesson's aggregate-metrics trap describes.

Sourceanthropic.com › multi-agent-research-systemfetched 2026-07-30

Human testing catches what automated evaluation misses

Even with the rubric-based LLM judge in place, Anthropic states plainly: "people testing agents find edge cases that evals miss... even in a world of automated evaluations, manual testing remains essential." This is the direct justification for stratified sampling of high-confidence extractions rather than relying purely on the model's own confidence signal — automated self-assessment (by the model, or even by an LLM judge) is not a substitute for human-verified ground truth on a genuine sample.

Sourceanthropic.com › multi-agent-research-systemfetched 2026-07-30

detected_pattern fields for systematic dismissal-pattern analysis

The exam guide's Task 4.4 (validation and retry loops) describes a companion technique worth knowing alongside calibration: "tracking which code constructs trigger findings (detected_pattern field) to enable systematic analysis of dismissal patterns." Applied to extraction review, this means tagging each low-confidence or human-corrected item with what specifically triggered the flag — not just that it was flagged — so that recurring failure causes (a particular field format, a particular document layout) surface as a pattern rather than a pile of individually-reviewed anomalies.

Sourceanthropic-partners.skilljar.com › partner-certificationsfetched 2026-07-30

Self-reported confidence alongside findings for calibrated routing

Task 4.6 (multi-instance and multi-pass review) names the specific skill this lesson's calibration pipeline depends on: "running verification passes where the model self-reports confidence alongside each finding to enable calibrated review routing." The word "calibrated" is load-bearing — the raw self-reported number only becomes useful for routing after it has been checked against a labelled validation set, exactly as this lesson describes. An uncalibrated confidence score is not a routing signal; it's an unverified guess.

Sourceanthropic-partners.skilljar.com › partner-certificationsfetched 2026-07-30

Explicit severity criteria repair trust after false positives

The related exam guide objective on precision (Task 4.1) records the consequence of skipping calibration: "the impact of false positive rates on developer trust: high false positive categories undermine confidence in accurate categories" — and its documented fix is not a blanket confidence threshold but "defining explicit severity criteria with concrete code examples for each severity level to achieve consistent classification." The lesson here applies the same logic to document extraction: a single automation threshold cannot fix a trust problem that is actually concentrated in specific segments.

Sourceanthropic-partners.skilljar.com › partner-certificationsfetched 2026-07-30

Quick Reference

Exam Traps

Practice Scenario

Build Exercise

Build a Confidence-Calibrated Review Router

Difficulty: Advanced (3/4)

50 minutes

1. Create a mock extraction system that outputs field-level confidence scores for different document types (invoices, receipts, scanned PDFs, international documents)

Why: Field-level confidence scores are the foundation of intelligent review routing. The exam tests that raw model confidence is not calibrated and must be validated against ground truth before use. Building the mock system gives you data to calibrate against.

You should see: An extraction function that returns each field with its value and a confidence score between 0.0 and 1.0. The system should process at least 4 document types with noticeably different confidence distributions per type.

2. Implement accuracy tracking broken down by document type and field segment — not just aggregate metrics

Why: The aggregate metrics trap is the most dangerous misconception in production extraction systems. 97% overall accuracy can hide catastrophic failure rates on specific document types because standard invoices dominate the volume. The exam tests that you must validate by document type AND field segment before automating.

You should see: An accuracy table showing each document type and field combination separately. Standard invoices should show 95%+ accuracy while handwritten receipts and international documents show 40-70%. The aggregate should look excellent (90%+) despite the poor per-type numbers.

3. Build a calibration module that takes a labelled validation set (ground truth) and produces calibrated confidence thresholds per field type per document type

Why: Raw model confidence scores are not calibrated. A model reporting 0.90 confidence might actually be correct 94% of the time on date fields but only 82% on amount fields. Calibration using labelled validation sets is required before confidence scores can drive automated routing decisions.

You should see: A calibration curve for each field type per document type, mapping reported confidence ranges to actual accuracy percentages. The curve should reveal that the same confidence score means different things for different field-document combinations.

4. Implement stratified random sampling that selects high-confidence extractions for ongoing verification, sampling proportionally across all document types

Why: High-confidence extractions are automated and not reviewed. If the model develops a novel error pattern affecting high-confidence items, only stratified sampling will catch it. Sampling only low-confidence items leaves you blind to systematic errors in automated extractions.

You should see: A sampling function that selects a representative subset from each stratum (document type and confidence band), including samples from the high-confidence automated extractions. The sample should be proportional to the volume in each stratum.

5. Build a review router that prioritises limited reviewer capacity on the highest-uncertainty items, dynamically reordering the review queue as new extractions arrive

Why: Human reviewers are expensive and limited. Spreading capacity evenly across all extractions wastes time on high-confidence items while leaving insufficient capacity for uncertain items that need human judgement. Dynamic priority ordering ensures the most uncertain items are always reviewed first.

You should see: A priority queue that orders items by uncertainty (lowest confidence first), dynamically reorders as new extractions arrive, and serves the next-highest-uncertainty item to each available reviewer. The queue should never serve items in chronological order.

Sources

• Claude Certified Architect Foundations Exam Guide — Domain 5, Task Statement 5.5 — Anthropic
• Agent SDK — Structured outputs — Anthropic
• Anthropic Human-in-the-Loop Patterns — Anthropic
• How we built our multi-agent research system — Anthropic

Appendix A — Build Exercise Step Hints

Progressive hints revealed by the "Stuck? Get a nudge" control on each step.

Step 1. Create a mock extraction system that outputs field-level confidence scores for different document types (invoices, receipts, scanned PDFs, international documents)

Why: Field-level confidence scores are the foundation of intelligent review routing. The exam tests that raw model confidence is not calibrated and must be validated against ground truth before use. Building the mock system gives you data to calibrate against.

You should see: An extraction function that returns each field with its value and a confidence score between 0.0 and 1.0. The system should process at least 4 document types with noticeably different confidence distributions per type.

function mockExtraction(document) {
  const confidenceByType = {
    invoice: { vendorName: 0.98, date: 0.95, amount: 0.97 },
    receipt: { vendorName: 0.71, date: 0.60, amount: 0.55 },
    scannedPdf: { vendorName: 0.80, date: 0.72, amount: 0.69 },
    international: { vendorName: 0.63, date: 0.45, amount: 0.52 }
  };
  
  const base = confidenceByType[document.type];
  return {
    vendorName: { value: document.vendor, confidence: base.vendorName + (Math.random() * 0.1 - 0.05) },
    date: { value: document.date, confidence: base.date + (Math.random() * 0.1 - 0.05) },
    amount: { value: document.amount, confidence: base.amount + (Math.random() * 0.1 - 0.05) }
  };
}

import random

CONFIDENCE_BY_TYPE = {
    "invoice": {"vendor_name": 0.98, "date": 0.95, "amount": 0.97},
    "receipt": {"vendor_name": 0.71, "date": 0.60, "amount": 0.55},
    "scanned_pdf": {"vendor_name": 0.80, "date": 0.72, "amount": 0.69},
    "international": {"vendor_name": 0.63, "date": 0.45, "amount": 0.52},
}


def mock_extraction(document: dict) -> dict:
    # An aggregate accuracy figure would hide the spread between these rows —
    # that is the whole point of measuring per type and per field.
    base = CONFIDENCE_BY_TYPE[document["type"]]
    jitter = lambda: random.uniform(-0.05, 0.05)
    return {
        "vendor_name": {"value": document["vendor"], "confidence": base["vendor_name"] + jitter()},
        "date": {"value": document["date"], "confidence": base["date"] + jitter()},
        "amount": {"value": document["amount"], "confidence": base["amount"] + jitter()},
    }

Step 2. Implement accuracy tracking broken down by document type and field segment — not just aggregate metrics

Why: The aggregate metrics trap is the most dangerous misconception in production extraction systems. 97% overall accuracy can hide catastrophic failure rates on specific document types because standard invoices dominate the volume. The exam tests that you must validate by document type AND field segment before automating.

You should see: An accuracy table showing each document type and field combination separately. Standard invoices should show 95%+ accuracy while handwritten receipts and international documents show 40-70%. The aggregate should look excellent (90%+) despite the poor per-type numbers.

function trackAccuracy(extractions, groundTruth) {
  const stats = {};
  for (const ext of extractions) {
    const key = ext.documentType;
    if (!stats[key]) stats[key] = { vendorName: { correct: 0, total: 0 }, date: { correct: 0, total: 0 }, amount: { correct: 0, total: 0 } };
    for (const field of ["vendorName", "date", "amount"]) {
      stats[key][field].total++;
      if (ext[field].value === groundTruth[ext.id][field]) stats[key][field].correct++;
    }
  }
  // Calculate per-type accuracy
  for (const [type, fields] of Object.entries(stats)) {
    for (const [field, counts] of Object.entries(fields)) {
      counts.accuracy = (counts.correct / counts.total * 100).toFixed(1) + "%";
    }
  }
  return stats;
}

FIELDS = ["vendor_name", "date", "amount"]


def track_accuracy(extractions: list[dict], ground_truth: dict) -> dict:
    stats: dict[str, dict] = {}
    for ext in extractions:
        key = ext["document_type"]
        per_type = stats.setdefault(
            key, {f: {"correct": 0, "total": 0} for f in FIELDS}
        )
        for field in FIELDS:
            per_type[field]["total"] += 1
            if ext[field]["value"] == ground_truth[ext["id"]][field]:
                per_type[field]["correct"] += 1

    # Calculate per-type accuracy
    for fields in stats.values():
        for counts in fields.values():
            counts["accuracy"] = f"{counts['correct'] / counts['total'] * 100:.1f}%"
    return stats

Step 3. Build a calibration module that takes a labelled validation set (ground truth) and produces calibrated confidence thresholds per field type per document type

Why: Raw model confidence scores are not calibrated. A model reporting 0.90 confidence might actually be correct 94% of the time on date fields but only 82% on amount fields. Calibration using labelled validation sets is required before confidence scores can drive automated routing decisions.

You should see: A calibration curve for each field type per document type, mapping reported confidence ranges to actual accuracy percentages. The curve should reveal that the same confidence score means different things for different field-document combinations.

function buildCalibrationCurve(extractions, groundTruth) {
  const bands = {};
  for (const ext of extractions) {
    for (const field of ["vendorName", "date", "amount"]) {
      const key = `${ext.documentType}-${field}`;
      const band = Math.floor(ext[field].confidence * 10) / 10; // 0.0, 0.1, ..., 0.9
      if (!bands[key]) bands[key] = {};
      if (!bands[key][band]) bands[key][band] = { correct: 0, total: 0 };
      bands[key][band].total++;
      if (ext[field].value === groundTruth[ext.id][field]) bands[key][band].correct++;
    }
  }
  // Convert to calibrated thresholds
  const thresholds = {};
  for (const [key, bandData] of Object.entries(bands)) {
    thresholds[key] = Object.fromEntries(
      Object.entries(bandData).map(([band, counts]) => [band, counts.correct / counts.total])
    );
  }
  return thresholds;
}

import math


def build_calibration_curve(extractions: list[dict], ground_truth: dict) -> dict:
    bands: dict[str, dict[float, dict[str, int]]] = {}
    for ext in extractions:
        for field in FIELDS:
            key = f"{ext['document_type']}-{field}"
            band = math.floor(ext[field]["confidence"] * 10) / 10  # 0.0 … 0.9
            counts = bands.setdefault(key, {}).setdefault(band, {"correct": 0, "total": 0})
            counts["total"] += 1
            if ext[field]["value"] == ground_truth[ext["id"]][field]:
                counts["correct"] += 1

    # Convert to calibrated thresholds: what a reported score actually means
    return {
        key: {band: c["correct"] / c["total"] for band, c in band_data.items()}
        for key, band_data in bands.items()
    }

Step 4. Implement stratified random sampling that selects high-confidence extractions for ongoing verification, sampling proportionally across all document types

Why: High-confidence extractions are automated and not reviewed. If the model develops a novel error pattern affecting high-confidence items, only stratified sampling will catch it. Sampling only low-confidence items leaves you blind to systematic errors in automated extractions.

You should see: A sampling function that selects a representative subset from each stratum (document type and confidence band), including samples from the high-confidence automated extractions. The sample should be proportional to the volume in each stratum.

function stratifiedSample(extractions, sampleRate = 0.05) {
  // Group by document type and confidence band
  const strata = {};
  for (const ext of extractions) {
    const key = `${ext.documentType}-${ext.overallConfidence >= 0.80 ? "high" : "low"}`;
    if (!strata[key]) strata[key] = [];
    strata[key].push(ext);
  }
  
  // Sample from EVERY stratum, including high-confidence
  const sample = [];
  for (const [stratum, items] of Object.entries(strata)) {
    const n = Math.max(1, Math.ceil(items.length * sampleRate));
    const selected = shuffleAndTake(items, n);
    sample.push(...selected.map(s => ({ ...s, stratum })));
  }
  return sample;
}

def stratified_sample(extractions: list[dict], sample_rate: float = 0.05) -> list[dict]:
    # Group by document type and confidence band
    strata: dict[str, list[dict]] = {}
    for ext in extractions:
        band = "high" if ext["overall_confidence"] >= 0.80 else "low"
        strata.setdefault(f"{ext['document_type']}-{band}", []).append(ext)

    # Sample from EVERY stratum, including high-confidence: those are the
    # automated, unreviewed ones where novel errors hide.
    sample = []
    for stratum, items in strata.items():
        n = max(1, math.ceil(len(items) * sample_rate))
        sample += [{**s, "stratum": stratum} for s in shuffle_and_take(items, n)]
    return sample

Step 5. Build a review router that prioritises limited reviewer capacity on the highest-uncertainty items, dynamically reordering the review queue as new extractions arrive

Why: Human reviewers are expensive and limited. Spreading capacity evenly across all extractions wastes time on high-confidence items while leaving insufficient capacity for uncertain items that need human judgement. Dynamic priority ordering ensures the most uncertain items are always reviewed first.

You should see: A priority queue that orders items by uncertainty (lowest confidence first), dynamically reorders as new extractions arrive, and serves the next-highest-uncertainty item to each available reviewer. The queue should never serve items in chronological order.

class ReviewQueue {
  constructor(calibrationData) {
    this.queue = [];
    this.calibration = calibrationData;
  }
  
  add(extraction) {
    const calibratedConfidence = this.getCalibratedConfidence(extraction);
    this.queue.push({ ...extraction, calibratedConfidence });
    this.queue.sort((a, b) => a.calibratedConfidence - b.calibratedConfidence);
  }
  
  getNext() {
    return this.queue.shift(); // Highest uncertainty first
  }
  
  getCalibratedConfidence(ext) {
    const key = `${ext.documentType}-${ext.fieldType}`;
    const band = Math.floor(ext.rawConfidence * 10) / 10;
    return this.calibration[key]?.[band] || ext.rawConfidence;
  }
}

import bisect


class ReviewQueue:
    def __init__(self, calibration_data: dict):
        self.queue: list[dict] = []
        self.calibration = calibration_data

    def add(self, extraction: dict) -> None:
        item = {**extraction, "calibrated_confidence": self.calibrated_confidence(extraction)}
        # Keep the queue sorted so the least certain item is always at the front.
        keys = [q["calibrated_confidence"] for q in self.queue]
        self.queue.insert(bisect.bisect_left(keys, item["calibrated_confidence"]), item)

    def get_next(self) -> dict | None:
        return self.queue.pop(0) if self.queue else None  # Highest uncertainty first

    def calibrated_confidence(self, ext: dict) -> float:
        key = f"{ext['document_type']}-{ext['field_type']}"
        band = math.floor(ext["raw_confidence"] * 10) / 10
        return self.calibration.get(key, {}).get(band, ext["raw_confidence"])

Appendix B — Interactive Study Prompts

Two prompts to paste into Claude. B1 drills the judgement the exam actually measures; B3 reviews the code you wrote for the Build Exercise above. The exam simulator between them is the interactive quiz on this page.

B1. Concept Check — Discrimination Drill

B2. Exam Simulator

B3. Build Coach — Code Review

The Build Exercise and its hint ladder are already on this page. This prompt is for the one thing the page cannot do: review the code you actually wrote.