Prompt Engineering & Structured Output flashcards

Every card in this deck

The whole deck as a list, for scanning or printing.

Where does the Messages API expect explicit review criteria to live?

In the top-level `system` parameter — there is no "system" role inside the `messages` array.

What happens to consecutive same-role messages in a request?

They are combined into a single turn, not rejected.

Is `max_tokens` a target or a ceiling?

An absolute ceiling only — the model may stop generating before reaching it.

Does `temperature: 0.0` make output fully deterministic?

No — even at 0.0, results are not fully deterministic.

What are the two prompt-design failure modes Anthropic names?

(A) Hardcoded, brittle if/else-style logic — fragile and hard to maintain. (B) Vague, high-level guidance — no concrete signal (e.g. "be conservative").

What is the "right altitude" target for a system prompt?

The minimal set of information that fully outlines expected behaviour — not a laundry list of every edge case.

How should categorical criteria (e.g. severity) be enforced structurally?

As a tool `enum` field or via structured outputs — not left to prose interpretation.

What eval-set size is enough to see the impact of a prompt change early on?

About 20 representative queries.

What is the documented "best form of feedback"?

Clearly defined rules for an output, plus an explanation of which rules failed and why (e.g. code linting).

What does a high false-positive rate in one review category do to trust in other categories?

It destroys it — trust is not category-specific, it bleeds across the whole output.

What is the correct fix for a category with an unacceptably high false-positive rate?

Temporarily disable that category, refine its criteria with concrete code examples, then re-enable it.

How should severity levels be defined for consistent classification?

With concrete code examples per level, never prose descriptions alone.

What is the first-choice fix when detailed instructions still produce inconsistent output formatting?

Few-shot examples — not more instructions, not confidence thresholds, not temperature adjustments.

How many few-shot examples should you typically use?

2-4 targeted examples. Fewer than 2 doesn't establish a pattern; more than 4 wastes tokens without proportional benefit.

What must a good few-shot example include beyond input and output?

Reasoning — why one action was chosen over plausible alternatives. This teaches generalisation, not just pattern-matching.

How does Anthropic's context-engineering guidance describe good few-shot examples?

"Pictures worth a thousand words" — curated, diverse, and canonical.

What does Anthropic explicitly NOT recommend for handling edge cases?

Stuffing a laundry list of edge cases into a prompt in an attempt to articulate every possible rule.

For classification tasks, what should pair with few-shot examples to guarantee valid labels?

A tool with an `enum` field listing valid labels, or structured outputs.

Why can piling in 8-10 few-shot examples hurt output quality?

"Context rot" — as token count grows, accuracy and recall degrade, diluting attention on the pattern that matters.

Malformed JSON output in ~5% of responses — which technique fixes it?

tool_use with JSON schemas, not few-shot examples (few-shot fixes consistency, not structural syntax).

Few-shot examples fix inconsistent output. What is the multi-agent equivalent of leaving them out?

Vague per-subagent task descriptions — no objective, output format, or boundaries — so subagents duplicate work and leave gaps. Same fix both times: show the expected shape explicitly instead of describing it.

Rank the structured-output reliability hierarchy.

1) tool_use with JSON schemas — eliminates JSON syntax errors entirely. 2) Prompt-based JSON — can produce malformed output.

List the four tool_choice values and their defaults.

`auto` (default with tools), `any` (must call a tool, model picks which), `tool` (forces a specific tool), `none` (default with no tools).

Does tool_choice "any" alone guarantee the tool's input matches its schema?

No — only that a tool is called. Add `strict: true` on the tool definition for the schema-validation guarantee too.

What is the newer Structured Outputs feature, distinct from tool_use?

`output_config.format` — constrains a plain-text JSON response via constrained decoding, no tool call required.

What model availability applies to the Structured Outputs feature?

Generally available on the Claude API for Claude 4.5 and later models only.

What must `additionalProperties` be set to in a structured-output object schema?

`false` — any other value is rejected.

Name four things NOT supported in structured-output JSON schemas.

Recursive schemas, external `$ref`, numerical constraints (minimum/maximum/multipleOf), string constraints (minLength/maxLength).

What latency cost applies the first time you use a new structured-output schema?

Extra latency while the grammar compiles; compiled grammars are then cached for 24 hours from last use.

Does using structured outputs change your token count?

Yes — it adds a hidden system prompt explaining the expected format, slightly raising input tokens on every call.

What happens if you prefill the last assistant turn on Claude 4.6+ / Mythos Preview?

The request returns a 400 error — prefill on the last turn is no longer supported on those models.

What is documented as prefill's migration path for forcing output formats?

The Structured Outputs feature — designed specifically to constrain responses to a given schema.

What does tool_use with JSON schemas eliminate — and what does it NOT eliminate?

Eliminates: JSON syntax errors. Does NOT eliminate: semantic errors — sum discrepancies, field placement, fabrication.

How do you prevent a model from fabricating a value for a field the source document lacks?

Make the field optional/nullable rather than required — a required field pressures fabrication; a nullable field allows an honest `null`.

What schema pattern handles genuinely ambiguous and extensible categories?

Add an "unclear" enum value for ambiguous cases, and "other" paired with a freeform detail string for extensible ones.

What three things must a retry-with-error-feedback message include?

The original document, the failed extraction, and the specific validation error.

What does the Claude API already do internally for invalid/missing tool parameters?

Retries the tool call 2-3 times with corrections before apologising to the user — the API-level precedent for retry-with-error-feedback.

What is the Messages API's general feedback-reporting contract for a failed tool execution?

Return `content` plus `"is_error": true` in the `tool_result`; Claude incorporates the error into its next response.

List error types retries CAN fix.

Format mismatches, structural output errors, misplaced values, and mathematical errors (e.g. a missed line item).

List error types retries CANNOT fix.

Information genuinely absent from the source document, data that exists only in an external document, or knowledge the model lacks.

What should you do with an unfixable extraction failure?

Flag it for human review, or return null if the schema allows — do not keep retrying.

What self-correction schema pattern flags total discrepancies automatically?

Extract both `calculated_total` (summed from line items) and `stated_total` (from the document); flag when they differ.

What does a `conflict_detected` boolean capture?

That the source document contains contradictory information (e.g. two different payment-terms statements) rather than the model silently picking one.

What does a `detected_pattern` field enable over time?

Systematic analysis of which code construct or document feature triggers dismissed findings, prioritising prompt refinement.

Fix for a pattern that fails repeatedly despite retries?

Add a formal rule or refine the prompt/schema for that specific pattern — an architectural fix, not another retry.

Distinguish schema syntax errors from semantic validation errors.

Syntax errors (malformed JSON, wrong types) are eliminated by tool_use. Semantic errors (wrong sums, misplaced values) require validation logic and retry loops.

What ranking does Anthropic give LLM-as-judge feedback vs rules-based feedback?

Rules-based (clear rules + which failed and why) is best; LLM-as-judge is "generally not a very robust method" with heavy latency tradeoffs.

Message Batches API cost discount?

50% of standard API prices, unconditionally.

Typical vs maximum batch processing time?

Most batches finish in under 1 hour; the hard maximum is 24 hours, with NO latency SLA guaranteed.

Message Batch size limit?

100,000 requests OR 256 MB, whichever is reached first — exceeding size returns a 413 request_too_large error.

custom_id format and purpose?

Matches `^[a-zA-Z0-9_-]{1,64}$`; used to match results to requests because results can return out of order.

Batch processing_status lifecycle?

`in_progress` → (optionally `canceling`) → `ended`, once every request has a terminal result and results are ready.

The four per-request batch result types?

`succeeded`, `errored`, `canceled`, `expired`.

Which batch result types are billed?

Only `succeeded`. Errored, canceled, and expired requests are all NOT billed.

Where and when are batch results available?

A `.jsonl` file at `results_url`, populated only once the batch has `ended`; the docs recommend streaming rather than bulk-downloading.

How long are batch results downloadable, and from when?

29 days, measured from `created_at` — NOT from `ended_at`.

Which two request params are rejected in a batch with a validation error?

`stream: true` (results come back as one file, not a stream) and Threads params `store` / `previous_thread_event_id` (Threads are stateful; batch requests are not).

When does batch param validation surface errors?

Asynchronously — only after the entire batch has ended, not at submission time. Dry-run a single request against the synchronous API first.

Recommended prompt-cache TTL for batch workloads?

The 1-hour cache duration, since batches often take longer than the default 5-minute cache window.

Do server tools (web search, code execution, etc.) work inside a batch?

Yes — the batch worker runs the same server-side agentic loop as the synchronous Messages API.

What genuinely cannot happen mid-batch, and what's the fix?

Turn continuation — if a result comes back `pause_turn`, you must submit a NEW follow-up request (batch or synchronous); it cannot continue inside the original batch.

Matching rule: synchronous vs batch API?

Synchronous for blocking workflows (someone/something waiting, e.g. pre-merge checks). Batch for latency-tolerant workflows (overnight reports, weekly audits).

A report has a 30-hour SLA and the Batch API window is up to 24 hours. When must the final batch go in?

At least 24 hours before the deadline — that is the hard processing ceiling. The remaining 6 hours are buffer for collecting requests, validating inputs, and absorbing delays; submit every 4-6 hours so a fresh batch is always in flight.

Correct batch failure-handling pattern?

Identify failures by custom_id, resubmit ONLY those with targeted modifications (chunking, simplified prompts, format-specific examples) — never resubmit the whole batch.

Why refine prompts on a sample set before a large batch run?

First-pass success rate compounds: 90% success on 1,000 docs = 100 retries; 60% success = 400 retries, 4x the resubmission cost.

Why does self-review in the same session underperform an independent instance?

The model retains its original reasoning chain and tends to confirm rather than challenge its own prior decisions.

Official documented reason a fresh session improves code review?

"A fresh context improves code review since Claude won't be biased toward code it just wrote."

Describe the Writer/Reviewer session pattern.

Session A generates the artefact; a separate Session B reviews it fresh; Session A then addresses the feedback.

What does a fresh adversarial reviewer subagent actually see?

Only the diff and the review criteria — never the reasoning that produced the change.

What distinguishes "verification by a second opinion" from a simple re-check?

The fresh model actively tries to refute the result, so the agent that did the work isn't the one grading it.

What architectural property underlies independent review instances?

Subagent context isolation — subagents use their own isolated context windows and only send relevant information back.

Three symptoms of attention dilution in single-pass multi-file review?

Inconsistent depth across files, missed bugs in middle files, and contradictory findings across files.

Correct fix for attention dilution?

Split into per-file local analysis passes plus a separate cross-file integration pass — NOT a bigger context window.

Why doesn't a larger context window fix attention dilution?

The problem is attention quality, not context capacity — more room doesn't stop uneven attention across files.

What does the cross-file integration pass check for?

Data flow inconsistencies between modules, contradictory findings across per-file passes, and API contract violations.

Confidence-based routing rule?

High confidence → report directly to developers; low confidence → route to human review.

Is raw self-reported model confidence reliable for routing on its own?

No — it's uncalibrated, effectively the model judging itself, and must be validated against labelled data before use.

How do you calibrate confidence thresholds correctly?

Run labelled validation sets (known-correct answers) through the system and measure how reported confidence tracks actual accuracy.

Documented principle for judging agent/review outcomes?

Judge whether the correct final state was reached, not whether a specific process was followed.

Example LLM-judge rubric dimensions from Anthropic's research system?

Factual accuracy, citation accuracy, completeness, source quality, and tool efficiency.