Skip to content

Verification & CI Gating

LLM Council's most distinctive surface: multi-model verification of code, documents, or any work product, with machine-actionable verdicts (ADR-034). Four models deliberate over your change; a chairman renders pass / fail / unclear with confidence, rubric scores, and blocking issues.

Three ways in

Surface Invocation Use for
MCP tool verify(snapshot_id, target_paths, tier, ...) agent sessions (Claude Code, Cursor)
CLI llm-council gate --snapshot <sha> [--tier ...] CI/CD pipelines (exit code 0/1/2, or 3 when nothing was reviewed)
HTTP POST /v1/council/verify services
llm-council gate --snapshot $(git rev-parse HEAD) \
  --file-paths src/module.py --tier balanced --rubric-focus Security

Exit codes: 0 PASS · 1 FAIL · 2 UNCLEAR · 3 nothing reviewed.

3 means the council did not run: no --file-paths were given and no reviewable file resolved from the snapshot (#704). It is deliberately outside 0–2, because llm-council-action turns 2 into a passing check with a warning, and a gate that reviewed nothing must not pass.

Without --file-paths, the subject is the files the snapshot changed. For a merge commit, which is what github.sha is on a pull_request event, that means the diff against its first parent: exactly the PR's changes. The output lists the files reviewed, so a CI step summary shows what the verdict is about.

Tiers

Tier Tier budget Verify deadline Max input Use
quick ~30s 60s 15K chars sanity checks, small diffs
balanced ~90s 180s 30K chars default — routine verification
high ~180s 360s 50K chars security-critical reviews
reasoning ~600s 1200s 50K chars complex architectural decisions

Max input is the size of the whole prompt: the review instructions, any evidence, and the files. The files get what is left, about 1,000 characters less than the figure in the table when no evidence is passed.

When the files do not all fit, they are taken in this order: source, tests, configuration, documentation. A file that does not fit is skipped, and smaller files after it are still considered. If the very first file is larger than the room available, it is cut to fit so that something is reviewed. Every file that was left out is named in expansion_warnings and appears in coverage.omitted with reason too_large; coverage.reviewed lists only what the council was shown. A left-out file that you named in target_paths turns a pass into unclear (see coverage accounting); one that came from the commit's diff does not.

verify runs to twice the tier budget — size MCP_TIMEOUT for that

run_verification wraps the pipeline in a global deadline of tier_deadline × VERIFICATION_TIMEOUT_MULTIPLIER (2.0), so stage 3 isn't starved on a slow day. The Verify deadline column above is the number your client transport has to accommodate — not the tier budget.

The MCP_TIMEOUT guidance in the MCP guide is sized for consult_council, which does use the tier budget. Carrying those numbers over to verify sets a timeout at half what a verify can legitimately take. Set at least:

Tier MCP_TIMEOUT for verify
balanced 240000
high 480000
reasoning 1500000

A client-side kill is worse than a server-side timeout: it returns no verdict, no transcript reference, and no unclear_reason, so none of the routing below applies and the run is indistinguishable from a hang. A server-side timeout, by contrast, still salvages a partial result with timeout_fired and completed_stages set.

LLM_COUNCIL_TIMEOUT_MULTIPLIER scales the tier budget (and therefore the verify deadline with it). VERIFICATION_TIMEOUT_MULTIPLIER is a fixed code constant, not an environment variable.

Reading an UNCLEAR verdict (ADR-047)

The exit code stays 2 for every UNCLEAR cause — that is a deliberate compatibility contract (ADR-047): existing automation keying on exit codes keeps working, and unclear_reason is the routing signal you layer policies on top of:

  • infra_failure — the chairman call itself errored (billing, auth, rate limit). Check your gateway/billing, then retry; never treat it as a review outcome. Exception: if retry_hint is set, the chairman call timed out — that is starvation, so follow the hint (below).
  • low_confidence — deliberation completed below the confidence threshold. Common policy: accept-and-audit when blocking_issues is empty.
  • timeout — the tier deadline fired. Retry at a higher tier or reduce input scope; retry_hint says which.

retry_hint: retry up, never down (#597)

When a run ran out of time — the global deadline fired, or the chairman call hit its own timeout — the response carries a retry_hint:

{"action": "escalate_tier", "suggested_tier": "high",
 "reason": "synthesis_starved", "message": "Retry at tier=high: ..."}

reason is synthesis_starved when peer review had finished (only the chairman ran out of time) and deadline_exhausted otherwise. At reasoning there is no higher tier, so action is reduce_scope and suggested_tier is null. Retrying at the same or a lower tier during a slow-provider window is self-defeating: a lower tier has a shorter deadline, so it starves the chairman harder. The hint is advice only; nothing is re-run automatically. A genuine infra failure (auth, billing, rate limit) gets no hint. - chairman_disabled — chairman_disabled=true (config or LLM_COUNCIL_CHAIRMAN_DISABLED) skipped chairman synthesis, so no verdict was ever computed; rationale carries the top-ranked peer response for reference only. Never treat this as a pass/fail review outcome — disable chairman_disabled for any BINARY-verdict use (council-verify, council-gate, CI approval). - incomplete_coverage (#556) — the coverage clamp fired: a pass was downgraded because a changed-or-explicitly-named file was not reviewed (see coverage.clamped). The council's verdict on what it did see was pass; it simply did not see everything. Review coverage.clamped, then either switch to content file selection so the file is reviewed, acknowledge its reason via LLM_COUNCIL_COVERAGE_ACK_REASONS, or accept the partial coverage.

Calibrated confidence (ADR-047)

Every response carries confidence (raw) and confidence_calibrated (raw passed through a monotonic mapping fitted against your recorded human dispositions). Build the mapping from your own transcript corpus:

llm-council calibration-report          # analyze .council/logs
llm-council calibration-report --fit    # fit mapping from dispositions

Both numbers are telemetry, not a calibrated probability (ADR-054 D3a): confidence is derived from reviewer agreement and the chairman's self-report, and no verdict threshold consumes the calibrated value — verdicts are decided by findings (structured path) or verdict extraction (legacy). The former LLM_COUNCIL_CALIBRATED_CONFIDENCE PASS gate was removed; do not build automation on confidence >= X as if the number were a probability.

Screening judge (ADR-047, opt-in)

A single quick-tier model can pre-screen easy changes (LLM_COUNCIL_SCREENING=shadow|active; default off). Blocking-capable requests (blocking evidence, security focus, risk-glob paths) are never screened — the full council always runs for those. Start with shadow and read .council/screening/decisions.jsonl before trusting active.

Evidence injection (ADR-042)

Feed upstream tool output (linters, scanners) as structured evidence; the council must disposition each item. strength: blocking items make the request blocking-capable.

Prompt-cache cost note (ADR-049)

Verification prompts are assembled stable-prefix-first and cached on Anthropic council members (0.1× read price on repeat rounds; verified on the OpenRouter route). Multi-round verify sessions on the same subject are therefore much cheaper than round 1. The verify path uses a 1-hour cache TTL by default (rounds typically land 3–11 minutes apart); LLM_COUNCIL_PROMPT_CACHE_TTL=5m|1h overrides it, and LLM_COUNCIL_PROMPT_CACHING=false disables injection entirely. input_metrics reports cached_tokens (reads), cache_write_tokens, and cache_session_id — zero reads across rounds means a broken prefix or a lapsed TTL.

Structured findings (ADR-051, opt-in)

By default the verdict is derived from the chairman's prose and blocking_issues is scraped from it — which historically left blocking_issues empty even on FAIL. With LLM_COUNCIL_STRUCTURED_FINDINGS=true the chairman instead emits a typed findings array (one entry per issue, each with a severity), and the host computes the verdict mechanically: fail iff any finding is critical, else pass (confidence may still soften a pass to unclear). The verdict is therefore a provable function of the evidence — it cannot decouple from findings.

  • findings — the full list across all severities.
  • blocking_issues — the critical subset (unchanged shape; now always consistent with the verdict).
  • diagnostics.verdict_source — mechanical when the flag drove the verdict, else legacy.

Consumer migration. Stop keying acceptance on blocking_issues == [] (empty even on real FAILs under the legacy path). Key on the verdict plus the findings/severity you care about — e.g. "block on any critical or major". This is the durable contract; the flag defaults off and this epic is non-breaking, but the legacy prose-scrape is the path being retired.

Controlling what gets reviewed (ADR-053)

verify decides which files enter the prompt through a layered filter. A compiled-in secret boundary always runs first and is never overridable: credential files (.env*, *.pem, id_rsa*, .npmrc, .aws/credentials, kubeconfig, secrets.y*ml, …) are never transmitted, even if you name one explicitly. Inspect it with llm-council ignore --print-defaults, or ask why a given path is or isn't reviewed with llm-council ignore --explain <path>.

File classification — LLM_COUNCIL_FILE_SELECTION:

  • content (default) — git's own text detection (a blob is text iff its first 8000 bytes contain no NUL), honouring the snapshot's .gitattributes. Reviews any text file regardless of extension — unlisted languages, LICENSE, CODEOWNERS, shebang scripts. Also omits linguist-generated/linguist-vendored paths and (unless LLM_COUNCIL_REVIEW_SVG=true) .svg.
  • allowlist — the historical TEXT_EXTENSIONS list; the one-var opt-out that restores pre-flip behavior exactly. A file whose extension isn't on the list (.zig, .tf, .dart, …) is dropped.

Default changed. Until the #557 flip, allowlist was the default and unlisted-extension text files were silently not reviewed — a gate could PASS without reading them. The content default is strictly stricter: nothing reviewed before stops being reviewed, but commits touching previously-unlisted files can gain findings (and modest prompt cost). Set LLM_COUNCIL_FILE_SELECTION=allowlist to restore the old behavior. - shadow — acts on the allowlist but records what content would add or drop, so you can measure the change before adopting it. Each shadow run appends one JSON line — {ts, snapshot_id, candidates, selected, would_add, would_drop}, paths only, never file contents — to .council/selection/decisions.jsonl (empty deltas included, so the delta rate is measurable). Recording is soft-fail and never affects selection.

Repo-owned exclusions — in content mode, verify honours the first present of .llmignore → .aiexclude → .aiignore → .cursorignore → .codeiumignore (read from the snapshot, gitignore syntax). It can only narrow what is reviewed; it can never re-admit a denied secret. llm-council ignore --init writes a commented starter file (it never overwrites an existing one, and never runs unless you ask).

Coverage accounting (ADR-053)

Every response carries a coverage receipt so a gate can tell what was reviewed vs skipped, and why, instead of parsing prose:

"coverage": {
  "reviewed": ["src/app.py"],
  "omitted": [{"path": "main.zig", "reason": "non-text", "origin": "discovered"},
              {"path": ".env",     "reason": "denied_secret", "origin": "discovered"}],
  "explicit_omitted": false
}

reason distinguishes a dropped source file (non-text) from a binary (binary) from a secret (denied_secret — the value is never recorded). Key CI logic on coverage, not on expansion_warnings.

The coverage clamp — LLM_COUNCIL_COVERAGE_POLICY:

  • warn (current default) — receipt only, no verdict effect. Byte-identical.
  • clamp — a pass over a changed-or-explicitly-named file the council did not review becomes unclear(incomplete_coverage) (see coverage.clamped). LLM_COUNCIL_COVERAGE_ACK_REASONS (default binary,generated,vendored,too_large,ignored,noise) acknowledges expected omissions, so the clamp fires only on the surprising ones (non-text, not_found, truncated, denied_secret).
  • fail — the same condition raises a hard 422 instead.

The clamp is the default. Since the #557 flip, a verify/gate pass over an unreviewed changed-or-explicitly-named file returns unclear(incomplete_coverage) instead, and gate refuses an explicit LLM_COUNCIL_COVERAGE_POLICY=warn (post-flip that means "ignore coverage", which is a foot-gun rather than the status quo). Because content file selection is also the default, unlisted-extension source files are reviewed rather than dropped, so the clamp's triggers are the genuinely surprising omissions — not_found, truncated, denied_secret, and non-text for anyone who opted back into allowlist selection. Expected omissions (binary, generated, vendored, too_large, ignored, noise) are acknowledged and never clamp. To opt out, set LLM_COUNCIL_COVERAGE_POLICY=warn — receipt only, no verdict effect.

Response fields

Every field on the verify response (VerifyResponse), the nested Finding, and the telemetry-only VerifyDiagnostics. All are additive — older clients that ignore unknown fields keep working.

Core verdict

Field Type Meaning
verification_id string Unique id for this run.
verdict string pass | fail | unclear.
exit_code int 0 PASS · 1 FAIL · 2 UNCLEAR (CLI/gate).
confidence float Raw council-agreement confidence, 0–1.
confidence_calibrated float? confidence after the fitted monotonic mapping (ADR-047); equals raw until a mapping is fitted.
retry_hint object? #597: set when the run ran out of time. {action: escalate_tier\|reduce_scope, suggested_tier, reason: synthesis_starved\|deadline_exhausted, message}; None otherwise. See retry_hint.
unclear_reason string? infra_failure | low_confidence | timeout | chairman_disabled | incomplete_coverage (#556, see above); None for pass/fail.
rationale string Chairman synthesis explanation.
transcript_location string Path to the full .council/logs/<id>/ transcript.
error string? Non-verdict error marker: input_too_large (#357) or no_reviewable_content (#704, nothing resolved to review); None for a real verdict.

Scores & findings

Field Type Meaning
rubric_scores object Per-dimension scores (accuracy/relevance/completeness/conciseness/clarity), 0–10 or null.
blocking_issues list Issues that caused FAIL — the critical subset of findings.
findings list of Finding Full structured findings across all severities (ADR-051; empty unless LLM_COUNCIL_STRUCTURED_FINDINGS).
diagnostics VerifyDiagnostics Telemetry-only (below) — not control flow.

A Finding has: severity (critical | major | minor | info), description (text), location (file.py:42, global, or null), and dimension (the rubric axis it maps to, when derivable).

Diagnostics (telemetry only — never gate on these)

Field Type Meaning
findings_source string structured (clean parse), fallback (missing/malformed → legacy path), or skipped (chairman_disabled=true — no synthesis to parse).
fallback_reason string? Why the fallback fired, when it did.
verdict_source string mechanical = policy(findings); legacy = prose parse; chairman_disabled = synthesis skipped, no verdict computed.
verdict_parse string How the chairman's BINARY verdict block parsed: ok, error (malformed — see verdict_parse_error), or absent (no structured verdict expected). Reported regardless of LLM_COUNCIL_STRUCTURED_FINDINGS. Distinct from fallback_reason, which describes the findings parser.
verdict_parse_error string? Exception type and message when verdict_parse == "error". Never contains the offending payload.
deliberation_agreement float? Stage-2 reviewer agreement, published under its real name. Measures how well the council reviewed, not how sure we are of the verdict. Gates nothing.
pass_blocked_by string? Why a mechanical pass was downgraded to unclear: deliberation_invalid (no well-formed chairman verdict, fewer than MIN_STAGE1_REVIEWERS, or a stage-3 error) or chairman_contradicts_findings (chairman rejected but labelled no finding critical). None otherwise.
findings_by_severity object Count per severity — surfaces severity mis-labelling over time.
verdict_evidence_mismatch string? Defensive invariant marker; None in normal operation (should never fire under the mechanical gate).
inner_verdict string? Structured verdict before UNCLEAR softening (nested so consumers can't parse it to bypass the low-confidence gate).
inner_confidence float? Confidence that accompanied inner_verdict.
inner_confidence_calibrated float? Calibrated form of inner_confidence.

Timeout, expansion & telemetry

Field Type Meaning
partial bool Result is partial (timeout/error) (ADR-040).
timeout_fired bool Global deadline was exceeded.
completed_stages list? Stages completed before a timeout (e.g. ["stage1","stage2"]).
expanded_paths list? Files included after directory expansion (#311).
paths_truncated bool? MAX_FILES_EXPANSION limit was reached.
expansion_warnings list? Warnings from directory expansion (skipped files, etc.). Human-readable; prefer coverage for machine use.
coverage object? #555 structural coverage receipt — which requested paths were reviewed vs omitted, typed. Additive; no verdict effect (the clamp is #556). Fields below.
coverage.requested list? Verbatim target_paths (None when the changed set was used).
coverage.reviewed list Paths whose contents entered the prompt.
coverage.omitted list Each omitted path as {path, reason, origin}.
coverage.omitted[].reason string denied_secret | garbage | non-text | binary | generated | vendored | noise | ignored | too_large | not_found. denied_secret records the path only, never the matched value.
coverage.omitted[].origin string explicit (caller named it) or discovered (directory/diff-tree expansion).
coverage.explicit_omitted bool True if a caller-named path was omitted — the load-bearing signal that a pass covers less than asked.
coverage.truncated bool MAX_FILES_EXPANSION capped directory expansion.
coverage.conservation_ok bool Invariant marker: reviewed and omitted are disjoint (ADR-051 C4 pattern; should always be true).
coverage.policy string? Active LLM_COUNCIL_COVERAGE_POLICY (clamp | fail | warn) — #556.
coverage.clamped list? Omissions that clamped the verdict to unclear(incomplete_coverage) — present only when the clamp fired (#556).
timing object? Per-stage and total timing in ms (ADR-041).
input_metrics object? Input-size metrics (content_chars, tier_max_chars, num_models, num_reviewers, tier, cache fields).
screening object? Screening-judge audit trail (ADR-047); present only when LLM_COUNCIL_SCREENING is shadow/active.
evidence_summary list? Per-evidence-item Council disposition (ADR-042); None when no evidence supplied.
evidence_warnings list? Structured warnings about evidence handling (truncation, format errors).

Operational tips

  • Scope target_paths to the files that changed — whole-file expansion of pre-existing code invites off-scope findings.
  • Repeated re-verification of the same scrutinized files hits diminishing returns; act on verdicts rather than re-rolling them.
  • Every run persists a full transcript under .council/logs/<timestamp-id>/ (the audit MCP tool retrieves them).