Skip to content

ADR-048: Council Quality Benchmark & Golden-Dataset Regression

Status: Implemented 2026-07-03 (epic #421, v0.32.0) Date: 2026-07-03 Decision Makers: Chris Joseph, LLM Council Council Review: 2026-07-03 (4 models, balanced) — feedback incorporated: spend caps, CLI spec, dataset governance, per-phase DoD Related: ADR-036 (Phase 3 — this implements the golden-dataset/eval-bridge slice), ADR-011 (cost ground truth), ADR-044 (router needs this data), ADR-047 (calibration corpus overlaps)

Context

The core product claim — a deliberating council beats a single frontier model — is unproven in our own repo, and the counter-question that matters commercially is sharper: when is it worth the cost? We are uniquely positioned to answer: heterogeneous-model ensembles are called out in the test-time-compute literature as underexplored, and (unusually) we hold per-query cost ground truth (ADR-011) next to quality metrics (ADR-036 CSS/DDI/SAS, rubric scores).

Meanwhile nothing guards council quality against silent drift: model pool changes, prompt tweaks, and routing features (ADR-044) all shift output with no regression signal. ADR-036 Phase 3 specified golden datasets and DeepEval/RAGAS bridges but was deferred.

Decision

Phase 1 — Golden dataset + drift regression harness

  • A small curated question set (~20–30 items across domains: coding, reasoning, factual, judgment) committed to the repo with expected-quality envelopes (rubric ranges, CSS floor, key-content assertions — not exact-string).
  • A harness (llm-council bench run) executes the set against a configured council and reports per-item metrics + aggregate deltas vs. a committed baseline snapshot.
  • Runs on demand and nightly (scheduled CI), never per-PR — each run costs real API spend; the harness prints its own ADR-011 cost total.
  • Spend caps (council feedback): a hard per-run cap (LLM_COUNCIL_BENCH_MAX_USD, default $2.00) enforced via the ADR-011 estimator before each item and actuals after; the run aborts gracefully at the cap with partial results marked partial. The scheduled workflow sets an explicit cap and a monthly guard (skip if month-to-date bench spend, summed from run artefacts, exceeds LLM_COUNCIL_BENCH_MONTHLY_USD, default $30).
  • CLI (council feedback): llm-council bench run [--config PATH] [--max-usd N] [--items id,…], bench baseline --set (snapshot current as baseline), bench report [--format md|json]. Exit codes: 0 within envelope, 1 drift beyond envelope, 2 aborted/partial.

Phase 2 — Council-vs-single-model benchmark (quality per dollar)

  • The same question set run across configurations: each council member solo, the full council, and (flag-on) ADR-044 graduated depth.
  • Output: quality-per-dollar table using ADR-011 actual costs — the empirical answer to "when does deliberation pay?", and the tuning data ADR-044's thresholds need.
  • Scoring: rubric scores from a fixed judge configuration, with the ADR-047 calibration caveats attached; publish methodology with results.

Dataset governance (council feedback): the dataset is versioned in-repo (bench/dataset/vN/), changes only by PR review; items must be original or verifiably licensed, contain no PII, and record provenance + rationale; envelope changes require a note justifying the new expected range (no silent goal-post moves).

Phase 3 — Publication + eval-framework bridges

  • Results page in the docs site, regenerated by the harness (methodology, costs, dates, model versions — reproducibility first).
  • Thin DeepEval/RAGAS adapters so external eval suites can drive the council as a target (ADR-036 P3 commitment).

Consequences

Positive: evidence for the core claim (or honest boundaries of it) in a crowding ecosystem; a drift alarm for every future routing/prompt change; the dataset doubles as ADR-047 calibration input and ADR-044 threshold-tuning input — three consumers, one asset.

Negative / risks: benchmark runs cost real money (mitigation: small set, scheduled not per-PR, cost printed via ADR-011); judge-based scoring inherits judge bias (mitigation: fixed judge config, publish raw outputs, ADR-047 calibration notes); a public "council loses at X" result is possible (mitigation: that is the point — publish honestly with cost context).

Definition of Done (per phase — council feedback: all phases covered)

  • P1: harness unit-tested with mocked councils (no live spend in CI unit tests); dataset v1 committed with governance doc; baseline snapshot; spend-cap abort tested.
  • P2: config matrix runs produce the quality-per-dollar table from ADR-011 actuals; methodology documented alongside results.
  • P3: docs-site page regenerated by the harness; DeepEval/RAGAS adapters with at least one round-trip example each.
  • All phases: README bench section, CLAUDE.md, CHANGELOG; scheduled workflow reviewed for both caps.

References

  • docs/roadmap-2026-h2.md item 5; ADR-036 §Phase 3; ADR-011 usage schema.