A run keeps the conditions, the raw answer, and the normalized observation apart

What is captured for every answer, and how a run folder is laid out

Each answer in a run is one question, asked once, on one engine. The record for it holds:

the benchmark version, the exact question, its class, and its weight;
the engine and product, the model or mode if known, and whether web access was on;
the date, the approximate time, and the geography when it matters;
the full answer, kept in raw/ and never edited;
whether the subject was mentioned, and the sentence where it was (substantive, incidental, absent, or ambiguous);
the citations, each with its URL and source type;
the alternatives the answer introduced; and
the apparent rationale and any notes.

For a named question, the record also holds a recognition judgment (accurate, partly accurate, inaccurate, or not found) and any conflicting names or dates the answer repeated.

Raw, normalized, and analysis files stay separate

Cleaning an answer never overwrites it. A later researcher can read the same raw answers differently, and the score can be recomputed from the normalized records without trusting anyone’s interpretation.

runs/<date>-baseline/
  manifest.yaml               versions: methodology, benchmark, engines, subject
  questions.yaml              the frozen benchmark, copied unchanged
  benchmark.lock.yaml         its SHA-256 and freeze date
  raw/                        every answer as captured
  normalized/results.jsonl    one line per answer
  normalized/assessments.yaml evidence ratings, chains, clarity and technical checks
  analysis/diagnosis.yaml     observation, hypothesis, experiment; the Competitive Inclusion Set
  evidence-gaps.yaml          the Evidence Gap Register
  interventions.yaml          what to change, and what would show it failed
  scorecard.yaml              written by archon score generate, never by hand
  executive.md, report.md     written by archon report generate

A future run gets its own folder, such as runs/2027-01-15-retest/, and its manifest names the methodology version, the benchmark version, the engine configuration version, and the subject configuration version it used.

Answers are captured by hand or through a permitted interface

Capture answers by hand, or through an interface the engine’s maker permits. Do not drive an engine’s website with browser automation that its terms forbid. The method works without it. The engine file records how each result was captured: manual_capture, permitted_api, vendor_export, or third_party_audit.

Generative systems vary from one run to the next, so run each question more than once on each engine where you can. The scorecard counts the share of runs that were substantive and lists the cells where runs disagreed.

Engines are configuration, not method

The engines go in one file, engines.yaml. Adding, retiring, or changing an engine edits that file and nothing else. A changed engine set is a disclosed comparison, and the comparison script prints the change. The durable pieces are the finder, the question, the evidence, the association, the observation, and the experiment.