Intellectual Frontiers

Question classes carry weights, and the benchmark is frozen before the first run

The ten classes, the rules against gaming, and the freeze

A finder choosing a provider counts more than a finder who already has a name

Someone asking "Who is X?" has already found X. Someone asking "Who can solve this problem?" has not. The second question is the real subject of Discoverability Engineering, so it counts for more. Weights are in config.yaml and are stored on every question so a reader can see them.

Class Weight Feeds The finder is asking

entity_lookup

0.25

Entity Recognition only

Who is X? What does X do?

educational_discovery

0.50

Coverage

How does this work? What should I know?

publication_discovery

0.75

Coverage, Category

Which books, guides, or papers cover this?

problem_discovery

1.00

Coverage, Problem

I have this problem, trigger, constraint, or failure. What now?

trust_validation

1.00

Coverage

How do I check that a provider is credible?

capability_validation

1.25

Coverage, Problem

Who can do this specific job?

category_discovery

1.25

Coverage, Category

What kinds of providers, tools, or methods exist?

provider_selection

1.50

Coverage, Category

Who should I choose?

comparison

1.50

Coverage, Category

How do these options compare?

buying_intent

1.50

Coverage, Category

I need this soon. Where do I start, and what does it cost?

entity_lookup is branded by definition. It never enters Finder Specification Coverage, so recognition cannot inflate the unbranded measures. The scorecard reports a "blended mention rate" across all classes at these weights as a diagnostic: it is the number a vanity dashboard would show, and it is shown only so it can be compared with the real one.

Questions come from the finder’s words

Do not ask "what queries might help this company rank?" That produces a benchmark that confirms its author’s positioning. The order is:

  1. Write the Finder Specification: the finder, role, trigger event, job, desired outcome, constraints, risks, terminology, selection criteria, and trust requirements.

  2. Collect real inputs: customer interviews, sales calls, search behavior, support conversations, RFP and procurement language, analyst questions, industry forums, public buying guides, and common professional questions.

  3. Write the questions in the finder’s words, one at a time, each tied to a finder specification.

  4. Only then let AI expand and normalize. AI works from the inputs and never decides which questions matter.

  5. Have a person review every question and record the review.

The subject must be removable from the process. A good corpus still makes sense if the subject disappeared tomorrow.

The validator enforces the rules against gaming

  • A non-lookup question may not contain the subject’s name, any alias, or any listed brand term. This is an error.

  • A question’s source.input_type names where the wording came from. The subject’s own marketing copy, slogans, and category pages are not an allowed source.

  • An AI-expanded question needs a named human reviewer.

  • Duplicate wording is a warning.

  • A question needs a class, a weight equal to its class weight, a finder specification, and a date added.

  • Do not write a question because the subject happens to answer it. Delete it.

A benchmark is frozen before the first run

archon benchmark record questions.yaml writes benchmark.lock.yaml with a SHA-256 of the canonical form of the file. After that:

  • archon score check fails a run if the questions no longer match the lock, if the lock is missing, or if the run began before the freeze date.

  • A question is never reworded, removed, or reweighted. New or changed questions are a new benchmark_version.

  • A retest runs the same benchmark. The old results stay comparable with each other.