Discoverability Benchmark Build

A skill for the Discoverability Operator that builds the question benchmark a Discoverability Score runs on, drafting finder questions from real inputs, reviewing them, and ending with a recommendation to freeze the benchmark or not.

This skill is Labor as Code™ for the measurement loop in Discoverability Engineering. Labor as Code is a skill that encapsulates a person's knowledge, judgment, taste, and skill in durable, inspectable, versioned form. The know-how is an editor's suspicion of any test set that its author wrote with the subject in mind.

The method comes from “A Discoverability Score Measures Whether Finders Who Don’t Know Your Name Meet You”, which defines the Finder Specification as the source of the corpus, the ten question classes and their weights, the rules against gaming, and the freeze. "A Finder Question Library Holds What Finders Actually Need to Know" gives the question fields. Read them for the argument. This skill runs the method on the user's own subject and never reuses a worked example's questions.

The toolkit does the arithmetic. This skill works with the Discoverability Score toolkit, the download on the book's online companion, run as archon score commands. Every path below is relative to the toolkit folder: templates/, schemas/, methodology/config.yaml, and docs/. A case folder holds entity.yaml, engines.yaml, finder-specification.yaml, benchmarks/<version>/, and runs/<run_id>/. The skill never computes a score, a weight, or a hash itself. The scripts do, and the same inputs give the same score.

A benchmark that confirms its author is worse than none. Never ask what queries might help the subject rank. Start from the finder. The subject must be removable from the process: the corpus still makes sense if the subject disappeared tomorrow.

Say surfaced, never recommended. An engine that names a subject has not endorsed it, ranked it, or recommended a purchase. The skill writes surfaced, included, associated, introduced, cited, and discoverable, and it never writes "recommended" or "best" about what an engine did.

Before running

Confirm these points with the user before the first draft:

1. Which subject. One subject per benchmark. Read entity.yaml, or ask for the canonical name, every alias, brand terms, and canonical URLs.
2. Which real inputs exist. List the files in inputs/. If there are none, say so now: every question will rest on the Finder Specification alone, the weakest permitted source, and the skill will say so again at the end.
3. Who is the Discoverability Operator. A named person reviews every question and approves the freeze. If nobody is named, stop and ask.
4. Which version. Name the benchmark version, such as v1. A frozen version is never edited.

Step 1: Draft the Finder Specification

Use templates/finder-specification.yaml. Run this prompt against the entity file and the inputs:

Draft Finder Specifications for the people who might legitimately need to find the entity in entity.yaml. Use templates/finder-specification.yaml. For each finder, state the finder, the role, the trigger event, the job to be done, the desired outcome, the constraints, the risks, the finder's own terminology, the selection criteria, and the trust requirements. Take every statement about a finder from the files in inputs/; write the source of each in evidence_basis. If the inputs do not support a finder, do not write one. Write what you do not know in unknown. Use the finder's words, not the entity's. Do not copy phrases from the entity's own pages. Do not add a finder because it would make the entity look more discoverable.

Check the output:

• Reject a finder with no source in inputs/.
• Reject terminology that appears only on the subject's own pages.
• Ask whether the specification still makes sense if the subject were deleted.
• The Operator decides who the finder is. The skill proposes and never decides.

Save finder-specification.yaml in the case folder.

Step 2: Generate the questions

Use templates/questions.yaml. Set each weight from methodology/config.yaml. Run this prompt:

Given these Finder Specifications, generate the natural questions each finder would ask before knowing the entity exists. Do not use the entity's name, any alias, brand language, slogans, product names, or wording copied from the entity's website. Cover problem discovery, category discovery, capability validation, provider selection, comparison, trust validation, buying intent, educational discovery, and publication discovery. Write each question the way the finder would type or say it, using the finder's terminology from the specification. For each question, name the finder specification it comes from, the class, the framing, and the real input its wording was based on. Add the entity-lookup questions last: who is the entity, what does it do, what is it known for, and what products or organizations are associated with it. Do not write a question because the entity happens to answer it.

Check the output:

• Every question names a finder specification, a class, a framing, and a real source.input_type. The subject's own marketing copy is never a source.
• Set ai_expanded: true on every question the AI wrote or reworded. The Operator records the reviewer in source.reviewed_by.
• Read each question aloud as the finder. Delete any the finder would not ask, and any that exists only because the subject happens to answer it. Also delete any question the subject should not be an answer to.

Save benchmarks/<version>/questions.yaml.

Step 3: Validate the draft

Run:

archon benchmark check --draft benchmarks/<version>/questions.yaml

Errors stop the work. They include a weight that differs from its class, a finder that does not exist, and an unbranded question that contains the subject's name or a brand term. Warnings are findings, not noise. Read each one. A finder with no question, a missing class, a duplicate wording, an AI-expanded question with no reviewer, and wording derived from the specification alone all appear here. Fix the errors and report every warning to the Operator.

Step 4: Review the corpus as an adversary

Use a different AI session from the one that wrote the questions, with no access to the drafting conversation. Run this prompt there:

Review this question corpus as an adversary who wants to find out whether it was built to favor the entity. List every question that is target-biased, that uses the entity's own positioning or vocabulary, that has a predetermined answer, that is unnatural phrasing no finder would use, that duplicates another question, that is obscure, or that only an insider would ask. List the finder specifications that have too few questions and the classes that are missing. For each problem, quote the question, state the problem in one sentence, and propose the smallest fix. Do not rewrite the corpus and do not add questions.

Check the output: the Operator accepts or rejects each finding. Fix before the freeze, never after. Save analysis/corpus-review-<date>.md.

Step 5: Freeze, only with the Operator's approval

Ask the Operator for an explicit yes. Then run:

archon benchmark record benchmarks/<version>/questions.yaml

The script writes benchmark.lock.yaml with a SHA-256 and the date. It refuses to change an existing lock. After the freeze no question is reworded, dropped, or reweighted. A change is a new benchmark version. Never freeze on the skill's own judgment.

Step 6: Conclude

End with a recommendation and its reasoning. Use one of three:

• Freeze. Every question has a real input and a named reviewer, the validator shows no error, the adversarial review's findings are resolved, and every finder has questions in every class that applies.
• Do not freeze yet. Name each condition that fails and who fixes it. A benchmark whose wording comes from the Finder Specification alone is not ready. Say so, and say what real input would settle it: an interview, a sales call, a search log, an RFP, a forum thread.
• The evidence is too thin to recommend either. Say what is missing and what would resolve it. Never freeze to sound useful.

State the number of questions by class, the finders covered, the sources used, and every warning left open.

Never

• Never put the subject's name, an alias, or a brand term in an unbranded question.
• Never copy wording from the subject's own pages.
• Never write a finder or a question the inputs do not support.
• Never freeze without the Operator's yes, and never edit a frozen benchmark.
• Never run a question against an engine. Capturing answers belongs to discoverability-score-run.