--- name: discoverability-benchmark-build description: A skill for the Discoverability Operator that builds the question benchmark a Discoverability Score runs on, drafting finder questions from real inputs, reviewing them, and ending with a recommendation to freeze the benchmark or not. Use this skill when the user wants to build, review, or freeze the question benchmark that the Discoverability Score runs on. Inputs include the subject's entity file, real finder inputs (interviews, sales calls, search data, support conversations, RFP language, forum threads, buying guides), and any existing Finder Specification. The skill drafts the Finder Specification only from those inputs, generates natural finder questions that never contain the subject's name or brand language, runs the validator and an adversarial review in a separate session, and freezes the benchmark with a hash only after the Discoverability Operator approves. It ends with a recommendation to freeze or not to freeze yet, with the reasons, and says plainly when the questions rest on the weakest permitted source. Triggers on requests like "build the discoverability benchmark," "write the finder questions for the score," "is this question set biased toward us," "freeze the benchmark," or "start a Discoverability Score baseline." --- # Discoverability Benchmark Build This skill is Labor as Code™ for the measurement loop in *Discoverability Engineering*. Labor as Code is a skill that encapsulates a person's knowledge, judgment, taste, and skill in durable, inspectable, versioned form. The know-how is an editor's suspicion of any test set that its author wrote with the subject in mind. The method comes from “A Discoverability Score Measures Whether Finders Who Don’t Know Your Name Meet You”, which defines the Finder Specification as the source of the corpus, the ten question classes and their weights, the rules against gaming, and the freeze. "A Finder Question Library Holds What Finders Actually Need to Know" gives the question fields. Read them for the argument. This skill runs the method on the user's own subject and never reuses a worked example's questions. **The toolkit does the arithmetic.** This skill works with the Discoverability Score toolkit, the download on the book's online companion, run as `archon score` commands. Every path below is relative to the toolkit folder: `templates/`, `schemas/`, `methodology/config.yaml`, and `docs/`. A case folder holds `entity.yaml`, `engines.yaml`, `finder-specification.yaml`, `benchmarks//`, and `runs//`. The skill never computes a score, a weight, or a hash itself. The scripts do, and the same inputs give the same score. **A benchmark that confirms its author is worse than none.** Never ask what queries might help the subject rank. Start from the finder. The subject must be removable from the process: the corpus still makes sense if the subject disappeared tomorrow. **Say surfaced, never recommended.** An engine that names a subject has not endorsed it, ranked it, or recommended a purchase. The skill writes *surfaced*, *included*, *associated*, *introduced*, *cited*, and *discoverable*, and it never writes "recommended" or "best" about what an engine did. ## Before running Confirm these points with the user before the first draft: 1. **Which subject.** One subject per benchmark. Read `entity.yaml`, or ask for the canonical name, every alias, brand terms, and canonical URLs. 2. **Which real inputs exist.** List the files in `inputs/`. If there are none, say so now: every question will rest on the Finder Specification alone, the weakest permitted source, and the skill will say so again at the end. 3. **Who is the Discoverability Operator.** A named person reviews every question and approves the freeze. If nobody is named, stop and ask. 4. **Which version.** Name the benchmark version, such as `v1`. A frozen version is never edited. ## Step 1: Draft the Finder Specification Use `templates/finder-specification.yaml`. Run this prompt against the entity file and the inputs: > Draft Finder Specifications for the people who might legitimately need to find the entity in `entity.yaml`. Use > `templates/finder-specification.yaml`. For each finder, state the finder, the role, the trigger event, the job to > be done, the desired outcome, the constraints, the risks, the finder's own terminology, the selection criteria, > and the trust requirements. Take every statement about a finder from the files in `inputs/`; write the source of > each in `evidence_basis`. If the inputs do not support a finder, do not write one. Write what you do not know in > `unknown`. Use the finder's words, not the entity's. Do not copy phrases from the entity's own pages. Do not add a > finder because it would make the entity look more discoverable. Check the output: - Reject a finder with no source in `inputs/`. - Reject terminology that appears only on the subject's own pages. - Ask whether the specification still makes sense if the subject were deleted. - The Operator decides who the finder is. The skill proposes and never decides. Save `finder-specification.yaml` in the case folder. ## Step 2: Generate the questions Use `templates/questions.yaml`. Set each weight from `methodology/config.yaml`. Run this prompt: > Given these Finder Specifications, generate the natural questions each finder would ask before knowing the entity > exists. Do not use the entity's name, any alias, brand language, slogans, product names, or wording copied from the > entity's website. Cover problem discovery, category discovery, capability validation, provider selection, > comparison, trust validation, buying intent, educational discovery, and publication discovery. Write each > question the way the finder would type or say it, using the finder's terminology from the specification. For each > question, name the finder specification it comes from, the class, the framing, and the real input its wording was > based on. Add the entity-lookup questions last: who is the entity, what does it do, what is it known for, and what > products or organizations are associated with it. Do not write a question because the entity happens to answer it. Check the output: - Every question names a finder specification, a class, a framing, and a real `source.input_type`. The subject's own marketing copy is never a source. - Set `ai_expanded: true` on every question the AI wrote or reworded. The Operator records the reviewer in `source.reviewed_by`. - Read each question aloud as the finder. Delete any the finder would not ask, and any that exists only because the subject happens to answer it. Also delete any question the subject should not be an answer to. Save `benchmarks//questions.yaml`. ## Step 3: Validate the draft Run: ``` archon benchmark check --draft benchmarks//questions.yaml ``` Errors stop the work. They include a weight that differs from its class, a finder that does not exist, and an unbranded question that contains the subject's name or a brand term. Warnings are findings, not noise. Read each one. A finder with no question, a missing class, a duplicate wording, an AI-expanded question with no reviewer, and wording derived from the specification alone all appear here. Fix the errors and report every warning to the Operator. ## Step 4: Review the corpus as an adversary Use a different AI session from the one that wrote the questions, with no access to the drafting conversation. Run this prompt there: > Review this question corpus as an adversary who wants to find out whether it was built to favor the entity. > List every question that is target-biased, that uses the entity's own positioning or vocabulary, that has a > predetermined answer, that is unnatural phrasing no finder would use, that duplicates another question, that is > obscure, or that only an insider would ask. List the finder specifications that have too few questions and the > classes that are missing. For each problem, quote the question, state the problem in one sentence, and propose > the smallest fix. Do not rewrite the corpus and do not add questions. Check the output: the Operator accepts or rejects each finding. Fix before the freeze, never after. Save `analysis/corpus-review-.md`. ## Step 5: Freeze, only with the Operator's approval Ask the Operator for an explicit yes. Then run: ``` archon benchmark record benchmarks//questions.yaml ``` The script writes `benchmark.lock.yaml` with a SHA-256 and the date. It refuses to change an existing lock. After the freeze no question is reworded, dropped, or reweighted. A change is a new benchmark version. Never freeze on the skill's own judgment. ## Step 6: Conclude End with a recommendation and its reasoning. Use one of three: - **Freeze.** Every question has a real input and a named reviewer, the validator shows no error, the adversarial review's findings are resolved, and every finder has questions in every class that applies. - **Do not freeze yet.** Name each condition that fails and who fixes it. A benchmark whose wording comes from the Finder Specification alone is not ready. Say so, and say what real input would settle it: an interview, a sales call, a search log, an RFP, a forum thread. - **The evidence is too thin to recommend either.** Say what is missing and what would resolve it. Never freeze to sound useful. State the number of questions by class, the finders covered, the sources used, and every warning left open. ## Never - Never put the subject's name, an alias, or a brand term in an unbranded question. - Never copy wording from the subject's own pages. - Never write a finder or a question the inputs do not support. - Never freeze without the Operator's yes, and never edit a frozen benchmark. - Never run a question against an engine. Capturing answers belongs to `discoverability-score-run`.