Check Every Piece of Evidence Against the Fourteen Ways It Can Mislead
Evidence challenge checklist
Companion to The Product Manager’s New Job Is Acceptance · Updated 2026-09-30
The chapter “Teams Misread Acceptance Evidence in Fourteen Predictable Ways” explains each mistake. Run this list on a record before a decision leans on it. Rank the issues you find by how much they could change the decision, not by how easy they are to fix.
| Mistake | Ask | Result |
|---|---|---|
| Selection bias | How were these people chosen, and who was left out? | |
| Survivor bias | What happened to the people who are not in this data? | |
| Proxy-user bias | Is anyone here actually the person who does this work? | |
| Buyer-user confusion | Whose satisfaction or behavior is this evidence about? | |
| Coerced-usage bias | Could these people have chosen not to use it? | |
| Demo bias | Would a customer see this on their own data? | |
| Training contamination | What could they do before the training? | |
| Instrumentation blindness | What do we know from behavior, and what only from what people said? | |
| Metric substitution | Is this the outcome, or something that usually goes with it? | |
| Incentive contamination | Would they have done this without the incentive? | |
| Funnel fragmentation | Did one customer go through the whole sequence? | |
| Confirmation bias | What would each team have said if the result were reversed? | |
| Novelty effects | What does use look like after the novelty is gone? | |
| Version drift | Which version did this evidence test? |
Seven conditions every metric needs
For which ICP? Under which product version? Through which channel? Compared with what? With how much assistance? Over what time? Was use voluntary?
If you cannot answer three or more, the number is not ready to support a decision.
Download