Protocol for H1 and H2
This protocol implements the practical tests of H1 and H2 in the program. It measures whether an outside person can reach a paper’s reported numbers from its shared code, how long it takes and why it fails. It doesn’t show whether the paper’s conclusions are right.
Before you start
-
Write down the sample frame: the journals, the publication window and the rule for "links to code". Write it before you search.
-
Write down what counts as reaching a reported number: the same value to the precision the paper reports, or a stated tolerance for results that use random numbers.
-
Write down the time limit for one rerun. The program’s H1 uses one working day of eight hours.
-
Choose the people who do the reruns. A person who reruns a paper must not be an author of it.
-
Record the computer, the operating system and the network access that each rerun uses.
Sampling
-
Search the journals in the frame for papers that link to a code repository or a code archive.
-
Exclude each paper whose code needs data that the rerunner cannot get under the paper’s own access terms. Record each exclusion and its reason.
-
Choose papers from the remaining set at random. Record the random seed.
Mechanical checks, before the rerun
Do these checks for each repository before anyone tries to run it. Record each answer as yes, no or cannot tell.
-
Is the release archived with a DOI, or another persistent identifier?
-
Is the environment pinned: a lockfile, a container definition or a list of package versions?
-
Does the data reference resolve to the data the paper cites?
-
Is there one documented entry point: a command or a numbered list of steps?
-
Are the outputs declared: does a file or a README map each table and figure to what writes it?
-
Is a licence stated?
Do not change the answers after the rerun.
The rerun
-
Start a clean environment with no software from earlier reruns.
-
Follow the repository’s instructions as written. Record each step you take that the instructions do not state.
-
Stop at the time limit.
-
For each table and figure in the paper, record whether the rerun reached the reported number.
-
If the rerun failed, record the first cause: code not reachable, environment not reproducible, data not reachable, code error, output differs, instructions missing, or time limit.
-
Write the result record before you look at the mechanical-check scores of other repositories.
Failed reruns are reported with the same detail as successes
Report the share of papers that reached all their reported numbers within the time limit, the share that reached some, the time each rerun took, and the count of each first cause. For H2, report how well the mechanical-check score separates the reruns that succeeded from those that failed. Report the failed reruns with the same detail as the successful ones.
H1 fails if more than half of the sampled papers reach all their reported numbers within the time limit. H2 fails if the mechanical-check score separates successes from failures no better than chance.