Each Rerun Gets One Result Record

Result record for the rerun protocol

This record goes with "A Rerun of a Paper’s Code Is Scored Before It Starts". Fill one for each paper in the sample, including the ones that failed or were excluded. It names the paper by its DOI. It holds no data from the paper and no data about patients.

Field	What to write
Paper	The DOI of the paper.
Repository	The address of the code and, where there is one, the DOI of the release.
Rerunner	An identifier for the person, and a statement that the person is not an author of the paper.
Date and environment	The date of the rerun, the computer, the operating system and the network access.
Mechanical checks	Yes, no or cannot tell for each of the six checks, recorded before the rerun.
Unstated steps	Each step taken that the instructions did not state.
Outputs reached	For each table and figure: reached, differs (with both values), or not produced.
Time	The hours from the first step to the last result, or "time limit".
First cause of failure	One of: code not reachable, environment not reproducible, data not reachable, code error, output differs, instructions missing, time limit. Empty when every output was reached.
Notes	Anything a second rerunner would need to know. No judgment of the paper’s conclusions.

Field

What to write

Paper

The DOI of the paper.

Repository

The address of the code and, where there is one, the DOI of the release.

Rerunner

An identifier for the person, and a statement that the person is not an author of the paper.

Date and environment

The date of the rerun, the computer, the operating system and the network access.

Mechanical checks

Yes, no or cannot tell for each of the six checks, recorded before the rerun.

Unstated steps

Each step taken that the instructions did not state.

Outputs reached

For each table and figure: reached, differs (with both values), or not produced.

Time

The hours from the first step to the last result, or "time limit".

First cause of failure

One of: code not reachable, environment not reproducible, data not reachable, code error, output differs, instructions missing, time limit. Empty when every output was reached.

Notes

Anything a second rerunner would need to know. No judgment of the paper’s conclusions.

A replication shares records, never code outside its licence

A replication of the study may share the result records, the sampling frame and the random seed. It shares a paper’s code only under that code’s own licence.