Reviewer's checklist
This checklist is for a reviewer who has a companion repository and limited time. It does not replace reading the paper. It is the reviewer’s half of "A companion repository holds six parts", and whether it helps a reviewer find errors is H4, which is untested.
-
Find the version DOI the paper cites. Use that release, not the latest one.
-
Check the six parts with the author’s checklist. Record each part that is missing.
-
Choose the result that the paper’s main conclusion depends on.
-
Start a clean environment from the repository’s environment record.
-
Follow the entry point to that one result. Record each step you had to guess.
-
Compare the number you get with the number in the paper.
-
If the numbers differ, or you could not reach the result, write that in the review with the step where it failed.
-
If you have more time, repeat the rerun for the next result the conclusion depends on.
For a private repository, ask the authors or the editor for access as an invited reviewer. Do not copy the code out of the repository.