Code
The scripts, notebooks or spreadsheets that produced every table and figure. A spreadsheet counts, with its formulas left in, and a short README that says the order of the steps.
Research program
Companion Executable Code Repositories
A companion executable code repository is the code that produced a paper's results, packaged so that someone else can run it and get the same numbers. It sits beside the paper and the paper's data, and each of the three points to the other two. This program asks what such a repository has to hold for a reviewer, a funder or a reader to check a result without rebuilding the whole analysis. It starts with diabetes and biomedical research, because that's where I work and where a wrong number can reach a patient, and its rules are written to work in any field.
What must a companion executable code repository contain, link to and prove for a reviewer to verify a paper's results without rebuilding the analysis?
A paper, its data and its code let anyone check a result
Claim: recommendation
A published paper tells you what the researchers did and why. A data repository holds what they measured. Neither shows how the raw data turned into the numbers in the paper's tables. A companion code repository fills that gap: it holds the steps themselves, in a form a computer can run again. Add a habit of releasing new versions as new data arrives, and you get a result that anyone can rerun today and test again next year.
This adds to peer review and to the data repositories researchers already use. It replaces neither. The journal still publishes the paper, and the data stays where the data rules say it goes.
What would show this claim is wrong
- Readers and reviewers given the paper, the data and the code check results no more often, or no more accurately, than readers given the paper and the data alone.
- Research groups that add a companion repository see their findings questioned or corrected no more often than groups that don't, because nobody reruns them.
Only the code shows how an analysis was done
Claim: inference
A methods section is a summary, written after the analysis ran. It can't list every rule for leaving out a patient, every unit conversion and every software version, and the result can depend on any of them. So a reviewer who wants to know whether a number is right has two choices: rebuild the analysis from the summary, which takes days nobody has, or judge whether the description sounds right. Most judge. The code that actually ran is the only complete statement of the method. When it isn't shared, a reader can believe the method but can't check it.
What would show this claim is wrong
- Independent teams given only a paper's methods section reproduce its main results about as often as teams given the paper and its code.
- When shared code fails to rerun, the cause is mostly missing access to data, not anything the code itself would have told the reader.
A companion repository holds six parts
Claim: recommendation
I recommend that a companion repository hold six things, and that nobody call a paper reproducible until all six are there. Each one closes a gap that otherwise stops a stranger from getting from the data to the paper's numbers.
The seven practical steps to set one up are in A companion repository takes seven steps to set up. The companion to this program has the checklists an author and a reviewer can use today.
What would show this claim is wrong
- Repositories that hold all six parts fail to rerun about as often as repositories that hold fewer.
- Reviewers given repositories with all six parts take no less time to confirm a result than reviewers given the code alone.
Environment
The exact software the code needs, down to the version numbers, written in a lockfile (a list of every package and its version) or a container definition (a recipe for a clean computer with that software installed), or both.
Data reference
The DOI of the exact copy of the data the paper used. A DOI (Digital Object Identifier) is a permanent address for a paper, a dataset or a software release that keeps working when websites move. Where the data may be shared, the copy itself goes in too, or a synthetic copy with the same structure.
One entry point
One documented command, or one numbered list of steps, that goes from the data to every reported result.
Declared outputs
A list of the paper's tables and figures, each matched to the file the entry point writes for it, so a rerun can be compared with the paper number by number.
Metadata and links
A DOI for the exact release the paper cites, a second DOI that stands for every version, and links from the code to the paper and the data and back again.
Patient-level data stays in a governed repository
Claim: recommendation
Data about individual patients, even with names and identifiers removed, belongs in a repository built to control who may see it, such as Vivli or the YODA Project, under the laws of the place the data came from. The companion repository then holds the code, the environment, the metadata and a link to the governed record. To let the code run end to end without the real data, it can also hold a synthetic copy: made-up records with the same columns and types.
Plenty of studies have no patient-level data at all: simulations, device bench tests, analyses of published aggregates, new statistical methods. For those, the code and the data can sit together in one repository with no extra governance. Splitting only where patient data is involved puts the burden where the risk is.
Three limits apply in every case. Clinical trial data stays under the rules that already govern it, and a companion repository for a trial shares the analysis with synthetic or aggregate data. Work under an industry contract can stay private, using the approach in A private repository lets a reviewer rerun code that can't be public. And institutional policy and data use agreements come before anything this program recommends; check with your research compliance office before you share anything about sensitive or restricted data. The reasoning is in Patient data stays governed and the code goes beside the paper.
What would show this claim is wrong
- Synthetic copies built to the governed data's structure don't exercise the parts of the code that produce the paper's main results.
- Reviewers invited into a private companion repository rerun it no more often than they rerun public ones.
Versioned releases let an analysis follow new data
Claim: recommendation
The version of the code a paper cites never changes. When new data arrives, such as real-world data from health records, registries, claims or devices, the authors can run the same method again and publish a new release. Each release gets its own DOI, and one more DOI, the concept DOI, stands for the whole line of versions. Real-world evidence is what that analysis of real-world data shows.
A reader can then see whether the finding still holds as evidence builds up, while the original stays exactly as the reviewers saw it. The Lovable microsite called this living evidence. I'd use that phrase only for a line of releases a person can rerun, version by version. A paper that's quietly edited after publication is the opposite of what's meant.
More on how this works, and on what it does to how medicine weighs evidence, is in Versioned releases keep the evidence current while the paper stays fixed and A result anyone can rerun counts for more than a famous name.
What would show this claim is wrong
- Few authors release a second version of a companion repository after publication, so the line of releases doesn't happen in practice.
- Readers and guideline writers don't use later versions, even when they exist.
Metadata should be a profile of four existing standards
Claim: recommendation
The Lovable microsite promised a new JSON Schema, a machine-readable list of the fields a file must have, for companion repositories. Standards already describe most of what's needed. CodeMeta describes software. The Citation File Format says how to cite it. RO-Crate packages code and data with the links between them. The DataCite Metadata Schema describes the DOI record. I recommend a profile: a short list of which fields of those four standards a companion repository must fill, written as a JSON Schema that checks them. A brand-new schema would be one more format for every journal and tool to learn, and it would drift away from the standards archives already read.
The one thing none of the four says is which file makes which table and figure. The profile adds that as an RO-Crate entry, and nothing else.
What would show this claim is wrong
- A field a reviewer needs to rerun the analysis can't be expressed in any of the four standards or in the one added entry.
- Archives and journals ignore the profile's required fields, so it changes nothing about what gets deposited.
Reproducibility review should check with rules first
Claim: recommendation
Most reviewers don't have days to rebuild an analysis. Automation can help, if it's built in the right order. I recommend three layers. First, rules a program can check without judgment: is the release archived, are the software versions pinned, does the data DOI resolve, is there one entry point, are the outputs declared. Each rule answers yes, no or cannot tell. Second, a model, used only where a rule can't answer, such as finding the steps in a README that never names an entry point; it says where in the repository it looked. Last, a person, who judges what the first two layers flagged. A lower layer's answer is never overruled by a higher one.
Instead of reconstructing a pipeline by hand, a reviewer gets a report: which checks passed, which outputs the rerun reached, where it stopped, and what the model flagged and why. The reviewer's time goes to the findings, not the plumbing. The reasoning, and why the order matters, is in Check reproducibility with rules first, a model second and a person last. The same ordering is what Evidence Credibility Engineering recommends for judging research in general.
What would show this claim is wrong
- A model given the whole repository predicts rerun success as well as the rules do, at the same cost, and its errors are no harder to find.
- Reviewers given the layered report find no more problems, and take no less time, than reviewers given the repository alone.
Policies ask for data and mention code only at the edges
Claim: evidence
Sources: the NIH Policy for Data Management and Sharing (NOT-OD-21-013) and its plan elements (NOT-OD-21-014), Executive Order 14303, "Restoring Gold Standard Science" (90 FR 22601), CONSORT 2025, SPIRIT 2025, the FAIR principles for research software and the TRUST principles, read on 2026-10-22 (source-check.md).
The NIH policy took effect on 25 January 2023. It requires a plan for managing and sharing scientific data, and it doesn't ask anyone to share the code of an analysis; the plan only names the tools a person needs to use the shared data. The executive order, signed on 23 May 2025, defines Gold Standard Science as reproducible, transparent and open to being proved wrong, among other things, and tells federal agencies, "to the extent practicable", to make public the analyses behind influential science, "including, as applicable, the source code". It binds agencies, not the researchers they fund. CONSORT 2025 and SPIRIT 2025, the reporting guidelines for clinical trials, ask authors to say where the statistical code can be found; they don't require sharing it. NIH published a plan for Gold Standard Science on 22 August 2025, which I couldn't open, so I say nothing here about what it asks of code.
A companion repository fits all of these and is required by none of them.
| Policy or principle | What it asks for | Where a companion repository fits |
|---|---|---|
| NIH Data Management and Sharing Policy | A plan, and shared scientific data | Adds the code that turns the shared data into the paper's results, which the policy doesn't ask for |
| Executive Order 14303 | Agencies make influential analyses public, with source code where applicable | Gives an agency, or a grantee it funds, the code in a form that can be rerun |
| CONSORT 2025 and SPIRIT 2025 | Trial reports say where the statistical code can be accessed | Gives that statement a permanent address with a DOI |
| FAIR principles for research software | Code that is findable, accessible, interoperable and reusable | The six parts and the metadata profile make a paper's code meet them |
| TRUST principles | Repositories that are transparent, responsible, user-focused, sustainable and technically sound | Applies to the archive that holds the release, such as Zenodo or Software Heritage |
| Institutional policy and data use agreements | Whatever the institution and the data owner require | Comes first, every time |
A journal code policy alone leaves most results unreproduced
Claim: evidence
Sources: Stodden, Seiler and Ma, PNAS 2018, Price and colleagues, BMJ Open 2018 and the American Journal of Political Science verification policy, read on 2026-10-22 (source-check.md).
Stodden and colleagues asked the authors of 204 computational papers in Science, published after the journal's 2011 data and code policy, for their code and data. They got them for 44% of papers and estimated that the findings of 26% could be reproduced.
A survey of reviewers at 17 BMJ journals found that 95% said they had seen supplementary material at least sometimes, which says what they saw and not what they checked. I found no study that measures whether peer reviewers open or run the code they're given. The American Journal of Political Science makes publication depend on a successful rerun, and specialists at two data archives do that work, not the peer reviewers. Asking for code isn't the same as rerunning it, and where rerunning happens it's a paid job.
Badges raise sharing, and citation gains from sharing code are correlations
Claim: evidence
Sources: Kidwell and colleagues, PLOS Biology 2016, Maitner and colleagues, Ecology and Evolution 2024 and Heumüller and colleagues, Empirical Software Engineering 2020, read on 2026-10-22 (source-check.md).
After Psychological Science started giving badges for open practices, the share of its articles reporting open data rose from about 2.5% to 22.8%, while comparison journals stayed near 3%. That study didn't measure citations. Of 1,001 ecology and evolution papers that cite R, about 6% shared code, and those papers gathered citations faster. In 789 software engineering conference papers, linking to the software had a small positive link with citations. All three are observational: papers that share code may differ in other ways that bring citations, so none shows that sharing causes them, and none is from biomedical research. Badges, featured placement and credit in grant reviews are reasonable incentives to try; whether they raise citations in medicine is not known.
Existing standards already describe most of what a companion repository needs
Claim: evidence
Sources: CodeMeta, the Citation File Format, RO-Crate 1.3, the DataCite Metadata Schema 4.7 and Zenodo's versioning help, read on 2026-10-22 (source-check.md).
CodeMeta, now at version 4.0, describes software in a codemeta.json file. The Citation File Format, at 1.2.0, says how to cite it in a CITATION.cff file. RO-Crate 1.3, published in June 2026, packages data, code and the links between them. The DataCite schema 4.7, from March 2026, describes the DOI record and now accepts a Software Heritage identifier. Zenodo gives each archived GitHub release its own DOI and a concept DOI for all versions. None of them states which file makes which table and figure, which is the part a rerun needs most.
H1. Most shared biomedical code can't be rerun in a day
Claim: hypothesis
Practical test: Take a random sample of papers in diabetes technology and biomedical journals that link to code. A person who didn't write each paper follows the rerun protocol in the companion. Measure how many reach the paper's numbers within one working day, how long each takes, and why each failure happened. If more than half reach the reported numbers, H1 fails.
H2. Mechanical checks predict whether a rerun will succeed
Claim: hypothesis
Practical test: Before the reruns in H1, score each repository on the rules a program can check without judgment: archived with a DOI, software versions pinned, data DOI resolves, one entry point, outputs declared. Compare the scores with the rerun results. If the checks separate the reruns that work from those that don't no better than chance, H2 fails.
H3. A group's first companion repository costs the most
Claim: hypothesis
Practical test: Ask research groups to log the hours they spend making a companion repository for each of their next three papers, using the author checklist in the companion. If the third takes as long as the first, H3 fails, and the cost is a standing cost per paper that someone has to fund.
H4. Reviewers who rerun the analysis find more errors
Claim: hypothesis
Practical test: Give matched reviewers the same papers, half with a companion repository that reruns and half without, and plant known errors in some of the analyses. Measure how many planted errors each group finds and how long it takes. If both groups find the same share, H4 fails.
Each kind of reader gets something different from it
Claim: recommendation
Research groups make companion repositories, so the program starts with them. Everyone else reads them, and each gets a different thing.
I recommend that funders and diligence teams treat a project without a companion repository as unverified until the authors show the code that produced it, and that they ask for a rerun before any decision that depends on a number. That's a recommendation about how to read evidence, not a finding that projects without one are wrong.
What would show this claim is wrong
- Funders or diligence teams who ask for reruns make no better decisions, measured by later replication or later write-downs, than those who don't.
- Asking for a companion repository drives away good work, because the groups best placed to do it can't spare the time.
| Reader | What they get | Use case |
|---|---|---|
| Research groups and data managers | A record of every analysis decision, tied to the paper, that answers a reviewer's question with code that runs | A research group can answer reviewers by pointing to code that runs |
| Journal editors and reviewers | A rerun report with each submission, so review time goes to the findings | A journal can give reviewers a rerun report with each submission |
| Funders and program officers | Analyses they can compare across grants, and a rerun before a renewal | A funder can ask for a rerun before renewing a grant |
| Clinicians and guideline writers | The exact definitions and thresholds behind a clinical finding, and how they held up on new data | A clinician can see exactly how a study defined its outcome |
| Industry and diligence teams | A finding they can rerun before they license it, invest in it or build on it | A diligence team can rerun a finding before licensing it |
Who pays for the time is not known
Claim: unknown
Hosting public code on GitHub, GitLab or Zenodo costs a research group little or nothing. The real cost is time: the researcher's, to clean up, document and archive the code, and the reviewer's, to rerun it. I don't know how many hours either takes in this field, or whether journals, funders, institutions or the groups themselves would carry them. H3 measures the first part. The second needs a funder or a journal willing to pay for reruns on a small set of papers and report what it cost.
Whether existing services already answer the question is not known
Claim: unknown
Services exist that package or run research code. Code Ocean bundles code with what it needs to run, and three Nature journals trialled it for code review in 2018. Binder builds a runnable environment from a repository. Whole Tale packages data, code and environment together. Software Heritage archives source code under an identifier that is now the standard ISO/IEC 18670. I haven't tested any of them against the six parts or the rerun protocol. If one of them already holds all six and reruns to the reported numbers on most biomedical papers, the program should narrow to what that service leaves out, or stop.
The first measurement is a rerun study of published papers
Claim: recommendation
I recommend starting with H1 and H2 together: a random sample of published papers in diabetes technology and nearby biomedical fields that link to code, each scored on the mechanical checks and then rerun by a person who didn't write it, using the protocol and result record in the companion. Success and failure are defined before the first rerun, and failed reruns are reported in the same detail as successful ones. Nothing is built for sale until this study shows what goes wrong and how often.
What would show this claim is wrong
- Too few papers in the field link to code for a sample to say anything.
- The code depends on data that outside reruns can't reach, so the study measures data access and nothing else.
Questions people ask about companion code repositories
- What is a companion code repository?
- It's the code that produced a paper's results, packaged with the exact software versions, a link to the exact data and one set of instructions, so someone else can run it and get the same numbers. It's archived with its own DOI and linked to the paper and the data in both directions.
- Does a companion repository replace the paper or the data repository?
- No. The paper says what was done and why, the data repository holds what was measured, and the companion repository shows how the numbers were made. Each keeps its job.
- Can I update the code after the paper is published?
- Yes. Publish a new release with its own DOI. The release the paper cites stays exactly as it was, and a concept DOI points to every version, so readers can see how the result held up on new data.
- Where should patient data go?
- In a governed repository such as Vivli or the YODA Project, even when it's de-identified. The companion repository holds the code, a link to the governed record and, if you like, a synthetic copy with the same structure so the code runs end to end.
- What if my study has no patient data?
- Keep the data and the code together in one companion repository and archive it with a DOI. Simulations, bench tests and analyses of published aggregates need no extra governance.
- How do I protect intellectual property and still let a reviewer check my work?
- Use a private repository and invite the editor or the reviewers you choose. Available to a reviewer is not the same as visible to everyone.
- What if my analysis was done in a spreadsheet?
- That counts. Include the spreadsheet with its formulas left in, the data files in open formats such as CSV, and a short README that says the order of the steps.
- Does the NIH require me to share my analysis code?
- No. The NIH Data Management and Sharing Policy, in effect since 25 January 2023, requires a plan and shared scientific data, and asks only that the plan name the tools needed to use that data. A companion repository goes further than the policy asks.
- How do I cite a companion repository?
- Archive a release on Zenodo or a similar archive to get a DOI, put that DOI in the paper, and add a CITATION.cff file to the repository so others cite it the same way.
- Does sharing code get a paper more citations?
- Papers that share code have been cited more in ecology and in software engineering, but those studies show a link, not a cause, and none is from biomedical research. In medicine it's not known.
- What does it cost?
- Hosting public code on GitHub, GitLab or Zenodo is free or close to it. The real cost is the researcher's time to clean up and document the code, which this program sets out to measure, and the reviewer's time to rerun it.
- What if my institution requires a specific platform?
- Institutional policy comes first. The six parts matter more than the platform that holds them.
- How is AI used to check reproducibility?
- Rules check the repository first and answer yes, no or cannot tell. A model reads only what rules can't, such as an unclear README, and says where it looked. A person judges what's flagged. A model never overrules a rule.
- What about clinical trial data?
- It stays under the rules that already govern it, such as FDA regulations and Good Clinical Practice. A companion repository for a trial can still share the analysis code, run on synthetic or aggregate data that protects participants.
- Can a journal or a funder require a companion repository?
- Yes, and some already ask where code can be found: CONSORT 2025 asks trial reports to say where the statistical code is. Any requirement should respect institutional policy and data agreements, and allow private repositories for code that can't be public.
- Couldn't data repositories just add code hosting?
- They could, and that would help. Repositories such as Vivli and YODA were built to govern patient data, not to version and rerun code. Until they do both, the practical answer is to keep the data frozen and citable where it is and keep the code on a platform built for code, linked by DOIs.
- Doesn't this add work for researchers?
- Some, mostly the first time. The checklists and a template cut it down, and a companion repository can save time later by answering reviewers' methods questions with code that runs. How much time it takes is something this program sets out to measure.
- How does it work when several groups wrote the code together?
- The same way software teams work: everyone contributes to one repository, the history records who changed what and when, and the citation file lists every contributor.
Research notes
- 22 October 2026
A clinician can see exactly how a study defined its outcome
A clinician deciding whether a study applies to their patients needs the exact definitions behind its conclusion: what counted as treatment response, which patients were left out, which thresholds were used. A companion repository shows them in the code that ran, with the data dictionary, and later releases show whether the finding held on new patients.
- 22 October 2026
A companion repository takes seven steps to set up
Setting up a companion code repository takes seven steps, and most research groups can do the first five in an afternoon once the analysis is finished. The steps go from storing the data in the right place to making the code rerun on new data, and the last one, automated checking, is optional.
- 22 October 2026
A diligence team can rerun a finding before licensing it
Before a company licenses a biomarker, invests in a device or builds on a published result, its diligence team can rerun the analysis behind the finding. A companion repository, public or shared privately under an agreement, turns a judgment about the authors into a check of the method.
- 22 October 2026
A funder can ask for a rerun before renewing a grant
A funder or program officer deciding whether to renew a grant usually reads the progress report and the papers. With companion repositories, the funder can also see the actual analysis pipelines, compare methods across the grants in a portfolio, and ask for a rerun of the result the renewal depends on.
- 22 October 2026
A journal can give reviewers a rerun report with each submission
A journal editor can ask authors to submit a companion repository with the paper, run automated checks and a rerun before review, and hand reviewers a short report: what ran, what matched the paper's numbers, and what was flagged. Reviewers spend their time on the science, and the journal can mark papers whose results rerun.
- 22 October 2026
A private repository lets a reviewer rerun code that can't be public
Code that can't be public, because of a patent plan, a contract or commercial value, can still be checked. Keep the companion repository private and invite the editor, the reviewers or the funder you choose. Software companies protect their code this way every day.
- 22 October 2026
A result anyone can rerun counts for more than a famous name
When evidence is thin or can't be checked, medicine leans on who said it: the senior author, the famous journal, the expert panel. When anyone can rerun the analysis and test it on new data, the question shifts from who said it to what the evidence keeps showing.
- 22 October 2026
A research group can answer reviewers by pointing to code that runs
A research team can publish a companion repository with its paper. When a reviewer asks how a number was produced, the answer is a release that reruns it. The team gets a citable software record, a precise history of every analysis decision, and a way to update the method as new data arrives.
- 22 October 2026
Check reproducibility with rules first, a model second and a person last
Most peer reviewers can't spend days rebuilding an analysis. Automation can take that load, but only if it's built in the right order: rules that a program checks without judgment first, a model only for what rules can't answer, and a person last. A higher layer never overrules a lower one.
- 22 October 2026
Patient data stays governed and the code goes beside the paper
Split a study's materials by risk: data about individual patients goes to a repository built to control access to it, and everything else, the code, the metadata and any data with no patients in it, goes in the companion repository beside the paper. The split puts the governance burden only where the risk is.
- 22 October 2026
A spreadsheet analysis can be a companion repository
Many published analyses were done in a spreadsheet, not in code. That's fine: a spreadsheet with its formulas intact, its data in open formats and a short README that says the order of the steps is a companion repository too.
- 22 October 2026
Versioned releases keep the evidence current while the paper stays fixed
A published paper is a snapshot. A companion repository can keep going: each time the same method runs on new real-world data, the authors publish a new release with its own DOI, and the release the paper cites never changes. Peer review keeps its job, and readers can see whether a finding still holds.
Related
- Research Code Sharing in Support of Gold Standard Science
- Evidence Credibility Engineering
- Eidolons: working digital reflections
- A companion repository takes seven steps to set up
- Patient data stays governed and the code goes beside the paper
- A private repository lets a reviewer rerun code that can't be public
- A spreadsheet analysis can be a companion repository
- Check reproducibility with rules first, a model second and a person last
- Versioned releases keep the evidence current while the paper stays fixed
- A result anyone can rerun counts for more than a famous name
- A funder can ask for a rerun before renewing a grant
- A journal can give reviewers a rerun report with each submission
- A research group can answer reviewers by pointing to code that runs
- A clinician can see exactly how a study defined its outcome
- A diligence team can rerun a finding before licensing it