AI Tech Inspire

ICLR’s Reproducibility Statement: What to Include and Where to Put It

  • Confusion around appendix and supplementary material submissions has led to desk rejections at other venues.
  • ICLR encourages a paragraph-long Reproducibility Statement at the end of the main text, before references.
  • The statement should not include reproduction details; it should point to where those details live (main paper, appendix, or supplemental materials).
  • Examples given: anonymous downloadable source code in supplementary materials; theoretical assumptions and full proofs in the appendix; dataset processing steps in supplementary materials.
  • The Reproducibility Statement is optional and does not count toward the page limit.

Reproducibility is no longer a nice-to-have in AI research—it’s a quality bar. And in submission season, a missing line, a misplaced appendix, or an unclear statement can be the difference between a smooth review and a desk reject. At AI Tech Inspire, we spotted ICLR’s guidance on the Reproducibility Statement, and it’s both simple and surprisingly easy to get wrong without a checklist.

What ICLR actually asks for

ICLR encourages authors to include a short, paragraph-long Reproducibility Statement placed at the end of the main text, just before the references. This statement is optional and explicitly does not count toward the page limit. The intent is clear: use that paragraph to direct reviewers and readers to the exact locations—main text, appendix, or supplementary materials—where the nuts and bolts of reproduction live.

Key takeaway: The Reproducibility Statement is a navigational aid, not a dump of technical specifics.

ICLR’s examples are pragmatic:

  • For novel models or algorithms: link to an anonymous, downloadable source code package in the supplementary materials.
  • For theory: keep assumptions and complete proofs in the appendix.
  • For datasets: put a full description of preprocessing and pipelines in the supplementary materials.

That division of labor keeps the main text crisp while ensuring the full artifact trail is available for anyone who wants to rerun experiments or verify claims. It also answers a common anxiety: yes, you can keep detailed scripts, preprocessing, and proofs outside the main narrative—just make them easy to find.

Where should details live? A practical split

Based on ICLR’s guidance, here’s a clean way to organize artifacts and references in your paper package:

  • Main text: Briefly describe your method, experimental setup, and high-level results. Reserve space for the one-paragraph Reproducibility Statement before references.
  • Appendix: Extended proofs, algorithmic derivations, ablations, extra qualitative results, and any detailed math that won’t fit in the main narrative.
  • Supplementary materials (separate files): Anonymized code, scripts to run experiments, requirements.txt or environment.yml, Dockerfile, dataset processing scripts, and large figures or tables.

The exact mechanics (file formats, what counts as supplementary vs. appendix) can vary by venue and year. If you’ve ever seen colleagues tripped up at submission time, that’s why. When in doubt, double-check the conference’s official author kit and submission portal—interpretation differences have caused desk rejections at other venues.

Does the statement count toward the page limit?

Per ICLR’s note, the Reproducibility Statement is not part of the main text length and does not count toward the page limit. It should appear right before the references. That’s a sensible place—reviewers naturally land there, making it an ideal pointer to your appendix and supplementary materials. Still, it’s wise to confirm details in the official style files and latest author instructions, especially around final camera-ready formatting.

How to write a high-signal Reproducibility Statement

Think of this as a one-paragraph router. It should answer: where do readers find the code, data processing, hyperparameters, proofs, and environment specs—without repeating them in the statement itself.

Try a structure like this:

Reproducibility Statement. Code for all experiments is provided as an anonymous archive in the supplementary materials. Section 3 details the model and training setup; hyperparameters and full ablations are listed in Appendix A. Dataset preprocessing steps and scripts are included in the supplementary package (see /scripts/data). For theoretical results, assumptions and complete proofs are provided in Appendix B. Environment details, including pinned package versions and CUDA/cuDNN requirements, are documented in the README.

That’s it. One paragraph, zero speculation, and every claim has a pointer.

Packaging code: anonymity and determinism

For code submissions, two pitfalls dominate: identity leaks and non-deterministic runs. A few concrete steps:

  • Make it anonymous: Remove names from git history if you include it, scrub author tags in notebooks, and avoid URLs that could reveal identity. When linking, consider a temporary, anonymous hosting option rather than a named repository.
  • Pin versions: Specify exact versions for frameworks such as TensorFlow or PyTorch, and explicitly note GPU stack requirements (e.g., CUDA and cuDNN versions).
  • Environment capture: Include requirements.txt or environment.yml, plus a Dockerfile if possible. A one-liner like docker build -t paper-repro . followed by docker run ... makes the path to reproduction clear.
  • Seeds and settings: Set random seeds across libraries and note any sources of nondeterminism (e.g., GPU kernels). Add a --deterministic flag if relevant.
  • External models and datasets: If you pull from hubs like Hugging Face, pin model snapshots and dataset versions. For large or popular models like GPT or Stable Diffusion, record the exact API/model version and any inference parameters.

A README.md with a numbered set of commands can be the difference between a reviewer running your code in minutes versus giving up.

Data processing: make it boring (in a good way)

Reproducibility often dies in data prep. If the paper uses custom preprocessing, put scripts in the supplementary materials and reference them in the statement. Include:

  • Exact download links or instructions to acquire raw data.
  • Clear, ordered scripts, e.g., 01_download.py, 02_clean.py, 03_split.py.
  • Hash checks for files where practical.
  • Documented assumptions (language, casing, tokenization, filtering thresholds).

If licensing or access requires approval, say so up front. Reproducibility also includes reproducible access.

For theory-heavy papers

ICLR’s guidance explicitly recommends placing assumptions and complete proofs in the appendix. That’s good news for readability and rigor. In the statement, just point to Appendix X. Consider including a short proof roadmap in the appendix intro—reviewers appreciate the signposting.

A quick keyboard-friendly checklist

  • 1 Place a one-paragraph Reproducibility Statement before references.
  • 2 Point to code (supplement), proofs (appendix), and data processing (supplement).
  • 3 Pin framework and GPU stack versions; document exact run commands.
  • 4 Anonymize archives and links; avoid identifiable URLs.
  • 5 Capture seeds and any nondeterministic components.
  • 6 Verify compliance with the latest ICLR author kit and submission portal.

Why this matters for developers and reviewers

For engineers who adopt published methods, a clean reproducibility trail translates directly into hours saved. It accelerates baselining, improves comparisons, and reduces the risk of hidden confounders. For reviewers, it clarifies scope: they can inspect math in the appendix, run code with minimal friction, and evaluate data processing without fishing through the main text.

And for authors, there’s a reputational upside: people build on what they can run. A tight Reproducibility Statement plus well-organized artifacts makes your work a dependable reference implementation, not just a citation.

Bottom line

ICLR’s message is straightforward: include a concise Reproducibility Statement that tells readers exactly where to find the materials needed to verify your claims. Put the statement before the references, keep it short, and keep the details where they belong—appendix and supplementary materials. Given ongoing confusion across venues about what goes in the PDF versus a supplemental zip, verifying the current author instructions is still essential. But if the statement reads like a reliable map, you’re already ahead.

At AI Tech Inspire, the practical angle is clear: the best research doesn’t just perform—it repeats. Treat the Reproducibility Statement as the hyperlink that makes your paper executable.

Recommended Resources

As an Amazon Associate, I earn from qualifying purchases.