13 December 2024
Andrea Chiarelli, Principal Consultant at Research Consulting
Reliable research is the most important ingredient for evidence-based decisions in policy, practice and our personal lives. Decisions based on unreliable research findings that are misguided or incorrect can waste resources, harm people or miss important opportunities. In a socio-economic context characterised by economic uncertainty and greater pressures than ever – including instability, multiple crises and rapid technological change – this is not something we can afford.
Traditionally, the research community has relied on peer review as the primary mechanism through which other researchers verify the claims in a manuscript that is being considered for publication. However, the seemingly never-ending wave of retractions in recent years suggests that the trust markers we previously relied on are no longer sufficient.
The concept of reproducibility is a concrete approach to assessing the reliability of research. This principle asserts that research should be conducted, documented and reported in a way that allows other researchers to obtain the original findings using the same methods and source information published alongside a manuscript.
However, most published works have not been checked for reproducibility. There are three main barriers to this, and there is a lot that publishers can do to help.
Barrier 1: Limited access to data and methods
One of the biggest obstacles to reproducing the findings from research articles is limited access to the original data, detailed methodologies and materials used in a study. Without these elements, other researchers simply cannot reproduce the original analysis described in an article. Many researchers don’t make their full datasets publicly available; this has to do with time availability and the fact that data preparation and documentation are specialised skillsets that not all authors possess.
That said, efforts to pursue transparent sharing of research objects other than articles have been growing for years, with several research funders now mandating these for grantees. Platforms like Zenodo, Dryad, the Open Science Framework, GitHub and Figshare make it easier than ever for researchers to archive and share these materials, which has significantly reduced the amount of effort needed to align with open sharing behaviours. Furthermore, initiatives like the Transparency and Openness Promotion (TOP) Guidelines are seeking to drive publisher support of the principles of openness, transparency, and reproducibility.
What can publishers do? Publishers can require authors to share their data, code and detailed methods as a condition for publication. They can partner with data repositories to streamline the process and ensure that all necessary materials are made publicly available. Enforcing clear open sharing policies and providing comprehensive guidelines for reporting methodologies can significantly improve access to the information needed to enable reproducibility.
Barrier 2: Insufficient incentives
The current recognition and reward mechanisms unfortunately do not encourage researchers to take their time to carefully document and share the intermediate steps that have led to their discoveries. On the contrary, an excessive focus on quantitative assessment is pushing researchers to publish as many articles as possible at the fastest possible rate – often at the expense of research integrity. This has already led to various issues, including journals shutting down, prominent figures resigning and public accusations becoming uncomfortably common.
Additionally, novel findings are typically valued more highly in publications and promotions, while studies focusing on reproducibility are seen as less impactful or valuable. Solving these challenges does require a greater degree of culture change (for more on this, you may explore the work of DORA and CoARA).
What can publishers do? Publishers can create incentives for reproducible research by recognising and rewarding authors who engage in these practices. For example, this can include featuring reproducible or reproducibility-focused articles or establishing awards for exemplary reproducible research. Publishers could also be more receptive to publishing reproducibility-focused contributions (not necessarily as peer-reviewed articles), showcasing their interest in methodological rigour as well as novel findings.
Barrier 3: Technical and methodological challenges
Reproducing someone else’s work can be technically difficult, especially when dealing with large datasets, specialised data formats or intricate statistical or computational analyses. In practice, many researchers and peer reviewers will lack the specific skills or tools needed to properly analyse and interpret complex data from other studies – even if these are in their area of expertise.
Variations in research approaches, software versions or even small imprecisions in how methodologies are reported can make reproduction even more challenging.
What can publishers do? Publishers can support researchers in overcoming technical and methodological challenges by providing detailed guidelines for authors and reviewers that clearly outline their expectations around methodological detail, data and code sharing and reproducibility. They can also train peer reviewers to assess manuscripts for reproducibility and experiment with reproducing key figures and charts in a manuscript during the review process. However, there is still no consensus on who should be responsible for reproducing submitted research. Options being discussed include peer reviewers, dedicated journal staff or even third-party reproducibility checks.
Towards reproducible research
Training, supporting and incentivising researchers with regard to reproducibility must be a shared endeavour across research performing organisations, funders, publishers and disciplinary communities. The above only seeks to highlight areas where publisher efforts are likely to be effective.
In practice, each stakeholder is embracing the various facets of open sharing and reproducibility at a different pace, and publishers cannot simply introduce new requirements without testing their acceptability to and appropriateness for their audience and authors. They can, however, gradually socialise some of the concepts covered above that apply to their target audience across different disciplines. Doing so will certainly minimise the risk of research integrity issues that can take years to be forgotten.
Examples of strong transparency requirements that are highly conducive to reproducibility are not uncommon today and are starting to apply across different fields of research:
- PLOS journals require authors to make all data underlying the findings described in their manuscript fully available without restriction.
- The American Economic Association (AEA) requires information about the data, code and other details of the computations, as well as information about access to data and code.
- Nature Portfolio journals require authors to make materials, data, code and associated protocols promptly available to readers, including in partnership with the Code Ocean platform.
The tide is turning. These requirements, which used to be perceived as too high a bar to set for authors, are being normalised in the interest of reliability, integrity and trust. After all, journals that prioritise transparency and reproducibility are not only protecting their long-term reputation but also attracting authors who value these principles.
The Slow Science Academy reminds us that “Science needs time to think. Science needs time to read and time to fail.” This concept of “slow science” aligns perfectly with the goals of reproducible research. It emphasises the need for careful, methodical work over rushed publications, and values quality over quantity. The path to reproducible research may be challenging, but it is an essential enabler of integrity and trustworthiness. By working together to create a system that values thorough reporting, transparent sharing and reproducibility, we can ensure that research continues to be a reliable foundation for evidence-based decision-making in all aspects of society.