On the one hand, research libraries are a stable entity when it comes to reliable information management for their institution and broader research communities. On the other hand, they are undergoing a constant transformation based on their changing environments as well as related roles and responsibilities for example in the area of open research. Based on continually emerging opportunities and challenges, libraries adopt and drive new or enhanced information practices. In recent years, the extent to which research findings can be trusted has been challenged based on the inability to reproduce or replicate research findings (the so-called “reproducibility crisis”),¹ which is often due to limited or no access to research data, lack of documentation in methodologies and/or limited or no availability of the software and code used to run an analysis. These issues, which are often considered as part of research integrity (Diaba-Offeh, 2021), apply to published research, and can severely affect a researcher’s career, ranging from minor errors to severe ones leading to retractions or even investigations of research misconduct.

We acknowledge that there is no universally agreed definition of “research reproducibility”, that is the related terms “replication”, “reproduction” and “repetition” are sometimes used interchangeably or as distinct concepts, depending on context and research discipline (Stanford Encyclopedia of Philosophy, 2018; National Academies of Sciences, Engineering, and Medicine [NASEM], 2018, 2019). For the purpose of this paper, we will refer to reproducibility as the ability to reproduce the findings of a study based on access to the original data and methods/procedures. This usually requires a detailed description of the methods used to process and analyse the data, access to any relevant datasets (and related documentation) and an ability to obtain and run computer code, where appropriate. It must be noted that definitions of reproducibility often focus on “computational reproducibility”. Findings based on qualitative methodologies (e.g. literature reviews, interviews, manual thematic analysis) will often not be fully reproducible based on subjective elements in assessment, classification, selection, contextualisation or interpretation processes.

Between 2020 and 2021, Knowledge Exchange (KE) investigated the publication of reproducible research outputs, with support from a group of international open science experts. The KE partners are six key national organisations within Europe tasked with developing infrastructure and services to enable the use of digital technologies to improve higher education and research: IT Center for Science (CSC) in Finland, Centre national de la recherche scientifique (CNRS) in France, the Danish e-Infrastructure Consortium (DeiC) in Denmark, the German Research Foundation (DFG) in Germany, Jisc in the UK and SURF in The Netherlands. The study was initiated by KE’s Open Access Expert Group, which focuses on the ongoing transformation of the scholarly publishing landscape.² One core finding was that ensuring the reproducibility of research is a multi-stakeholder effort, with challenges and opportunities for individual researchers and research communities, librarians, publishers, funders and service providers. These emerge at various steps of the research cycle, and, in particular, at the publication stage.

The output of this work was a public report (Chiarelli et al., 2021a, 2021b), which was based on an extensive literature review of almost 130 sources and a mix of interviews and focus groups with 51 stakeholders from 12 countries. In this landscape study, stakeholders were considered at the micro, meso and macro level based on the KE Open Scholarship Framework (Neylon et al., 2019, cf. Figure 1).

Across its levels, the Open Scholarship Framework considers researchers and research groups, their disciplinary communities and institutions as the content creators (micro level); academic libraries and their institutions as well as publishers and infrastructure providers in supporting and strategic roles (meso level); and research funders and policymakers as those who influence the broader ecosystem and establish recommendations and requirements (macro level).

In this context, the exact role that librarians should or could play remains unclear. For example, why should librarians care about the reproducibility of (published) research findings? Through what mechanisms could they enable and foster best practices in research and publishing, leveraging the services and activities that they manage in collaboration with other stakeholders? In this paper, we have sought to translate the results of the KE study into practical tools for research libraries.

Our paper begins with a background section that provides context and foundational information on research reproducibility, including with regard to the role of libraries. This is followed by an analysis of stakeholder roles and key touchpoints for librarians, enriched with relevant quotes. The paper ends with a summary of our findings and a series of recommendations to enhance the role of research libraries in the reproducibility landscape.

Libraries’ engagement with reproducibility as an activity and set of skills has multiple perspectives: 1) From the perspective of collections management, libraries select, manage, curate and ensure the discoverability of the published record of research outputs. In the academic and societal environment in which libraries operate, challenges such as the “reproducibility or replication crisis” can affect the trust in these collections. 2) In the context of research support, libraries directly engage with researchers by providing support and infrastructures for research data management as well as training in data (science) literacy and its relationship to open science (e.g. FAIR and open data) (Lyon, 2016; Quan, 2021). Building up services for the management and curation of research software is a relatively new yet emerging responsibility for libraries (Chassanoff et al., 2018; Martinez-Ortiz et al., 2023). 3) Institutional publishing services develop publication policies, workflows and infrastructures and provide support for transparency and openness (e.g. Nosek et al., 2023). This includes guidance on data publishing and citation, e.g. via own or external repositories, availability statements and licensing. 4) Across all these service areas, libraries contribute to awareness raising, training and implementation support for responsible research practices at various stages of the research and scholarly communication cycle as a safeguard for reliable research.

Over the last few years, the research reproducibility landscape has been rapidly evolving, as discussions around open access, FAIR and open data, research integrity, open source and open infrastructures continue to converge. More specifically, it has been pointed out that open science provides a means to improve the quality, transparency and reproducibility of research (Munafò et al., 2017) leading to more accountability of research. This also holds for the step of publishing research outcomes: “executable research papers” support reproducibility through a combination of text, raw data and code that (re)creates the analysis and figures, and offers readers means to interact with or manipulate the code (Tsang & Maciocci, 2020). For researchers, taking up this approach for writing papers can turn out as an insightful learning experience and journey through open science (Lasser, 2020). The approach is however far from becoming the established norm as this would require a cultural shift of the scientific community towards making data and code openly available and a recognition of the skills and efforts needed to implement all necessary steps.

To support and foster reproducibility across the research cycle, many stakeholders including researchers, scholarly societies, funders and publishers have created guidelines and recommendations (e.g. KNAW, 2018; Munafò et al., 2017; National Institutes of Health, 2014; Nosek et al., 2023; Wellcome Trust, 2017).³ Moreover, collaborative initiatives led by researchers have formulated documentation standards in support of reproduction (e.g. Project TIER Protocol),⁴ hands-on guides (e.g. Turing Way Handbook⁵ (The Turing Way Community, 2022)), reviews of existing infrastructures (e.g. Konkol et al., 2020), assessed the reproducibility of research results in funding programmes (e.g. European Commission et al., 2022a), and are co-creating activities and policy recommendations (e.g. Ross-Hellauer et al., 2022). These documents provide guidance for researchers but also point out infrastructures and tools as well as support services that are provided by a range of stakeholders. From a conceptual perspective Lyon (2016) proposed a 3-dimensional model or spectrum for open science, which includes the 3rd axis of transparency. In this model librarians act as ‘transparency agents’ that advocate, promote and demonstrate particular behaviours and good practices throughout the research lifecycle, which over time will lead to cultural change towards a more open science environment. Sayre and Riegelman (2019) compared a selected set of reproducibility guidelines and have mapped libraries’ expertise and services to key themes of these guidelines. LaPolla et al. (2022) investigated motivations, approaches and lessons learned from libraries’ engagement – typically in collaboration with other departments – in teaching rigor and reproducibility in the health sciences.

When it comes to reproducibility, research libraries, publishers and other stakeholders need to navigate a web of interconnected roles and responsibilities, which includes not only the advice and support they traditionally provide, but also an ever-broadening range of tools and workflows that span from choosing a repository for the deposit of data and code to preparing research compendia⁶ or executable research articles (Packer, 2020). Positively, research libraries have taken up innovative new roles, such as the “data librarians/stewards” that are becoming more common across Europe – in particular in the context of implementing the FAIR data principles (Wilkinson et al., 2016) – and the “reproducibility librarians” that are emerging in the USA and operate at the interface between the various facets of open science (Steeves, 2017).

The role of libraries and research support services with regard to reproducibility is also changing in terms of digital infrastructures. Researchers require more and more support in navigating the fast-paced world of technological solutions emerging to support open science practices, thus leading to uncertainty and a constant need to upskill both the research base and support staff. Whether internally or in partnership with external providers, research libraries also continue to develop institutional and thematic repositories, which can be interconnected with a web of digital objects to enable reproducible research to take place. The European Open Science Cloud (EOSC) is an example of ongoing efforts to build a trusted distributed digital research infrastructure that supports verifiable and reproducible research (European Commission, 2022b).

Finally, the broader context of the COVID-19 pandemic has highlighted the role of research reproducibility in enabling trust in the published record. For example, it was found that rapidly emerging preprints tackling important public health questions didn’t always meet high transparency criteria and could improve their data sharing and scientific reporting practices (Sumner et al., 2020). Research libraries can play a dual role in this regard, as they can be both the custodians of research objects and advisers to local academics who seek to work in a reproducible way and enhance the quality of their institution’s publications.

This work presents a secondary analysis of the findings that were gathered during the 2020–2021 KE exercise on the Art of Publishing Reproducible Research Outputs.⁷ This study included a literature review, stakeholder consultation and was presented at the 2022 LIBER conference.

To provide an updated view of the reproducibility landscape, we have built on the literature review conducted as part of the KE exercise as a starting point. This literature review included almost 130 literature sources from across academic disciplines and multidisciplinary sources and provided an overview of definitions, problems, recommendations and solutions or initiatives in the context of reproducibility (Chiarelli et al., 2021a; Loffreda & Chiarelli, 2021). In the present study, we have revisited the literature findings to focus particularly on the key roles and responsibilities for research libraries. We have then conducted a brief review of literature to reflect any recent developments in the field of reproducibility that may affect or relate to these roles, such as the establishment of new initiatives and networks.

In addition to the updated review of literature, a secondary analysis of the interview transcripts gathered as part of the original exercise (Chiarelli et al., 2021b) has been conducted. Transcripts that have been reviewed include interviews with 41 international stakeholders, including research librarians, researchers, publishers and funders, and focus groups with a further 10 infrastructure providers. Text searches were conducted on these sources, including searches for “library” and “steward”, to identify examples of those in research support roles supporting the publication of reproducible research. Quotes from interviews are used throughout this paper to illustrate our key findings.

Finally, based on the above-mentioned methodology, we have considered how the findings of the updated landscape analysis align to LIBER’s 2023–2027 strategy (Frey et al., 2022). This approach has informed the development of the three main recommendations presented in the final section of this paper.

We note that our updated literature and landscape analysis has been conducted during January and February 2023, thus representing a snapshot of the reproducibility landscape at a given point in time. It should also be noted that the original analysis has been underpinned by thematic coding. By nature, this relies on the subjective interpretation of the researcher.

Due to the multifaceted nature of research reproducibility, roles and responsibilities are inevitably complex and must be shared between different stakeholder groups at the aforementioned micro (researchers and disciplines), meso (institutions, libraries, publishers, software providers) and macro levels (research funders, policymakers). In particular, we note that cooperation is essential throughout the research lifecycle, as the publication of a reproducible research output requires a range of considerations from the inception of a research project all the way through its delivery and dissemination. One academic interviewee highlighted this point:

One of the key challenges in research reproducibility is the need to share a broad range of research objects (as applicable) alongside one’s outputs, including detailed methods, pre-registrations, research data, any relevant code/software and information about the software environment used. Notably, in listing these research objects we do not mean that reproducibility is inherently quantitative or computational in nature. It is, however, essential to acknowledge that the terminology we employed above may be misunderstood or misinterpreted in some disciplines, which highlights the importance of stakeholders speaking a shared language. For example, reproducibility guidance may be inappropriate at the funder or publisher level and much more helpful at the funding council or journal level, so as to add an extent of tailoring that will allow the target audience to understand any requirements. This sentiment was also expressed by another academic interviewed as part of the landscape study:

When it comes to the sharing of the above-mentioned research objects, it is clear that researchers play a key role: they are responsible for designing, delivering and disseminating the research and hold the information on relevant research objects (including metadata). At the same time, researchers can act as peer reviewers in cases where journals have reproducibility requirements, meaning that they may be responsible for checking that someone else’s work is reproducible. In the case of reproducibility, it is also helpful to think of researchers as members of disciplinary communities: in most cases, reproducibility is unlikely to become a widespread concern unless the (sub)discipline establishes that this should be the case. For example, some disciplines show widespread data and code sharing (with implications on reproducibility), particularly where physical or digital infrastructure is shared (e.g. particle physics, astronomy), while others are currently not engaging with the concept. It is, however, recognised that reproducibility “is relevant across scientific disciplines and cultures, be it in the humanities, engineering, life science, natural sciences, or social sciences”, although projects in different disciplines may take “demonstrably different approaches, varying from enhancing step-by-step reproducibility through code-based transparency to tracing the origin of an argument through publication lines” (Rahal et. al., 2022). Furthermore, the role of individual reproducibility champions (including with the support of national reproducibility networks) in raising the profile of practices underpinning reproducible publication in disciplines where these remain uncommon is broadly recognised (Chiarelli et al., 2021b; Ross-Hellauer et al., 2022). This is where the potential for institutions to make a difference is highest, as fostering such efforts and providing related support is conducive to the broader uptake of new practices. As one academic interviewee highlighted:

Arguably, libraries are uniquely placed to provide support around research reproducibility, as they are the institutional reference point for both authors and publishers and are typically well connected with other professional services such as IT, software engineers, ethics, data governance and data protection. Clearly, the remit of librarians varies based on national and institutional contexts, meaning that the provision around research reproducibility can take a range of different forms. For example, New York University and Oregon Health & Science University provide dedicated research data management and reproducibility librarians,⁸ while at the Delft University of Technology reproducibility is within the remit of research data stewards (Martinez-Lavanchy et al., 2022). In other cases, reproducibility may be supported by a mix of roles, including research object curators or subject librarians, as was suggested by a research support librarian:

A wide range of initiatives is being introduced as part of grassroots efforts to support research reproducibility, and these are often advertised and signposted within institutions by libraries in collaboration with researchers, IT and research services. These include community-initiatives such as ReproHacks,⁹ ReproducibiliTea,¹⁰ The Carpentries¹¹ and the Turing Way Handbook⁵ (The Turing Way Community, 2022) and the national Reproducibility Networks (e.g. UK, Germany, Sweden, Portugal)¹² as well as local awareness raising and capacity building efforts (e.g. Rethlefsen et al., 2018). These initiatives are complemented by platforms, software and technical tools to support the management, sharing and execution of relevant research objects, and a list of examples is available in Chiarelli et al. (2021b). At the high level, we note the importance of tools for reproducibility checking, software containerisation, code sharing and documentation, reproducible environments, research compendia and more.¹³ In most cases, individual researchers will not be familiar with these tools: given the great and continually increasing pressures on researchers’ time, it is essential for institutions (and, in most cases, libraries) to provide an extent of training and/or support around these systems if reproducibility is considered as a strategic priority. One institutional librarian noted:

In practice, research libraries are only beginning to engage with reproducibility, including because this is not currently a priority in all disciplines and is lagging behind other portions of the open research agenda (e.g. open access, open data), partly due to the mix of complex practical requirements and expectations. This is where the role of research funders and government players becomes apparent. To date, minimal pressure has been perceived around reproducibility at the policy level, and institutions have consequently not seen this to be an area for strategic investment in most cases. This is not due to a lack of interest in reproducibility, but rather in acknowledgement that library (and, more broadly, institutional) budgets continue to be under growing pressure and areas of expenditure have to be prioritised. An interview with an institutional librarian touched on this topic, and they noted:

As noted above, publishers in some disciplines are beginning to introduce reproducibility requirements, however, meaning that strategic pressures that might often arise at the macro level are actually emerging at the meso level and as part of micro-level grassroots movements led by researchers and disciplines

This paper set out to identify the rationale for libraries to engage with the reproducibility of (published) research findings and to outline a range of mechanisms and practices that would enable this. The discussion so far indicates that libraries have a natural role to play in raising awareness about responsible research practices – including efforts that aim at increasing transparency and enhancing reproducibility of research – and for providing support for their day-to-day implementation. For this, libraries can build on their good connections to academic communities and are well aware of disciplinary needs. To cover all steps of the research and scholarly communication cycle, collaboration with other stakeholders such as research funders and publishers is essential.

In sum, we would like to propose three recommendations for the the role of research libraries in the reproducibility landscape: librarians should team up with researchers (and other stakeholders) for the promotion of an (institutional) research culture that supports and rewards the principles of research transparency and reproducibility; build on and expand existing research infrastructure and support; and engage and invest in raising awareness and the creation of skills and capacities.