The project aimed to:

Accordingly, we pose the following research questions:

The project team at the Royal Library, Denmark, was brought together after a visit to Finland in December 2019. The library at the University of Helsinki had recent experience in implementing an AI-search tool in academic search practices and developing services around the tool. As the leadership at the Royal Danish Library wished to develop support in systematic search and review, we wanted to learn more from the University of Helsinki about the strengths and weaknesses of the AI-search tool, their services, users’ needs, use-cases and licence negotiations. Two members of the project team were part of this “innovation-visit” but neither had specific competencies or previous experience in AI before the visit to Helsinki. Two members joined the team after the trip, chosen for their experience in systematic reviews and systematic searching. All four had a background in Library and Information Science, but with no specific skills in the technical aspects of AI. The project group represented different departments at the Royal Library and different disciplinary expertise: Business and Law, Health, Social Science and Humanities and Researcher Support.

The project team reported to a “reference-group” to whom midway reports, milestones and deliverables were presented and discussed. The reference group consisted of individuals with relevant search methodological expertise and technical skills in data analysis, including technical knowledge of search algorithms and information retrieval.

Over the course of the project, the team has published and communicated the methodology, results and implications of the study. The collected output is available on the Zenodo site: Artificial Intelligence and Literature Seeking¹. Content published on Zenodo includes reports of:

In the reports we address the applied methods, their limitations and consequences in detail. These methods and all results are therefore not covered in detail in this paper. We refer instead the reader to the Zenodo site.

Results from the project have been communicated at conferences such as: INCONECSS, Virtual Conference (Spring 2022), LIBER Conference in Odense, Denmark (Summer 2022) and the Business Library Association (Spring 2023). The slides are published on our Zenodo site. Further, two articles were published at the very start of the project. The first describing our visit to Helsinki and the second concerning the impact of AI on literature search services at the library. Both articles were published in the Danish Journal “REVY” (Kjær et al., 2020; Lyngsfeldt et al., 2022).

In the following sections, we provide a summary of the applied methods, Section 3, and major findings from our investigation, Section 4. We provide a critical reflection of our observations and consequences for academic searching and librarianship in Section 5. Section 6 provides the conclusions and Section 7 the limitations of our study.

We identified 16 AI-search tools marketed for academic application. Each tool was examined according to our requirements that is to which extent they indeed supported academic literature search and research integrity, including transparency, reproducibility and data security. Two products, Iris.ai and Yewno.discover, met the posed requirements. The remaining AI-search tools failed on aspects such as data security, academic searching, maturity, content that could be searched and the possibility of documentation. A detailed analysis of the AI-search tools considered for inclusion in our project are presented in Wildgaard et al. (2020).

A report of the results of the Think-aloud tests, methodological framework and set of tasks is published in Zenodo (Wildgaard et al., 2021). In the following, we provide a summary of the main results.

Ten information specialists were invited to take part in the Think-aloud tests in May and June 2021. Nine completed the tests, resulting in five tests in Iris.ai and four in Yewno.discover. The participants were from the natural and technical sciences, business and social sciences, and arts and humanities. They described themselves as good (n1), intermediate (n3) and expert (n5) in a systematic review and systematic search. All were new to the tested AI-search tools.

The participants agreed that Iris.ai and Yewno.discover have the potential to be a supplement to the databases and search systems that the university libraries already offer. They considered the AI-search tools to be especially useful at the start of a project for both students and PhD students, as the tools can be used to explore and investigate concepts, find cross-disciplinary topics and thus generate ideas. Further, they pointed out that AI-search tools may contribute to innovation at the universities, as they “break new ground” in the way they search and present results.

The AI-search tools present search results as a graphical presentation similar to a sociogram, where the searcher can move through hierarchical nodes and links between nodes to explore topics and find literature. This form of display is very different from the bibliographic list of results that is produced in databases employed in academic searches. The graphical display was assessed as valuable by the information specialists. They suggested that the AI-search tools could in this regard be useful in the pre-award stages of a funding application, as the user gains a visual insight into evidence clusters and evidence deserts, trends, disciplinary perspectives and interdisciplinary connections in relation to a research question.

However, the AI-search tools challenged the information specialists’ approach to the search. In traditional bibliographic databases information specialists use Boolean, proximity and field search operators. Whereas the search features in the AI-search tools encourage the searcher to explore connections between results and to investigate the metadata describing the results to identify concepts and related works. As a result, the information specialists found that using the AI-search tools required in depth knowledge of the functionality of the tool and in-depth knowledge of the specific domain of the search query to be able to assess the success of the search. They commented that in traditional databases, they typically carry out all the search technicalities on behalf of the researcher (i.e., identification of search terms and search string setup). However, in the AI-search tools, they need to be able to make qualified relevance assessments at an early stage in the search process, which in turn form thoughtful exploration of the concepts and literature discovery together with the researcher.

In the AI-search tools, a semantic algorithm assesses the “aboutness” of the texts and links the texts to a list of automatically generated concepts. Dependent on the tool, these concepts can be grouped together to define broader topics and can be labelled with either a name and short definition or just by a number. At the time of testing, both tools worked with concepts, meaning that search terms were not highlighted in the retrieved text, but rather text snippets that are “about” the concept in a specific context were shown to the searcher. The information specialists were sometimes confused by the suggested concepts and how they represented their query statement. They were unable to remove irrelevant concepts from the search. They disagreed with the tools’ “aboutness” assessments, and they were critical of how concepts were defined in the tool.

It was emphasized that both Iris.ai and Yewno.discover could be useful tools for navigating the available Open Access literature as both tools searched open access resources. However, the information specialists were not satisfied with the academic quality of the identified papers in these resources. Overall, the information specialists were not satisfied with the transparency and relevance of the search and the extent to which the search could be documented.

The assessment of the data collected during the Hackathon and the qualitative and quantitative analyses were completed in March 2022. Hackathon group demographics are presented in Table 1. The main observations from the Hackathon are presented in Section 4.3.1. The results of the qualitative and quantitative analysis are presented in Section 4.3.2, and Section 4.3.3.

Two of the research team groups searched using the AI-search tool, respectively Iris.ai (Group 1), and Yewno.discover (Group 2). The information specialist group (Group 3) searched in both Iris.ai and Yewno.discover. Group 4, the third research team group, searched in a set of traditional resources, and thus functioned as a control group. They searched in PubMed, Google Scholar and Web of Science.

The results of the expert analysis and quantitative assessment indicate that the AI-search tools provided papers of lower scientific quality. Without any knowledge of which system, the retrieved publications came from, the expert group unanimously gave the publications identified in the AI-search tools a lower quality rating. The expert group came from a wide range of disciplines and academic profiles. Consequently, we do not consider the expert group biased toward one form of communication or research methodology over another.

The control group retrieved primary studies, whereas the AI-search tools retrieved meeting reports, case studies, reports, letter to the editor and opinion papers. The expert review of these papers questioned the rigor of the academic approach (Tables 6, 7, 8 in Johnsen et al. (2022)). However, this wide range of publication types is not necessarily a weakness of the AI-search tools. Yewno.discover includes grey literature and non-primary works that can be very accessible and easier to digest during the discovery phase of the search, especially to a novice in a research area. Further, as pointed out by the Institute for Work and Health (IWH); “…because they [grey literature] aren’t tied to a conventional structure, they can be longer and provide more detail.” (Institute for Work & Health, 2008). Specifically, once grey literature and studies lower down the pyramid of evidence are properly understood, it is possible to appreciate how these can further the goal of literature discoverability (Polonioli, 2020).

The quantitative assessment also showed that the AI-search tools support the visibility of Open Access publications, which is another area of strategic interest in our University Library. Furthermore, the publications found by the AI-search tools experimented with other forms of peer review. These forms of peer-review mean that papers can potentially be published more quickly, the peer review report can be accessed, and the peer review process is more dialogue-based and transparent, thus bringing new research to the forefront in the AI-search tools. However, it is important to determine whether or not the innovations in peer review and the speed with which papers are published can affect their quality and content (Barroga, 2020).

Results from our Think-aloud tests, Hackathon and surveys show that it is difficult for both information specialists and researchers to understand how the AI-search tools “think”. It is not possible for the everyday searcher to understand the level of precision of the search, recall, how and why the AI-search tool clusters the results together or labels the clusters in the way that they do. This lack of understanding may lead to distrust, and questions about the reliability and usefulness of the tools. Our combined results show that transparency of the search tool as well as trust in the scientific content in the retrieved articles are significant factors for the test-takers continued use of and engagement in AI-search tools. Trust is further discussed in Beller et al. (2018) and Gasparini and Kautonen (2022).

An academic search should always be conducted with integrity and honesty. The search needs to be assessed as an appropriate method to find literature in the selected sources and to answer the research question. The search and the sources should be assessed for bias. Bias assessments in an academic search include assessing the risk of publication bias, database bias and selection bias and the effect bias can have for the retrieval of results. Attention should be paid to bias in the search as bias in the search may result in bias in the results and summary of data in the review synthesis. In an AI-powered search, an assessment should also be undertaken of the extent bias in the applied algorithms effect the search. We did not have the technical skills to assess bias in the search tools in our study.

Understanding the system architecture and intention of the AI-search tool is essential if information specialists aim to provide teaching, support, and high-quality professional search services that support research. Increasing our algorithmic literacy throughout the project extended our awareness of what an academic search could be. We began to think beyond systematic search methods, which are developed to support the responsible conduct of research. At the start of the tests our assumption was, that the search methods should be as transparent as possible and documented in a way that enables the search to be validated, evaluated and to some extent, reproduced. We learned however, through working with the AI-search tools that in an academic search there can be other goals apart from methodological transparency, accountability and reproducibility. An academic search can be enriched, particularly in its early phases, by discoverability and serendipity. Discoverability and serendipity are perhaps restricted in traditional Boolean searches and there can be multiple goals and trade-offs between the AI-search tools and the traditional databases involved in the process of the search.

Both Iris.ai and Yewno.discover support the discovery of different semantic connections between concepts in the text corpus. By generating different perspectives to which literature and concepts may be related, idea generation and creativity may be enhanced, divergent thinking increased, and innovative solutions to a research problem may become more perceptible. In addition, through utilising graphical interfaces and node-graphs rather than lists of index terms or bibliographic references, the tools provide a visual entrance to the literature that invites the user to discover “unintended knowledge”. Unintended knowledge is when unexpected connections to the searched topic are identified during the search (Hofman–Apitius et al., 2009). The graphical interfaces in the AI-search tools have the potential to challenge our preconceptions of a concept or topic, reducing our cognitive bias, and may offer a form of serendipity that is lacking in traditional databases. The interdisciplinarity of the search may be increased.

The AI-search tools enabled the searchers in our Hackathon to put their cognitive biases and pre-understanding of a search query to one side. In Iris.ai and Yewno.discover the searchers had to write a research statement which in turn fed words and phrases to the tools’ algorithms and generated concepts and pathways through the literature. Whereas, in our control group which searched traditional search databases, they formulated their queries using the keywords and Boolean logic (Azzopardi, 2021). When searching for literature in the traditional databases, the searcher may be influenced by the conformity of how to search and the desire to confirm the research question (Azzopardi, 2021) rather than discover new connections, as displayed in the AI-search tools, that may identify new perspectives and papers that refute their question and/or provide alternative solutions.

Our participants in both the Think-aloud tests and in the Hackathon could see a potential in using the AI-search tools in the exploratory phase of a literature search. Yet to adhere to the good conduct of research, the explorative phase of the search needs to be documented and rationalised. How to document an explorative search in the project documentation or in the search protocol is at the current time unclear. A case in point is the systematic search and review process. When referring to the PRISMA-S guidelines for reporting literature searched for systematic reviews (Rethlefsen et al., 2021) it is evident that a reproducible and transparent search strategy is mandatory for documenting searches in bibliographic databases and internet search engines. But there are in PRISMA –S (Rethlefsen et al., 2021) so far, no requirements for reporting the initial searches and exploratory methods. In protocols for the JBI evidence synthesis (Aromataris & Munn, 2020) it is recommended to document that initial searches were performed to find articles on the topic and use them to find relevant free text words and index terms for the final search strategy. When reporting on a Cochrane Systematic Review of Interventions any specific methods used to develop the search strategy should be noted (Higgins et al., 2022) and in the Technical Supplement to Chapter 3, by Lefebvre et al. (2022), it is suggested to use text mining tools to objectively identify terms.

After completing our study, we were approached by several AI-search tool developers, eager to engage in feedback on their tool from the perspective of the library. Through meeting with developers, we found that typically, information specialists are not part of the software development team, yet the products are marketed libraries. This means that knowledge of reference management, search processes, search and review methodologies, review workflows and requirements for research integrity may be underdeveloped in tool functionalities.

A collaboration between developers and libraries is also discussed in the literature surrounding AI tools and academic searching. For example, developers are keen for libraries to assess the applicability of the AI-search tool to search different fields, across different types of literature and different languages; semantic and topic development in a search (Clark et al., 2021; Khalil et al., 2022), interface development (Marshall & Wallace, 2019), and user-support, particularly the role of the librarian/library in providing support and expertise (Gasparini & Kautonen, 2022).

Beller et al. (2018) address the barriers and facilitators AI-search tools bring to the search process. They define facilitators as places where libraries and AI software developers have the potential to work together with benefits for both parties (Beller et al., 2018). Gasparini and Kautonen (2022) also argue that AI-search tools provide libraries with the voice to act as a facilitator. Acting as a facilitator can give libraries a new or more proactive role, providing not only a collaborative space for AI-search tool development but also a space to provide technically literate search support, to critically assess the quality of the search AI-search tools produce, and to create awareness about the implications AI-assisted searches can have on the responsible conduct of research. Other topics the library could address with developer and research communities are: commerciality vs. trustfulness and the academic credit of the tool (Arno et al., 2022; Gasparini & Kautonen, 2022), bias and transparency in the search (Beller et al., 2018; Clark et al., 2021), the quality of the metadata (Gasparini & Kautonen, 2022), sorting and filtering algorithms (ACRL Research Planning and Review Committee, 2020; Henry, 2019, pp. 47–65), and the compatibility of the AI-search tool with other tools/workflows commonly used by review writers such as reference management, screening and data extraction programmes (Beller et al., 2018).

The AI-search tools we tested challenged conventional approaches to academic searching and the searcher expectations to search methodologies. However, the tools also encouraged new forms of collaboration and dialogue around the search activity. The control group in our Hackathon, searched in PubMed, Google Scholar and Web of Science, and succumbed to the conformity of how to search as discussed in the Reflection 3. They divided responsibilities according to group member profiles: information specialists conducted the search, and researchers judged the relevance of search terms and results. However, the researchers and information specialists who searched using the AI-search tools explored topic areas, discussed literature and search strategies together. This novel form of collaboration could of course be caused by the “newness” of the AI-search tool, which the participants were curious to explore and evaluate. Yet it is worth considering if discovery systems such as Iris.ai and Yewno.discover do require an increased interaction, discussion and collaboration within a research “team”.

Combining AI-search tools and traditional databases in different phases of the search process may mitigate and manipulate cognitive biases to improve the impartiality of the search, the search behaviour of the searcher and outcomes of the search. By mediating the value and limitations of different approaches to literature retrieval in traditional and AI-search tools, the information specialist can equip searchers with the skills they need to use them in combination with traditional approaches to the search. We need to adjust our expectations to what a search on an AI-search tool can deliver; it does not provide a single search solution even though the search is conducted across thousands of multidisciplinary documents.

The role of the information specialist in identifying and clarifying core aspects of the search does not cease with introduction of AI-search tools at the library. If anything, the role of the librarian is rejuvenated, and we are given the opportunity to develop methods for searching in AI-search tools and develop protocols for documenting AI supported approaches to academic search. Asemi et al. (2021), Bethard et al. (2009), and Ewing and Hauptman (1995) predicted the extinction of information specialists in the scholarly ecosystem when professional expertise and behaviour are transferred to automated systems (Gasparini & Kautonen, 2022). On the contrary, our results show the increased need for information specialist involvement in the AI-search process and in AI-search tool development. Gasparini and Kautonen (2022, p. 8) go so far as to suggest, that information specialists need to. “abandon their old paradigms, practices, and workflows,.” Abandonment of existing professional approaches to the search is perhaps extreme. Knowledge and skills in AI-search tools enable information specialists to define new professional roles for themselves and nurture new roles and collaborations in the search and review process with students, researchers, IT and enterprise units, (ACRL Research Planning and Review Committee, 2020; Nolin, 2013) as cited in Gasparini and Kautonen (2022).