Critiques of world university rankings and their processes, indicators and proxies, including prejudices and blind-spots in relation to the measurement tools and data sources have grown since 2007. For example, the databases providing publication and citation data consist of predominantly English language, elite publication sources whose scope precludes research output from many countries and marginal communities. As universities are driven by the rankings to compete, their focus becomes less on national and local needs, a major shift (Hazelkorn, 2018), reducing publication in local sources that are accessible and relevant to their communities. Rankings persist, creating their own reality and paradoxically changing university strategies and performance (Gadd, 2021).

The Curtin Open Knowledge Initiative is a strategic research project that aims to move beyond university rankings, to promote openness in institutions and research production (Montgomery et al., 2021). The project aggregates and analyses multiple dimensions of research output that can facilitate dialogue and encourage different ways of thinking about research impact. Central to the project’s approach is gathering together publicly available data that is often fragmented. This includes bibliographic, affiliation, citation, funder and workforce demographic data from a range of public sources to understand research performance including open access, collaborations, citation advantages, publishing, funding and institutional diversity.

The paper falls into four parts. The first discusses the rationale and background to the project including a critique of world university rankings and the genesis and development of our dataset and dashboard tools. In the second section we discuss methods used in developing the project’s dataset, data sources and analysis, and a feedback survey undertaken with academic libraries. The third section discusses two key aspects of the project: open access research analysis and gender balance in academic workforces, with some examples. We conclude by discussing collaboration with academic libraries and other stakeholders in developing dashboard tools and a community open knowledge coalition.

Leonelli et al. (2017, p. 194) discuss the notion of “data shadows” – data excluded fully or partially and the strategic or methodological reasons for absences. Our project critiques the selective use of data to stratify institutional performance by major world university rankings Academic World Ranking of Universities (AWRU), the Times Higher Education (THE) World University Ranking and Quacquarelli Symonds (QS). For example, they depend on publication and citation data extracted from commercial bibliographic databases Scopus (Elsevier) and Web of Science (Clarivate Analytics) whose coverage favours prestigious, English language, Global North publications with more emphasis on sciences, medicine and technology than the humanities and social sciences (Aksnes & Sivertsen, 2019; Tennant, 2020). Publication in these sources devalues diverse research from non-dominant and Indigenous populations, some countries and in disciplines such as the creative arts and political activist scholarship where journal publications are not the primary output format (Shahjahan & Wagner, 2019). This leads to a “geopolitics of knowledge” and a persistent coloniality (Stack & Mazawi, 2021, p. 226). These limited proxy measures with embedded epistemologies and questionable ontological assumptions distort the wider picture, and are open to institutional measures to influence outcomes and policy (McCormick, 2017).

The presence of rankings pressures university managements to develop strategies and practices in order for their institutions to compete on the ranking ladder and achieve status (Marginson, 2014). This occurs despite the limitations and exclusions of methodologies and the measurements rankings used as proxies for performance (Hazelkorn, 2018). For example, reputation and research performance, two key concepts of the AWRU, THE and QS (Selten et al., 2020) are built on citation analysis from the commercial databases discussed above. The use of “celebretization” measures such as Nobel Prize winners as a proxy of excellence and to rank reputation, perpetuates the dominance of white male demographics in universities (Stack, 2020, p. 5). These practices contradict the equity, diversity and inclusion (EDI) policies which many universities promote.

A key difference between our project and commercial datasets and analytical services is our approach to understanding scholarly communication from a critical research perspective. Our research emphasises the importance of asking how datasets might influence the questions asked about a university’s performance, as well as the answers that are found. For example, a comparison of bibliographic data sources and their implications for university rankings finds significant differences in coverage of disciplines, publications and date ranges (Huang et al., 2020a). Our project identifies and analyses public data sources to examine more deeply the outcomes of EDI policies and how this may affect research production, and institutional progress in achieving open access to knowledge. Such analysis can assist higher education institutions, authorities and researchers in understanding the extent and nature of their research performances, to become open to alternative ways of viewing research impact, to value openness in output and in research production. We aim to change the narrative about the role of research and knowledge production within institutions. The dataset takes a wide geographic scope and includes research publication data from countries in Asia, Africa and Latin America whose coverage by major commercial publishers and databases may be restricted, and across all disciplines. We engage with communities in Africa and Indigenous researchers in Australia to increase the visibility and accessibility of research within their communities, countries and the world.

In this section we discuss the project’s dataset sources and analysis; the collection and analysis of diversity data; and methods used in the dashboard feedback survey. We use open data sources, including Microsoft Academic Graph, Unpaywall, Crossref metadata, Open Citations, ORCID and the Research Organization Registry (ROR) to analyse bibliographic, open access, affiliation, funder, publisher and citation data. The dataset includes more than 12 trillion items from over 140 countries and thousands of research institutions. Through our collection pipeline, we gather data about research publications using unique Digital Object Identifiers (DOIs) from multiple sources, synthesise the datasets and create research performance calculations, including open access for each country and institution (see Data Availability statement).

To identify higher education institutional workforce demographic data relating to diversity we searched for publicly available regional and national web-based sources of statistics. Terminology varies and includes ‘higher education’, ‘tertiary education’, ‘university/universities’, ‘ministry’, ‘department of education’, ‘statistics’, ‘gender’, ‘ethnicity’, ‘race’ and ‘disability’ (Wilson et al., 2022a). Gender statistics are the most widely available, although primarily binary (men/women, male/female). We acknowledge limitations in how data are collected and reported, and identify the need for further exploration of the intersectionality of gender or sex and ethnicity, race or indigeneity, age and disability in understanding equity and diversity (Harris & Patton, 2019; Rice et al., 2019).

To gather feedback on the dashboard tool we undertook a survey in our region to obtain: (1) comments on the dashboard layout, navigation and the data incorporated therein; (2) insights into how institutions, library and research staff, researchers, executives and administrators might use the data, and associated behaviour around such use in the academic environment. In February 2020, we invited CAUL (Council of Australian University Librarians) and CONZUL (Council of New Zealand University Librarians) members via email to register to access the dashboard, review their institution’s data, and to participate in an anonymous online Qualtrics survey, plus an optional follow-up telephone interview (Curtin University Human Research Ethics Committee approval HRE2020-0086). Twenty-seven universities out of 47 member institutions (39 CAUL and eight CONZUL) registered to access the dashboard. Five responses came from universities in Aotearoa New Zealand and 12 from Australia. Ten were fully completed, four partially and three minimally, with two follow up telephone interviews. Some more survey respondents indicated they would participate in a telephone interview, but as the responses were anonymous as specified in the ethics approval, we could not follow up.

The online survey included twelve questions:

Most questions were multiple choice and allowed for textual comments. One free text question asked respondents for additional comments and suggestions on the dashboard. The final survey question offered participants the option to expand on their responses in a follow up telephone interview with eight questions aimed to build qualitatively on the survey responses. They included two institutional questions, one related to the interviewees’ work role, and five semi-structured questions focused on their responses to the usefulness and relevance of the data and dashboard (Wilson et al., 2022b).

In developing the dashboards, the project uses different visual software, including Google Data Studio and Kibana, an open source user interface for the ElasticSearch search analytics tool and web-based software.

A key focus of the project is understanding the trajectory of open access institutional research output across the globe. The Budapest Open Access Initiative (BOAI, 2002) produced the first formal statement on open access (OA) free to read, download and share, with two options for opening research output: self-archiving into repositories and publication in OA journals. Self-archived (Green) OA is research free to read through an institutional or disciplinary repository or a preprint server. Published (Gold) OA is free to read either with an open publisher (no fee to publish), or via a paid publisher (with article processing charges or APCs), and distributed with an open licence. These options have expanded as commercial publishers entered the OA landscape. Hybrid OA is free to read from a subscription-based publisher (with APCs) and distributed with an open licence. Gold/Hybrid and Green can overlap when a self-archived copy exists as well as a published version.

Different self-archiving options include institutional and disciplinary repositories, and pre-print servers such as ArXiv for physics, mathematics and computer science, a pioneering model followed by other disciplines. Publishing models include digital only publishers who are exclusively OA, both non-profit (e.g. the Public Library of Science (PLOS)) and for profit (e.g. BioMed Central), as well as existing originally paper-based publishers who provide an OA option, usually involving APCs (Tennant et al., 2016). With rising costs to researchers of publishing OA commercially, research organisations and universities began to cancel institutional journal subscriptions. This has led to transformative or ‘read and publish’ OA agreements between publishers and libraries or library consortia in many countries and regions, combining the costs of OA publishing for institutional authors, who generally retain copyright under a Creative Commons licence, with costs for reading publications (Hinchliffe, 2020).

Many OA publishing policies, statements and funder mandates are in place across the globe. Policies by country or region include Europe’s Plan S policy requiring publicly and privately funded research from 2021 to be immediately available via OA journals, platforms or repositories (cOAlition S, n.d.). Institutional mandates or policies may specify OA through local repositories (Green) or by publisher (Gold); and funder mandates require research they fund to be available as open access. Publisher Gold OA fees can be prohibitive and out of reach for many researchers and regions, producing global inequities. The option of repository-based OA (Green) is cost free to authors, although it may be more complex to negotiate (Björk, 2017). In some parts of the world challenges with reliable Internet access, technology infrastructure, policies, expertise and funding impact the development and maintenance of institutional repositories (Wilson et al., 2020). However, our OA analysis highlights strong performance and initiatives at some regional and institutional levels.

Our OA analysis from 2010 to 2020 across six regions (Asia, Europe, North America, Latin America, Africa, Oceania) indicates the growth and shifts in Green (Repository) and Gold (Publisher) output over the ten-year period. These numbers reflect developing national policies and programs. In 2010, Latin American institutions (brown) appear with the highest Gold and Green output (Figure 1). Brazil and Chile developed the Scientific Electronic Library Online (SciELO) journal network in 1998 as an open platform for Latin American research output, to provide greater visibility of research publication than the major commercial databases such as Web of Science (Packer, 2020). An additional 13 countries, including Portugal, Spain and South Africa, have joined the network.

Figure 2 shows Latin American institutional research output moving further towards the Gold OA axis by 2020. The growth since 2010 of European institutions (green), clustering between 35% and 60% on the Green and Gold OA axes, reflects EU and UK policies that promote OA publishing, although these vary by country and institution. A small number of institutions in Asia (orange) reaches towards 80% Gold OA and up to 60% Green OA. Some institutions in Africa (pink) show high Green OA and Gold OA output. United States institutions (blue) lean towards Green OA with some reaching higher Gold OA levels, reflecting variations by institution and funder, and suggesting variant policies and practices. Australian institutions (black) hover between 20% and 40% Green OA and Gold OA, reflecting a lack of national policy although the two major public research funders have OA mandates. Figure 3 shows the animated version of OA Gold and Green trajectories for the years 2010-2020.

The Curtin Open Knowledge Initiative encourages universities to centre the principles of openness, to embrace digital open access, and to build diversity, equity and inclusion within institutions and in society (see Figure 4). Internal diversity includes demographic dimensions of researchers, teaching, management and administrative workforces, with diverse researchers as knowledge agents engaged across disciplines and communities. Structural diversity within institutions is achieved through equitable policies and practices and the sharing of data and knowledge. External diversity encompasses diverse research perspectives and knowledge sources, languages, geographical collaborations at regional, national and international levels, as well as diversity in funding sources (Montgomery et al., 2021).

Greater diversity in research workforces challenges structural and scholarly inequalities in the topics and perspectives investigated through institutional research, and awareness of the gaps in knowledge output and research funding is fundamental to addressing inequalities (Kozlowski et al., 2022). In relation to internal diversity, our project analyses public institutional demographic data in order to understand who is producing research and how knowledge is shared within and across disciplines, between universities and wider communities (Montgomery et al., 2021). We discuss gender, the public higher education data the most widely available.

The Curtin Open Knowledge Initiative shares the dataset, analysis and visual dashboards with scholarly communication communities. We seek feedback and engagement from those who understand and use the data to make their own analyses such as academic libraries and research offices. They produce and disseminate information about publishing and data sharing, promote open knowledge and skill development through the use of analytical tools, reporting and educational functions, advise and assist researchers and are well-placed to review data analysis and tools.

In 2020, we engaged Council of Australian University Librarians (CAUL) and Council of New Zealand University Librarians (CONZUL) members in a survey to review the dashboard interface and the relevance of the data analysis within their working environments. Respondents indicated the dashboard provided new perspectives on existing data, and access to new and previously unavailable data. In particular, they noted the value of institutional level OA publication analysis, compliance with research funder OA mandates and OA citation advantages which were difficult to obtain from other sources, including institutional repositories. Respondents indicated the analysis of OA citation figures would help researchers to appreciate the value of OA publishing and how it can benefit them, beyond simply following institutional policy imperatives to self-archive research. They expressed interest in access to more analysis, including underlying data and the OA performance of other institutions for comparative and benchmarking purposes. Overall, they indicated the data and its understanding was strengthened by being brought together in one location in a visual and interactive format (Wilson et al., 2022b).

This feedback provided valuable insights into the dashboard format and layout and researcher scholarly publishing practices and attitudes. First, the cognitive advantages of the visual dashboard interface indicate the strength of this method in communicating complex data and related narratives. Second, the participants’ positive responses to the data content and dashboard, its accessibility and format encouraged us to continue building tools. Third, the interest expressed in comparative data contributed to our development of a public Open Access Dashboard (https://open.coki.ac). This dashboard provides publication, OA and citation data for 142 countries and 5,117 institutions with at least 1,000 research outputs. Users can search and display analysis by institution and country. Full details of data collection, the dataset code and data sources are available (Hosking et al., 2022). Figure 6 provides an example of an institutional dashboard entry with research performance analysis of total publications, the OA percentage and total citations.

In Figure 7, the institutional OA percentage is broken down into ‘Closed’, ‘Publisher Open’ (in an OA Journal, sometimes referred to as Gold OA, Hybrid) and ‘No Guarantees’, (or Bronze); ‘Other Platform Open’ (deposited in an institutional repository, a disciplinary repository, a preprint repository or a general repository, sometimes referred to as Green OA), and ‘Both’ (‘Publisher Open’ and ‘Other Platform Open’). ‘Both’ indicates an item may be published open simultaneously with a copy in a repository, representing two pathways for creating open and accessible research. We introduce new terminology to clarify the sometimes-confusing colour labelling for OA types (such as Green, Gold, Hybrid, Bronze).

The insert box data in Figure 7 provides details of research output in the year 2001. The longitudinal data indicate changes over the two decades. These data and images are available for all institutions and countries included in the dataset at present are those with over 1,000 research outputs. Development of the dashboard continues, adding more search features.

Working from a critical research perspective the Curtin Open Knowledge Initiative project has built a dataset and created dashboard analytical tools that provide new insights into the research performance of higher education institutions. Using our data and analysis, institutions can explore how open are the knowledges they produce and how their workforces reflect the diversity of student bodies, communities and wider populations. The analysis can provide indications of diverse research perspectives that may be lacking as a result of employment practices and preferences. The data analysis reveals complex national and regional patterns of research performance across the world, in particular OA output.

The project’s goals are to deliver accessible, shareable data, resources, analysis and to open conversations about ways to open knowledge. Contributions and feedback received from academic libraries, researchers, funders and publishers in the scholarly communication community validate and confirm the need for accessible and neutral data that form the basis and origins of this project. We regard information as a community asset, and highlight an opportunity to move beyond commercial data towards the construction of resources that are governed by the higher education and research community. Collaboratively we have built a community coalition to meet a vital need for maintaining access to complex research performance analysis and extending open knowledge production. The coalition includes stakeholders from higher education consortia, funders, research managers, libraries and publishers. Central components of the Open Knowledge community coalition include building and maintaining the data asset we have developed, a community of practice around making change, sharing expertise, software and data on evaluation and openness. Through sharing data and dashboard code with coalition members the Open Knowledge project expands its scope and analysis of institutional openness and knowledge making, diversity and inclusion in knowledge production, and evaluation of open knowledge performance.