<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE article PUBLIC "-//NLM//DTD JATS (Z39.96) Journal Publishing DTD v1.0 20120330//EN" "JATS-journalpublishing1.dtd">
<article article-type="research-article" xml:lang="EN" xmlns:xlink="http://www.w3.org/1999/xlink">
<front>
<journal-meta>
<journal-id journal-id-type="publisher-id">LIBER</journal-id>
<journal-title-group>
<journal-title>LIBER QUARTERLY</journal-title>
</journal-title-group>
<issn pub-type="epub">2213-056X</issn>
<publisher>
<publisher-name>openjournals.nl</publisher-name>
<publisher-loc>The Hague, The Netherlands</publisher-loc>
</publisher>
</journal-meta>
<article-meta>
<article-id pub-id-type="publisher-id">lq.11726</article-id>
<article-id pub-id-type="doi">10.53377/lq.11726</article-id>
<article-categories>
<subj-group subj-group-type="heading">
<subject>Article</subject>
</subj-group>
</article-categories>
<title-group>
<article-title>Multi-Stakeholder Research Data Management Training as a Tool to Improve the Quality, Integrity, Reliability and Reproducibility of Research</article-title>
</title-group>
<contrib-group>
<contrib contrib-type="author">
<contrib-id contrib-id-type="orcid">https://orcid.org/0000-0001-5927-3781</contrib-id>
<name>
<surname>Rantasaari</surname>
<given-names>Jukka</given-names>
</name>
<email>jukka.rantasaari@utu.fi</email>
<xref ref-type="aff" rid="aff1"/>
</contrib>
<aff id="aff1">University of Turku, &#x00C5;bo Akademi University</aff>
</contrib-group>
<pub-date pub-type="epub">
<month>06</month>
<year>2022</year>
</pub-date>
<volume>32</volume>
<fpage>1</fpage>
<lpage>54</lpage>
<permissions>
<copyright-statement>Copyright 2022, The copyright of this article remains with the author</copyright-statement>
<copyright-year>2022</copyright-year>
<license license-type="open-access" xlink:href="http://creativecommons.org/licenses/by/4.0/">
<license-p>This is an open-access article distributed under the terms of the Creative Commons Attribution 4.0 International License (CC BY 4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. See <uri xlink:href="http://creativecommons.org/licenses/by/4.0/">http://creativecommons.org/licenses/by/4.0/</uri>.</license-p>
</license>
</permissions>
<self-uri xlink:href="https://www.liberquarterly.eu/article/10.53377/lq.11726"/>
<abstract>
<p>To ensure the quality and integrity of data and the reliability of research, data must be well documented, organised, and described. This calls for research data management (RDM) education for researchers. In light of 3 ECTS Basics of Research Data Management (BRDM) courses held between 2019 and 2021, we aim to find how a generic level multi-stakeholder training can improve STEM and HSS disciplines&#x2019; doctoral students&#x2019; and postdoc researchers&#x2019; competencies in RDM. The study uses quantitative, descriptive and inferential statistics to analyse respondents&#x2019; self-ratings of their competencies, and a qualitative grounded theory-inspired approach to code and analyse course participants&#x2019; feedback. <bold>Results</bold>: On average, based on the post-course surveys, respondents&#x2019; (n &#x003D; 123) competencies improved one point on a four-level scale, from &#x201C;little competence&#x201D; (2) to &#x201C;somewhat competent&#x201D; (3). Participants also reported that the training would change their current practices in planning research projects, data management and documentation, acknowledging legal and data privacy viewpoints, and data collecting and organising. Participants indicated that it would be helpful to see legal and data privacy principles and regulations presented as concrete instructions, cases, and examples. The most requested continuing education topics were metadata and description, discipline specific cultures, and backup, version management, and storage. <bold>Conclusions</bold>: Regarding to the widely used criteria for successful training containing 1) active participation during training; 2) demand for RDM training; 3) increased participants&#x2019; knowledge and understanding of RDM and confidence in enacting RDM practices; and 4) positive post-training feedback, BRDM meets the criteria. This study shows that although reaching excellent competence in a RDM basics training is improbable, participants become aware of RDM and its contents and gain the elementary tools and basic skills to begin applying sound RDM practices in their research. Furthermore, participants are introduced to the academic and research support professionals and vice versa: Stakeholders will get to know the challenges that young researchers and research students encounter when applying RDM. The study reveals valuable information on doctoral students&#x2019; and postdoc researchers&#x2019; competencies, the impact of education on competencies, and further learning needs in RDM.</p>
</abstract>
<kwd-group>
<kwd>Research data management</kwd>
<kwd>Training</kwd>
<kwd>Competencies</kwd>
<kwd>Early career researchers</kwd>
<kwd>PhD students</kwd>
<kwd>Doctoral students</kwd>
<kwd>Postdoc researchers</kwd>
</kwd-group>
</article-meta>
</front>
<body>
<sec id="s1">
<title>1. Introduction</title>
<p>During the second decade of the 2000s, many international, national, and institutional principles and policies and an increasing number of funders and publishers started recommending or mandating researchers to write data management plans (DMPs) and share data (e.g., <xref ref-type="bibr" rid="r2">Academy of Finland, 2019</xref>; <xref ref-type="bibr" rid="r5">&#x201C;Amsterdam call for action on open science&#x201D;, 2016</xref>; <xref ref-type="bibr" rid="r23">European Commission, 2018a</xref>,<xref ref-type="bibr" rid="r24">b</xref>; <xref ref-type="bibr" rid="r25">European University Association, 2017</xref>; <xref ref-type="bibr" rid="r40">National Science Foundation, 2011</xref>; <xref ref-type="bibr" rid="r69">UNIFI, 2016</xref>; <xref ref-type="bibr" rid="r72">Wellcome, 2017</xref>). Researchers need education, guidance, and support in research data management (RDM) to help fulfil this task. At the core of these principles, policies and demands is to obtain data of publicly funded research openly accessible and reusable, when possible. In principle, research data, or at least metadata, should be Findable, Accessible, Interoperable and Reusable (FAIR<xref ref-type="fn" rid="fn1">1</xref>). Sound RDM practices advance the integrity of data, reliability of research results, transparency of the research process, and reproducibility of research (e.g., <xref ref-type="bibr" rid="r16">Chiarelli et al., 2021</xref>). However, research transparency and data reuse may only be fully realised if data is opened and shared (<xref ref-type="bibr" rid="r6">Borghi et al., 2018</xref>). Shared research data also avoids the gathering of duplicate data and enables combined efforts to find solutions to complicated interdisciplinary research issues like climate change and pandemics (<xref ref-type="bibr" rid="r22">Doucette &#x0026; Fyfe, 2013</xref>; <xref ref-type="bibr" rid="r63">Shearer, 2009</xref>). Moreover, sharing research data can significantly shorten the time it takes to move from an initial scientific discovery to practical applications (<xref ref-type="bibr" rid="r26">Federer, 2016</xref>). Nevertheless, it is only useful to share well-documented, described, and organised data (<xref ref-type="bibr" rid="r6">Borghi et al., 2018</xref>; <xref ref-type="bibr" rid="r58">Rieser, 2018</xref>) that provides clear data sharing parameters, including intellectual property rights (IPR) and agreements. (<xref ref-type="bibr" rid="r51">Rantasaari, 2021</xref>).</p>
<p>Though RDM<xref ref-type="fn" rid="fn2">2</xref> is perceived as important or very important by researchers and graduate students (<xref ref-type="bibr" rid="r44">Pasek &#x0026; Mayer, 2019</xref>; <xref ref-type="bibr" rid="r66">Thielen et al., 2017</xref>), many researchers are not managing their data according to recommended RDM guidelines. For example, graduate students are often given substantial data management responsibilities in research projects though they usually have received little or no education in RDM (<xref ref-type="bibr" rid="r27">Goben &#x0026; Griffin, 2019</xref>; <xref ref-type="bibr" rid="r33">Krahe et al., 2020</xref>; <xref ref-type="bibr" rid="r36">Maienschein et al., 2019</xref>; <xref ref-type="bibr" rid="r75">Wiley &#x0026; Kerby, 2018</xref>). Thus, they tend to develop ad hoc solutions with the trial-and-error method (<xref ref-type="bibr" rid="r67">Thielen &#x0026; Hess, 2017</xref>; <xref ref-type="bibr" rid="r79">Wright &#x0026; Andrews, 2015</xref>). Therefore, RDM practices are often unstandardised, and IPR and contract issues may be unfamiliar. Also, documentation made to carry out the ongoing research that does not consider other uses and users does not enable data sharing and reusing, undermining research reproducibility (<xref ref-type="bibr" rid="r51">Rantasaari, 2021</xref>).</p>
<p>In this article, our goal is to find how generic, multi-stakeholder training can improve participants&#x2019; competencies and further comprehension of the relevance of sound research data management practices to the quality and integrity of data and reliability of the research.</p>
<p>In practice, we will report the outcomes of the Basics of Research Data Management (BRDM) course over three years (2019&#x2013;2021), held at two Finnish universities. The learning objectives and contents of BRDM were developed based on an interview study on doctoral students&#x2019; RDM competencies and learning needs (<xref ref-type="bibr" rid="r51">Rantasaari, 2021</xref>; <xref ref-type="bibr" rid="r52">Rantasaari &#x0026; Kokkinen, 2019</xref>), discussions with the leader of the biostatistician team of the University of Turku (UTU), and research literature and lessons learned from previous RDM trainings (e.g., <xref ref-type="bibr" rid="r49">Piorun et al., 2012</xref>; <xref ref-type="bibr" rid="r50">Qin &#x0026; D&#x2019;ignazio, 2010</xref>; <xref ref-type="bibr" rid="r66">Thielen et al., 2017</xref>; <xref ref-type="bibr" rid="r74">Whitmire, 2015</xref>; <xref ref-type="bibr" rid="r79">Wright &#x0026; Andrews, 2015</xref>).</p>
<p>We aim to answer the following questions:</p>
<list list-type="bullet">
<list-item><p>RQ1: How did course participants self-rate their RDM competencies before and after BRDM course&#x003F;</p></list-item>
<list-item><p>RQ2: What kind of educational impact did the course have on participants&#x2019; RDM competencies (knowledge, skills, and abilities) based on participants&#x2019; self-ratings and collected and categorised feedback&#x003F;</p></list-item>
<list-item><p>RQ3: What kind of further learning needs did the respondents express after the course&#x003F;</p></list-item>
</list>
<p>After the introduction, we will discuss specific contents and lessons learned in previous RDM basic trainings directed specifically for graduate students or researchers. The methods section will describe BRDM&#x2019;s objectives, structure, and learning methods. Research methods used to answer research questions RQ 1 to 3 will be described. Section four contains the results of the study, and section five the discussion and conclusions.</p>
</sec>
<sec id="s2">
<title>2. Literature Review</title>
<sec id="s2a">
<title>2.1. Common Contents in RDM Education</title>
<p>We collected the information of 30 RDM trainings from research articles and conference proceedings directed at graduate students or researchers between 2010 and 2021. Using the trainings&#x2019; descriptions, the author categorised their contents as RDM topics and listed the topics handled in each training (<xref ref-type="table" rid="tb003">Table 1</xref> in <xref ref-type="app" rid="app1">Appendix A</xref>). The number of trainings addressing each topic is charted below (<xref ref-type="fig" rid="fg001">Figure 1</xref>). The most common topics covered in over 50% of the trainings were &#x201C;Planning data management and organisation&#x201D; (27); &#x201C;Sharing and reuse&#x201D; (25); &#x201C;Storage, backup, and security&#x201D; (21); &#x201C;Metadata and data description&#x201D; (21); &#x201C;Preservation&#x201D; (21); &#x201C;Legal and ethical issues&#x201D; (17); and &#x201C;Quality and documentation&#x201D; (17).</p>
<fig id="fg001">
<label>Fig. 1:</label>
<caption><p>RDM topics handled in 30 trainings held between 2010 and 2021.</p></caption>
<graphic xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="figures/LIBER_2022_32_Rantasaari_fig1.jpg"/>
</fig>
<p>Overwhelmingly, the most common topics were &#x201C;Planning data management and organisation&#x201D; and &#x201C;Sharing and reuse&#x201D; which is understandable as the need for RDM became widespread after big funders like <xref ref-type="bibr" rid="r72">Wellcome (2017)</xref> and <xref ref-type="bibr" rid="r40">National Science Foundation (2011)</xref> began mandating DMPs and recommending data sharing in funding applications. Finnish major research funder Academy of Finland has required DMPs and data sharing in principle since 2015 (<xref ref-type="bibr" rid="r2">Academy of Finland, 2019</xref>). In RDM educational programs, data have been noted as a validator of research. Also, the reuse of data, as well as the policies, permits, and licenses demanding the data sharing, and the importance of becoming familiar with data sharing culture and infrastructure, have been discussed (<xref ref-type="bibr" rid="r49">Piorun et al., 2012</xref>; <xref ref-type="bibr" rid="r55">Read et al., 2019</xref>; <xref ref-type="bibr" rid="r56">Research data service, n.d.</xref>; <xref ref-type="bibr" rid="r79">Wright &#x0026; Andrews, 2015</xref>).</p>
<p>Besides the informational type of contents, some courses and workshops include more technically oriented RDM topics such as data analysing and visualising, wrangling, merging, cleaning, and publishing data sets, as well as building and using relational databases for data gathering, organising, and querying (<xref ref-type="bibr" rid="r11">Carpentries, n.d.</xref>; <xref ref-type="bibr" rid="r43">Pascuzzi &#x0026; Sapp Nelson, 2018</xref>; <xref ref-type="bibr" rid="r50">Qin &#x0026; D&#x2019;ignazio, 2010</xref>; <xref ref-type="bibr" rid="r55">Read et al., 2019</xref>; <xref ref-type="bibr" rid="r56">Research data service, n.d.</xref>; <xref ref-type="bibr" rid="r79">Wright &#x0026; Andrews, 2015</xref>).</p>
</sec>
<sec id="s2b">
<title>2.2. Lessons Learned in RDM Education</title>
<p>Though a lack of comprehensive and specific reporting of the results of educational RDM efforts exists (<xref ref-type="bibr" rid="r27">Goben &#x0026; Griffin, 2019</xref>; <xref ref-type="bibr" rid="r45">Perrier et al., 2017</xref>), the feedback and results appear to be satisfactory or good. Typically, the attendants have been reported to have given good feedback (e.g., <xref ref-type="bibr" rid="r15">Chew et al., 2021</xref>; <xref ref-type="bibr" rid="r74">Whitmire, 2015</xref>), with their satisfaction varying from medium to high (<xref ref-type="bibr" rid="r38">Muilenburg et al., 2014</xref>).</p>
<p>As a result of training, competencies usually improve by one step, typically from &#x201C;no competency&#x201D; to &#x201C;little competency&#x201D; or from &#x201C;some competency&#x201D; to &#x201C;good competency&#x201D; (<xref ref-type="bibr" rid="r50">Qin &#x0026; D&#x2019;ignazio, 2010</xref>; <xref ref-type="bibr" rid="r59">Schmidt &#x0026; Holles, 2018</xref>; <xref ref-type="bibr" rid="r79">Wright &#x0026; Andrews, 2015</xref>). According to <xref ref-type="bibr" rid="r47">Peters and Vaughn (2014)</xref>, based on the participants&#x2019; self-assessment (n &#x003D; 65) after the NECDMC workshop, the competencies were mostly good. In a survey after five RDM courses held in 2013&#x2013;2017, 77% of the respondents (n &#x003D; 31) considered the course useful, and 58% said they were interested in advanced education when available (<xref ref-type="bibr" rid="r77">Wiljes &#x0026; Cimiano, 2019</xref>). In the feedback of the four clinical RDM workshops, respondents (n &#x003D; 113), who were mainly project coordinators, faculty members, and managers, expressed a need to learn RDM from many viewpoints and aspects like IPR, data security and privacy, and data curation (<xref ref-type="bibr" rid="r54">Read, 2019</xref>).</p>
<p>Participants have typically requested more practical exercises, discipline-specific cases, hands-on learning, and interactivity to concretise generic RDM principles to develop the training and deepen their competencies (<xref ref-type="bibr" rid="r3">Adamick et al., 2013</xref>; <xref ref-type="bibr" rid="r10">Byatt et al., 2013</xref>; <xref ref-type="bibr" rid="r15">Chew et al., 2021</xref>; <xref ref-type="bibr" rid="r43">Pascuzzi &#x0026; Sapp Nelson, 2018</xref>; <xref ref-type="bibr" rid="r77">Wiljes &#x0026; Cimiano, 2019</xref>). Nevertheless, fictitious cases not closely connected to participants&#x2019; own research have been stated as uninteresting in feedback (<xref ref-type="bibr" rid="r47">Peters &#x0026; Vaughn, 2014</xref>). Participants were most interested in learning more about data types and formats, archiving and long-term preservation, and metadata in the post-NECDMC workshop survey by <xref ref-type="bibr" rid="r47">Peters and Vaughn (2014)</xref>, as well as data sharing, IPR, and legal issues. Participants were also interested in gaining more information on metadata and data security issues in the post-course survey of the NECDMC application by <xref ref-type="bibr" rid="r38">Muilenburg et al. (2014)</xref>. In general, interactivity, discussion, peer supporting, and letting students apply generic principles in their own data are ways of concretising RDM (<xref ref-type="bibr" rid="r47">Peters &#x0026; Vaughn, 2014</xref>; <xref ref-type="bibr" rid="r55">Read et al., 2019</xref>; <xref ref-type="bibr" rid="r79">Wright &#x0026; Andrews, 2015</xref>).</p>
<p>Educational interventions in RDM are usually coordinated and led by libraries in academic institutions. Ideally, they begin with contextualising the education and determining the researchers&#x2019; practices and needs via interviews, surveys, work shadowing, or focus groups (<xref ref-type="bibr" rid="r31">Kafel et al., 2014</xref>; <xref ref-type="bibr" rid="r41">Oliver, 2017</xref>; <xref ref-type="bibr" rid="r50">Qin &#x0026; D&#x2019;ignazio, 2010</xref>). In some cases, there has been a multi-professional steering group or committee, under which library is leading, and usually also carrying out the implementation (<xref ref-type="bibr" rid="r31">Kafel et al., 2014</xref>; <xref ref-type="bibr" rid="r49">Piorun et al., 2012</xref>). The library has been the main, and many times, the only actor arranging and implementing education on RDM. However, in interviews and surveys with students and researchers, the fact that data management needs are unrestricted to informational and consulting services typically delivered by the library has become evident (<xref ref-type="bibr" rid="r30">Joo &#x0026; Peters, 2020</xref>; <xref ref-type="bibr" rid="r41">Oliver, 2017</xref>). Examples are creating RDM guidance, helping with data management plans, and planning and implementing education. Librarians may lack expertise in technical RDM assistance or using data science tools for data analysing, visualising, coding, cleaning, and database building (<xref ref-type="bibr" rid="r14">Cerny, 2021</xref>; <xref ref-type="bibr" rid="r54">Read, 2019</xref>). Librarians are not necessarily the best advisors on ethical and legal issues or safe and secure storage, either (<xref ref-type="bibr" rid="r14">Cerny, 2021</xref>; <xref ref-type="bibr" rid="r47">Peters &#x0026; Vaughn, 2014</xref>). Thus, many educators are planning an increased collaboration in data management training and support with researchers, libraries, research IT, legal services, research funding, and research offices (<xref ref-type="bibr" rid="r13">Castle, 2019</xref>; <xref ref-type="bibr" rid="r21">Cox &#x0026; Pinfield, 2014</xref>; <xref ref-type="bibr" rid="r30">Joo &#x0026; Peters, 2020</xref>; <xref ref-type="bibr" rid="r34">Latham, 2017</xref>; <xref ref-type="bibr" rid="r41">Oliver, 2017</xref>; <xref ref-type="bibr" rid="r47">Peters &#x0026; Vaughn, 2014</xref>; <xref ref-type="bibr" rid="r54">Read, 2019</xref>; <xref ref-type="bibr" rid="r57">Revez, 2018</xref>; <xref ref-type="bibr" rid="r70">Verbaan &#x0026; Cox, 2014</xref>; <xref ref-type="bibr" rid="r78">Wittenberg &#x0026; Elings, 2017</xref>; <xref ref-type="bibr" rid="r80">Yu, 2017</xref>).</p>
</sec>
</sec>
<sec id="s3">
<title>3. Methods</title>
<sec id="s3a">
<title>3.1. Course Backgrounds</title>
<p>Our research goal is to find how generic, multi-stakeholder training can improve participants&#x2019; competencies and further comprehension of the relevance of sound research data management practices regarding the quality and integrity of data and reliability of the research. The methods section will describe how we aimed at these goals with the versatile expertise of the course designers and teachers, as well as the learning objectives, course structure, and contents. We will also describe how we analysed the results of the training.</p>
<p>The analysed BRDM course was developed and implemented at the University of Turku (UTU), the third-largest research-intensive university in southwestern Finland with eight faculties, five independent units, and 21,000 students including 2,000 doctoral students and 3,300 staff members. The data policy of UTU (2016) motivated the planning of the studied course, according to which researchers would be offered training and support for writing DMPs and data managing during the project&#x2019;s lifecycle. The OpenUTU project group, containing members from the research office, library, research computing services, legal affairs, and communications unit of UTU, created the data policy. The library was responsible for creating and coordinating trainings and support in RDM for researchers. Because developing education for all researchers was impossible, the head of library services (the author) suggested starting with doctoral students (DSs) and postdoc researchers (PdRs) in a prime position to learn sound RDM practices from the beginning of their career. The author interviewed 35 doctoral students, supervisors, and biostatisticians in UTU to learn the perceived importance of RDM competencies and doctoral students&#x2019; current competencies (<xref ref-type="bibr" rid="r51">Rantasaari, 2021</xref>; <xref ref-type="bibr" rid="r52">Rantasaari &#x0026; Kokkinen, 2019</xref>). Data management planning, documentation of data processing, and managing IPR and contract issues contained the most profound skills gaps. However, participants also lacked knowledge of different issues throughout the data lifecycle. Therefore, the author, with the leader of UTU&#x2019;s biostatistician team, set up a working group and invited researcher-teachers from different faculties, a grant writer, data librarians, lawyers, a data security officer, and an IT computing specialist to plan and teach a course on RDM for DSs and PdRs. In 2020 we extended the course to Turku&#x2019;s other university &#x2013; &#x00C5;bo Akademi University (&#x00C5;AU) &#x2013; the only Swedish language multi-faculty university with 5,500 students and 1,100 staff members in Finland and with whom UTU has a long tradition of joint projects.</p>
</sec>
<sec id="s3b">
<title>3.2. Learning Objective, Course Structure and Data Management Plans in BRDM</title>
<p>A participant&#x2019;s learning objective was to familiarise themselves with RDM&#x2019;s central concepts and develop a high-class research plan and data management plan (DMP). After completing the course, a participant comprehends the significance of well-documented FAIR data for the ongoing study and other potential use and users, applying safe and secure practices in collecting, producing, handling, storing, sharing, and preserving the data, and acknowledging IPR, privacy, and sensitivity considerations when needed.<xref ref-type="fn" rid="fn3">3</xref></p>
<p>Though BRDM is a generic and introductory course, we separated the course for different tracks. The preliminary idea behind the track-based division was that the data management actions needed and applied depend partly on the type of the data, partly on research methods, and partly on discipline (<xref ref-type="bibr" rid="r4">Aker &#x0026; Doty, 2013</xref>; <xref ref-type="bibr" rid="r30">Joo &#x0026; Peters, 2020</xref>; <xref ref-type="bibr" rid="r35">Lefebvre et al., 2018</xref>; <xref ref-type="bibr" rid="r60">Scholtens et al., 2019</xref>; <xref ref-type="bibr" rid="r73">Weller &#x0026; Monroe-Gulick, 2014</xref>). These underlying factors delineate what kind of contracts, usage rights, storing solutions, processing, reuse, and preserving is needed or possible. For example, the methods in the clinical health sciences are usually experimental or observational<xref ref-type="fn" rid="fn4">4</xref>; data are often identifiable, confidential, and highly sensitive. In the natural sciences, methods are typically experimental, observational, or simulation-based<xref ref-type="fn" rid="fn5">5</xref>; data is largely not confidential and sensitive. However, there can be other rigorous demands for handling, storing, and preserving large data sets. In survey and qualitative research, the data and its needed and possible actions can vary greatly, depending partly on discipline and partly on each respondent&#x2019;s or interviewee&#x2019;s answers, the study subject&#x2019;s activity, and so forth.</p>
<p>In 2019, the first year, the BRDM course consisted of three tracks (Clinical Health Sciences, Survey Research, and Natural Sciences), with seven face-to-face modules in Finnish for DSs and PdRs at UTU. In each track, participants were to prepare a shared DMP together during the course. A DMP was based on a fictitious research plan delivered by the faculty teacher-researchers in Module One. The participants learned by familiarising themselves with pre-class materials and preparing assignments on Moodle, after which they attended a lecture on the module.</p>
<p>In 2020, the course began with a joint introductory lecture with all the four tracks (Clinical Health Sciences, Survey Research, Interview Research, and Natural Sciences). The course was developed for DSs and PdRs of UTU and &#x00C5;AU. Clinical Health Sciences and Survey Research tracks were held in Finnish, whereas Interview Research and Natural Sciences were held in English. The course was turned fully online via Moodle after the three first modules because of the COVID-19. Instead of preparing a fictitious research plan and DMP, everyone created their own research plan and a DMP. Course modules were linked by mapping each module with the sub-section(s) of the General Finnish DMP template<xref ref-type="fn" rid="fn6">6</xref> and adding an assignment to prepare and update a relevant section of the DMP before and after each module&#x2019;s workshop session. The last assignment was to return the DMP and give an anonymous peer review of another participant&#x2019;s DMP. Finally, the author of this article assessed and rated each DMP and gave a general level feedback of all the DMPs using Finnish DMP Evaluation Guidance (FDEG) (<xref ref-type="bibr" rid="r1">Aalto et al. 2021</xref>). Otherwise, the learning followed the 2019 pattern, consisting of pre-class activities followed by a lecture on Zoom.</p>
<p>In 2021, the course was online from beginning to end and adapted a flipped classroom method for teaching. The course continued with the same four track structure used in 2020 except the Interview Research track was turned to the Qualitative Research track. In each module, participants introduced themselves with the modules&#x2019; pre-class materials in Moodle and drafted a relevant section of their DMP for themselves. The participants also added questions to the discussion forum based on the pre-class materials and their own data. After pre-class activities, the module&#x2019;s Zoom workshop session was reserved for discussion based on the questions that participants had written beforehand or asked during the workshop. As in 2020, the modules&#x2019; post-class assignment was to update a DMP&#x2019;s relevant section, informed by the discussion in the modules&#x2019; workshop. Each participant returned their DMP and peer-reviewed another participant&#x2019;s DMP as a final assignment for the course, after which the author assessed and scored the DMPs (<xref ref-type="table" rid="tb001">Table 1</xref>).</p>
<table-wrap id="tb001">
<label>Table 1:</label>
<caption><p>The structure, contents, and responsible teachers of the four-track BRDM course.</p></caption>
<table frame="hsides" rules="groups">
<tbody>
<tr>
<td align="left" valign="top"><img src="figures/LIBER_2022_32_Rantasaari_fx1.jpg"/></td>
</tr>
</tbody>
</table>
</table-wrap>
</sec>
<sec id="s3c">
<title>3.3. Formative Assessment: Feedback</title>
<p>Following each module, participants were asked to give formal feedback through an online form (<xref ref-type="app" rid="app2">Appendix B</xref>). Module-based feedback was used as a formative assessment to control the participants&#x2019; learning, receive information on experienced challenges, and collect proposals for improving the course. Hence, feedback produced ongoing information for the teachers to edit and enhance the modules and their contents. Moreover, halfway through the course, the author compiled the feedback with answers and information about remedies that were made or would be made.</p>
<p>The author used the grounded theory-inspired approach to code and analyse the feedback for this study (<xref ref-type="bibr" rid="r9">Bryant &#x0026; Charmaz, 2016</xref>; <xref ref-type="bibr" rid="r12">Cassidy, 2012</xref>; <xref ref-type="bibr" rid="r68">Timonen et al., 2018</xref>). Sub-categories were created based on the topics that emerged from the coded comments. Grounded theory as an analysing approach is well-suited for processing and analysing the feedback as the aim was to let the feedback data speak for itself and not use an existing theoretical framework and categories formulated according to the framework.</p>
</sec>
<sec id="s3d">
<title>3.4. Summative Assessment: Survey</title>
<p>The participants were asked to participate in a survey to self-rate their competencies before and after the course. In 2019, the survey was carried out twice &#x2013; before and after the course &#x2013; on a scale from 1 to 5: 1 &#x003D; no competence, 2 &#x003D; some competence, 3 &#x003D; good competence, 4 &#x003D; very good competence, 5 &#x003D; top competence (<xref ref-type="app" rid="app3">Appendix C</xref>). In 2020&#x2013;2021, we performed a post-course survey in which participants were asked to self-rate their competencies before and after the course on a scale from 1 to 4: 1 &#x003D; no competence, 2 &#x003D; little competence, 3 &#x003D; somewhat competent, 4 &#x003D; very competent (<xref ref-type="app" rid="app4">Appendix D</xref>). Participants were also asked to give a course rating from 1&#x2013;100, if they would recommend the course to other DSs and PdRs, and choose the topics about which they would like to have more education.</p>
<p>The survey served as a summative assessment for collecting participants&#x2019; perceptions of their learning, the quality of the course, and further education needs.</p>
<p>The respondents&#x2019; self-ratings of their competencies were analysed using JMP Pro 16 to produce descriptive and inferential statistics with medians, custom quantiles, Wilcoxon signed-rank test (one group), Wilcoxon rank-sum test (two independent groups), and Steel-Dwass test (multiple comparisons). Frequencies and Chi-square test were used for announcing further learning needs. A significance level of 0.05 (two-tailed) was used. Also, module- and course-based feedback comments were coded and categorised in NVivo 12.</p>
</sec>
<sec id="s3e">
<title>3.5. Summative Assessment: Data Management Plans</title>
<p>In Module One, each participant created their own research plan. During the course and based on their research plan, they wrote a DMP using a Finnish General DMP template and guidance. The course participants&#x2019; DMPs will be analysed in a later study.</p>
</sec>
</sec>
<sec id="s4">
<title>4. Results</title>
<sec id="s4a">
<title>4.1. Participants</title>
<p>Of the 386 enrolled participants in 2019&#x2013;21, 346 (90%) were DSs, 37 (10%) were PdRs, and 3 (1%) were university employees. Of those who completed the full course with 3 ECTS credits, 154 (91%) were DSs, 14 (8%) were PdRs, and 1 (1%) was a university employee. Of the participants who did not complete the full course but (on average) half the modules, 72 (80%) were DSs, 17 (19%) were PdRs, and 1 (1%) was a university employee. In 2019, participants who did not complete the full course performed (on average) 3 out of 7 modules; in 2020, 3 out of 8; and in 2021, 4 out of 8. Performing only part of the modules does not mean that participants interrupted the course but that the modules were performed evenly between modules 0 (introduction) and 8 (final assignment). PdRs, in particular, picked modules according to their interests, without needing to earn the 3 ECTS credits (<xref ref-type="table" rid="tb002">Table 2</xref>).</p>
<table-wrap id="tb002">
<label>Table 2:</label>
<caption><p>Enrolled, fully completed, and (on average) approximately half the modules performed in 2019&#x2013;2021 courses.</p></caption>
<table frame="hsides" rules="groups">
<tbody>
<tr>
<td align="left" valign="top"><img src="figures/LIBER_2022_32_Rantasaari_fx2.jpg"/></td>
</tr>
</tbody>
</table>
</table-wrap>
<p>The largest disciplines represented regarding the number of participants in 2019&#x2013;2021 (n &#x003D; 259) were Health Sciences with 77 (30%) participants; Social, Business and Economics with 66 (26%) participants; and Science and Engineering with 63 (24%) participants. Markedly fewer participants came from Humanities with 32 (12%) participants; Education with 7 (7%) participants; and Law with only 2 (1%) participants. (<xref ref-type="fig" rid="fg002">Figure 2</xref>; <xref ref-type="table" rid="tb004">Table 2</xref> in <xref ref-type="app" rid="app1">Appendix A</xref>).</p>
<fig id="fg002">
<label>Fig. 2:</label>
<caption><p>The number of all participants by discipline in 2019&#x2013;2021 courses (n &#x003D; 259).</p></caption>
<graphic xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="figures/LIBER_2022_32_Rantasaari_fig2.jpg"/>
</fig>
</sec>
<sec id="s4b">
<title>4.2. Feedback</title>
<p>We asked participants to fill in a feedback form after each module on the Moodle course platform. In 2019, the module-based feedback was a mandatory course assignment. This task was voluntary in 2020 and 2021, echoed in the number of feedback forms we received: 133 forms in 2019, 114 in 2020, and 69 in 2021. In 2019 and 2020, participants were given a time slot at the end of the classes or workshops to provide feedback; in 2021, we simply reminded participants to give feedback after the live Zoom workshop sessions.</p>
<p>The feedback form contained three main categories:</p>
<list list-type="order">
<list-item><p>What are the three things you have learned&#x003F;</p></list-item>
<list-item><p>How will the things you have learned change your practices&#x003F;</p></list-item>
<list-item><p>How would you suggest the module be developed&#x003F;</p></list-item>
</list>
<p>Under these main categories, the author created sub-categories and sorted the comments using a grounded theory-inspired approach. The five biggest sub-categories stand for 90 to 100% of all comments in the main categories (<xref ref-type="fig" rid="fg003">Figures 3</xref> to <xref ref-type="fig" rid="fg005">5</xref>; <xref ref-type="table" rid="tb005">Tables 3</xref> to <xref ref-type="table" rid="tb007">5</xref> in <xref ref-type="app" rid="app1">Appendix A</xref>). Because a respondent&#x2019;s comment in a feedback form could include several aspects, it could be placed accordingly in two or more sub-categories. For example, the comment &#x201C;I am now more aware of IPR issues and GDPR, which enables me to plan my next research in more detail&#x201D; has been placed in the sub-categories &#x201C;I will pay notice to IPR, agreements and licenses&#x201D;, and &#x201C;I will pay more notice to data privacy and data security&#x201D;. Hence, the total number of comments in different sub-categories is bigger than the number in the original four main categories.</p>
<fig id="fg003">
<label>Fig. 3:</label>
<caption><p>Five top sub-categories based on the feedback given in the main category &#x201C;What are the three things you have learned&#x201D; in 2019&#x2013;2021&#x003F;&#x201D;.</p></caption>
<graphic xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="figures/LIBER_2022_32_Rantasaari_fig3.jpg"/>
</fig>
<fig id="fg004">
<label>Fig. 4:</label>
<caption><p>Five top sub-categories based on the feedback comments given in the main category &#x201C;How will the things you have learned change your practices&#x003F;&#x201D;.</p></caption>
<graphic xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="figures/LIBER_2022_32_Rantasaari_fig4.jpg"/>
</fig>
<fig id="fg005">
<label>Fig. 5:</label>
<caption><p>Seven top sub-categories based on the feedback comments given in the main category &#x201C;How would you suggest the module be developed&#x003F;&#x201D;.</p></caption>
<graphic xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="figures/LIBER_2022_32_Rantasaari_fig5.jpg"/>
</fig>
<sec id="s4b1">
<title>4.2.1. What are the Three Things you have Learned&#x003F;</title>
<p>Based on the feedback given in this main category, we identified 252 separate comments in 2019, 194 in 2020, and 119 in 2021. The sub-category &#x201C;What, why and when in RDM&#x201D; contains comments of RDM essentials such as the learning of the rationale and tools to plan data management, the central concepts of RDM, and the different phases such as storing, documenting, preserving, and sharing of data:</p>
<disp-quote>
<p><italic>&#x201C;(I learned) ways to conceptually approach Research Data Management and the practices and perspectives related to it.&#x201D;</italic> (Module 2, Qualitative Research, ID 16/2021).</p>
</disp-quote>
<p>The number of the comments concerning data management planning and documentation grew in 2020 and 2021:</p>
<disp-quote>
<p><italic>&#x201C;Documentation of data in a clear and readable form is a crucial step in the data management and processing.&#x201D;</italic> (Module 5, Natural Sciences, ID 91/2020).</p>
</disp-quote>
<p>At the same time, the percentage of the comments belonging to the sub-category &#x201C;The importance of legal considerations&#x201D; dropped from 25% in 2019 to 8% in 2021. (<xref ref-type="fig" rid="fg003">Figure 3</xref>; <xref ref-type="table" rid="tb005">Table 3</xref> in <xref ref-type="app" rid="app1">Appendix A</xref>).</p>
</sec>
<sec id="s4b2">
<title>4.2.2. How will the Things you have Learned Change your Practices&#x003F;</title>
<p>Based on the feedback given in this main category, we identified 87 comments in 2019, 132 in 2020, and 63 in 2021. The comments concerning improving data management planning and documenting practices increased from 21% to 49%, whereas the comments about the intention to focus more on IPR, agreements, and license issues decreased from 29% to 8%. The following quotation illustrates the increased number of comments concerning documentation&#x2019;s importance:</p>
<disp-quote>
<p><italic>&#x201C;I learned a lot about the importance of documentation and metadata as well as publishing datasets. I will apply the FAIR principles when my research work needs to be checked and will review the data management all the time.&#x201D; (Module 7, Qualitative Research, ID 68/2021</italic>).</p>
</disp-quote>
<p>At the same time, the percentage of data privacy comments (e.g., data privacy notice, informed consent, and GDPR), and data security comments (safe and secure data storing platforms), increased from 15% to 25%. (<xref ref-type="fig" rid="fg004">Figure 4</xref>; <xref ref-type="table" rid="tb006">Table 4</xref> in <xref ref-type="app" rid="app1">Appendix A</xref>).</p>
</sec>
<sec id="s4b3">
<title>4.2.3. How would you Suggest the Module be Developed&#x003F;</title>
<p>We identified 90 (2019), 136 (2020), and 101 (2021) proposals to develop the modules. Most of the respondents wished for practicality such as more discipline-specific instruction, checklists, and cases, along with the clarification and standardisation of course practicalities, schedules, and course platforms, and how to balance the workload between different modules. More practicality and concreteness were desired, especially in law-related modules three and four:</p>
<disp-quote>
<p><italic>&#x201C;All the law-related sections could explain things in less of a law-speech manner as law speech is generally really vague and does not provide any practical knowledge. In general, the wideness of topics was really good.&#x201D; (Post-Course Survey, Natural Sciences, ID 53/2021</italic>)</p>
</disp-quote>
<p>In 2019 and 2020 (but not in 2021), many comments expressed a desire for more interactivity and discussions. Unlike in 2020&#x2013;2021 courses, participants preparing their own research plan and DMP in 2019 &#x2013; visible in the 7 (8%) comments &#x2013; was impossible. For the first time in 2021, we received 12 (12%) answers that the module is good as is. (<xref ref-type="fig" rid="fg005">Figure 5</xref>; <xref ref-type="table" rid="tb007">Table 5</xref> in <xref ref-type="app" rid="app1">Appendix A</xref>).</p>
</sec>
<sec id="s4b4">
<title>4.2.4. The Overall Score</title>
<p>In the post-course surveys after the 2020&#x2013;2021 courses, participants were asked to score the course between 0 and 100. After the 2021 course, participants were also asked if they would recommend the training to other DSs or PdRs. Based on the survey respondents&#x2019; general score of 68 out of 100 in 2020 (n &#x003D; 53) and 74 out of 100 in 2021 (n &#x003D; 64), the course lived up to the reasonable expectations of a general level introductory education. Equally, 92% of the post-course survey respondents in 2021 expressed they would recommend the course to other DSs and PdRs.</p>
</sec>
</sec>
<sec id="s4c">
<title>4.3. Surveys 2019&#x2013;21: Competencies in RDM before and after the Course</title>
<sec id="s4c1">
<title>4.3.1. BRDM 2019</title>
<p>Participants were asked to rate their current RDM competencies on a five-point scale from 1 to 5 before and after the 2019 course (<xref ref-type="app" rid="app3">Appendix C</xref>). Hence, 45 (82%) enrolees answered the pre-course survey, and 17 (41%) of those who completed at least part of the modules answered the post-course survey. Before the course, participants&#x2019; median self-rating of their RDM competence was 1.96 (Q1:1.82, Q3:2.09). After the course, participants&#x2019; median self-rating of their competence was 2.32 (Q1:2.12, Q3:2.84). The improvement was statistically significant, p &#x003D; 0.003 (Wilcoxon rank-sum test), or 0.36 points. (<xref ref-type="fig" rid="fg006">Figure 6</xref>; <xref ref-type="table" rid="tb008">Table 6</xref> in <xref ref-type="app" rid="app1">Appendix A</xref>).</p>
<fig id="fg006">
<label>Fig. 6:</label>
<caption><p>Respondents&#x2019; median self-ratings of their competencies before and after the BRDM 2019 course.</p></caption>
<graphic xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="figures/LIBER_2022_32_Rantasaari_fig6.jpg"/>
</fig>
</sec>
<sec id="s4c2">
<title>4.3.2. BRDM 2020&#x2013;2021</title>
<p>The surveys in 2020 and 2021 (<xref ref-type="app" rid="app4">Appendix D</xref>) differed from the 2019 survey related to the contents and execution:</p>
<list list-type="bullet">
<list-item><p>Instead of pre- and post-course surveys, we only had a post-course survey.</p></list-item>
<list-item><p>The competencies were specified to respond more closely to the learning objectives of the modules (<xref ref-type="bibr" rid="r53">Rantasaari et al. 2021</xref>).</p></list-item>
<list-item><p>The scale was 1 to 4 instead of 1 to 5.</p></list-item>
</list>
<p>The combined response rate to the surveys was 49% (106 respondents out of 217 participants) after the 2020&#x2013;2021 courses. On the 1 to 4 scale, the median self-rated competence before and after the courses was 1.97 and 3.03, respectively. Thus, the median self-rated competencies improved statistically highly significantly, p &#x003C; 0.0001 (Wilcoxon signed-rank test), or 1.06 points (<xref ref-type="fig" rid="fg007">Figure 7</xref>; <xref ref-type="table" rid="tb009">Table 7</xref> in <xref ref-type="app" rid="app1">Appendix A</xref>).</p>
<fig id="fg007">
<label>Fig. 7:</label>
<caption><p>Based on median self-ratings, respondents&#x2019; competencies related to the specified learning objectives before and after the BRDM 2020&#x2013;2021 courses. Light blue bars represent the competencies before the courses and dark blue bars after the courses. Full descriptions of the learning objectives can be found in the survey form (<xref ref-type="app" rid="app4">Appendix D</xref>).</p></caption>
<graphic xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="figures/LIBER_2022_32_Rantasaari_fig7.jpg"/>
</fig>
<p>Regarding the variance in the results between disciplines and course tracks, differences were statistically insignificant at the level of total medians, although some were found concerning a few specific competencies before the course. First, respondents in the &#x201C;Qualitative Research&#x201D; track and the &#x201C;Humanities, psychology, and theology&#x201D; discipline self-rated their competence higher than those in the &#x201C;Clinical Health Sciences&#x201D; track in identifying the data life cycle and recognising a DMP&#x2019;s components (p &#x003D; 0.02, Steel-Dwass). Second, respondents in the &#x201C;Qualitative Research&#x201D; track and the &#x201C;Humanities, Psychology, and Theology&#x201D; and the &#x201C;Social Sciences, Business, and Economics&#x201D; disciplines self-rated their competence in applying anonymisation higher than those in the &#x201C;Natural Sciences&#x201D; track (p &#x003D; 0.01, Steel-Dwass) and the &#x201C;Science and Engineering&#x201D; discipline (p &#x003D; 0.02, Steel-Dwass). Third, respondents in the &#x201C;Social Sciences, Business, and Economics&#x201D; discipline and the &#x201C;Qualitative Research&#x201D; track self-rated their competence in applying data privacy higher than those in the &#x201C;Science and Engineering&#x201D; discipline and the &#x201C;Natural Sciences&#x201D; track (p &#x003D; 0.02, Steel-Dwass). All differences after the course were insignificant.</p>
</sec>
</sec>
<sec id="s4d">
<title>4.4. Subjective Educational Needs in 2020&#x2013;2021: What would you like to Learn more about&#x003F;</title>
<p>Participants were asked to choose the topics they wanted to learn more about in the post-course surveys. As much as 102 respondents (96%) expressed interest in advanced training. Six topics receiving over half (261) of all mentions (471) were &#x201C;Metadata and description&#x201D; (55), &#x201C;Discipline-specific cultures&#x201D; (44), &#x201C;Backup, version management, storage&#x201D; (42), &#x201C;Ethics and legal considerations&#x201D; (40), &#x201C;Quality and documentation&#x201D; (40), and &#x201C;Visualisation and representation&#x201D; (40). However, interest for advanced training in &#x201C;Discovery and acquisition&#x201D; (21) and &#x201C;Data curation and reuse&#x201D; (26) were the lowest. Differences were statistically insignificant related to respondents&#x2019; discipline or course track. The frequencies of mentions for further learning needs are illustrated in <xref ref-type="fig" rid="fg008">Figure 8</xref>.</p>
<fig id="fg008">
<label>Fig. 8:</label>
<caption><p>The topics respondents would like to learn more about.</p></caption>
<graphic xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="figures/LIBER_2022_32_Rantasaari_fig8.jpg"/>
</fig>
</sec>
</sec>
<sec id="s5">
<title>5. Discussion and Conclusions</title>
<sec id="s5a">
<title>5.1. How did the Course Succeed&#x003F;</title>
<p>In this article, our goal was to find how generic, multi-stakeholder training could improve participants&#x2019; competencies and further comprehension of the relevance of sound research data management practices to the quality and integrity of data and reliability of the research. Furthermore, the questions we aimed to answer were as follows: RQ1) How did the course participants self-rate their RDM competencies before and after the course&#x003F; RQ2) What kind of educational impact did the course have on participants&#x2019; RDM competencies (knowledge, skills, and abilities) based on participants&#x2019; self-ratings and the collected and categorised feedback&#x003F; RQ3) What further learning needs did the respondents express after the course&#x003F; These questions will be discussed with the help of the criteria for successful training as created by <xref ref-type="bibr" rid="r42">Oo et al. (2021)</xref>.</p>
<p>Based on the systematic review of 28 RDM trainings between 2012 and 2019, <xref ref-type="bibr" rid="r42">Oo et al. (2021)</xref> introduced a four-part criterion for successful training consisting of 1) active participation during training; 2) demand for RDM training; 3) increased participants&#x2019; knowledge and understanding of RDM and confidence in enacting RDM practices; and 4) positive post-training feedback. How BRDM matched these criteria will be discussed below.</p>
<p>Concerning the participation during training, BRDM was based on active learning: Participants read and listened to course materials, completed assignments, developed their own research plan and a DMP, peer-reviewed each other&#x2019;s DMP, drafted questions based on course materials and their own data management issues, and participated in the workshop discussions. The activities sought to help participants link the principles and other theoretical contents to their research practices (see also <xref ref-type="bibr" rid="r66">Thielen et al., 2017</xref>; <xref ref-type="bibr" rid="r74">Whitmire, 2015</xref>; <xref ref-type="bibr" rid="r77">Wiljes &#x0026; Cimiano, 2019</xref>; <xref ref-type="bibr" rid="r78">Wittenberg &#x0026; Elings, 2017</xref>). Judging by the feedback during and after the 2021 course, we succeeded in bringing interactivity and discussion to modules. However, there was still a demand for turning, especially legal and data privacy principles and regulations, into concrete instructions, cases, and examples when possible.</p>
<p>After completing the course, almost all respondents expressed interest in further education for RDM training. The most frequently mentioned topics for further learning were &#x201C;Metadata and description&#x201D;, &#x201C;Discipline-specific cultures&#x201D;, &#x201C;Backup, version management, storage&#x201D;, &#x201C;Ethics and legal considerations&#x201D;, &#x201C;Quality and documentation&#x201D;, and &#x201C;Visualisation and representation&#x201D;. Metadata, ethics, and legal issues were also the most wanted topics for continued learning in the courses of <xref ref-type="bibr" rid="r38">Muilenburg et al. (2014)</xref> and <xref ref-type="bibr" rid="r47">Peters and Vaughn (2014)</xref>. Conversely, despite emphasising FAIR principles, as well as data sharing and reuse throughout BRDM, advanced training in &#x201C;Discovery and acquisition&#x201D; and &#x201C;Data curation and reuse&#x201D; were the least preferred topics. Minor interest in these might be comprehensible concerning cultures of practices in many disciplines where researchers&#x2019; primary interest is getting their current project through and obtaining results from the data rather than long-term preservation and the possible data reuse in future projects (<xref ref-type="bibr" rid="r32">Kowalczyk, 2017</xref>; <xref ref-type="bibr" rid="r51">Rantasaari, 2021</xref>).</p>
<p>Concerning increased knowledge, understanding, and confidence in enacting RDM practices, participants highlighted that they had learned RDM essentials such as understanding the rationale and learning the tools to plan data management, RDM&#x2019;s central concepts, and storing, documenting, preserving, and sharing data. Moreover, participants learned legal and data privacy issues and how to use REDCap and NVivo in data collecting and organising. Correspondingly, they reported that the training would change their current practices in planning research projects, managing and documenting data, acknowledging legal and data privacy viewpoints, and using REDCap and NVivo in data collecting and organising. The median self-rated improvement in RDM competencies was 0.36 points in 2019 and 1.06 in 2020&#x2013;2021 &#x2013; one level up from &#x201C;little competence&#x201D; to &#x201C;somewhat competent&#x201D;. One-step improvement during a generic RDM course is a typical result that has been documented in several post-course surveys (e.g., <xref ref-type="bibr" rid="r50">Qin &#x0026; D&#x2019;ignazio, 2010</xref>; <xref ref-type="bibr" rid="r79">Wright &#x0026; Andrews, 2015</xref>).</p>
<p>As far as respondents&#x2019; disciplines or course tracks are concerned, the differences were statistically insignificant at the level of total medians. However, some significant differences were found concerning a few specific competencies before the course. Respondents in the &#x201C;Qualitative Research&#x201D; track and the &#x201C;Humanities, Psychology, and Theology&#x201D; and &#x201C;Social Sciences, Business, and Economics&#x201D; disciplines self-rated their anonymisation competencies before the course as higher than those in the &#x201C;Natural Sciences&#x201D; track and the &#x201C;Science and Engineering&#x201D; discipline. Likewise, respondents in the &#x201C;Social Sciences, Business, and Economics&#x201D; discipline and the &#x201C;Qualitative Research&#x201D; track self-rated their data privacy management competencies before the course as higher than those in the &#x201C;Natural Sciences&#x201D; track and the &#x201C;Science and Engineering&#x201D; discipline. These differences are comprehensible because data in qualitative research and social sciences, more often than in the natural sciences and engineering, contain personal or even sensitive contents. However, the differences had disappeared after the course. This indicates that the course had bridged the gaps in respondents&#x2019; competencies regarding their disciplines and course tracks. On applying the RDM principles in participants&#x2019; own data management planning, the results of the assessment and rating of the returned DMPs in the BRDM 2020&#x2013;2022 courses will be reported in another upcoming article.</p>
<p>Pertaining to feedback, the course was perceived as a solid and important introduction to RDM&#x2019;s different aspects. Teachers &#x2013; including a grant writer, researchers, data librarians, lawyers, a data privacy officer, a data archive specialist, a biostatistician, and an IT professional &#x2013; were appreciated as real domain experts. However, regarding propositions for course development, respondents asked for a more down-to-earth approach, concretising, and examples, especially in legal and data privacy issues. Clarification of the course platform and course practicalities were also requested.</p>
<p>BRDM can be determined as one of the few trainings (so far) that meet all parts of the four-part criteria for successful training as defined by <xref ref-type="bibr" rid="r42">Oo et al. (2021)</xref>. However, because of BRDM&#x2019;s limited number of participants, we cannot generalise our study&#x2019;s results and the factors affecting them outside the studied group. Furthermore, we cannot know the long-term impact of the participants&#x2019; self-rated competencies on their RDM activities without follow-up. Still, 319 returned module-based feedback forms, and 168 survey responses revealed valuable, indicative information of doctoral students&#x2019; and postdoc researchers&#x2019; competencies, the impact of the education on competencies, and further learning needs in RDM.</p>
</sec>
<sec id="s5b">
<title>5.2. The Value of BRDM and Lessons Learned</title>
<p>BRDM is an educational effort bringing value to RDM training. So far, academic libraries have been the main, and many times the only, actor arranging and implementing education on RDM in research-intensive universities. As a further development need, educators have often mentioned a need for collaboration with multiple stakeholders (<xref ref-type="bibr" rid="r13">Castle, 2019</xref>; <xref ref-type="bibr" rid="r21">Cox &#x0026; Pinfield, 2014</xref>; <xref ref-type="bibr" rid="r30">Joo &#x0026; Peters, 2020</xref>; <xref ref-type="bibr" rid="r34">Latham, 2017</xref>; <xref ref-type="bibr" rid="r41">Oliver, 2017</xref>; <xref ref-type="bibr" rid="r47">Peters &#x0026; Vaughn, 2014</xref>; <xref ref-type="bibr" rid="r54">Read, 2019</xref>; <xref ref-type="bibr" rid="r57">Revez, 2018</xref>; <xref ref-type="bibr" rid="r70">Verbaan &#x0026; Cox, 2014</xref>; <xref ref-type="bibr" rid="r78">Wittenberg &#x0026; Elings, 2017</xref>; <xref ref-type="bibr" rid="r80">Yu, 2017</xref>). In BRDM, using versatile expertise in planning and teaching has been embedded from the beginning: Academic and research support experts planned and taught the course. Second, the contents of BRDM were wide-ranging containing most of the phases of data life cycle, beginning from the writing of a high-class research plan &#x2013; which makes this course unique &#x2013; to the sharing and long-term preservation of the data. However, limited resources excluded more technical data science contents such as analysing, visualising, cleaning, merging, and programming data. Third, participants applied sound RDM principles in their data management by writing a DMP during the course. Hence, assessing BRDM&#x2019;s results is based not only on the feedback and self-rating of the participants&#x2019; competencies with further learning needs (typical measures of success in many previous trainings) but the returned DMPs. Fourth, a flipped classroom approach that is rarely used as a teaching method in previous RDM training (<xref ref-type="bibr" rid="r28">Griffin, 2020</xref>; <xref ref-type="bibr" rid="r29">Johnston &#x0026; Jeffryes, 2015</xref>; <xref ref-type="bibr" rid="r37">Mithun &#x0026; Luo, 2020</xref>), was adapted in the BRDM 2021 course. Fifth, many previous RDM trainings have been criticised for inadequate reporting (<xref ref-type="bibr" rid="r27">Goben &#x0026; Griffin, 2019</xref>; <xref ref-type="bibr" rid="r45">Perrier et al., 2017</xref>). In this study, we aimed for extensive and precise reporting.</p>
<p>Next, we will present some concrete lessons that we have learned during planning, implementing, and analysing the results of BRDM in 2019&#x2013;2021.</p>
<p>Planning and implementing training with multiple RDM stakeholders enable acknowledging all relevant aspects of the data life cycle. Participants will get an overall view of the numerous factors affecting RDM, while stakeholders&#x2019; overall understanding of the RDM and the challenges doctoral students and postdoc researchers confront increase. The downside of a large working group and many teachers is the administrative burden in coordinating the training. Moreover, the pedagogical skills of multi-professional specialists can be diverse. Therefore, keeping the training coherent by reaching a consensus on the learning objectives, teaching methods and contents, course practicalities, and deadlines with all the teachers and working groups is paramount.</p>
<p>RDM is an organic part of a research project &#x2013; from planning the goal and research questions and proceeding to the methods of collecting, producing, processing, storing, sharing, and preserving the data. Thus, recalling and updating or, preferably, rewriting a research plan is important when developing a DMP. Otherwise, research and data management plans can be asynchronous for example regarding data types to be collected, produced, and reused in a project.</p>
<p>Collecting feedback throughout training serves as a formative assessment to control the participants&#x2019; learning, receive information from experienced challenges, and gather proposals to quickly improve the training.</p>
<p>Though planning and implementing the flipped classroom approach takes a lot of work from teachers, it pays back by increasing flexibility and helping activate participants. Still, quizzes or follow-up tasks are essential to show that participants learned the pre-class materials, as <xref ref-type="bibr" rid="r37">Mithun and Luo (2020)</xref> have pointed out.</p>
<p>Measuring the learning results should not be based solely on the participants&#x2019; self-assessment or feedback, but on the assessment of assignments such as DMPs developed during training. Moreover, a follow-up intervention would be needed to collect empirical evidence on how the planned actions in DMPs have been applied in research practice (see also <xref ref-type="bibr" rid="r45">Perrier et al., 2017</xref>).</p>
<p>A modular structure enables cherry-picking the training, reducing the dropout rate. For example, PdRs do not necessarily need credits or certification by completing a training but want to bridge their knowledge gaps by choosing the modules that interest them.</p>
<p>Finally, according to this study and many others (e.g., <xref ref-type="bibr" rid="r15">Chew et al., 2021</xref>; <xref ref-type="bibr" rid="r43">Pascuzzi &#x0026; Nelson, 2018</xref>; <xref ref-type="bibr" rid="r77">Wiljes &#x0026; Cimiano, 2019</xref>), there is never too much IPA (Interaction, Practice, and Application) in training. Still, participants achieving excellent competence in a basic training are improbable. Instead, discipline, data type or research method specific workshops with fewer participants will help deepen the elementary skills (e.g., <xref ref-type="bibr" rid="r48">Petters et al., 2019</xref>; <xref ref-type="bibr" rid="r54">Read, 2019</xref>; <xref ref-type="bibr" rid="r67">Thielen &#x0026; Hess, 2017</xref>). However, as highlighted in research literature, training without synchronised incentives, policies, processes, and infrastructure is insufficient to bring about behavioural change (<xref ref-type="bibr" rid="r15">Chew et al., 2021</xref>; <xref ref-type="bibr" rid="r46">Perrier et al., 2020</xref>). A realistic target for a generic training could be that participants become aware of RDM and its contents and gain the elementary tools and basic skills to begin applying sound RDM practices in their research processes. Moreover, introducing participants to support services of multiple RDM stakeholders is important. That stakeholders learn what kind of challenges researchers and research students encounter when applying RDM is equally important.</p>
</sec>
</sec>
</body>
<back>
<ack>
<title>Acknowledgements</title>
<p>I would like to thank my supervisors, Professor Gunilla Wid&#x00E9;n at the &#x00C5;bo Akademi University and Professor Isto Huvila at the Uppsala University, who read and commented on several drafts of this study; Biostatistician Eliisa L&#x00F6;yttyniemi at the University of Turku for discussions on statistical analyses; and Information Specialist P&#x00E4;ivi Kanerva for acting as a responsible teacher of the BRDM course.</p>
</ack>
<sec>
<title>Availability of Data</title>
<p>The quantitative data underlying this study can be accessed through Zenodo: <ext-link ext-link-type="uri" xlink:href="https://doi.org/10.5281/zenodo.6526121">https://doi.org/10.5281/zenodo.6526121</ext-link>.</p>
</sec>
</back>
</article>
