The pure collected data in the database is just raw material. Metrics help to comprehend and visualize the collected data in a more user-friendly way. This makes the data more usable and gives a better insight into understanding the user base. The first thing, when thinking of creating the metrics, is to determine which information is actually required. Next, the collecting of the raw data for the metrics is introduced and after following the usage for a while, the first trends can be created from the metrics.

For the process of developing the metrics we used the Plan-Do-Check-Act model, which aims for a systematic process improvement with four simple steps. Despite critique, e.g. by Cole (2002) who claims that the PDCA-model is a bit too rigid for rapidly changing environments, we felt that a light-weight application of the model was suitable, because it gives a structure for the planning of the metrics and is quite straightforward to implement.

Sokovic, Pavletic and Kern (2010) explain the PDCA-model (Figure 1), which was originally developed by Edward W. Deming, by describing the phases:

An important part of the PDCA-process improvement is the idea of the development as a cycle. This aspect has important consequences for the usability of the method, and it allows stakeholders to get involved in different phases of the development.

In the case of the Centre for Preservation and Digitisation (CPD), the discussion surrounding the need of metrics started some time before the launch of the new version of the presentation system of the digitized collections. The initial need was to see how crowdsourcing attracts the interest of the users and what the general consequences of its implementation are. Then, interestingly, the metrics also appeared in the discussions about the project, where copyrighted materials were planned to become accessible for more users via negotiations with copyright organizations and news media companies — both of these parties were interested in the metrics, too. So, in a way, providing metrics could actually be of service both to the library and to the news media companies we collaborate with.

The aim was to develop only a few very basic metrics covering the quantity of clippings made over time: how many keywords are there, and to which materials in the clippings are they targeted to? Another aspect was also to follow the end-user feedback as it exists so that people can directly give input about the features and the contents, or even about extra wishes that they might have.

The initial planning gave enough input for developing the metrics. The main purpose was to first get the basic direction, and then to further develop the metrics. This is the process, which is also used in agile software development — first getting a basic understanding about the task at hand with the ­stakeholders and then evolving until the end result provides the highest possible gain with the minimum effort needed. After initial versions of the reports were developed (Do-phase), they were shown to the stakeholders and then developed further to answer their requirements (Check-phase). The scheduling of the reports was also considered, but at least for the time being the reports work in real-time. It seemed that it might be beneficial to develop the automation further, so that the report can be subscribed by anyone within the unit, who can then send the report to their email.

One of the positive things for the metric development was that a few of the core library metrics had already existed for a number of years, so the basic data collection features were already in place. Therefore, the largest part of the development could focus on the metrics themselves — extracting the data and developing comparisons and visualizations, which should be both functional and usable.

As typically happens in computer science or software development, when there is an interesting feature, the requirements for the metrics and reporting increase. From the original ideas and needs, a new set of metrics for crowdsourcing is formed. Interestingly, the content of the metrics is now starting to change from the library-oriented metrics towards the content metrics. Crowdsourcing also gives a new boost to the traditional content metrics, and illustrates how different kinds of materials are, and could be used by new types of users.

These new users actually want to participate, and make their own interpretations of the digitized collections provided (Bernstein, 2006). This can be considered as the first step towards the remix culture (Duncum, 2013), where existing works are taken and utilized either by mixing different materials together, or by utilizing existing content in completely different contexts.

The National Library of Finland has been enthusiastically experimenting with crowdsourcing activities from early on. In 2011, Digitalkoot 1.0 was launched, and with it the OCR recognition errors were targeted to be fixed with the help of users via a game (also known as the “molegame”) (Bremer-Laamanen, 2014). The system used the typical PBL gamification elements, meaning Points, Badges and Leaderboards. All-in-all, there were nearly 110 000 participants who completed 8 million fixing tasks in total (Digitalkoot, n.d.). Media visibility was also quite considerable, as the news of the crowdsourcing effort went viral and it was picked up by major news sources both globally and nationally by different media, such as The New York Times, Wired, Helsingin Sanomat. Throughout the game process the corrections were stored and even recently the words corrected were used to help in the analysis of the current OCR quality of the newspaper corpus (Kettunen et al., 2014).

The new #Digitalkoot crowdsourced project was launched on 13^th of May 2014. In this case, the aim of the project differed from the earlier project (gamified OCR correction of newspaper text). The idea was to let people use and collect material interesting for them via digital clippings. The clippings could basically be any content from the newspapers, journals or ephemera: for example advertisements, news stories or pictures.

What needs to be highlighted is the fact that making a clipping requires end-users to follow quite a rigorous process with specific steps. In the following section, the clipping workflow is described in order to explain the process, which has led to certain kinds of metrics to be collected.

One of the most basic metrics is the number of clippings that has been made by a user. This shows his/her overall interest toward the crowdsourcing capabilities but it can also show how this interest changes over time. For example, we hypothesized that in the beginning the crowdsourcing metrics would start slow, but that they would increase over time, when a) there would be returning users, who had formed a habit of the clipping creation or b) when the general knowledge and communication about the usefulness and the fun of digging through the newspapers and creating clippings would become apparent.

The benefits of this metric is that our presentation system http://digi.kansalliskirjasto.fi stores the created clippings, with the date of the creation and all related data. So, with a simple database query the data can be collected and analysed.

In the beginning we also considered whether the 1% rule of the internet culture would be visible in the NLF crowdsourcing effort. The 1% rule states that in any internet community there are 1% of creators (highly active users who participate wholeheartedly), 9% of commentators, and 90% of spectators (who use the system, but not in a deeper way). This 1% rule can be interpreted in different ways: is it the percentage of all the page visitors who have registered, or is it the percentage of registered users who become very active? We used the latter approach in our metrics in order to follow how the new functionalities impact individual users.

After six and twelve months the results of the clippings quantity by user were as described in Figure 3.

The purple line illustrates the situation after six months and the red line after twelve months. Points of interest are the most active users who have made nearly 1000 or nearly 3000 clippings respectively. This actually repeats the same finding as reported by Holley (2010a), that ‘super’ volunteers outshine the effort of others. So getting the few highly active users is very significant to the crowdsourcing as a whole in terms of quantity. The second point of interest is the midrange of users who have made a few hundred clippings. An interesting observation is that over the months this midrange has increased and expanded, which seems like a positive sign for the functionalities as a whole. Finally, the long tail of low-activity users is another main factor, and their amount has nearly tripled between the six month and the twelve month phase. There are a lot of users who might try the clipping once or twice: maybe they found the content they looked for and are happy with that, but they do not form a habit of using the crowdsourcing as such. The super users tend to follow a theme, when the single clipping makers might find one particular gem — but it requires a bit more research to see real patterns of usage.

Another metric that we considered in order to understand the different usage purposes of the materials is the keyword or tag usage. This was studied by simply counting and adding the keyword quantities across the date when the keywords were added, see Figure 4. In short, there are only a few very extensive clipping keyword clusters, but there is a long tail of more specific and rarer keywords also present here. The clusters of keywords tell about a specific theme that the users have created — it can either come from one or two active users, or it can be formed by several people who participate in the same theme.

In the keywords a blue dot represents a keyword in newspapers, red in journals and green in ephemera. As it can be seen from the graph, newspapers form a clear majority, superseding journals and ephemera. There is, however, also one interesting cluster for journals.

When checking the graph it is clear that there are some peaks for certain keywords. In December and January car related topics were searched, and they are highly visible in the keywords. A second main cluster is formed by the dance-theme: various dance events, dance schools that are visible in the digitized newspapers. The cluster of journals is mainly focused on word puzzles and quizzes, which were collected in August 2014. It is interesting to note that, like in the case of the tagging feature of the Historic Australian Newspapers (Holley, 2010b), also in our case the top users have made or added tags or keywords to nearly 1000 clippings for their theme of interest. As far as user amount goes, Australia and Finland are quite different due to the size of the language area (mainly Finnish and Swedish content compared to English), the profile of user interests seems to be in the same range even in this early phase of crowdsourcing.

There is also a constant usage of the keywords, so despite the fact that the users change and vary, they still follow the unwritten community rules and attempt to add at least one or a few keywords to the clippings they create. We have been happily surprised with the multitude of clippings and keywords made; people seem to be thoughtful with the keyword selection and it has highlighted the various user groups of the digitized collections. The clippings feature has given us insight into the users’ interests: there are local historians focusing on certain regions, theme-based users who collect material, e.g. from dancing or from technological advancements, and, expectedly, family and history researchers.

One of the ideas was also to compare the user-picked keywords to the full listing of the keywords provided by the Finnish Ontology, which the http://digi.nationallibrary.fi service uses and to see whether users use those or develop their own ones. We have also noticed that the Finnish generic ontology has been utilized quite well by users and that its keywords are used. Users have found in this ontology the required concepts and this has maintained a good structure of the keywords in the clippings. In addition to ontology words, end-users can also define their own keywords and therefore new concepts were also created. The ontology gives structure to the tagging, but it might still be that in the long run some clean-up of typos, or singular/plural form corrections might be needed for the clippings. Nevertheless, even now any user can fix erroneous tags, which gives them the opportunity to participate in the creation of larger clipping collections by creating, updating or deleting keywords. From the administration side, however, it would be nice to have a tool to do simple tag fixes for multiple clippings.

One thing on which we also kept an eye for was the user feedback that we received after launching the new version. At the beginning we decided to wait and see what kind of feedback we got, so that we could prepare the necessary user manuals, the frequently asked questions and the remaining appropriate help material.

The changed search facility was clearly one thing that has shown up in the feedback. There are those who really like the changes and those who were quite reserved towards the implemented changes. In order to help with the search a new help page and instructional videos were created and the feedback was answered by giving example custom searches that solved the particular questions of a user. In fact, after adding a link to the frequently asked questions (FAQ), the amount of search related questions has gone down, so probably the FAQ and the added help materials have been useful.

Another thing extracted from the feedback has been that some participants have contacted us and told us about the things they have looked for and why. This has partly been because they have a certain feature request in their mind for which they want to give us background information or because they would like to get access to more materials via the public web. Partially it has also been pure enthusiasm where they have wanted to share their findings or their work done with the library. The open website shows material until 1910 and when people doing research reach that year, they wonder where they should seek for additional material — one alternative being, e.g., the legal deposit libraries.