The DNB’s collection mandate covers all publications in written, visual and audio form that have been published in Germany, in German, as a translation from German, or about Germany since 1913. The media works that are subject to collection include all publications in physical form such as books, periodicals, newspapers, maps, music, standards, music recordings, and audio books. Since 2006, the collection mandate also includes publications without a physical medium, such as e-books, e-journals, e-papers, digital audio books, music publications, or websites.

The media works currently used for automatic subject cataloguing are those parts of the DNB collection that have machine readable text available, for example full texts or digital tables of contents, blurbs, or metadata such as the title of a publication. Image-based media or audio files are not included in the workflow. As many methods for automatic cataloguing are language-specific and therefore do not work with cross-lingual collections, the language in which a publication is written is a fundamental criterion for processing. Currently, German and English publications are processed. The publications’ full texts and tables of contents are available as EPUB or PDF files. They have to be converted into plain text in order to be readable for further processing steps.

The vocabularies for knowledge representation are used for both intellectual and automatic subject cataloguing. Therefore, all publications that have previously been catalogued intellectually can be utilised to form the gold standard for the purpose of training and evaluating automatic subject cataloguing processes. We use publications with at least one label of the DDC Subject Categories or regular DDC numbers (which can be used to establish a DDC Short Number if required) or a descriptor from the GND (Table 1).

To prepare the data for automatic subject cataloguing, it is necessary to analyse and to pre-process the data in multiple steps. How much of each kind of text is available within the gold standard? What kind of metadata is available to structure the collections for different use cases? In detail, the data management has to perform tasks such as processing the bibliographic metadata, extracting the text from PDF/EBUB to plain text, as well as data-cleaning and splitting.

In the DNB’s catalogue system, more than 49.7 million bibliographic records and approximately 10 million GND records are hosted and encoded in the internal format PICA+. PICA+ is a data format with multiple fields and subfields. To handle the large amount of data in an efficient way and to convert the needed information into data formats of lower complexity, we implemented the toolkit pica-rs (Wagner, 2024). It is written in Rust and enables very fast processing and conversion of PICA+ data to tabular representations. It also includes comprehensive functionalities to determine and process elementary and statistical values from the metadata. We use pica-rs as base software for data analysis and it forms the entry point for the automation of metadata workflows in our data management.

The requested texts are retrieved from the DNB repository and extracted from the PDF/EPUB format in plain text. Another step is data cleaning, such as separating out broken texts. Pre-processing steps, such as shortening full texts or adding a title to another kind of text, may be necessary (Serrano, 2021, p. 387). According to the classification and indexing use cases, the data must be compiled into a bundled dataset that can serve for training or evaluation. A feasible approach to validating machine learning systems is the three-way-hold-out method of splitting up the dataset into train, validate, and test corpora (https://huyenchip.com/machine-learning-systems-design/toc.html). However, splitting bibliographic records has its pitfalls: many publications arise as reprint or new edition of a previous work, others are parallel editions in print and digital representation. Inserting different manifestations of the same work into training and test data may result in unwanted bias during performance estimation.

Due to the large amount of data, it was necessary to set up a specialised management environment to meet the requirements of the DNB. One objective was to automate the underlying workflows in order to make their management traceable and less error-prone. The combination of pica-rs, Git (https://github.com/), and Data Version Control (https://dvc.org) as well as various Python scripts formed the base to set up a processing pipeline that makes it possible to select, structure and process the data.

The pipeline structure is described in the configuration files. The use of Git allows version control to manage source code and configuration files so that they can be reproduced at any time. The combination of Git with DVC as a workflow management and pipelining tool provides a complex toolset to handle metadata and large text files on a huge scale in a deterministic way. It establishes the possibility of sharing and synchronising pipelines and data within the team.

The DNB currently uses two main approaches for automatic classification and indexing. The first one is a supervised machine learning approach, also known as associative approach. Here, a model is trained using labelled data in order to make predictions (assign descriptors, subject categories, etc.) about unknown data. Labels, such as notations of a classification system or descriptors of a controlled vocabulary, that have been assigned intellectually to a publication, are learnt by an algorithm using the words (features) from the publication. The trained model is then applied to make predictions for a new publication. In this kind of supervised learning approach, only labels that occur during training can be predicted in productive use. This approach is suitable for notations and descriptors that have a high occurrence in the training data.

In the second approach, also known as the lexical approach, terms (words or parts of words) from a digital publication are compared with the terms (labels) of a vocabulary (e.g. the GND descriptors). If a term from the text and a term from a descriptor match, the descriptor can be assigned to the publication from which the term was taken. Unlike the supervised machine learning approach, the lexical approach does not require training material. It is also suitable to predict rare or previously unused descriptors.

Various machine learning and lexical approaches are available in the open-source toolbox Annif (Suominen et al., 2024). Machine learning approaches in Annif are TF-IDF, fastText, Omikuji and SVC. Lexical approaches in Annif are MLLM, STWFSA, and YAKE. To combine the advantages of the methods, there are three fusion/ensemble backends in which individual results from the methods can be merged together: ensemble, PAV and nn_ensemble (Inkinen, 2024, section Backends/Algorithms).

To decide whether a model performs well, it is necessary to measure its performance. A good level of quality must be guaranteed for productive models and configured methods in operation. The quality of a model is not always trivial to measure. Various metrics and appropriate tools for evaluation have to be used (Serrano, 2021, p. 177). Standard metrics are e.g. precision (usefulness of the result), recall (completeness of the result) (“Precision and recall”, 2024), F1 score (harmonic mean of Precision and Recall) (“F-Score”, 2024) and NDCG (“Normalised Discounted Cumulative Gain”, 2024). The Annif toolbox already includes an evaluation command that can be used to calculate various evaluation metrics such as precision, recall, and F1 score. In order to obtain more detailed information about the usefulness of the predictions and to continuously evaluate and improve the quality of the results, regular feedback from our subject cataloguing experts is an important component of the performance analysis. This also means that creating and improving the models is an ongoing process. Several times a year, newly trained models based on current vocabulary and training material are put into production.

The DDC Subject Categories were introduced in 2004 in order to structure the German National Bibliography. They are mostly based on the top two notational levels of the Dewey Decimal Classification (DDC). Since 2012, the machine-based assignment of DDC Subject Categories has been applied for daily incoming digital publications based on the full text and selected print publications based on the table of contents.

The results and findings from the automatic classification based on the DDC Subject Categories gave reason to examine whether this approach could also be suitable for complete DDC numbers. Because of the large amount of possible DDC numbers and the small amount of available training material for so many numbers, initial investigations did not lead us to satisfactory results. In order to still enable DDC numbers to be used in automated processes, the DDC Short Numbers were developed in 2015. These abbreviated DDC numbers are selected from sets of valid regular DDC numbers and represent specific topics with more general classes. Thus, DDC Short Numbers are the trade-off between broad subject categories and the full descriptive capability of the DDC classification system. The short numbers are selected intellectually by domain experts. Their selection is primarily oriented towards the library’s holdings and collection mandate on the basis of subject-specific aspects (Mödden, 2022).

Two separate models are trained for automatic classification using DDC Subject Categories, one for German-language publications and one for English-language publications. The model for the machine-based assignment of DDC Subject Categories to German-language publications is described in more detail below.

The first challenge on the way to create a model is to identify publications that can be used for training and testing the model. The publications for training must be written in German, have a digital kind of text (full text, table of contents, blurb or bibliographic metadata such as the title) and have to be classified intellectually into a DDC Subject Category. These publications form the gold standard.

After taking these criteria into account, in 2023 around 3.8 million objects were available as material for training and testing the models (see also Table 1).

For the development of a classification model, the set of full texts was randomly split into 85 percent for training, another 10 percent for testing and the remaining 5 percent for validation. Thus, they form disjoint sets. The individual DDC Subject Categories occur with varying frequencies in these text corpora, but the frequency distribution of the individual subject categories is approximately the same in all three text corpora types. The decision to include only full texts in the test and validation text corpora is based on the later intended primary use of the model for the classification of full texts.

However, there is one important constraint applied to the full text data. The processing of full text data which can often comprise several hundreds of thousands of characters can become a significant cost driver as it leads to an increasing computational and time-consuming effort. In order to keep these costs as minimal as possible without affecting the F1 score and other relevant metrics too much, many iterations of tests were performed. For this use case, the best balance between a reduced, fast processable full text and a sufficiently high F1 score that we found is to take the first 30,000 characters for further processing and cutting off the rest of the text.

Table 2 gives a first impression of how the individual categories are represented in the training text corpora. It contains an overview of the five largest and the five smallest subject categories. As can be seen, a lot of training data is available for the subject categories “Medicine” and “Law”, while very little training data is available for “History of other regions” or “Manuscripts and rare books”. Depending on the machine learning method used, these large differences in quantity can have a major influence on the performance of a model for each category.

Two machine learning approaches in Annif, Omikuji and SVC, proved to be suitable for solving our classification tasks. SVC is an implementation of a support vector machine algorithm (“Support vector machine”, 2024) and the Omikuji algorithm is based on decision trees (“Decision tree learning”, 2024). A large amount of testing and analyses was needed to find the best performing approach.

Over all DDC Subject Categories, the SVC model for German-language publications was able to achieve a document average F1 score of about 0.774 on a test corpus of 29,872 full texts. The Omikuji model achieves a document average F1 score of about 0.766. The result covers all DDC Subject Categories.

Hence, to get more stratified results, it is worth to have a more detailed look into the data. Table 3 provides an overview of the 10 subject groups with the highest F1 score. Some individual DDC subject categories score significantly higher than the average F1 score. As stated before, one reason for the differences is the number of available training data for each subject category which has an important impact on the performance. The category’s content definition must also be taken into account. In addition to the subject categories with a large proportion of training material (such as categories 340, 610, and 230), there are also some in the top ten for which only a small amount of training material is available. It seems that the number of documents in the training material is not the only criterion for achieving better results. The F1 scores show that SVC performs slightly better than Omikuji in our use case. Therefore, the decision was made to use SVC for the DDC Subject Categories.

How does the SVC model for German perform in productive use? The model has been running since the beginning of August 2023. The period looked at here spans from the beginning of August 2023 until March 2024. In this time range 58,828 German-language digital publications were processed with this model and thus classified with a DDC Subject Category. To verify how well the model has predicted the respective subject category, publications are required that have intellectually assigned categories as well as automatically assigned categories. Of the 58,828 digital publications, 6,571 publications have an intellectually assigned subject category that is taken from the printed parallel edition in addition to the automatically assigned category. This means a sample of around 11% of the processed texts is available for analysis.

For 5,289 publications, there is a match between the subject category assigned by machine and the one assigned intellectually, i.e. the subject category assigned by the machine was correctly assigned to 80.49% of the sample.

As Table 4 shows, the F1 score of 17 subject categories lies between 1 and 0.85. These 17 subject categories comprise a total of 3.036 publications and thus account for 46.20% of the total sample. The most frequently assigned subject categories are “Law” with an F1 score of 0.95 and the category “Medicine, Health” with an F1 score of 0.85. For a further 26 categories, totalling 2,414 publications, the F1 score is between 0.85 and 0.7. 51 subject categories have an F1 score of less than 0.7 but with a total number of 1,121 publications, they account for only 17% of the sample.

One challenge concerning the analysis of automatically assigned subject categories for digital publications is the composition of each sample that is analysed. It always depends on the type of publications and which of these publications have an intellectually assigned subject category. This can mean that not all subject categories are always represented (not available) in the evaluations. It is therefore important to carry out the evaluations at regular intervals and over a long period of time in order to make long-term statements about which subject categories perform with a satisfying F1-Score.

The German Integrated Authority File GND currently contains about 9.6 million standardised German descriptors and is growing continuously. A subset of about 1.4 million GND descriptors is marked for usage in subject indexing. As the GND is a joint vocabulary cooperatively managed and enriched by multiple institutions, not every single descriptor of this 1.4 million GND subset is necessarily linked to one or more publications in the DNB’s collection and can be used for training. As already shown in Section 2, currently around 218,000 records with digital publications (full text), around 745,000 records with digital tables of contents and around 292,000 records with digital blurbs in German language have been intellectually linked to at least one GND descriptor. In addition, various parts of the metadata such as the title can be used for training.

From a machine learning perspective, the use of up to 1.4 million GND descriptors as controlled vocabulary for automatic subject indexing can be interpreted as an extreme multi-label classification (XMLC) problem (Dasgupta et al., 2023). XMLC problems are characterised by a large set of classifying labels, from which only a small subset is used with a high frequency. The large majority of the labels is rarely or never used. The visualisation of these distributions is better known as the ‘long tail’ (Jain et al., 2016).

Table 5 provides an overview of the frequency of occurrence of GND descriptors in the library’s collection. It shows the ‘long tail’ in absolute and relative numbers: Over 1.18 million GND descriptors are not linked in the DNB’s collection at all. 167,126 GND descriptors are linked between 1 and less than 10 times (tail labels). It means that not even 5% of the GND descriptors are used more than 10 times. Only 997 GND descriptors are linked at least 1,000 times. They represent the head labels.

While supervised machine learning is suitable for labels with medium or high frequency of occurrence in the training data, other approaches can help to assign labels with few or no occurrences. Lexical approaches are able to assign rarely or never assigned GND descriptors and can basically use all 1.4 million GND descriptors. In order to make use of the advantages of different approaches, it is useful to fuse their results in an ensemble.

A useful machine learning approach is letting the data for training and the data for prediction follow similar data-generating distributions in order to avoid unfavourable model bias and loss of performance in productive use (Goodfellow et al., 2016). We found this fact to be of particular importance with regard to the selection of the kind of texts. A series of experiments for the indexing use case showed that machine learning models deliver more accurate results if the kind of texts for which predictions are to be provided also correspond to the kind of text that the models were trained on. For example, it can have a positive effect if a model trained with titles is applied to titles. A model trained on tables of contents should be applied to tables of contents, etc. Otherwise the models could learn from a feature distribution that may be different in the data for prediction. In particular, mixing training material of various kind of texts (e.g. titles, tables of contents, blurbs, full text) may confuse models. We found that it is beneficial to train and apply models on a single kind of text only and combine predictions at a later fusion stage as described in Toepfer and Seifert (2020).

The ensemble for automatic indexing with the GND descriptors for German-language digital publications consists of four differently configured models. Three models use the supervised learning approach Omikuji and one model uses the lexical approach MLLM. The MLLM method was parameterised and provided with the entire GND, i.e. about 1.4 million descriptors. The first Omikuji model (Omikuji-1) was trained with the full texts of about 218,000 German-language digital publications. They were pre-processed by cutting them down to 50,000 characters and always adding the title of a publication to the beginning of the text. The second Omikuji model (Omikuji-2) was trained with about 1.4 million German-language titles. Finally, the third Omikuji model (Omikuji-3) was trained with about 720,000 German-language digital tables of contents.

With Annif, it is possible to train each model with a distinct training corpus, in particular we can define model-specific corpora for each kind of text. Thus for example we can train a model for the kind of text “title” with titles. Several specialised models can then be used for prediction in an ensemble. All models in an Annif ensemble, however, can only be given one specific kind of text for prediction, i.e. at inference time it is not possible to pass on different kinds of text to each model. For our purposes, ideally, we want to be able to direct different kinds of texts to each of the models simultaneously. Annif provides only the functionality of the ‘transform limit’ parameter to cut the text to an individual length separately for each underlying model (Inkinen, 2023). To approximate different kinds of text we can cut the text length with the transform parameter. We evaluated that the maximum number of characters of 90% of all titles in the DNB collection is about 150. 90% of the DNB’s tables of contents have a maximum size of 10,000 characters.

We do not use the transform parameter in Annif for training (see Table 6). We do use it for prediction for new unknown publications to approximate the length of a title and the assumed length of a table of contents for the ensemble (see Table 7). The text for prediction consists of a full text that is cut down to 50,000 characters beforehand and a title added at the beginning of the full text. This text then forms the base text. Corresponding to the different models, the transform parameter of Annif is now applied: the entire base text is transferred to Omikuji-1 for prediction. Only the first 150 characters are passed to Omikuji-2. To Omikuji-3, 10,000 characters are given for prediction. Last but not least, MLLM gets the entire base text.

We tested the ensemble of Omikuji-1, Omikuji-2, Omikuji-3, and MLLM in an experiment with 10,682 digital German-language publications (full texts).

To measure the quality of the results, the following evaluation metrics were obtained for each model: precision, recall, F1 score, and NDCG (Table 8, all calculated as document averages). In the experiment, each publication was assigned a maximum of seven GND descriptors with a minimum confidence score of 0.04. The metrics were obtained by comparing the machine-assigned GND descriptors with the intellectually assigned GND descriptors from the gold standard.

The individual MLLM method produces an F1 value of 0.2402, the Omikuji-1 model an F1 value of 0.3902, the Omikuji-2 model an F1 value of 0.4142 and the Omikuji-3 model an F1 value of 0.3573. All four combined in an ensemble yield a joint F1 value of 0.4738. It can be seen that the fusion of the results from the MLLM lexical approach and Omikuji’s supervised learning models in an ensemble leads to the best results.

In addition, for each model the precision-recall curve and the derived area under the curve “PR AUC” (Boyd et al., 2013) were calculated from a total of 10,682 publications. The precision recall curve visualises the various possible trade-offs between precision and recall, when varying a models decision threshold (minimum confidence score). High thresholds lead to predictions of high precision and low recall (upper-left area of the diagram), low thresholds achieve high recall but low precision (lower-right area). For every publication, the first 25 GND descriptors with the highest confidence value were included in the ranking. The ensemble reaches the best result with a PR AUC of 0.5232. As can be seen in Figure 1, the ensemble can outperform the precision of the best performing Omikuji model by more than 0.15 at a fixed recall level of 0.5.

Another way to measure the quality of automatic indexing is through assessment by subject indexing experts (see also the suggestions for evaluating quality in Golub et al. (2016)). What is assessed is the degree of correlation between the machine-assigned GND descriptors and the topic(s) of a publication. Is the topic of a publication described correctly and meaningfully by the individual GND descriptors? Is the GND descriptor useful to describe the topic? Each GND descriptor is assigned a value on a 4-point scale: “very useful”, “useful”, “slightly useful”, or “wrong”. In order to assign one of the three “useful” levels, the GND descriptor must match a topic of the publication or describe a partial aspect of the topic.

In an earlier assessment involving subject indexing experts, a sample of 702 German-language digital publications was evaluated with GND descriptors from an ensemble of Omikuji and MLLM (Uhlmann & Grote, 2021). The analyses led to the result that 38% of the descriptors fell into the “very useful” category, 30% were rated as “useful” and 22% as “slightly useful”. 10% of the machine-assigned GND descriptors were labelled as “wrong”. A positive trend can be observed from the first models we used productively in the DNB to assign GND descriptors in 2014 to the present day. The proportion of descriptors rated as “very useful” and “useful” has increased and the proportion rated as “wrong” has decreased (compare the older results from 2013 in Uhlmann (2013) and 2018 in Mödden et al. (2018)).

Finally, the experts added missing GND descriptors in order to correct and complete subject indexing of a publication. Thus, these analysed and revisited sample datasets form new gold standard. The datasets then get joined with the other training data to improve our next models. This closes a circle and the work of the subject indexing experts becomes indispensable in the sense of “human-in-the-loop” (Monarch, 2021).

Requirements engineering is an essential aspect and a prerequisite in the development of professional software. Therefore, the first step to what would later become the EMa system, was formulating a set of main requirements that outlined the questions “What features do we need?” and “What should the software do?”.

The basic feature set for the EMa project includes the following items:

The automatic subject cataloguing system has to be designed to work on DNB’s hardware resources. The generic, design-related requirements of maintainability, expandability, and interoperability were given a high priority beside the above mentioned. As any automatic subject cataloguing is a highly technology-driven task, the implementation of more innovative features in the future has to be planned for. In order to keep the system at the state of the art, implementation costs for new features have to be kept low by designing the system in an expandable way.

This leads us to the design principle of a modular and service-based architecture, where new services and “backends” for automatic subject cataloguing can easily be added with an acceptable degree of maintenance. Interoperability from an intrasystem perspective stands for reusable and exchangeable processes and methods, from an intersystem perspective it stands for a seamless fit in the organisation`s system infrastructure.

In 2024, it can be stated that most of the requirements are satisfied by the EMa system and the main features are implemented.

In the following, we provide a high-level introduction of the productive EMa system.

As Figure 2 shows, the EMa environment (1) is the backbone and the workflow engine of the entire EMa system. Every day it retrieves a stack of identifiers that represent newly arrived publications which have been selected for automatic cataloguing. This component implements for every publication the flow control of automatic cataloguing in interaction with the EMa services, the catalogue system and the text storage. It comes with a command line interface and gets triggered by automated scheduled processes (cronjobs).

The main parts of the workflow engine’s configuration and the connected services are held in the central yaml configuration file. The file is easy to edit and contains basic configuration items such as parameters for the Annif models like the maximum number of returned descriptors and thresholds. It is provided by the EMa config service (2) and is visible in the entire system.

Every publication has to pass a processing pipeline of several stages. In the first stage, the EMa environment calls the Text providing service (3). It carries out a plain UTF-8 text extraction from PDF and EPUB files out of the repository. For PDF files we use the open source tool PDFBox (https://pdfbox.apache.org/), for EPUBs we use a software developed in-house. A basic method to get relevant content is to cut the text at its beginning and end.

The PICA record service (4) next gets the metadata needed for an object (e.g. the title). The Language guesser (5) determines the most probable language for a text and returns it as ISO 639-2/B codes. The results from this service are important for the selection of the models for classification and indexing. The language code becomes also part of the catalogue’s data record, if there is no other language code available. The Language guesser currently encapsulates two models of freely available software: Lingua (Stahl, 2024) and Apache Tika (https://tika.apache.org/).

At that point in time, all of the required information for the automatic suggestions is available. The Classification and indexing service (6) defines an important layer to encapsulate the underlying suggestion tools and toolkits for indexing and classification from the EMa environment. Basically, the service hands the extracted German or English text data (title, table of contents, full text – if available) to the suggestion backends. Currently, we only connect the Annif toolkit for our purpose of use. In the future, we will also have the ability to make other algorithms, software, and services available for the EMa in a flexible manner.

The connected Annif service (7) suggests the DDC Subject Categories, DDC Short Numbers, and GND descriptors as a key functionality of the EMa. Annif provides a REST interface that suits the flexible service-based architecture of the EMa system. Inside, for every use case, we provide an Annif project in a separate Docker container that we configure with Docker Compose files. Furthermore, we have developed a semiautomatic workflow to transfer trained Annif models, ensembles, and their configurations in the productive system in a comfortable and deterministic way.

Finally, the PICA cataloguing service (8) converts the automatically generated subject cataloguing results into the PICA+ format and the PICA record import service (9) writes them back into the catalogue system. After that, the next publication’s identifier gets processed until the stack of identifiers is done.