Performance management is about controlling the relationship between effort and delivery of the desired outcomes (Van Dooren, Bouckaert, & Halligan, 2015). All organisations, private and public, have to be able to show how well they are fulfilling their purpose. The purpose is relative to the viewpoint of interest and the changes in the environment (op cit.). Libraries, having the essential role in facilitating information to people, are influenced by the attitudes and norms addressed to public services.

Since the late 1980s, there has been an increasing demand for public sector reforms, which has affected all service providers with a public role. The New Public Management ideology has adopted elements from the private sector that aim at improving the effectiveness of public service provision (Hyndman & Lapsley, 2016; Osborne, Radnor, & Nasi, 2012). Despite some criticism (e.g., Lilburn, 2017), management and measurement in the public sector is an area of continuing interest to both scholars and practitioners (Düren, Landøy, & Saarti, 2017; O’Flynn, 2007; Stenvall & Virtanen, 2017; Talbot, 2010).

In recent years, the studies on public service management have shifted the focus from mechanistic performance budgeting towards a value-based evaluation of the performance. The Public Value theoreticians position the public value creation into the core of public sector performance (Guthrie, Marcon, Russo, & Farneti, 2014; Scott, DeLone, & Golden, 2016; Talbot, 2010). This new paradigm emphasises the role of public managers in securing, balancing, and advocating public values (Moore, 2014; Spano, 2014). Moreover, Public Value management is not just about controlling the resources, but also about controlling the created outcome, i.e., the values and benefits for the citizens and all other stakeholders of public services (O’Flynn, 2007). This approach is not far from the performance and impact evaluation that libraries are familiar with: the “fitness for purpose” evaluation (Poll & te Boekhorst, 2007).

Libraries have experience in systematically collecting qualitative and quantitative data through web analytics, surveys (such as LibQUAL), and some more ethnographic methods (e.g., Carlsson, 2016; Renard, 2007; Sinikara, 2006). Libraries consider the Balanced Scorecard (BSC) approach an inspiring framework in the context of their services, as BSC takes a broader view of performance (Corrall, 2015; Passonneau, 2013; Tanner, 2012).

However, it seems to us that the existing evaluation models, frameworks and standards are focusing on the value of libraries’ collections and services rather than on the value of their operations. Besides, design activity has its unique characteristics that require consideration in the management and evaluation of operations (Harpum, 2004). We do not, therefore, consider these models and frameworks applicable to the assessment of UCD performance as such.

To address this gap, we apply the Public Value approach and, specifically, Moore’s (2014) Strategic Public Management triangle (Figure 2), to the performance evaluation of library services. According to Moore (2014), strategic management of public services consists of three conceptual entities: 1) legitimation and support, 2) operational capacity and 3) public value. Each element is in dynamic relation to the others; stakeholders provide an organisation with legitimation and support to operate. Through operational programs, policies and procedures organisations build their capacity to create value. The outcome of this value-generating process is the public value, which, according to Moore (2014), takes the form of individual and social values.

The cultural-psychological time is an utterly essential element influencing our actions and world-view. There is a growing body of literature on temporalities in the organisational settings (e.g., Rubin, 2007b; Whipp, Adam, & Sabelis, 2002), and particularly, in future studies (e.g., Voros, 2003, 2006). In the scholarly discourses of design or library studies, temporalities have remained under-theorised. Nevertheless, aspects of the time, i.e., past, present and future, are interwoven on our everyday discourses.

Temporal information gives managers a basis for decisions concerning the efficient and effective use of design resources. They can use existing data from past projects or estimation data for future projects in prioritisation between individual design projects (Best, 2010; Karat, 2005). Ex-ante estimations and ex-post calculations can also help with choosing between various design procedures and methods (Mayhew & Tremaine, 2005).

In the context of design, comparisons with temporal data may be more challenging. Good design can bring positive results, which were not anticipated in the initial goals and, therefore, not taken into consideration in the evaluation (Hirsch, Fraser, & Beckman, 2004; Whicher, Raulik-Murphy, & Cawood, 2011). On the other hand, managers who are responsible for the development may not be accountable for future costs affected by the lack of design (Siegel, 2003). Thus, there can be significant temporal gaps between inputs and outcomes of design. From a holistic and value-oriented viewpoint, the power of design lies in moving from short-term problem solving to long-term improvements (Cockton, 2006, 2008; Design Council, 2015; Friedman, Kahn, & Borning, 2008; Sikorski, 2008).

The internet has redefined our perceptions of time by increasing the speed of communication. It is also vital for libraries to envision the different futures, the possible as well as the preferable ones (cf. Voros, 2003). Studies on libraries’ ability to meet the requirements of the future society have warned that the traditional approaches to strategy and management may become disconnected from the rapid societal and technical change (Batt, 2015; Kallinikos, Hasselbladh, & Marton, 2013). The concepts of time can help managers to better frame and make sense of organisational change (Wiebe, 2011). In our current society, the temporal structures of work time are seen to become more layered than linear (Rubin, 2007a). When discussing different temporal aspects of time, it is important to use flexible ‘timescapes’ and allow context-dependent interpretations (Keenoy, Oswick, Anthony, Grant, & Mangham, 2002).

Building on these notions, we argue that the influence of time (in the psychological meaning) should be explicitly expressed in the conceptualisation of UCD performance management. Furthermore, we embrace the orientation to the future and consider it essential to take a foresight approach to performance management and evaluation.

As described in previous chapters, our theoretical examination builds on the Public Value approach, complemented with the theoretical perspective of temporality. The Strategic Public Value triangle by Moore (2014) serves as a primary framework, but it lacks the aspect of time, which we considered essential to conceptualise for contemporary performance management. By extending Moore’s triangle with the concept of time, we construct a strategic public value diamond that consists of four interrelated theoretical concepts: 1) legitimation and support, 2) temporality 3) public value and 4) operational capacity.

Figure 3 illustrates the Strategic UCD Management diamond, which constitutes the integrated theoretical framework for our study. We bind the theoretical framework in the context of our research with four intermediary empirical key concepts. First, the digital library Stakeholders give the legitimation and support for the UCD activities. Second, the design performance is managed and evaluated through different temporal Phases. Third, the public value gained in design can be operationalised through Benefits that are received from the different outcomes of design. Finally, Design activities, which can have a variety of possible approaches and processes, constitute the operational capacity of design. Table 1 displays the empirical and the theoretical key concepts of the integrated model, and their appearance in the literature.

The ability to evaluate and compare operations lies in the core of management functions. However, some design theoreticians pose strong arguments against justifying incremental benefits of different design activities (Hirsch et al., 2004; Siegel, 2003). Instead, they point out that design should be considered more as a culture that, if persistently cultivated, accrues significant benefits in the future.

For this research, we decided to treat the empirical Design activities with a similar “black-box” approach. Thus, the concept has its role in the framework as a strategic entity, but not as a target of in-depth empirical examination. This approach allows using the framework in the context of diverse design approaches, independent of their processual and methodological differences.

The first element of the conceptual model is the Stakeholder, which is related to the theoretical concept of Legitimacy and support in the Strategic UCD Management diamond. By including this element in the conceptual model, we give an active voice to all people who contribute to or have interest in the design of a service.

In the context of digital library services, the benefits and value of the design activities are not only relevant to the organisation responsible for the development of the service, but also to various internal and external actors. As Tanner (2012) emphasises, only stakeholders can transform the understanding of output into a significant outcome. We argue that this also applies to the outputs of UCD activity. Therefore, our model engages stakeholders in the joint evaluation and promotes open and public conduct of the evaluation process.

Identification of relevant stakeholders requires explicit positioning of the centre of dominant interests. Most often, stakeholder analysis is organisation-centric (Friedman & Miles, 2006), and therefore it may take extra effort to objectively identify stakeholders from the viewpoint of the service or the benefits it accrues. Applying these notions to the classic definition of stakeholders (Friedman & Miles, 2006), a stakeholder in our model is an individual or a group who can affect or is affected by the activity of user-centred design (UCD).

In the interest of feasibility, the range of identified individuals should be segmented into purposeful stakeholder groups. The grouping should be context sensitive, and therefore we recommend relying on the empirical material or some existing stakeholder map (e.g., Eskerod & Jepsen, 2013). Design theories or guidebooks also provide feasible classifications for design-centred analysis of involved people (Bryson, 2004; Löwgren & Stolterman, 2005). Figure 5 illustrates exemplary stakeholder groups, who can be engaged in the evaluation.

The second element of our conceptual model is Benefits, which is related to the concept of Public Value in the theoretical framework. Literature implies that individuals and groups representing various fields or domains may have conflicting values and interests about public services (Botero, Paterson, & Saad-Sulonen, 2012; Wijnhoven, Ehrenhard, & Kuhn, 2015). Thus, we can assume that we may also find differences in their perception of values and benefits of design. The element Benefits in our model aims to capture Stakeholders’ divergent views of the goals of design performance.

The literature suggests a plethora of possible benefits and values which organisations can pursue in their design activities. The objectives of design can and should be analogous to the objectives of the service itself (Krippendorff, 2006). Altogether, we identified from the literature nineteen distinct benefit categories for UCD, such as cost-savings, efficiency, innovation and trust. These categories are partly overlapping, some of them apply better for a particular type of services and in a particular context. Moreover, the literature presented some of them only for private sector services, e.g., the value of intellectual property rights. Still, we considered all benefit categories as potential in the context of digital library service provision and therefore included them in our examination. Table 2 presents the nineteen categories of benefits of UCD derived from the literature.

The third and final key element of our conceptual 360°TB Model is Phases, which concretises the theoretical concept of Temporality. This element builds better understanding of the temporal connection between the design efforts and their outcomes. In other words, we argue that if the desired benefit can be located in a particular phase of the life-cycle, it can thus be more precisely evaluated and managed.

Another aspect of temporality found in the literature concerns the relation between past, present and future. Design activity is considered to be dominantly future-orienting, and it may disrupt present stabilities (Krippendorff, 2006). The outcomes of design activity can increase value and benefits soon or long after the outcome has been achieved or even during the implementation process. Table 2 displays the temporal aspects of each benefit category that we interpreted from the literature. Some benefits are considered to become apparent during a particular phase in the life-cycle of the service, while others can accrue variously along the process.

Applicability in the context of service provision requires splitting continuous time into useful phases. We identified from the literature three more or less overlapping periods of time, when different benefits of user-centred design may be perceived: during the process of developing or producing the service, during the use of the service, or in the undefined future (cf. Table 2). Encouraged by these notions in the literature, we divide the element Phases into three distinct parameters that point from the present to the future: process-time, use-time and (undefined) future.

To better comprehend the parameters in the context of digital public service development, we map them to the life-cycle of service development. Alter (2008, 2013) has captured the different conditions of service systems into the Work System Life Cycle (WSLC) model. The model represents the dynamics of work systems in four phases from the practitioners’ viewpoint: 1) initiation, 2) development, 3) implementation, and 4) operation and maintenance. User-centred design can take place in each phase of the life-cycle. The WSLC model acknowledges the iterative and repetitive nature of development processes and is to be understood as a realisation of cyclic and continuous activity.

The first two phases of the WSLC model (development and implementation) correspond to the parameter process-time of the 360°TB Model. The third WSLC phase, operation and maintenance, is parallel to the second parameter of our model, use-time. It is noticeable that use-time can occur during the WSLC phase initiation or as long the old service is still in use. The third parameter of our model, future does not have a counterpart in the WSCL model, because the latter model ends with the extinction of the service. As the theory and literature indicate, design of public services can derive benefits long after the service has ceased to exist (e.g., Service Design Network, 2016). Thus, we consider it essential to include the aspect of undefined future in the conceptual model. Figure 6 illustrates the comparison of the temporal elements of the WSCL model and the 360°TB Model on a single timeline.

Figure 7 illustrates the three key elements of our 360 Degree Temporal Benefits Model. We can best describe the relations between the elements through a narrative sentence: A Stakeholder anticipates Benefits of the design in different Phases. The evaluation process is predictive since it takes measures from the present moment in order to derive benefits for the future in three phases: process-time, use-time and (undefined) future.

The 360 Degree Temporal Benefits Model serves as an empirical conceptualisation of the integrated theoretical framework that we depicted in Figure 4. The purpose of the model is to capture the key elements and their relations in the public digital service provision landscape. Following the conceptual model and using the available metrics and data, design managers in libraries can have better control of their UCD efforts towards the desired goals. The model itself does not define which metrics and data sources are appropriate or adequate but recommends using material that is reasonable in the given context.

The predictive nature of the model allows using it already at the beginning of a design project, but it can also be applied later in the service life-cycle, even repeatedly. The utility of the model lies in its ability to detect similar or potentially conflicting goals of different actors and stakeholders of a service. In comparison to existing evaluation frameworks, our model suggests a 360-degree view to relevant stakeholders of a service. This approach should aid design managers in better guaranteeing the legitimation of and support for their design decisions. On the other hand, stakeholders can require that design activities are evaluated following the 360°TB Model to avoid an organisation-centric assessment. Using the model can equalise the power-balance in the strategic debate on the values and desired goals of services’ design.

Because the 360°TB Model defines the phases, when UCD benefits are anticipated to become effective, managers can use the collected assessment data for more accurate prediction and evaluation. The model can be used for UCD performance evaluation by indicating the status of current design activities (“Are we doing the right things?”) or UCD performance management by setting the future goals (“What should we do in the future?”).

The first step in using the model is the identification of relevant stakeholders for the service design case. After that, the person conducting the analysis should contact representatives of each stakeholder group and ask their opinion of UCD benefits. The list of UCD benefits derived from the literature in Table 2 can serve as a starting point, but the analyser can complement it with case-specific benefits expressed by stakeholders. Further, s/he should consider the relationship between benefits and their temporal aggregation (phasing). Depending on the case, some benefits may be relevant only in the process-time, some in the use-time, some in the (undefined) future and some at all times. Finally, we recommend that the results be opened to public discussion and stakeholders are allowed to form their joint understanding of UCD benefits. Figure 8 shows the recommended steps of using the model in practice.

To improve the rigour of the results, the benefits that stakeholders consider essential can be further concretised with appropriate metrics. The multitude of data that organisations already collect about their performance and services can be used for this purpose. Eventually, an evaluation conducted following the 360°TB Model may reveal defects in the current measurement practices.

Evaluation activities tend to consume time and effort, less in a simple case, more when the settings are complex, and the scale of the service is big. Furthermore, the frameworks of evaluation can be complex or intensive, e.g., by including many phases or procedures (cf. International Organization for Standardization, 2014; Tanner, 2012). The lack of standard definitions and parameters for the design discipline can also make the evaluation process overly time-consuming (Whicher et al., 2011). Some authors suggest that only big investments should be involved in an extensive analysis (Hirsch et al., 2004).

For these reasons, we suggest that practitioners should follow the principle of simplicity when they use the 360°TB Model in real cases. The following rules can help keep the application of the model within the constraints of time and resources:

Still, depending on the number of stakeholder groups, the evaluation conducted using the model can yield a multitude of benefits for the UCD of a service. If each stakeholder anticipates three different benefits (one for each phase), and these benefit-triplets vary between stakeholders, the outcome of the evaluation is a diverse set of expectations for the design activity. Figure 9 demonstrates how evaluation using the 360°TB Models may provide a multitude of benefits that stakeholders anticipate individually or share with each other.

As a tentative example of potential uses of the conceptual 360°TB Model, we applied the model to two existing cases. Both cases represent a digital library service development in which user-centred approaches were applied.

The first case is the National Digital Library (NDL), which joins together the repositories of practically all libraries, archives and museums in the country. The NDL includes a joint online user interface for all participating organisations’ materials, as well as a platform service for organisations to build their interfaces. By the end of 2016, all academic libraries and several public libraries in the country had joined the service with their collections and electronic materials. From the very beginning of the project, an in-house team of interaction designers has been responsible for the user-centred design of the service. Besides, various other stakeholders have taken part in the design activities (Hormia-Poutanen, Kautonen, & Lassila, 2013).

The second case differs from the previous one in its scale and scope. The project began as a typical library system acquisition, which included an online user interface for the end-users of a medium sized special library, serving visually impaired people in the country. Eventually, the library split the procurement and made a contract with two partners. They outsourced the interface design to a small enterprise that had expertise in the required user interface technology. During the development process, end-users have been involved in usability tests that were conducted by another firm of required expertise. The new service has been in use from the beginning of 2017.

These two projects provided the data for testing our conceptual model. The data sources were existing documentation and semi-structured interviews of key informants representing relevant stakeholder groups for the projects. Before interviewing informants, we needed some background information about the current status of each project, and particularly the goals set for user-centred design.

As the first step (see Figure 8), we browsed project documentation to identify relevant stakeholder groups for both projects. To capture stakeholders’ perception of benefits that should be received through user-centred design in the case project, we decided to interview the stakeholders. To enable more structured data gathering, we asked the informants to fill an online questionnaire during the interview (step 3, see Figure 8). We also induced informants to suggest suitable metrics for the benefits they chose as most important. For questionnaire options, we used the benefit categories (see Table 2) and temporal phases (process-time, use-time and future, see Figure 7), both identified from the existing literature (step 2, see Figure 8).

We conducted and recorded seventeen interviews altogether. Each informant filled the online questionnaire during her/his interview. Ultimately, the interviews comprised of over ten hours of recording, which we transcribed and combined with informants’ comments derived from the questionnaire. When we had conducted all interviews and compiled data, we sent each informant’s data to her/him for review and validation. Once all informants had reviewed their answers, we published the results to them (step 4, see Figure 8).

In Case 1 we interviewed eight informants, who represent a wide spectrum of stakeholders. Three of these informants represent internal groups, who are responsible for the digital library service in different roles. Two informants represent external partners; one is a representative of a customer organisation and two of end-users of the NDL.

From all 21 available benefits, Case 1 informants chose sixteen different options altogether. Two informants considered that the given options were not entirely satisfactory and described their benefits that were combinations of the proposed benefits. Figure 10 displays informants’ choices in a pie chart on the left-hand side and in a tabular format on the right-hand side.

The results from Case 1 show a notable variation of benefits anticipated by the stakeholders. All informants picked a different first priority benefit for the process time. For the use-time, they expressed three shared benefits: quality for a more extensive group of audience or citizens (Q1), potential for innovations among end-users (I2) and increased use (U). Similarly, innovation among organisations and developers (I1) and trust to the organisation or society (T) were shared by two distinct informants as number one benefits for the (undefined) future.

The heterogeneity of aims increases the challenges of design performance, particularly for the near future. The use-time benefits/targets seem ambitious and partly conflicting: UCD of the digital library should generate more use, provide a good user experience for broad audiences as well as for specific target groups, and improve these end-users’ efficiency and enable their innovativeness. The benefits expected in the distant future are also ambitious: UCD is considered as a means for improving competence and commitment of involved persons and groups, as well as a means for enabling innovation, “common good” and enhancing trust in the public services and society.

In the interest of space, we only present here the metrics/indicators, which informants in Case 1 considered applicable for measuring innovations (benefits I1 and I2). We chose this example because innovation is one of the strategic objectives for European public services, including libraries (European Commission, 2010; LIBER, 2017). Particularly open innovation, which involves various stakeholders in the design and development of services, is considered as an opportunity for the public sector organisations to improve their productivity and performance (Daglio, Gerson, & Kitchen, 2014).

Altogether, four informants chose either end-user innovation (I3) or developer innovation (I1) as their preferred benefit for proximate or distant future. The informants’ ideas for measuring achievements in innovation included quantitative and qualitative data of the service usage and users’ habits, as well as field-studies that involve end-users. One informant addressed the importance of detecting new users and new forms of using the service. Also, one informant pondered the option of measuring developer communities’ innovativeness through some gamified activities. Currently, only usage logs and usability studies generate data for this purpose. Other data sources for detecting innovation in the processes or use of the DL have not been established.

In Case 2 we interviewed nine informants. Five informants represented internal stakeholders, two external partners, and two end-users. One external partner was also the lead designer of the digital library user interface.

The same 21 benefit options were available for the Case 2 informants, who chose 14 of them (see Figure 11). Three informants considered increased competence and learning among involved parties (C/L) as the most important benefit to the process-time. Altogether, five informants preferred the quality for a broad group of end-users and citizens (Q2) in use-time. For the future benefits, two informants prioritised societal problem-solving (SP) and three citizens’ trust to the organisation or society (T). Thus, Case 2 informants showed more similar disposition to UCD design benefits than Case 1 informants.

Because of the relative consensus among informants in Case 2, we saw here more uniformity in their first priority benefits. Further, all use-time benefits/targets focus on the end-users: UCD of the digital library should increase the use (U), improve the quality for a broad audience (Q2) or the target group (Q1) or end-users’ efficiency (E3). While the informants expressed that the process-time benefits refer to the challenges of developing the digital library in the proximate past, the future benefits/targets reflect universal values, such as social problem-solving and trust in the society.

Again, we only present one example of the potential metrics/indicators expressed by the informants. In this case, we picked the most appreciated benefit, which was the quality of the service to a broad audience or citizens (Q2). The informants listed currently existing methods of gathering evidence: usage statistics, user feedback via surveys and user studies. They emphasised the importance of finding indicators for user satisfaction and user experience. Also, the informants mentioned the number of users and the extent of the user base as relevant indicators of quality.