Evaluation as a Tool for Developing the Quality of Academic Libraries. Case Study at a Large and Traditional Research Oriented Scandinavian University

Kaisa Sinikara

University of Helsinki, P.O. Box 33, 00014 Helsinki, Finland
Kaisa.Sinikara@helsinki.fi

Comments by:

Kai Ekholm, National Library of Finland, Unioninkatu 36, 00014 Helsinki, Finland, Kai.Ekholm@helsinki.fi

Gunnar Sahlin, Kungl. Bibliotheket, National Library of Sweden, P.O. Box 5039, 1002 41 Stockholm, Sweden, Gunnar.Sahlin@kb.se

The framework of the quality work

The leaders of our universities will ask today if we offer those services that a top university requires and whether the proposed resources for the libraries are worth investing. Establishing the European Higher Education Area (EHEA) requires the establishing of quality systems. Quality assurance systems will be introduced throughout the entire European academic sector and will certainly have significant impact on the development of academic organizations. The collaborative effort to evaluate European universities is coordinated by ENQA, the European Association for Quality Assurance in Higher education (Standards, 2005). Most European countries have developed a national quality assurance system. The solutions of different countries can be divided into three different classes: 1) Auditing countries[1] (institutional audit, quality audit, evaluation of quality assurance systems, enhancement-led institutional review): England, Scotland, Ireland, France, Finland, 2) Accreditation[2]: Holland, Germany and 3) Combination or another system: Norway, Switzerland, Denmark, Sweden (Audits, 2006).

The focus and methods are different when speaking about the quality of services, quality management or the quality assurance system.

	The Quality of the Services	Quality Management	The Quality Assurance System
FOCUS	Do the library services work well? What are the customers' needs? Are the customers satisfied?	Do the libraries provide the right services? Are they updated and cost-effective?	Focus in the entire quality system of the university
METHODS	LibQual User surveys	Peer review evaluation Self-assessment Benchmarking BSC CAF[3]	Auditing Accreditation

The focus of the article is in the possibilities of the evaluation. As an example will be used the wide evaluations that have been carried out at the University of Helsinki.

The customer viewpoint is fundamental when defining quality. The traditional customers are students and researchers, i.e. the users, but the university itself and the society can also be considered customers since they are the stakeholders who provide funding and define the policy and values for the library and information services of the university. The objective of quality management is to strengthen the comprehensive quality of user and internal services as well as the organisation and cost efficiency of the operation, such as the welfare and expertise of the staff.

Evaluation as a tool of quality management

Evaluation is the systematic determination of the value (evaluation) or the comparison of objectives to the measurement of performance (assessment) based on criteria set in the relation. Evaluation is a process that aims to raise development needs and proposals. Effective planning is impossible without an evaluation component, and evaluation has little utility unless its findings are integrated into the planning process.

Evaluation embraces change and encourages libraries to treat change as a positive force. By engaging in planning and research, librarians have a better idea of the future and they can meet that future with relevant, effective, and efficient services and activities. Furthermore, they can meet new challenges and innovations, and libraries will continue to play an important and positive role in the information society (Hernon & McLure, 1990, 238-239; Hernon & Altman 1996, 15-16).

The evaluation model consists of 4 parts:

1. external evaluation organisation,
2. self-assessment,
3. peer-review with its evaluation visits and
4. public evaluation report.

In the evaluation of the study and teaching of the university and the libraries peer-review evaluation has been used. To determine the best practices the evaluation of both the assessment processes and its results is required.

University of Helsinki

The University of Helsinki was founded in 1640 and it is one of the leading European research-intensive universities.[4] Within the Finnish higher education sector, it is by far the most comprehensive and offers a wide range of possibilities for students, researchers and collaboration partners. There are 21 universities in Finland.

At the University of Helsinki, there are 11 faculties and 20 independent institutes. About 4 300 degrees are awarded every year, of which about 400 are doctoral degrees. The number of students is 38000, and the number of staff is 7600, of which 4700 are researchers and teachers.[5]

Evaluation processes of the library institution of the university of Helsinki

The university libraries in Finland are open for everyone, which is not the case in every European country. The libraries are valuable not only for education and research at the University, but also for the society.

1993 Internal evaluation

The target-oriented development process of the library institution of the University of Helsinki began in earnest after the internal evaluation in 1993.[6] The year 1993 was dramatic from the point of view of the University of Helsinki and Finnish society as a whole: the economic depression cut resources significantly, the Ministry of Education was bringing a new budgeting model into use, and the university had to analyse its operations. All the faculties and independent institutes, including the library, rushed to be examined by the evaluation team leaded by the Vice-Rector Mustajoki. The evaluation produced more than 70 development proposals

The library institution included more than 160 separate units. The group included some rather large libraries, like the Helsinki University Library, but the majority comprised small institution libraries and faculty libraries. The internal evaluation encouraged the creation of a presentation on four information centres and a plan for staff reductions required by the reduction in resources. The premises of the libraries were mainly old and scattered. Little attention was paid to the suitability of the library premises. The Vice-Rector Arto Mustajoki who led the internal evaluation team became the chairman of the Library Committee in 1995. The first common strategy of the library institution was accepted by the University Senate in 1998. It was decided to carry out an international evaluation in 2000.

2000 International evaluation of the Libraries of the HU

A total evaluation of the library activities at the University of Helsinki took place in the year 2000. The members of the panel were University Librarian Ian R.M. Mowat (University of Edinburg, United Kingdom), Library Director Hans Geleijnse (Tilburg University, the Netherlands), University Librarian Göran Gellerstam (University of Lund, Sweden) and Library Director emerita Kyllikki Ruokonen (Helsinki School of Economics and Business, Finland). Both the International Assessment Panel and the Steering Committee of the assessment, chaired by Professor Maija-Leena Huotari, presented extensive action plans in their assessment reports (Geleijnse et al, 2000), and the follow-up on their realisation fell to the Library Committee.

The university reserved moderate resources for the evaluation and a planning officer employed for the entire year and a part-time evaluation officer. In the evaluation several methods were used:

• statistical and other background material;
• self-assessment of libraries;
• user inquiries performed on different groups;
• the panellists' visits to the libraries (1 week + 2 days).

The evaluation panel interviewed more than 80 persons (mainly as group interviews) including decision-makers, users and actors, university management, leaders of the university administration, members of the steering group, professors, students, leaders of libraries and staff.

The most important targets for development raised in the assessment were:

• clarity in defining the National Library and the University Library sectors,
• strengthening the coordination and strategic management of the University's libraries,
• creating a common personnel strategy for all the libraries,
• improving quality,
• creating centralised supporting services.

In January 2001, immediately after the assessment was over, the Library Committee approved an action plan, which was to form the basis for developing library activities. Its relevant points were discussed in the University Senate in August 2001. At that time the Senate made a decision in principle to establish the post of Director of Information and Library Services Development and agreed on a number of other definitions of strategic policies.

The follow-up evaluation in 2004

The goal of the evaluation was to assess the process of development within the library and information services at the University of Helsinki since the evaluation of 2000. The members of the International Panel were Library Director Hans Geleijnse (Tilburg University, the Netherlands), Library Director Gunnar Sahlin (National Library of Sweden,Sweden) and University Librarian emerita Sinikka Koskiala (Helsinki University of Technology Library, Finland). The process differed from that of the previous evaluation 2000. The steering committee was led by Vice-Rector Hannele Niemi, chairman of the Library Committee. The fact that the chairman of the steering group and majority of the members did not belong to the library committee in 2000 was to some extent problematic.

The follow-up evaluation was carried out more lightly than was the 2000 evaluation. The panel was supported only by the evaluation officer, who only worked part-time. The panel conducted interviews, but to considerably smaller group than in 2000. The self-assessment of libraries was an important document, as were the enquiries of the deans, researchers, teachers and students. Discussions based on the evaluation report (Geleijnse, Koskiala & Sahlin, 2004) took place in the faculties and the campuses under the chairmanship of the steering committee. A wide hearing was then arranged.

Diagram evalution process

The library institution in 2006

The library institution, which functions within the sphere of the university, comprises in 2006 two large entities:

• the National Library of Finland (formerly the Helsinki University Library)
• the University of Helsinki Libraries, a coordinated network.

The University of Helsinki Libraries 2006 are located on four campuses:

• City Centre Campus (faculties of arts and humanities, behavioural sciences, law, social sciences and theology),
• Meilahti Campus (faculty of medicine)
• Kumpula Campus (faculty of science)
• Viikki Campus (faculties of agriculture and forestry, biosciences, pharmacy, veterinary medicine).

Libraries in the four campuses

The library institution of the university has nowadays excellent, new or renovated premises. The libraries with the Library Coordination Unit have developed a customer feedback system in a joint project (Vakkari, 2006).

'Homework' of the Library Committee and the University of Helsinki after the follow-up evaluation

The Library Committee drew up the plan of action for which the statements were requested from the faculties. The tasks of the Coordination Office were defined in the ruling of the University Administrative Office decided upon by the Senate in 2005. The Director of Information and Library Services was permanently employed and the temporary vacancies of some centralised services were permanently filled.

A lot of work has been done for the structural reorganisation of the University library staff. A plan of action to 2011 was prepared in 2006. About 25% of the personnel will retire in the coming five years. The total number of the library staff is about 500, about 50% of whom constitute the personnel of the National Library.

A big step has been taken when the reform principles of the reorganisation of the library institution with two opportunities were approached in May 2006:

• the library institution as a uniform library organisation with one managing director
• an entity with larger coordination in the administration office and with four campus libraries (one coordinating director and four campus library directors).

The model will see wide discussion during 2006. Some decisions on structural reform will be made by the Senate of the University in August-September 2006.

Difficult topics from the point of view of decision-makers

It will come as no surprise that the central problem with the faculties is how the decision-making and financing will be reorganised. As a result of historical development, these are to some extent different in each faculty. The significance and value of the library services to the humanities and to the social sciences differs from their value and significance to the sciences and medicine. This difference makes the development of the uniform library institution more difficult. For this reason, certain fears of both lines are directed at the common library institution.

The solutions depend on whether the library is considered its own special branch with well-knit connections to both research and educational functions of the university or whether it is seen only as a support service belonging to the faculties. The expansion of the electronic library has strongly changed the work of libraries even now, but it remains difficult for the faculties to accept mutual dependence on decision-making and financing.

Challenging topics from the point of view of the library directors

The biggest concern from the point of view of the library directors is connected to the freedom of action and to the independence of the library units, and especially to staff questions. Several working groups have been created for the e-library and portal development, Information Literacy, staff training, quality issues and communication. Throughout 2005, the library directors have been working with a consultant to create a common vision for the future. Important tasks have included the development of the new staff structure and the identification of common centralised tasks for the libraries.

The library coordinator's point of view

For the University of Helsinki, permanently appointing the coordinating leader to develop the library entity has been a big step. Development of the electronic library and a reasonable staff structure for the future, as well as the backup of the quality of the library services, will not succeed, if the library organisation is too scattered. At the same time the organisational model based on the coordination, is quite demanding because it includes much responsibility but little authority to carry out changes in practice. Placing the coordinator in the University Administration Office is a significant advantage because in this way, the development of the libraries can be linked to definitions of policy at the university level. On the other hand, this placing has some disadvantages, because some feel that the coordination tasks are a part of the central administration and not a part of the library services.

The financial and strategic periods of the university are quite short only three years. Every three year certain amount of the deans and committee members will change. This means possibilities but also a lot of risks for the development of the libraries.

Conclusion

The evaluation processes have served as a primus motor for an extensive, multidimensional and long-lasting development process in the libraries of the University of Helsinki. Without the key viewpoints on development raised in the evaluations, the same process would have been far slower to set in motion, and would have been far more difficult and burdensome to carry out than the currently ongoing process.

It can safely be stated that the library evaluations play a vital role in the development of the information services and the organisation of the University of Helsinki. On the other hand, we all know, that lot of 'homework' remains to be done after the evaluations. The process will go on.

Notes

1. The audit is an independent external evaluation that clarifies whether the quality assurance system functions according to the objectives, and whether it is efficient and suitable for a particular purpose.[1]
2. Accreditation (lat ad + credere): reliable and credible, presenting and confessing in relation to externally set criteria.[2]
3. Balance Score Card -- this method takes into consideration effectiveness, processes, resources and know-how, and is well-suited as an instrument for drafting a development strategy. Common Assessment Framework (CAF) provides tools for the analysis of operations (enablers) and of results (results). See: European Institute of Public Administration (EIPA)[3].
4. Internationally, the University of Helsinki is known for its outstanding research output for which it was selected as one of the founding member of the League of European Research Universities (LERU). Several international university rankings place the University of Helsinki among the leading universities in Europe. For example, the Times Education Supplement ranked the University of Helsinki number 18 in Europe in 2005 and the Jiao Tong ranking of the top-500 universities in the world places the University of Helsinki at number 22 in Europe. [4]
5. The University of Helsinki: research funding 189 million euro, involved in 24 National Centres of Excellence and in 3 Nordic Centres of Excellence, 24 national postgraduate schools, and in 55 postgraduate schools organised by other universities. The share of the University of Helsinki of the universities in Finland: budget funding 23%, professors 21%, master's degrees 18%, doctoral degrees 27% and National Centres of Excellence 57%.[5]
6. Reports: Universitas renovata 1993, Universitas renovata continuata 1994.[6]

References

Audits of Quality Assurance Systems of Finnish Higher Education Institutions, Audit Manual for 2005-2007. 2006, 37 p. (Publications of the Finnish Higher Education Evaluation Council 4). http://www.kka.fi/pdf/julkaisut/KKA_406.pdf

Geleijnse, Hans, Göran Gellerstam, Ian R.M. Mowat and Kyllikki Ruokonen; introduction by Aimo Virtanen: Helsinki University Libraries - Report of an Assessment panel. Helsinki : University of Helsinki, 2000. (Evaluation Projects of the University of Helsinki, 7).

Geleijnse, Hans, Sinikka Koskiala and Gunnar Sahlin: Follow-up Evaluation of the Library and Information Services 2004. Helsinki, 2004. (Evalution Projects of the University of Helsinki, 8).

Hernon, Peter and Charles R McLure: Evaluation and Library Decision Making. Norwood, N.J. : Ablex Pub.. 1990.

Hernon, Peter and Ellen Altman: Service Quality in Academic Libraries. Norwood, N.J. : Ablex Pub., 1996.

Libraries - The Heart of the University. Development Strategy for the Libraries of the University of Helsinki 1998-2005, Helsinki : University of Helsinki, 1998. (Reports and reviews by Committees and Working Groups of the University of Helsinki, 72)

Mäkinen, Riitta and Aimo Virtanen: Helsingin yliopiston kirjastot - hakua vai vientiä. Arviointiprojektin loppuraportti. Helsinki : University of Helsinki, 2000. (Helsingin yliopiston arviointiraportteja, 8).

Standards and Guidelines for Quality Assurance in the European Higher Education Area. Helsinki : European Association for Quality Assurance in Higher Education, 2005. http://www.enqa.eu/files/ENQA%20Bergen%20Report.pdf

Universitas Renovata. Helsinki : University of Helsinki, 1993. 162 p. (Helsingin yliopiston arviointiryhmän raportti ja ehdotukset).

Universitas Renovata Continuata. Helsinki : University of Helsinki, 1994. 93 p. (Helsingin yliopiston arviointiryhmän raportti).

Vakkari, Mikael: Ei pidä syyttää peiliä, jos kuva ei miellytä - Kirjastojen käyttötutkimushanke. Helsinki : University of Helsinki, 2006. 70 p. (Helsingin yliopiston kirjastojen yhteishanke 2004–2006). http://www.helsinki.fi/julkaisut/aineisto/hallinnon_julkaisuja_28_2006.pdf

Web sites referred to in the text

CAF - Common Assessment Framework. http://www.eipa.nl/CAF/CAFmenu.htm

EIPA - European Institute of Public Administration. http://www.eipa.nl/index.asp

ENQA - European Association for Quality Assurance in Higher Education. http://www.enqa.eu/

LERU - League of European Research Universities. http://www.leru.org