Impact/outcome measures for libraries
Roswitha Poll
INTRODUCTION
In times of dwindling resources and a general demand for accountability, libraries have to show the results of the funding
spent on their services. Such results can on the one hand be documented by output data like number of issues or reference
answers, on the other hand by qualitative indicators that assess the speed, correctness, or cost-effectiveness of service
delivery and the users' satisfaction with the services. But neither the quality of library services nor the quantity of their
use gives evidence as to the impact on users. What funding institutions would actually like to know is whether the existence
of a library can further the goals of the institution or community it belongs to, whether learning and research, professional
work or communal life benefit by library use. To find answers to such questions, libraries should be able to demonstrate not
only their output, but also their outcome. [1]
OUTCOME OF LIBRARIES
Assessing the outcome of libraries means assessing the effect of library services on individual users or user groups:
| • |
"Outcomes ... are the ways in which library users are changed as a result of their contact with the library's resources and
programs." (ACRL, 1998). |
| • |
"Outcomes can be seen as the eventual result of using library services, the influence the use had, and its significance to
the user." (Revill, 1990, p. 316). |
Such effects on users could be immediate or long-term; the effects could be intended by the library, but they might also be
quite unexpected effects; outcomes could be actual or potential, and - although libraries would of course wish to assess positive
outcomes - there might also be negative effects of library use. What libraries are meant to further, can be summarized thus:
knowledge, information literacy, democracy, social inclusion, local identity, lifelong learning, individual well-being.
Short-term results of even a single library visit/library use could be:
| • |
information gained (e.g. factual or bibliographical information) |
| • |
problems solved |
| • |
time saved (in studies or professional work) |
| • |
information seeking skills improved |
| • |
IT skills improved |
Long-term results of using library services might be:
| • |
information literacy |
| • |
improved academic success |
| • |
better career prospects |
| • |
changes in behaviour (e.g. frequency of reading, competent use of information) |
Outcome of libraries can also be shown as constituting an economic value, as well for the individual user as for commercial
firms, communities or institutions. And, lastly, libraries can effect benefits for the future, by storing information and
making it accessible for use in future years.
ASSESSING OUTCOME
Any assessment of a library's effects on users should start from a consideration of the mission and goals of that library:
What are the objectives of that library? What is the primary clientele that it should serve? Benefits users could derive from
their library use vary in accordance with the type of library. The effect e.g. of saving time by the help of library services
will be more important in the library of a commercial firm than in a school library.
As yet, we have no internationally agreed or tested methods for assessing the different aspects of outcome, but there have
been quite a number of projects on this topic. The most evident difficulty is that in many cases the relation between a certain
library service or activity and an effect on a user cannot be verified, though it can be assumed. The influences on users
are complex: if there is an improvement in information literacy, can we indeed trace it back to the library's influence? Statistically,
valid results are difficult to obtain, even if statements in user surveys or interviews indicate a direct influence of the
library. The 'anecdotal evidence' is prevalent.
Other problems include:
| • |
The benefits vary as to certain user groups. |
| • |
Different stakeholders may see values differently. |
| • |
The data that have been collected are as yet not comparable, due to differing ways of data collection. |
| • |
In many cases, users cannot exactly define their intention when using library services; the result obtained may be accidental,
and the user statement of having profited can have different meanings. |
Trying to measure outcome is indeed coping with "six impossible things before breakfast" (Cram, 1999). We may have to use
surrogate measures in order to assess outcome, and among all the methods that have been tested, there is none that is not
time-consuming. But as the issue of proving libraries' benefits is an urgent one, we must try to find consensus on 'measures'
and 'indicators' in the same way as it has been successfully done for proving the quality of library services. Assessing outcome
is most difficult when one tries to measure the effect of the library as an entity: effects achieved by the existence of a
certain library. It is somewhat easier to measure outcomes of certain services, e.g. offering a children's library or chat
reference, and effects become tangible when they are achieved by services that are limited in time: if a library offers a
freshman course or a story-telling event, direct effects might be measured. The same applies to services that are newly introduced:
changing from closed access to free access shelves or introducing e-books in a medical undergraduate collection will probably
allow to assess certain changes in information behaviour.
POSSIBLE METHODS OF ASSESSING OUTCOME
Projects of assessing outcome of libraries have been and are going on in the following fields:
| • |
Academic/professional success: the goal is to establish a relation between the use of library services and academic or professional success. |
| • |
Information literacy/information retrieval: assessing the impact of library services, especially user education, on users' information skills and their success in information
retrieval. |
| • |
Social impact: assessing the imputed value of library services, e.g. by social audits. |
| • |
Economic value: assessing the market value or proxy price of library services. |
Academic or professional success
The methods that have been tested try to assess a direct outcome of library services on the academic success of students and
academics or on the results of professional work.
Academic 'success' is expressed by:
| • |
short duration of studies |
| • |
high grades in examinations |
| • |
student persistence (retention) |
| • |
employment rates after examinations |
| • |
highly reputed publications (e.g. citation impact) |
Such success is then set in relation to:
| • |
the frequency of library use:- number of loans- number of visits- average time per week spent on using the library |
| • |
the variety of library services used (e.g. reference service, ILL, user training) |
| • |
the kind of materials used (undergraduate collection, e-journals, specialized material) |
| • |
the attendances at user training lessons |
Studies on the correlation of library use and academic success have been conducted for several years (Barkey, 1965; Kramer
& Kramer, 1968; Hiscock, 1986; Self, 1987). The project described here was carried out in the Department of Information and
Library Studies at the University of Cape Town (Jager, 2002).
The project compared:
| • |
the loans out of 'short loan material' (core material for the curriculum) |
| • |
the loans out of the general collection (more specialized documents) |
| • |
the academic success (duration of studies, grade in the final exam) |
The data were taken from the university records and the library system. There was no direct correlation between success and
loans out of the short loan material, but students with grade 1 proved to have frequently borrowed more specialized documents.
Similar results were found in other studies, though not always statistically valid. This method seems promising, but the data
might be difficult to obtain, and again the question is: Was the high grade indeed due to library use?
Different methods have been used to prove the library's influence on research or professional work. One of them is to compare
the citations in the publications of an institution with the collection in the institution's library. The percentage of material
cited in academic publications that was - or could have been - retrieved via the local library (print and electronic collection)
is seen as indicating the benefit derived for research. Such studies have been carried out e.g. in Helsinki (Ahtola, 2002)
with doctoral dissertations, and in the University of Georgia Libraries (Smith, 2003) with theses and dissertations. The
University and Regional Library of Münster is conducting a project on citations in doctoral dissertations in four subjects, combined with a questionnaire
sent to the authors asking for their ways of procuring information for their thesis. Thus, the statistical results - the percentage
of cited material being available via the library - can be validated by the authors' estimate of the library's role.
The users' estimate of the library's importance for their work has been used in several other projects. In the research department
of the UK Ministry of Defence, the members of the department - after writing a report - were asked whether using the information
resources department (the library) had influenced the production of the report (Stewart & Thornton, 2002).
| • |
What services were used when producing a report? |
| • |
What influence had this use? |
| • |
How could a similar effect be achieved without the existence of the information centre? |
The results clearly demonstrated the importance of the information centre. As the authors state, the results might have been
influenced by the interviewee's fear that the centre might be given up. But even that would point to a high estimate of the
information centre's importance.
Information literacy and information retrieval
Today, information literacy, the competence and skills necessary for finding, using and rating information, is indispensable
for study, research and education. The growing amount of available information and the 'Google mentality' of students make
it necessary to integrate information literacy training in the curricula. When universities stress outcome-based education
or accreditation models, information literacy plays an important role.
The Association of College and Research Libraries has summarized the competences of information literacy in standards. Such competences for information literate students are for instance the ability to:
| • |
define and articulate the kind of information they need (e.g. formulate a topic) |
| • |
identify types and formats of potential sources for information (e.g. differentiate between popular and scholarly resources) |
| • |
consider the costs and benefits of acquiring the needed information (e.g. decide on when to use other than local information) |
| • |
select the most appropriate method for information retrieval (e.g. know the scope and content of information retrieval systems)
|
| • |
evaluate the information and its sources (e.g. compare information from various sources as to its reliability) |
Are library services helping to gain such competences?
Libraries have tried several methods to assess the direct outcome:
| • |
surveys after instruction ('reaction data') |
| • |
self-assessment of users |
| • |
behavioural observation |
| • |
assessing the quality of bibliographies in students' papers |
| • |
tests (before and after instruction). |
One example of a survey asking students for self-assessment of their information skills was conducted in 2001 at the University
of Louisville/USA [2]. The students rated the effects of library use and training on:
| • |
their skills in using PC's and the Internet |
| • |
their ability to find and select information sources |
| • |
their academic success |
The results showed a high estimate of the library's influence on users' skills. Of course, a generally friendly attitude towards
the library may have influenced the answers. A more difficult problem is, that self-assessment is often not reliable. In one
example 90% of students rated their library skills as adequate, but in a test of competences, only 53% proved "minimally competent"
(Ware, 1986, p. 9). Tests for assessing information literacy are widely applied (Bober & Vileno, 1995; Rabine & Cardwell,
2000; Brown & Krumholz, 2002). They range from a 'mini-quiz' to elaborate tests in several steps. The results show the competences
and skills of students, but again it is difficult to track those skills back to library use and user training.
Another way to prove the effects of library services is to assess the library's actual and potential role in the information
seeking process. Electronic media and new ways of communication have established information channels outside libraries. Therefore,
libraries must redefine their role in information seeking and information retrieval. The questions are:
| • |
What part of information seeking and information provision is still carried out via libraries? |
| • |
Where could libraries step in? |
| • |
What is their role for information seeking and provision as to certain subjects or professions? (e.g. for chemists, psychologists,
nurses, teachers) |
Quite a number of studies have tried to assess information seeking behaviour (Hiller, 2002). A project in Germany, sponsored
by the Deutsche Forschungsgemeinschaft (German Research Association), was set up to evaluate the German Special Collection
Programme from the users' perspective (Poll, 2003). The study tried to assess users' information seeking behaviour in the
following subjects: English studies, Economics, Biology, History, and Mechanical engineering. In a survey 5000 academics were
asked about their:
| • |
preferred ways of finding information on their subject (information seeking) |
| • |
experience with ways of procuring information and documents (information provision) |
| • |
expectations and wishes as to an optimal supply of information |
The results show clear differences between the subjects: academics in the Humanities and Economics rely much more on library
collections, especially if they have to use historical sources. In Biology and Engineering electronic journals play an important
role. But in all subjects, academics stressed the problems of information overload and the amount of time necessary for finding
the information they need. Their expectations point to the library's new role of selecting information for different subjects
and offering personalized services for special needs. Tracking the information-seeking behaviour can thus be used as an indirect
method of showing the effect of library services on research.
Social impact
Projects conducted in this area have tried to find out whether the existence of libraries - or of one specific library - furthers
the well-being of individual users or the quality of societal life. This includes features like:
| • |
democracy |
| • |
social inclusion |
| • |
cultural life |
| • |
local identity |
| • |
life-long learning |
Methods used to assess the preciousness of library services for the individual (direct benefits) or society in general (indirect
benefits) are:
| • |
interviews (e.g. 'street surveys') |
| • |
questionnaires |
| • |
telephone surveys |
| • |
focus groups |
In most cases, users are also asked for their sociodemographic data (age, gender, ethnic origin, income, employment status,
academic status) and the frequency of library use.
Projects to assess the social impact by social audits have been carried out e. g. in public libraries in Great Britain (Bohme
& Spiller, 1999). In a survey conducted in several towns frequent users, occasional users and non-users were asked to rate
the:
| • |
direct benefit for the individual user by a library visit |
| • |
indirect benefits achieved by the existence of the library (democracy, cultural life in the town, leisure time value etc.) |
| • |
potential benefit of being able always to use a library ('contingency value') |
| • |
value for potential future users |
Frequent users rated direct as well as indirect benefits high, for occasional users and non-users the indirect or future benefits
were most important. Especially interesting was the rating for certain functions of public libraries. There was a general
consensus on the following priorities:
| • |
children's literacy |
| • |
establishing reading habits |
| • |
leisure reading |
| • |
cultural meeting point |
| • |
access to information |
| • |
help in finding information |
| • |
job and training information |
Results like these might be used to demonstrate 'outcome' by showing the public estimate of the value of library services.
But the assessment is, of course, subjective; evidence is often anecdotal.
Economic value
Assessing library outcome as an economic value does not mean to calculate the costs of a library or of one case of library
use. This has been dealt with in several handbooks (Snyder, 1997; Ceynowa, 2003), and calculating costs is, of course, necessary
when evaluating the efficiency of libraries. When assessing outcome, calculating the financial value means assessing the actual
or potential benefits to users quantified in money. This can be done by:
| • |
calculating 'proxy prices' ('shadow prices'): prices that would be paid for a library service on the free market, |
| • |
finding evidence of libraries directly affecting the economics of their institution or community, |
| • |
assessing the value a commercial firm or an institution puts into account for library use by calculating the time spent on
such use by members of the firm or institution. |
All these methods try to show return on investment.
One way to calculate 'proxy prices' is to let users estimate the economic value of library use. The example given here is
taken from a study in public libraries in Great Britain (Morris etc., 2001).
When returning a book 557 users in 4 libraries over 4 weeks were asked:
| • |
for fiction (374 titles): Did you like the book? |
| • |
for non-fiction (219 titles): Did you profit by the book? |
If users answered positively (90%), they were asked to name the price they would have been willing to pay for borrowing; users
could choose between 8 categories of prices (between 0 and more than 3 £). The average price was:
| • |
for fiction 62,2 Pence |
| • |
for non-fiction 66,3 Pence |
| • |
all books 63,7 Pence |
This average value was compared to the purchase price of the books. The result was that the value of one issue constitutes
about 7-8% of the purchase price. Other studies that tried to compare the purchase price of a book with a 'lending value'
showed the lending value to be between 7 and 25% of the purchase price (Newhouse & Alexander, 1972). The assumed value of
63,7 Pence is clearly lower than the real costs of lending one issue (about 1,41 £); this shows that users underestimate the costs of the lending service. The problem of this method is: Would users indeed
pay the price they named for borrowing a book? Another method to establish 'proxy prices' is the comparison with commercial
prices for certain services, e.g. by comparing with prices of commercial lending libraries or information brokers.
The method assessing the 'willingness-to-pay' has also been used in order to calculate the economic value of the library in
total. The St. Louis Public Library Services asked their users, what sum they would be willing to pay more in taxes if all
libraries were to be kept open, and for what amount of reduced taxes they would be willing to have the libraries closed (Holt,
1999). The result - like other such investigations - showed the 'willingness-to-pay' to be lower than the 'willingness-to-accept':
| • |
Users wanted 9 $ payback to every 1 $ in current taxes if they agreed to close the libraries. |
| • |
Users agreed to pay 1 $ more to every 1 $ in current taxes if all libraries would be kept open. |
A last method trying to establish a financial value of library services is 'replacement value of time'. Users must spend time
in order to reach a library and in using library services. The hypothesis is, that the value that they place on library use
is as great as their consumption of time, and this time is calculated in financial terms by average salaries. This has been
tested in public and special libraries. St. Louis Public Library Services compared the economic value that users placed on
the consumption of a service to users' costs (in time and travelling expenses) and got a 'consumer surplus' of 3 : 1. The
Parliamentary Library New Zealand set the 'time costs' of users in comparison to the purchase costs of the assets used to
provide a library service and found out, that the services had a value of between 2 and 20 times the annual library budgets
(MacEachern, 2001).
Calculating the replacement value of a user's time proved most efficient in special libraries with a defined clientele. Libraries
of commercial firms measured the time the members of the firm spent on library services and calculated a price for the service
used by average salaries. This indicates the value that the firm or institution puts on library services (Griffiths & King,
1993).
Calculating proxy prices, assessing a consumer surplus or the added value - all these methods try to show, that libraries
do not only create immaterial value, but that a market value can be proved and that there is indeed a return on investment.
The results are especially interesting to the financing authorities.
USER SATISFACTION AS AN INDICATOR OF OUTCOME?
In the discussion upon outcome assessment, experts disagree about the role of user satisfaction. Can high satisfaction of
users been seen as an outcome of library services, as a direct benefit for users?
| • |
"Satisfaction on the part of a user is an outcome. So is dissatisfaction." (ACRL, 1998). |
| • |
"Customer satisfaction ... is neither outcome nor output. Rather, it is a qualitative assessment of library outputs." (Cram,
1999). |
User satisfaction surveys can ask
| • |
for the user's positive or negative experience during one library visit, |
| • |
for users' general experience in a certain library, |
| • |
for expectation compared to experience. |
The results of satisfaction surveys must be evaluated with care:
| • |
Previous experience often affects the perception of services. Good experiences in other libraries can lead to high expectations,
bad experiences to low expectations. |
| • |
Loyalty to the home library may influence the answers (e.g. in judging friendliness of staff). |
| • |
The opinion expressed may be a momentary 'snapshot', influenced by criteria outside the library's responsibility (like building
noise). |
And - most important -
| • |
Users may be quite satisfied without any tangible benefits. |
Satisfaction certainly indicates that the library has been effective in transporting the idea: It is worthwhile to visit/use
this library. But does that indicate that users indeed benefited, that library use led to a change in skills, attitudes, behaviour,
and knowledge? User satisfaction might be seen as a foundation for such changes, furthering the receptiveness of users and
therewith their ability to benefit from the libraries services.
THE NEXT STEPS
There is indeed a broad range of methods that have been tested in order to prove and substantiate the outcome of library services.
All projects demanded a high input of resources, especially as most methods rely on the cooperation of users in the form of
interviews, surveys, or tests. The growing number of such studies shows, that libraries see the urgent need of demonstrating
and proving the benefits effected by their activities.
Though not all results of the present projects have been convincing or statistically valid, they have shown up ways to assess
an economic value of library services, the social value imputed by users and non-users of libraries, or the outcome on information
literacy, information retrieval, and even on academic and professional success. By international cooperation and tests on
a more comprehensive basis, the methods should be validated and improved in order to render comparable and reliable results
possible.
Are libraries making a difference? We are challenged to prove it.
REFERENCES
Ahtola, A.Anneli:
How to Evaluate and Measure the Impact of the Library's Collection on the Learning Outcome? 68th IFLA Council and General Conference, August 18-24, 2002
http://www.ifla.org/VII/s2/conf/ahtola.pdfBarkey, P.: "Patterns of Student Use of a College Library". College and Research Libraries, 26 (1965), 115-118.
Bober, C., L. Vileno: "Evaluating Library Instruction in Academic Libraries: A Critical Review of the Literature 1980 - 1993".
The Reference Librarian, 51/52(1995), 53-71.
Bohme, S., D. Spiller (Eds.): Perspectives of Public Library Use. Loughborough : LISU, 1999.
Brown, C., L.R. Krumholz: "Integrating Information Literacy into the Science Curriculum". College & Research Libraries, 63(2002), 111-123
Ceynowa, K., A. Coners: Cost Management in Academic Libraries. München : Saur, 2003.
Cram, J.: "Six impossible Things Before Breakfast. A Multidimensional Approach to Measuring the Value of Libraries". In: Proceedings of the 3rd Northumbria International Conference on Performance Measurement in Libraries and Information Services, Newcastle upon Tyne, 1999, pp. 19-29.
Griffiths, J.M., D.W. King: Special Libraries: Increasing the Information Edge. Washington D.C.: Special Libraries Association, 1993.
Hiller, S.: "How Different Are They? A Comparison by Academic Area of Library Use, Priorities, and Information Needs at the
University of Washington".
Issues in Science and Technology Librarianship, Winter 2002.
http://www.istl.org/istl/02-winter/article1.htmlHiscock, J.E.: "Does Library Usage Affect Academic Performance?" Australian Academic andResearch Libraries, 17(1986), 207-213
Holt, G. E., D. Elliot, A. Moore: "Placing a value on public library services". Public Libraries, March/April 1999, 89-108.
Jager, K. de: "Impacts & Outcomes: Searching for the Most Elusive Indicators of Academic Library Performance". In: Proceedings of the 4th Northumbria International Conference on Performance Measurement in Libraries and Information Services, Washington, DC, Association of Research Libraries, 2002, 291 - 297.
Kramer, L.A., M.B. Kramer: "The College Library and the Dropout". College and Research Libraries, 29(1968), 310-312.
MacEachern, R.: "Measuring the Added Value of Library and Information Services". IFLA Journal, 27(2001), 232-236.
Morris, A., M. Hawkins, J. Sumsion: "Value of Book Borrowing from Public Libraries". Journal of Librarianship and Information Science, 33(2001), 191-198.
Newhouse, J. P., A. J. Alexander: An Economic Analysis of Public Library Services. Santa Monica, Calif. : Rand, 1972.
Poll, R.: "Assessing the Library's Role in Information Seeking". Performance Measurement and Metrics, 4(2003) [in print]
Rabine, J., C. Cardwell: "Start Making Sense. Practical Approaches to Outcomes Assessment for Libraries". Research Strategies, 17(2000), 319-335.
Revill, D.: "Performance Measures for Academic Libraries". In: Encyclopedia of Library and Information Science, Vol. 45, Suppl. 10 (1990).
Self, J.: "Reserve Readings and Student Grades: Analysis of a Case Study". Library and Information Research News, 9(1987), 29-40.
Smith, E.T.: "Assessing Collection Usefulness: An Investigation of Library Ownership of the Resources Graduate Students Use".
College & Research Libraries, 64(2003), 344-355.
Snyder, H., E. Davenport: Costing and Pricing in the Digital Age. London : Library Association, 1997.
Stein, Joan, Martha Kyrillidou, and Denise Davis (eds.). Proceedings of the 4th Northumbria International Conference on Performance Measurement in Libraries and Information Services:
"Meaningful Measures for Emerging Realities". Washington, DC : Association of Research Libraries, 2002, 364 p.
Stewart, C., S. Thornton: "The Use of an Impact Survey on a Measure of Special Library Performance". In: Proceedings of the 3rd Northumbria International Conference on Performance Measurement in Library and Information Services, Newcastle upon Tyne, 1999, pp. 243-246.
Ware, S.A., D.J. Morganti: "A Competency-based Approach to Assessing Workbook Effectiveness". Research Strategies, 4 (1986).
Notes
[1] In the existing literature, terms like 'impact', 'value', or 'benefit' are also used to describe the effect of library services.
[2] The questionnaire was given to the author by H. Rader, Dean of University Libraries at the University of Louisville in Kentucky.
LIBER Quarterly, Volume 13 (2003), No. 3/4