BACKGROUND

The Universitäts-und Landesbibliothek, (ULB) Münster, has a substantial background of research and publication in performance measurement and evaluation of library and information services (Poll and te Boekhorst) and has conducted and published internal studies of various aspects of its services. (Obst 1995; Buch 1996). My contact with the Library goes back to late 1996 when I spent two weeks at ULB on a British Council exchange scheme with North Rhine Westphalia which gave me some understanding of how the Library operates and some knowledge of the technical vocabulary of German academic librarianship. In 1998 I directed a qualitative study of the use being made of Glasgow Caledonian University Library’s recently opened Electronic Information Floor (Crawford 1999). A literature review undertaken prior to the study showed that surprisingly little research had been done in this area and that one of the earliest studies was that undertaken by Dr Oliver Obst at Münster (1995). Coincidentally ULB is a partner in the EQUINOX Project, led by CERLIM, which aims to develop a set of standardised performance measures for the electronic library and was interested in collecting some qualitative data as a background to the Project. After email discussions with ULB it was decided that I should visit the Library again to undertake a qualitative study of the use of electronic information services there which would act both as a comparative study with Glasgow Caledonian University and a background study to the much larger, CERLIM led, EQUINOX Project.

PREVIOUS STUDIES

The findings of previous studies, have, on the whole, been pessimistic. There is lack of agreed performance indicators (Wynne and Brophy 1998) and, as the Glasgow Caledonian University study shows, the lack of agreed performance measures is compounded by users’ confusion about the difference between IT services generally and electronic information services specifically. Obst (1995) found that email and the Internet were the services mostly used by students and these mainly for private purposes. A survey of web use by undergraduate business school students at the University of Sunderland (McKeever 1998) was undertaken in May 1997. This showed that most respondents experienced such difficulties in navigating the web that its potential use value was greatly diminished. There was a conspicuous lack of awareness about web information gateways. ‚Entertainment purposes’ also accounted for a significant proportion of web use; 81 % of respondents encountered problems frequently or occasionally. Students customarily sought help from other students. Search engines were not found to be very helpful.

The Glasgow Caledonian University study confirmed these findings. A mixture of qualitative methods was used (Crawford 1999) consisting of 47 semi structured interviews and three independently facilitated focus groups. The questionnaire method was not used because of a lack of clearly defined performance issues on which to base questionnaire questions.

Some conclusions from the study were as follows:

• The distinctive mission of the Electronic Information Floor (EIF) was not clear to users who simply view it as a collection of computers located in the Library

• Much of the use was unsophisticated centring around email, the Internet and word processing. Electronic information services are less used.

• There is a good deal of non-curricular use centring around using email and the Internet for recreational purposes.

• Levels of IT skills are low especially among non traditional students.

• Much of the learning was from other students and not from Library or other staff.

The study also highlighted general issues requiring further study. Users did not appear to distinguish between electronic services generally like email and word processing packages and specific electronic information services like Science Citation Index. They saw the matter more in terms of ‚things you can do on a computer’. The low qualitative levels of use identified suggested that electronically gathered quantitative statistics would not reveal qualitative levels of use and would not be very meaningful. They might even measure services which the Library does not aim to provide such as the Internet based Hotmail service, rather than University authorised email. Low levels of IT skills also highlighted another problem. Traditional user orientated evaluation assumes that users understand the service provided well enough to comment on it intelligently. Clearly, in the measurement of electronic information services, this is an uncertain assumption. More positively the study raised the possibility of geographically widespread, perhaps even worldwide performance measurement if the same electronic information services are widely used in different countries.

It should, however be noted that Glasgow Caledonian University is something of a special case in that 30 % of our students are part timers and both the EIF study and a previous focus group based study of part time students (Crawford 1997) has shown levels of IT skill among part time students to be extremely low. A comparative perspective, preferably an international one was therefore required.

METHODOLOGY

As indicated above previous contacts with ULB Münster suggested that it would be a good base for a comparative study. Münster University is very different from Glasgow Caledonian University which has some 14,500 students, many drawn from relatively poor family backgrounds (60 % of our students do not pay fees). The University was founded as recently as 1992 as a result of a merger of two polytechnic type institutions and most of the courses are vocationally orientated. Glasgow Caledonian University has only two campuses, one much smaller than the other. By contrast Münster University is the fourth largest university in Germany with some 50,000 students, all of whom are at least nominally full time and are more middle class in social composition. The University teaches a much wider range of courses and is much more research orientated than Glasgow Caledonian University. It is not a campus university and its various departments straggle over a distance of about three kilometres from the old medieval city centre up to the ring road where the university hospital and related departments, including the medical library are situated. This geographical dispersion has encouraged the creation of many departmental libraries which range in size from small rooms containing a collection of textbooks and other monographic materials to large professionally staffed collections, offering a wide range of services, both conventional and electronic. A further consequence of this geographical dispersion is that some departments are not yet on the university network. (This inhibited discussion about the remote access issue).

A qualitative study, extending over a two-week period was therefore agreed and ULB agreed to provide participants. As ULB had no previous experience of qualitative methodologies the exercise is methodologically interesting to staff there. The focus group method was initially considered but rejected in favour of one to one interviews. It was felt that an overseas interviewer, especially one not intimately familiar with local circumstances, would have to spend too much time in a focus group clarifying details and getting to know the participants for the exercise to yield a lot of useful information. The one to one interview, with an open ended time scale, would give interviewer and interviewee time to get to know one another and discuss issues arising in the depth they required. Another advantage of this method is that it allows the interviewee to raise issues which he or she considers important, thus expanding the scope of the study, which, in this case, is what happened.

Ten people were interviewed, four academic staff, five students and one external user. The students were all part time student library assistants and had a good understanding of the issues involved. The staff included a professor. Interviewee interest was high. Three interviewees expressed a strong desire to be interviewed, one even telephoning the Library to make an appointment. Questions to be put to the interviewees were formulated in conjunction with ULB staff and likely issues and problems were discussed in advance. Questions of direct relevance to management of the Library were included. Academic staff were interviewed in their offices and students and the external users were interviewed in a meeting room at ULB. For practical reasons interviews with students and the external user were audiotaped but those with staff were not. All interviews were in English but technical terms were verified in German where appropriate. The results and analyses of the interviews were written up immediately after the interviews had taken place. After the interviews had been concluded, factual issues, although not of course, interviewees’ opinions were verified with ULB staff, thus guaranteeing the accuracy of the data.

INTERVIEWING THE STAFF

The four academic staff interviewed naturally had some knowledge of the administration of ULB and therefore took a more ‚top down’ approach than the students. They focused primarily on a number of issues

• The subject librarian system. All spoke highly of this. Communication between academic staff and subject librarians was considered to be good even if the lecturer had never met the subject librarian. Email correspondence was considered to be a satisfactory means of communication.

• Search engines and gateways were relatively little used. One interviewee regularly uses a search engine but seemed to be untypical. The ULB web pages and collections of bookmarks seemed to be preferred substitutes. One interviewee had effectively created her own gateway for herself and her students by creating a website listing relevant sites in her subject area.

• Only a small number of bibliographic databases are known or used by interviewees. Most interviewees used one or two ‚standards’ such as Medline or the MLA database and supplemented them with specialist news groups, email discussion lists and specialised websites.

• In view of ULB’s heavy investment in electronic journals interviewees were asked about their preference for paper or electronic journals. There was a general preference for paper over electronic journals even among the highly computer literate. One spoke of the ‚dignity of printed media’ and another described visits to the library to scan current journals as ‚inspirational’. One interviewee did not know that the Library subscribed to electronic journals and was delighted when these were identified on the ULB web pages.

• The high cost of journal subscriptions in whatever format raised the question of using inter library loans as a substitute and interviewees were asked about their use of the paid inter library loan service (JASON) available in North Rhine Westphalia (but not throughout the Bundesrepublik). All agreed that inter library loans were not an acceptable substitute for journal subscriptions. There were mixed opinions about the question of payment but questioning revealed that there is a range of avoidance mechanisms in place. These include:
  1. Writing to the article author to ask for a copy
  2. Visiting the holding library >
  3. Getting the Department to pay

• Interviewees expressed a strong interest in access to higher level monographic material and this led to expressions of interest in transparent access to a wide range of university library OPACs, a union catalogue in fact. One interviewee said that this should cover all university libraries in the Bundesrepublik. The existing union catalogue interfaces for cooperative cataloguing systems (Verbunddatenbanken) were commended as a useful first step.

• Students’ IT skills. Interviewees were generally pessimistic about their students IT skills and two were equally pessimistic about their colleagues. One had given IT skills training to students and had run a poorly attended course for colleagues and commented ‚Nobody cares for computers’ [in my department]. Another mentioned a survey he was completing of his students’ IT skills. This shows that there is no relationship between computer ownership and IT skills. Although 90 % of his students own computers, only 20 % – 30 % know how to use them. Between 70 % and 80 % have access to the Internet but do not know how to use it. This pessimistic view was partially borne out by interviews with students.

• Interviewees were asked about the Library’s programme of information seeking skills training. There was general ignorance about it and no one had any opinions about it except that it was ‚a good thing’.

• The use of Institute (departmental) versus central libraries. Interviewees tend to use the Institute libraries first because of their specialised stocks and convenient location. ULB was used for more general material and visited less frequently sometimes because of the distance from the interviewees department.

INTERVIEWING THE STUDENTS

While the academic staff interviewed generally recognised and accepted the agenda of the interview the students ranged more widely, raising issues on their own initiative. All were at (at least) senior undergraduate level. All speak English fluently and have either visited or lived in an English speaking country. All (at the time of the interviews) worked in ULB for between 9 and 15 hours a week. Two worked on public service points and two had used British academic libraries while studying at UK universities. This latter factor had a noticeable effect on their perceptions. The main issues discussed were:

• Information sources used. The main source is textbooks. This led to a discussion about the OPAC which is appreciated as a union catalogue as it holds the stocks of Institute (departmental) libraries as well as that of ULB. Interviewees used it for author and/or title (i.e. known item searches). Only one mentioned keyword searching. The reservation option was also mentioned. This became linked to a student led discussion about shelf arrangement (See below – Wider issues). They were also interested in the idea of searching across a range of university library OPACs but less so than academics. They make little use of the charged for JASON document delivery service but do not appear to be deterred by the cost (3 DM per item, about £1 sterling). All the interviewees are heavy users of photocopying. The use of electronic information databases was restricted. Like the academics students rely on a small number of ‚standards’ like the MLA database or the Internationale Bibliographie für Zeitschriftenliteratur (incidentally about the only non English language database mentioned). Only one interviewee used a range of electronic information sources.

• Other uses of computers. This was, on the whole, unsophisticated. Email is used mainly for recreational purposes. Internet use is limited and unsophisticated. Only one student mentioned using a search engine and no one seemed to know about gateways. Four of the interviewees have computers at home but only one has a modem so for the other three their computers functioned largely as electronic typewriters. Word processing for essays and major projects was a key area of use.

• Interviewees were asked what they thought of the ULB’s information services. As this was a broad question some interviewees found it rather confusing but usually referred to the OPAC and then went on to talk about the Enquiry Desk (Auskunft) services which they rated highly. Two issues were mentioned here:
  1. Problems with computers
  2. Directional questions – See below (Wider issues)

• ULB’s information seeking skills training programme. Interviewees did not know much about this. Only one had received training on an electronic information database. Three had attended or knew about introductory tours and thought them useful.

• The relationship between ULB and the Institute (departmental) libraries. This is viewed as a complex issue.

ULB is respected for:

1. Its large stock;

2. OPAC access to both ULB and Institute stocks;

3. The OPAC makes retrieval faster. (Some Institute libraries are not computerised).

(However items are not always on the shelves and have to be reserved.)

The Institute libraries are respected for:

1. Their specialised stocks;

2. They are small and user friendly;

3. The materials sought are always available.

The relationship between ULB and the Institute libraries seems to be a complementary, almost symbiotic one.

INTERVIEWING THE EXTERNAL USER

The external user is a Slavonic studies expert and openly admitted to a lack of IT skills although he has no difficulty using the ULB OPAC. His main source of information is books and, unlike the other interviewees, he is a heavy user of inter library loans although this may be related to the fact that he makes little use of scholarly journals. His main sources of information, other than books, are TV and the quality press which he reads partly for book reviews. He rates help from subject librarians very highly. He uses more libraries (5) than any of the other interviewees (ULB, three Institute libraries and the Stadtbibliothek) but has no preference or favourite although he rates highly the arrangement of books by subject which does not exist at ULB (See below <– Wider issues). He was the only interviewee to mention the Stadtbibliothek (town library).

COMPARING STAFF WITH STUDENTS AND EXTERNAL

• Students are much less aware of the subject librarian system than staff or externals. Students are more aware of the information (Auskunft ) service.

• Both staff and students are largely unaware of search engines and gateways but for different reasons:

1. Staff go straight to specialised websites,

2. Students know few websites, other than those of personal interest, or want to.

• Both staff and students are ignorant about the Library’s information seeking skills training programme.

• Apart from the external user little use is made of directly paid for inter library loans.

• Staff or students, so far, do not make heavy use of electronic journals. The students’ references to photocopying show them to be very paper orientated.

• Both staff and students are interested in the possibility of electronic union catalogue developments at a Bundesrepublik level but staff more so.

• Staff use email more for work while students use it more for non curricular purposes but neither group uses many electronic information databases.

• Staff have low expectations of student IT skills which is partially borne out by the student interviews.

• ULB and the Institute libraries. All interviewees see an essential complementary relationship here.

• ULB shelf arrangement. Staff did not mention this. Students and the external did.

• All interviewees asked for a copy of the report on the study.

WIDER ISSUES AND QUESTIONS RAISED BY THE STUDY

It is axiomatic that a qualitative study is essentially a collection of stories told by individuals or groups and that qualitative evidence needs to be tested further by quantitative means. The qualitative data summarised here raises several questions:

SHELF ARRANGEMENT AT ULB AND ITS RELATIONSHIP TO SUBJECT SEARCHING ON THE OPAC

At ULB books are shelved in order of receipt, not by subject, and books on the same subject are scattered throughout the Library. (The reason for this is purely historical. ULB was built as a library with closed stacks and not opened to public access until 1989. Over 80 % of the stock is now on open access). Some of the students were critical of this, describing the arrangement as unhelpful and shelf marks as confusing. (The external user also made this point, drawing attention to the helpful, subject based, shelf arrangement at the Stadtbibliothek.) In this context the role of the OPAC in subject searching is clearly crucial and, indeed, ULB has invested a good deal of time and money in enriching bibliographic records with subject headings but the evidence of the interviews does not suggest much use of these. Instead it suggests that the OPAC is mainly used for know item searching. Further quantitative data seems to be needed here to verify if this is the case.

DIRECTIONAL ENQUIRIES AND LIBRARY LAYOUT

Students who worked on public services points mentioned that they frequently received directional enquiries from users and this issue may be linked to the OPAC use issue. Although the Auskunft (Enquiries desk) records statistics of enquiries it does not record directional enquiries because ULB does not consider this to be a relevant indicator of the quality of the information service provided by the Auskunft. This seems a pity, as directional enquiries are the largest single area of enquiries in many academic libraries. Statistical information about directional enquiries would show whether users genuinely have difficulties in finding their way around the Library. If they do remedial help such as floor plans could be issued to students.

INFORMATION SEEKING SKILLS TRAINING

The Auskunft (Enquiries desk) service was commented on more and used more by students than the specialised training offered by subject librarians. Basic induction tours were valued more than specialised training. One student also raised the question of the availability of IT skills training. (This is provided by the University computing staff). This raises the question as whether there is a need for more, basic, information seeking skills training, especially as there may be orientation and OPAC issues and less concentration on higher level training. Currently the teaching departments provide basic information seeking skills training in their courses for beginners.

THE LIBRARY’S AIMS

The above is a broader question than it might appear because it raises the question of ULB’s market penetration aim. Is it to involve fewer students at a higher level or more students at a lower level? Are some students put off using the Library or indifferent to it? One student interviewee quoted the example of a senior undergraduate who had lived in Münster for several years but did not know where the Library is, despite the fact that it lies at the heart of the old university quarter. The orientation and item finding issues may contribute to discouraging students. One interviewee spoke of ‚Schwelle’ (literally – threshold) and the idea of ‚crossing the threshold’ something which she herself found easier as result of having used a British academic library.

THE ZWEIGBIBLIOTHEK MEDIZIN STUDY

These findings can, to some extent, be triangulated against a relatively recent user satisfaction study of the Medical Branch Library of ULB Münster (Jungnickel and Obst 1997). It found users’ preferences, in order, to be

  1 Opening hours,
  2 Photocopying services,
  3 The lending service,
  4 Textbooks,
  5 Enquiries service,
  6 PC workstations,
  7 Open access areas,
  8 Inter library loans,
  9 Catalogue,
10 User education,
11 The Internet.

This seems to broadly reflect the agenda of the interviews, especially those of the students. An analysis of main patterns of use showed the order to be:

1	going directly to the shelves (the classification is NLM and therefore browsable)
2	consulting the (at the time) microfiche catalogue,
3	searching for journal articles in databases such as MEDLINE,
4	scanning key journals.

Again the agenda of the interviews seems to be present.

Satisfaction with databases was high, despite the fact that the number of electronic information databases available has actually been reduced. Only 5 % of respondents asked for more databases. Electronic services were asked about in the questionnaire and it emerged that 52 % of respondents had never used the Internet.

A comparison between ULB and the Medical Library in four key areas: General; Opening Hours; Books and journals and Textbooks showed the Medical Library to be rated higher in all these areas, all of which constitute much of the basic information agenda of the undergraduate.

The survey courageously sought areas of overt dissatisfaction and found it in inter library loans service which was considered to be too slow. Although solutions to this are being considered and adopted it is interesting to compare this finding with the evidence of avoidance mechanisms in the interviews. There may be an issue here requiring further investigation especially if the Library hopes to replace journal subscriptions with inter library loans.

USE OF ELECTRONIC INFORMATION SERVICES AT ULB COMPARED WITH OTHER STUDIES

Most of the work quoted as background to this study is qualitative or limited in scope and the data reported here represents the views of ten individuals. The results have to be used with caution pending the completion of the large scale research projects which are obviously necessary. Nevertheless the degree of agreement across the studies is notable. The same general points seem to emerge:

• The use of electronic services is unsophisticated centring around email, word processing and the Internet.

• A limited number of electronic information databases are used. Users tend to settle for one ‚standard’.

• There is a good deal of recreational use of electronic services in a supposedly curriculum related context.

• Levels of IT skill are variable and frequently low.

• Search engines and gateways are not widely used.

Further support for these findings comes from an elaborate study at Cornell University (Payette and Rieger 1998) which used a combination of different techniques such as observation, semi structured interviews, a questionnaire and focus groups. This found a wide ignorance of the electronic sources available and how they are accessed. Staff typically only used two or three databases and none of the students used the library provided web gateway to access databases although they did use internet search engines to locate information for course work. Staff and students both wanted swift access to relevant material with minimal investment in learning and searching time.

THE IMPLICATIONS FOR PERFORMANCE MEASUREMENT

The study has sought and found evidence of common performance issues and confirmed the problem of recreational use which raises the danger of measuring things which either do not matter or could be actively misleading to measure. But there is clearly scope for international comparison. A limited number of databases are being widely used and could be readily compared. The future seems to lie with the comparison of English language databases which are clearly dominant except in some social science areas where the native language is still preferred for publication. Social sciences seem to be an example of this, especially as the Internationale Bibliographie fur Zeitschriftenlitteratur was about the only non English language database mentioned. (IBZ covers all non science and technology subject areas). Overall the picture seems to be that the possibilities of wide ranging performance measurement for the electronic library are enormous but so are the implementational pitfalls. If easy, widespread comparative evaluation of electronic information services becomes possible then the implications for the future of some of those databases may be considerable. Performance measurement for the electronic library may prove too have greater accounting possibilities than previous forms of performance measurement.

REFERENCES

Crawford, John C. (1997) Report on the general satisfaction survey conducted at Glasgow Caledonian University Library, February/March 1997 and a linked focus group investigation. SCONUL Newsletter, no.11, Summer/Autumn. 11-16.

Crawford, John C. (1999) A qualitative study of the use of networked software and electronic information services at Glasgow Caledonian University Library. Education for information, vol.17, no. 2 , June, 101-111.

Jungnickel, Lydia and Obst, Oliver (1997) Benutzerzufriedenheitsstudie der Zweigbibliothek Medizin der ULB Münster. AGMB aktuell. Oct. no. 2, pp. 17-19.

McKeever, Lucy (1998) Intellectual resource or digital haystack? web experiences of undergraduate business school students. [online] IRISS ‘98 Conference papers. SOSIG. Available from: ttp://www.sosig.ac.uk/iriss/papers/paper20.htm [Accessed August 1998].

Obst, Oliver (1995) Untersuchung der Internetbenutzung durch Bibliothekskunden an der Universitäts- und Landesbibliothek (ULB) Münster. Bibliotheksdienst, vol. 29, no.12, pp. 1980-1998.

Payette, Sandra D. and Rieger, Oya Y. (1998) Supporting scholarly inquiry: incorporating users in the design of the digital library, Journal of academic librarianship, March, pp. 121-129.

Poll, Roswitha and te Boekhorst, Peter (1996) Measuring quality: international guidelines for performance measurement in academic libraries. München: Saur.

Wynne, Peter M. and Brophy, Peter (1998) Performance measurement and management information for the electronic library. Proceedings of the 2nd Northumbria International Conference on Performance Measurement in Libraries and Information Services. Newcastle upon Tyne: Information North. pp. 295-303.

ACKNOWLEDGEMENTS

I am grateful to Dr. Peter te Boekhorst for his strong support of all aspects of the project. Also Harald Buch and Dr. Oliver Obst for their help and support. The work was funded by grants from the Carnegie Trust for the Universities of Scotland and the Library Services Trust.

Dr. John C. Crawford
Glasgow Caledonian University
Caledonian Library and Information Centre
Cowcaddens Road
Glasgow G4 0BA
J.Crawford@gcal.ac.uk