Developments in Surrogating Methods

Hans van Dormolen

Research & Development Division, Koninklijke Bibliotheek, National library of the Netherlands
P.O. Box 90407, 2509 LK The Hague, Netherlands
hans.vandormolen@kb.nl

Introduction

In this paper, I would like to talk about the developments in surrogating methods for preservation. My main focus will be on the technical aspects of preservation surrogates. This means that I will tell you something about my job as Quality Manager Microfilming for the Netherlands’ national preservation program, Metamorfoze, which is coordinated by the National Library. I am responsible for the quality of the preservation microfilms, which are produced for Metamorfoze. Firstly, I will elaborate on developments in preservation methods in relation to the following subjects:

In the closing paragraphs of this paper, I would like to tell you something about the methylene blue test. This is an important test for long-term storage of preservation microfilms. Also, I will give you a brief report on the Cellulose Acetate Microfilm Conference that was held in the British Library in London, May 2005.

Preservation microfilms

There are three questions which need to be answered before anyone can attempt to produce surrogates; the questions ‘what’, ‘why’ and ‘how’. What do you want to achieve? Why do you want to achieve this? And how are you going to achieve it? Are your goals attainable and realistic?

What do you want to achieve?

The Metamorfoze program aims to ensure that each specific detail of an original image is also visible in the microfilm version - not only in the first generation film, but also in the second (working copy) and third generation (user copy). It does not really matter whether the user copy is a film or a digital image; it should contain all the information, which the original carries. This aim surpasses legibility. We want to preserve the whole item.

Why are you doing this?

The paper-based originals are in a bad condition. Paper documents dating from the period 1840 to 1940 suffer from a form of decay that gets worse over the years. The decay is caused by the presence of acid and lignin in the wood pulp from which the documents are made. Eventually, the paper will become so brittle that it will be almost impossible to read the documents without damaging them. So, we have to protect the originals. Users in reading rooms have to consult the microfilm surrogates instead of the originals.

How are you going to achieve this?

How can we make sure that the surrogates are as good as the originals? Is this an attainable and realistic goal? In order to achieve this goal, we have set up guidelines for preservation microfilming, and we ensure that the preservation microfilms are produced according to these guidelines. Later on, I will tell you more about these guidelines. In order to keep the guidelines up to date, we continually conduct research. This is an absolute must; we operate in a dynamic environment in which new techniques - digitisation, scanning from microfilms and the computer-output-microfilm (COM) - emerge constantly. New techniques raise new questions. Through research, we can keep up with all these new developments. We are constantly trying to optimise the technical quality of the surrogates and their use that is why we need to rewrite parts of the guidelines every six or seven months. By doing so, we turn the guidelines into a living document that is always up to date.

Besides developing guidelines, we also offer advice and training to companies that produce surrogates as well as institutions that work with surrogates. We share our findings and knowledge. It is important for us to check at a regular interval whether the produced microfilms comply with the guidelines we have developed. Every two or three weeks, I visit the microfilm companies to perform a technical check up, and sample one out of five first generation microfilms to check the technical quality. During these visits, I discuss different preservation techniques with the staff. When staff members encounter problems and/or difficulties in working according to the guidelines, they contact me.

Guidelines preservation microfilming metamorfoze

The Guidelines Preservation Microfilming Metamorfoze distinguish four main technical aspects in producing preservation microfilms.

Density, resolution and illumination are commonly known and widely accepted as the three main technical aspects. The gamma value is rather new; that is why I would like to focus mainly on this particular aspect.

The gamma value

The gamma value gives you an indication of the relationship between the amount of grey tones of the original and that of the amount of grey tones in the film. In other words: when you have ascertained the gamma value, you will be able to calculate the amount of grey tones you are working with in your microfilm in relation to the amount of grey tones in the original. As soon as you have established the amount of grey tones in the microfilm you are working with, you will be able to calculate the amount of grey tones that is lost. A gamma value 1 means you are working with all the grey tones, so no grey tones are lost. A gamma value 2 means that you are working with 50% of the amount of grey tones, so 50% is lost. A gamma value 3 means that you are working with 33.33% of the amount of grey tones in relation to the original; 66.66% is lost. The gamma value can be used for all generations of the microfilm by just multiplying the gamma values of the different generations with one another. For instance, when the gamma value of the first generation is 2 and the gamma value of the second generation is 2 also, you simply have to multiply these figures. This means that for the second generation you are working with a gamma value 4. Gamma value 4 means one-fourth of the original amount of grey tones, so 25%. When filming manuscripts a 75% loss is quite dramatic.

Gamma and resolution

Gamma and resolution are connected. You cannot work with resolution without knowledge of the gamma value. Digital images demonstrate this; when you change the gamma value you change the resolution, the sharpness, of your image as well. So, the gamma plays a very important role not only for the black and white preservation microfilm, but also for preservation scanning and the Computer Output Microfilm.

Calculation of the gamma value

The gamma value depends on several aspects; first of all, on the film you are using. Each film has its own ″character″ or ability to represent grey tones. Secondly, there is the developer, combined with the temperature and developing speed. Typically, the gamma value is calculated with the help of the so-called ″s-curve″. This is a graph in which the character of the film is expressed. The s-curve is divided into three parts: the foot, the linear part and the shoulder of the curve. Each part offers information about the character of the film. My aim is not to give you a detailed description of the s-curve here; the main point I want to get across is that the curve contains a lot of information, but working with it - drawing and calculating the gamma value - is very complex and time consuming. Fortunately, there is also a much easier and less time consuming way of calculating the gamma. Before I tell you about this other method, however, I want to emphasize that for calculating and working with the gamma value, one needs a complete understanding of the s-curve. The gamma is well known in the photographic science called sensitometry. Sensitometry is the science of measuring the response of photographic emulsions to light. If you want more information about sensitometry then take a look at the Kodak website.

Calculation of the gamma value with the Kodak Gray Scale

The Kodak Gray Scale (Figure 1) enables you to calculate the gamma very precisely within 15 seconds. The Kodak Gray Scale is an oblong piece of thick paper containing patches. The scale comes in two different sizes, both with an identical lay out: the Q-14 and the Q-13. The Kodak Gray Scale Q-14 is 14 inches long, the Q-13 is 8 inches long. To the left you can see patch A, which is totally white (density 0.05). To the right you can see patch number 19. This patch is totally black (density 1.95). Between Patch A and 19 are patch 1 to 18, from grey (patch 1, density 0.15) to almost black (patch 18, density 1.85). With every step from patch A to19, the density increases 0.10. If you want to establish the gamma value, simply measure the density in patch A and in patch 3 on the film. Next, you have to calculate the difference in density between patch A and patch 3 and divide this by three.


Figure 1

When microfilming manuscripts in particular, it is very important to keep the loss of grey tones as limited as possible. Microfilm is black and white only so all colour information has already been lost. And that’s the only thing we are willing to lose. Apart from the original colours, we won’t accept any more loss of information. Consequentially, it is very important to keep the gamma value as low as possible in the first, second and third generation. Whether you are scanning from the film or using the film, you have to keep your gamma value as low as possible.


Figure 2

As stated earlier, the gamma value is calculated in the linear part of the s-curve. In the first generation microfilm, this is the part between patch A and 3 on the Kodak Gray Scale. This area represents the grey tones of pencil writing. I wrote something down with a pencil (Figure 2), and when you take a close look at the grey tones of the writing - I wrote down ‘the gamma’- you are able to see that the tones are all around patch 1 and 2. So, all the information that is written down with a pencil is in that area, between A and 3; this is the exact part of the Kodak Gray Scale in which we calculate the gamma. We work with gamma values 1.4 to 2, for low contrast materials, such as manuscripts. When we are filming in a high contrast mode, for materials such as books and newspapers, we use gamma values from 2.5 to 3.5. The guidelines I wrote for low contrast filming in February of 2004 were based on the quality of the films we produced in 2003 and 2004. The gamma value of those films was around 1.4 to 2. After the introduction of the gamma method in February of 2004, the microfilm companies became aware of the fact that a low gamma value is very important. Nowadays, two microfilm companies that work for Metamorfoze are able to produce microfilms with a gamma value from 1 to 1.4, so around 1. A method leading to two important results: the microfilms we are producing nowadays are twice as good as they used to be. There is no loss of information, or very little, and this is a great achievement.

If you want to know more about this method of calculating the gamma value, read the Metamorfoze Preservation Microfilming Guidelines (Dormolen, 2004).

Density and gamma value

When changing the density, you are not changing the gamma value. The amount of steps on the Kodak Gray Scale depends on the gamma value, which is being used. By measuring the density, one can check if there was a proper exposure, and ascertain that the range of visible patches clearly starts with Patch A. If the density is too high then the image is overexposed. This means that the range of visible patches starts with patch 2 or even 3. The result is a loss of information. For microfilmed manuscripts, a loss such as this can be really dramatic. When the density is too low then the image is underexposed and one cannot work with the full length of the Kodak Gray Scale. There will be a loss of information in the darker parts of the Kodak Gray Scale. This can be really dramatic for microfilmed dark drawings.


Figure 3

Scanning preservation microfilms

Figure 3 shows a drawing by Alexander Ver Huell. He was a famous illustrator in his time. He was born in 1821 and lived and worked in Leiden, a small town in Holland. He made many drawings of the inhabitants of Leiden. Here you can see what people in Leiden used to do on a Sunday afternoon in the winter. They went ice-skating. In a later stage of his life, Alexander Ver Huell suffered from depression. He lost his sense of humour, and his drawings became darker and darker. I estimate that, over the years, Metamorfoze has produced approximately 30,000 microfilms. The collection of Alexander Ver Huell is fairly small, consisting of 12 microfilms in total. Right now the microfilms of this collection are being scanned. Figure 4 shows one of the scanning results. I hope Alexander Ver Huell was not that depressed.


Figure 4

So, what went wrong? Here’s the answer: the scans were made from a film with the wrong gamma value. We used a film with a positive polarity as a second-generation copy in order to produce a diazofilm with positive polarity as a user copy. However, this second generation film has a gamma value of 2, and 2 times 2 is 4. This means we are working with one fourth of the tonal range; the result is an image that is totally black. This is not the result of bad scanning, but the result of scanning from a microfilm with a gamma value that is simply too high. At the moment we are test-scanning from a microfilm with gamma value 1. The results are much better. Again, you can see the importance of keeping gamma values as low as possible.


Figure 5

Newspapers

Here is another example of working with the wrong gamma value. In this case, the issue is not the wrong gamma value of the microfilm, but scanning with the wrong gamma value! Figure 5 shows a digital image of a newspaper. This newspaper was microfilmed in 1999. A few years later, in 2003, bitonal scans were made from these microfilms. Clearly visible here are the gutter shadow and, at the top of the image, a shadow caused by the filming in 1999. Nowadays, gutter shadows are no longer allowed. We advise that the original is disbanded if filming without gutter shadows is not possible. Of course, disbanding is only allowed with permission of the owner. As you can see, the shadow causes a lot of information loss in the digital image. The scans were used in an OCR project but, after a long period of OCR testing, the project staff became really frustrated with the bad retrieval results. They believed that the shadow caused the loss of information, but this was only part of the problem. The real loss was caused by the bitonal scanning. When you take a closer look at the digital image, Figure 6, you will see that it is not possible to read the entire column. The print quality of the original 1920 newspaper was rather poor, and caused differences in the tones of inking. The characters are either black, grey or hardly visible. All the different tones are clearly visible on the high contrast microfilm, but after scanning, they are no longer visible on the bitonal image. When you are scanning bitonal, you are doing the opposite of what you do when you are microfilming. While microfilming we keep the gamma value as low as possible. The newspapers were filmed with a high contrast gamma value, around 3. The gamma value of a bitonal image is infinite. In the case of the newspapers, the gamma value is not infinite, but even more than that. The first generation has a gamma value of 3. So, you have to multiply 3 with the gamma value of the second generation (which was 1: and 3 multiplied by 1 is 3), then you have to multiply 3 with infinite. So, actually, one was working with a gamma value of 3 times infinite. Three times infinite ... well, that causes a lot of loss! That’s amazing, 3 times infinite. Infinite or 3 times infinite is the same of course. I just want to emphasize the fact that bitonal scanning causes an overwhelming amount of loss. It is not possible to calculate the amount of loss. In this case, the loss must be two times infinite at least!


Figure 6

Results

The bad results of the scanning of the Ver Huell collection and the bitonal scanning of the newspapers microfilms led us to draw the following conclusions:

Research

After our negative experiences, everyone was a bit anxious about the further scanning of microfilms. The question on all of our minds was: what to do? First microfilm the original and then scan the microfilm? Or perhaps both microfilm and scan the original? But exposing the originals twice can cause damage to the originals. Plus, it is expensive. In order to carry out a large-scale project to microfilm and digitise the Dutch parliamentary papers (1815–1995), research was clearly needed. A test was conducted with bi-tonal printed originals from the period 1992-1995. The originals were microfilmed, the microfilms were grey-scale scanned and next an OCR was made. The OCRs where compared to OCRs made of grey-scale scans from the originals. The test comprised of 63 images, and about 200,000 characters and 35,000 words. The results were very interesting: the accuracy of the scans of the originals was about 99.89%, while the accuracy of the scans of the microfilms was about 99.90%. Following these test results, we decided to microfilm the originals and scan the microfilms in grey-scale. Research such as this is very important. We are discussing setting up a similar test with between 30 and 50 pages of newspaper from the period 1840–1950, in order to prove that Metamorfoze’s preservation microfilms can be scanned perfectly. I hope this will end the discussion on the scanning of microfilms.

Preservation scanning

Does preservation scanning already exist? Does it exist on the same quality level as preservation microfilming? As I said before: colour is definitely the only information we are losing in today’s preservation microfilms. And we accept that loss. But can we be certain about an equal and acceptable amount of loss in a preservation scan? That is the question we are faced with today. We don’t know the answer yet, but we are going to conduct research and plan to carry out two preservation scanning pilot projects to find the answers. The main underlying question behind our research is: Is it possible to reach the same solid and verifiable quality criteria for preservation scanning as we have established for preservation microfilming? There are six technical aspects connected to preservation scanning which we consider to be very important. In our research, we will be focusing mainly on these particular aspects: tonal reproduction, colour reproduction, detail reproduction, illumination, noise and digital image artefacts. Research activities started in August 2005, and we hope to publish our findings in the summer 2006. For now we stick to microfilming and scanning.

Computer output microfilm (com)

The COM is often compared to a preservation microfilm. The same technical requirements and guidelines apply to a COM and a preservation microfilm. The same Quality Index is the technical aspect that carries the highest priority. In my opinion, one cannot compare a computer output microfilm to a preservation microfilm. They are quite different. There is not a direct link between the preservation microfilm and the COM. The preservation microfilm is based on a paper document. The computer output microfilm is based on a digital image. Consequently, the technical quality of the COM depends on the technical quality of the digital image. The only way to check the quality of a COM is to scan it, and then compare this digital image to the original digital image.

So, what are the technical requirements of this first digital image and what is left of it in the scanned COM image? These are the questions we have to answer. First, we need to set up guidelines for preservation scanning. Then, we have to look at the quality of the COM. This summer we started doing COM research and will continue to do so alongside the preservation scanning research. In both research projects, I have teamed up with Robert Gillese, Digitization Quality Manager of the Memory of the Netherlands. Up to now we have not seen a COM with an acceptable amount of loss compared to the initial digital image. To be honest I don’t even believe it is possible. Working in two worlds, the digital and analog one is very difficult. People should realize that working in the analog world only, and producing preservation microfilms on the high quality level we demand, is very difficult. Sometimes library material has been filmed twice. Lots of dedication and technical knowledge is required from all the people working in the production area. It is like professional sport. And that is just working in the analog world. It is only a single step from a paper original to a preservation microfilm. Producing COM, switching from digital to analog and back to digital again later, are two, even three steps. For each step full technical attention is required. Perhaps it is possible to achieve an acceptable technical quality level in an optimized test area; in a production area, it is definitely not.

Methylene blue test

The methylene blue test is a test used to measure the amount of thiosulfate in a developed microfilm. Thiosulfate is a chemical used in a fixer. After the fixing bath, one cleans the film with water in the next bath. This water washes out the thiosulfate. Thiosulfate causes damage to the microfilm in the long term which is why the maximum permitted amount of thiosulfate is written down in an ISO standard. ISO 18901:2002(E) states that the permitted amount of thiosulfate for a microfilm with LE 500 (life expectancy of 500 years) is 1.4µg/cm². If a film is cleaned with an insufficient amount of water after a fixing bath, it may possibly contain too much thiosulfate.

The amount of thiosulfate left in a microfilm is very important, and something that has to be tested regularly. But I’m not happy with the current way of testing. In my opinion, the methylene blue test used nowadays applies more to the developing machine than the microfilms: right now the test is carried out once every week with a little piece of microfilm, cut off especially for this purpose. With hygienic gloves, this piece of microfilm is taken out of the developing machine and shipped directly to a specialized lab. The test can, for instance, be carried out and completed in a single Tuesday afternoon. However, the microfilms I would like to have checked are developed on Thursday, Friday and Monday. If the developing machine is in the exact same condition (with the tap on) as it was during the test, there is nothing to worry about, but this is something I can only assume. The methylene blue test also has to be completed within two weeks after developing the film and it is a very difficult and complex test which can be only be carried out in a specialized lab, and the results of the test are not available until 2 to 3 weeks later, sometimes even longer. It’s a quantitative result.

Right now we are testing the Agfa Structurix Thio test. This test is designed for use on radiographic films. It is a very simple test, it is not limited by time, it can be carried out anywhere and you can test as many films as you like. The day of development is not an issue anymore and the results are available in only two minutes. Clearly this test offers many advantages. It is a test related directly to microfilms. However, there is also a drawback. The test is designed for radiographic films and the permitted concentration of thiosulfate for radiographic films is 2µg/cm². This means that the test - a spot test with a colour indication (semi quantitative result) - is too rough. Some fine tuning and testing has to be done before we can decide to skip (or partly skip) the methylene blue test and introduce the Agfa Structurix Thio test as a new and reliable testing method. In this research project, I am working closely with the conservation scientist of our National Library and a chemist of Agfa-Gevaert, Belgium.

Cellulose acetate microfilm conference

The Cellulose Acetate Microfilm Conference (Cellulose, 2005) was held in the British Library in London, May 2005. Different strategies in dealing with the vinegar syndrome in large quantities of acetate microfilms were discussed. The vinegar syndrome, which derives its name from the vinegar-like odour it produces, is a slow form of chemical deterioration that affects the plastic support of acetate film. When the storage conditions of acetate films are not optimal over a longer period of time - for instance a room temperature of 21ºC and a Relative Humidity of approximately 50% over a period of 50 to 60 years - the vinegar syndrome will emerge. When the films start to smell like vinegar, they are close to reaching the so-called ‘autocatalytic point’. From this point on the deterioration will feed itself and speed up. After the autocatalytic point, the time that is left for duplication is shortened dramatically. Therefore, the deterioration must be prevented from reaching the autocatalytic point. A widely accepted strategy and approach to this problem is to reach a good knowledge level of:

When this information is gathered, a strategy can be developed, which will include duplication on polyester based microfilms and cold storage. Cold storage allows you to win time and take action. More information can be found on the website of the Image Permanence Institute, and the website of the Australian Network for Information about Cellulose Acetate called ANICA.

References

Cellulose Acetate Microfilm Conference. Liber Quarterly, 15(2005)2. http://liber.library.uu.nl/

Dormolen, Hans van. Metamorfoze Preservation Microfilming Guidelines. Den Haag : Bureau Metamorfoze, March 2004. http://www.metamorfoze.nl/publicaties/richtlijnen/english/guidelines.pdf

Web sites referred to in the text

ANICA - Australian Network for Information about Cellulose Acetate http://www.nla.gov.au/anica/

IPI - Image Permanence Institute. http://www.imagepermanenceinstitute.org/index.html

Metamorfoze: http://www.metamorfoze.nl/index.html

Metamorfoze Richtlijnen. http://www.metamorfoze.nl/publicaties/richtlijnen/richtlijnen.html

Sensitometry. http://wwwnl.kodak.com/US/en/motion/students/handbook/sensitometric.jhtml?id=0.1.4.9.6.&lc=en http://www.tpub.com/content/photography/14208/css/14208_40.htm