Digitizing Genealogy -- Understanding and Using Scanned File Formats

When a scanning device or a camera creates a digitized file, that file must be stored on the device in some kind of file format. The most common format is referred to as the JPEG format. There are, however, dozens of different image file formats. The most common file types include the following;

• TIFF -- Preferred archival format
• JPEG -- A lossy file format
• PNG -- Lossless but compressed
• GIF -- A compressed file format, not preferred by archivists
• RAW -- Used by higher end digital cameras
• BMP -- Microsoft proprietary format
• PSD-- Used by Photoshop

At this point the post could become highly technical. But the issues for genealogists, when they have a choice, should be to save their image files in the most accepted archival file format available. Unfortunately, many scanning devices and most digital cameras default to JPEG format.

My preferred source for information about digital preservation is the Library of Congress. The main reference is a section of the LOC.gov website entitled, "Sustainability of Digital Formats, Planning for Library of Congress Collections." I will refer you to the document itself, which is somewhat technical, but the summary of the summary is that TIFF images are preferred over any other type of file format. Most of the other file formats listed above are acceptable, but not preferred. The two file formats the Library of Congress deems unacceptable are the RAW format and the Photoshop file format PSD.

The issue here is sustainability. Will the file format be used in the future and will images stored in that format be able to be used by new computer devices in the future? The important terms here are "lossy" and "lossless." These are complicated terms and deal with the fact that lossy file formats lose information (quality, resolution etc.) as they are edited. Lossless file formats, such as TIFF files, do not lose information. Additionally, compressed files also involve a trade-off between file size and quality. Compressing a file so that it takes less storage room on a hard disk or other device, necessarily affects image quality. There is a way to avoid losing quality with compression and the JPEG 2000 file format claims lossless compression, although you may have some trouble finding a program that supports this file format.

What this boils down to for genealogists is that we need to use the most common and most highly supported file formats available to us and we also need to be aware of the file formats supported by the programs we use to store our data.

There are dozens (hundreds?) of image programs available today. Many of these programs such as the high end Adobe Photoshop and the lower end Photoshop Elements, can save files in a variety of file formats. Many other programs will convert images from one format to another. Just remember, you can't get blood out of a turnip. If your image was created in a lossy file format such as a JPEG file, subsequently saving the file as a TIFF file may help, but the quality of the image is determined at the time the image is created by the scanner or the camera and the file format will not improve that quality. It may help to preserve or not help depending on the circumstances, but the initial file creation is what is important.

Here is a list of the previous posts in this series.

http://genealogysstar.blogspot.com/2015/03/digitizing-genealogy-flatbed-scanner.html
http://genealogysstar.blogspot.com/2015/03/digitizing-genealogy-introduction-to.html
http://genealogysstar.blogspot.com/2015/03/digitizing-genealogy-understanding-dpi.html
http://genealogysstar.blogspot.com/2015/03/digitizing-genealogy-beyond-resolution_8.html
http://genealogysstar.blogspot.com/2015/03/digitizing-genealogy-beyond-resolution.html
http://genealogysstar.blogspot.com/2015/03/digitizing-genealogy-resolution-is.html
http://genealogysstar.blogspot.com/2015/02/digitizing-genealogy-scanners-vs-cameras.html
http://genealogysstar.blogspot.com/2015/02/digitizing-genealogy-what-is.html
http://genealogysstar.blogspot.com/2015/02/digitizing-your-genealogy-files.html

Digitizing Genealogy -- Flatbed Scanner

CanoScan LiDE40 is Canon's A4 USB CIS flatbed image scanner.By Qurren (Qurren's file) [GFDL (http://www.gnu.org/copyleft/fdl.html) or CC-BY-SA-3.0 (http://creativecommons.org/licenses/by-sa/3.0/)], via Wikimedia Commons

The term "scanner" covers a huge assortment of electronic devices for digitizing all sorts of original documents. In fact, the term has now been extended to 3D scanners that digitize three dimensional objects for 3D printers or additive manufacturing. Wikipedia has ten different types of devices listed on its disambiguation page for the term "scanner."  Some of the devices, known as book scanners, actually incorporate high resolution cameras into the scanning process. The devices using the label "scanner" can cost under $100 or cost many tens of thousands of dollars. As a genealogist, unless you intend to get into the commercial business of making scanned images as a service, you will probably settle for one of the many models of "flatbed" scanners such as the one illustrated above. This type of device is sometimes called an "image scanner" to differentiate it from other types of scanners.

A flatbed scanner has a flat glass plate, similar to a copy machine, and the same type of copy machine light source. Some of the flatbed scanners today use a charge-coupled device (CCD) or a contact image sensor (CIS) as the image sensor. The light source for flatbed scanners is usually a xenon, LED or cold cathode fluorescent type. An alternate type of scanner uses a contact image sensor (CIS) scanning consists of a moving set of red, green and blue LEDs strobed for illumination and a connected monochromatic photodiode array under a rod lens array for light collection. See Wikipedia: Image scanner.

The main issue in using a flatbed scanner is the time it takes to make one digital copy of a document or photograph. It can become extremely tedious to raise and lower the lid and replace the documents or photos to be copied. The image quality is usually very good, depending on the software supplied by the manufacturer. Most of the flatbed scanners available today, scan in color, black and white an gray scale. The newer models have all moved well beyond the needs of high quality archival standard scanning. I will be talking a lot more about scanner resolution in future posts.

You could spend a considerable amount of time comparing flatbed scanner models, or you could simply walk into the nearest store selling the devices and buy the first one that caught your eye. You would probably end up with a perfectly adequate scanner. But I suggest a mid-road approach. Spend some time reading reviews and thinking about the following considerations:

• How many documents do you expect to scan?
• How long will you spend scanning?
• Do you intend to use the scanner for purposes other than genealogy?
• Are you concerned about archive quality or do you want quick and dirty?

As you think through the physical process of scanning, you must also consider what you are going to do with the computer files created by the scanner. I will also be posting more information about understanding and using the different file formats. It does no good to purchase a scanner and then fail to learn how to transfer the files and use them as media attached to your genealogical database whether online or on your own computer.

Depending on the amount of paper you need to scan and the time you have to spend doing the scanning, you may wish to look into the possibility of purchasing a sheet-fed scanner. This is also a topic for another post. As you can probably tell, scanning documents for use in a genealogical context can be a somewhat technical area. I only have to go online and look at the quality of the images uploaded to family tree websites to know that many researchers lack a basic understanding of the way to scan an image or how to handle the digitized image file. Of course, the scanned image can be no better than the original, but sometimes there is a high quality scanned image already available of the ancestor and someone has uploaded a very, very poor quality copy of exactly the same image. There are a lot of considerations in making images for genealogical purposes besides those involved in the technology of the various scanning devices.

Here are two images of exactly the same photograph from the FamilySearch Memories section. Both images are copies from an original photograph. A high quality scan of the original is available on the same page as these two photos.

This is apparently the photo from which the two copies were made.

There are a lot more considerations about image quality and choice than just the technology involved in making scanned images. 

Here is a list of the previous posts in this series:

http://genealogysstar.blogspot.com/2015/03/digitizing-genealogy-introduction-to.html
http://genealogysstar.blogspot.com/2015/03/digitizing-genealogy-understanding-dpi.html
http://genealogysstar.blogspot.com/2015/03/digitizing-genealogy-beyond-resolution_8.html
http://genealogysstar.blogspot.com/2015/03/digitizing-genealogy-beyond-resolution.html
http://genealogysstar.blogspot.com/2015/03/digitizing-genealogy-resolution-is.html
http://genealogysstar.blogspot.com/2015/02/digitizing-genealogy-scanners-vs-cameras.html
http://genealogysstar.blogspot.com/2015/02/digitizing-genealogy-what-is.html
http://genealogysstar.blogspot.com/2015/02/digitizing-your-genealogy-files.html

Digitizing Genealogy -- Introduction to Scanners

CanoScan LiDE40 is Canon's A4 USB CIS flatbed image scannerBy Qurren (Qurren's file) [GFDL (http://www.gnu.org/copyleft/fdl.html) or CC-BY-SA-3.0 (http://creativecommons.org/licenses/by-sa/3.0/)], via Wikimedia Commons

Scanners are becoming ubiquitous and have been inexpensive for some time. Genealogists really need to invest in a scanner and use it regularly. The idea here is to not only provide a link to a document or other source, but also include a copy of the original. This is quite easy to do if the document is online, unless it happens to be copyright protected or watermarked in some way. Usually, right clicking on the document or image will let you save a copy to your own computer or other device. Most of the larger online genealogical database programs also have a way to download an original copy of the documents in their collections. The image can then be attached as a media item to an ancestor in your own desktop genealogy program or to an online family tree. There is also a program from RecordSeek.com, that will make a copy of the online documents, with a citation to the source, and allow you to transfer that file to FamilySearch.org Family Tree.

But here, in this post, I am exploring the world of scanners for those times when you need to scan your own documents or photos. Here is an example of a scanned photo from my Great-grandmother, Margaret Godfrey Jarvis Overson's Photographic Collection:

This particular image was scanned with a Canon CanoScan 8800F. The latest version of this scanner is the CanoScan 9000F Mark II for about $200. This model is considered to be a film and negative scanner but will also do an excellent job of any document or photo. An entry level Canon scanner costs about $50 retail and will do just as good a job of scanning a document or photo as the more expensive film and negative scanner. It is just slower and has fewer features. Here are a few of the current models and prices of the different manufacturers' less expensive, flat-bed scanners. You may be able to find a better deal online for the same models.

• Amazon.com's #1 Best Seller is the Epson Perfection V600 Color Photo, Image, Film, Negative & Document Scanner for $212.12.
• The Canon CanoScan 4507B002 LiDE110 Color Image Scanner is $59.99 from USA.Canon.com
• HP Scanjet G4050 Photo Scanner from the HP Store at Store.HP.com for $179.00

Scanners are going to vary in price by the following criteria:

• The speed of the first scan after turning on the scanner
• The speed of the scanner usually in pages per minute
• The resolution claimed by the manufacturer
• The size of the scanning bed from 8.5 inches by 11 inches up to 11 inches by 17 inches or even larger
• The color depth of the scans, i.e. 16-bit up to 96-bit color
• Sheet feed capability
• Whether the scanner will do negatives and slides
• Networking capability

The machines fall into several categories depending on functionality. 

• Hand-held scanners
• Flat-bed scanners
• Sheet fed scanners
• Large format scanners
• Work group scanners (networked for sharing)

If you want to explore the capabilities of the different types and models, it is best to go to the individual websites. Here is a list of the major manufacturers. You may have to look through some menus or search for the scanners.

• Epson.com
• USA.Canon.com
• Store.HP.com
• Fujitsu.com
• Kodak.com

By the way, all of them claim to have the best scanner and there are websites out there that support any one of the manufacturer's claims. I am not sure anyone has tried all the new models and compared them feature by feature. I have had good use out of both my Canon and Epson scanners over the years. Presently, I have and use scanners from both manufacturers.

Here is a list of the previous posts in this series:

http://genealogysstar.blogspot.com/2015/03/digitizing-genealogy-understanding-dpi.html
http://genealogysstar.blogspot.com/2015/03/digitizing-genealogy-beyond-resolution_8.html
http://genealogysstar.blogspot.com/2015/03/digitizing-genealogy-beyond-resolution.html
http://genealogysstar.blogspot.com/2015/03/digitizing-genealogy-resolution-is.html
http://genealogysstar.blogspot.com/2015/02/digitizing-genealogy-scanners-vs-cameras.html
http://genealogysstar.blogspot.com/2015/02/digitizing-genealogy-what-is.html
http://genealogysstar.blogspot.com/2015/02/digitizing-your-genealogy-files.html

Digitizing Genealogy -- Understanding DPI, PPI and LPI

Apple Dot Matrix Printer By AppleMacReporter (Own work) [GFDL (http://www.gnu.org/copyleft/fdl.html), CC-BY-SA-3.0 (http://creativecommons.org/licenses/by-sa/3.0/) or CC BY-SA 2.5 (http://creativecommons.org/licenses/by-sa/2.5)], via Wikimedia Commons

There is a whole lot of jargon thrown around about the subject of digitization. I suspect that many genealogists are somewhat stymied by the seemingly large number of options, especially the huge selection of models of scanners and cameras. One of the basic issues faced with any digitization project is the need to understand the difference between the three basic terms: DPI or dots per inch, LPI or lines per inch and PPI or pixels per inch. Of course, depending on the measurement system in force in your country, they may also be expressed in millimeters or centimeters. There seems to be an invalid assumption that "everybody" knows what these terms mean. In fact, they are really quite technical in nature and are often used incorrectly in advertisements and common usage.

DPI
The most common of these three terms is a reference to the original computer driven printers that used arrays of wire print heads hitting on an ink ribbon to make dots on a piece of paper. These devices were generically called "Dot Matrix Printers." The "matrix" was the array of wires used to make the images and the "dots" were the little ink marks on the paper (or other substance). In some devices, small arrays of heat transferring wires were used to make the dots on heat sensitive paper. This was actually my introduction to the process when I acquired a Canon Pocketronic introduced in late 1970.

In all these devices, the "resolution" of the print was accomplished by adding wires to the printhead and adapting the printing patterns. Here is an example:

Close-up of text from a dot-matrix printer. This is representative of the most basic output from a 9-pin printer (or one in draft mode), without NLQ. Note the enlargement of single letter for detail.By Fourohfour (Own work) [GFDL 1.2 (http://www.gnu.org/licenses/old-licenses/fdl-1.2.html) or CC BY-SA 2.5 (http://creativecommons.org/licenses/by-sa/2.5)], via Wikimedia Commons

As monitors were developed to view computer output, some of the same terminology used by the printer manufacturers was used to describe the image quality of the monitors (modified TV screens).

Simulated screenshot of a TRS-80 Color Computer showing all available alphanumeric characters and semigraphics characters.By Mmiller2 (Own work) [CC BY-SA 3.0 (http://creativecommons.org/licenses/by-sa/3.0)], via Wikimedia Commons

Rather unfortunately, the terminology used to describe these early printers and survived into the 21st Century and been applied to technology that has progressed well beyond this simplistic view of printer and monitor resolution. We are still using the term "dots per inch" or DPI to refer to technologies that bear little resemblance to the original.

The big jump in the dot matrix technology came with the introduction of so-called laser printers that substituted a beam of light for the wire array in making images. The light activated a light sensitive drum borrowed from the developing copy machine technology and allowed much higher quality printing. The same dot matrix technology is essentially in effect today in two different types of printers being sold; the laser printer and the inkjet printer.

An HP LaserJet 4200 dtns printerBy Combuchan. Combuchan at en.wikipedia (Own work) [CC BY 2.5 (http://creativecommons.org/licenses/by/2.5)], via Wikimedia Commons

By substituting sprayed drops of ink or melted plastic toner for the original inked ribbon image, the printers were able to increase the "resolution" or decrease the size of the dots to the point where the individual dots could not be seen without magnification.

This image shows an opened Canon S520 ink jet printer.By André Karwath aka Aka (Own work) [CC BY-SA 2.5 (http://creativecommons.org/licenses/by-sa/2.5)], via Wikimedia Commons

If you have a printer attached to your computer either directly or by WiFi, you probably have either a laser printer or an inkjet printer.

Describing either the image created by a scanner or a digital camera in DPI is inappropriate. It is quite stretch to claim that an image is created by dots per inch when the resolution claimed climbs into the thousands. But there does not seem to be any better way of expressing the idea, especially when consumers are generally conditioned to believe that the higher the DPI the better the quality of the image.

PPI
On computer screens and ultimately on any kind of transmitted light device, the resolution is sometimes referred to in terms of pixels per inch or PPI. The pixels being the tiny dots of light produced by the mechanism of the monitor or TV. Just as with the ubiquitous DPI, the technology has progressed to the point where the discrete pixels are all but invisible to the naked eye.

Dieses Bild zeigt die Farbentstehung an einem Röhrenmonitor.By Johannes Waschke (Own work) [CC BY-SA 2.5 (http://creativecommons.org/licenses/by-sa/2.5)], via Wikimedia Commons

The title of the above image translated into English says, "This picture shows the color formation on a CRT monitor." The pixels here are individually created, discrete elements. The similarities with images produced by dot matrix printers are obvious, but the technology is entirely different. The mechanism for producing the dots of light have evolved dramatically and there is little in common between the original images created by a Cathode Ray Tube or CRT and the newest images created by light emitting diodes (LED) or other technology.

Pixels per inch or PPI are expressed as the total number of pixels in a square unit of measurement. This is different than the measurement used by DPI originally, which was a linear measurement. So a measurement in DPI originally was nothing more than a single number. We used the term, 60 DPI dot matrix printer, for example. However, the new use for DPI and PPI refer to the array and you get measurements such as 1060 x 480 or other such numbers. The inference is that the higher the numbers the greater the resolution. Explaining why this is not always the case gets into some really technical explanations about the way light works.

Essentially, the relationship between PPI and DPI depends entirely on how you count the "dots." You also get a different measurement when you start to talk about digital cameras and you start hearing about "Megapixels." This term is just a way to reduce the total number of pixels to a single number by dividing the total by 1 million. Arguably, the higher the number the greater the resolution, but actual image quality is dependent on a number of other factors such as the quality of the camera lens and the contrast of the image, just to mention two of many issues. I will get into more particulars when I start discussing scanners and cameras in more detail.

LPI
This term, lines per inch, pre-dates both DPI and PPI. It is a measurement of the quality of printed media. The term originated with the process of making "halftone" images.

When you start talking to archivists and other technical folks, you immediately get introduced into the world of LPI. The standard way to measure the quality of an image is to examine it under high magnification and then compare it to a "standard" printed image. I have mentioned the standard images in a previous post in this series. Here is the example again:

EIA Resolution Chart 1956

Usually, the archivist is looking for a resolution in a scan or photo where fine lines have as least two pixels (dots). This is especially true for scans or photos of documents.

In all this it is important to remember that a digitized copy of an original document or photograph cannot possibly have a greater resolution than the original. More about this in later posts.

Here are the previous posts in this series:

http://genealogysstar.blogspot.com/2015/03/digitizing-genealogy-beyond-resolution_8.html
http://genealogysstar.blogspot.com/2015/03/digitizing-genealogy-resolution-is.html
http://genealogysstar.blogspot.com/2015/02/digitizing-genealogy-scanners-vs-cameras.html
http://genealogysstar.blogspot.com/2015/02/digitizing-genealogy-what-is.html
http://genealogysstar.blogspot.com/2015/02/digitizing-your-genealogy-files.html

Digitizing Genealogy -- Beyond Resolution to Standards Part Two

What are the archival standards for digitizing both documents and photographs? In the past posts, I have mentioned that fact that advertisements for scanners include claims for extraordinarily high "dpi" resolution. It is interesting to note that the manufacturers of high quality photographic printers no longer talk about the "dpi" or dots per inch of their prints, but merely claim high quality. You can find out what the dpi is, but you have to look for it carefully. The resolution claimed for the high-end printers is now about 5760 x 1440. In essence, the printer manufacturers have started using the total pixel count in a way that is similar to how monitors and TVs are now sold. Most monitors are also sold with advertisements of the number of pixels. You see claims for monitors of up to 1080p and higher. See Understanding HDTV Resolution for more details.

What do the archivists say about resolution? What is the current standard?

In this case, I always start with the Library of Congress' Preservation Directorate. The preservation efforts of the Library of Congress address issues with Audio-Visual materials, Books, Paper including manuscripts, drawings, newspapers, prints, posters, maps etc. and Photographs. Most of the standards they support involve the preservation of the original documents or books. But that is beyond the scope of this series. The pertinent information from the Library of Congress is in the digital preservation section. Their general guidelines are simple:

• Identify and select what to save
• Organize the files selected to be saved
• Save copies on at least two different storage media (e.g., USB drive and external hard drive) and keep these in separate physical locations
• Migrate saved copies to a current storage medium about every five years

They also make the following statement about the resolution of digital images:

What resolution should I use when digitizing? 

Note that resolution is not the only consideration when digitizing. See Technical Guidelines for Digitizing Cultural Heritage Materials. In short, it depends on what is being digitized and the intended use of the digitized image. See pp. 49-67 of the guidelines above.

If you are involved in scanning or photographically digitizing your genealogical documents, you should, at least, be familiar with the standards offered in the linked Technical Guidelines above. Over the years, I have heard many genealogical presentations and read a number of articles by genealogists who provide guidelines for "standards' for scanning and photographs, but few of them refer to or are apparently aware of the national standards. They often throw out numbers for dpi or whatever without any qualification or support. I have read and heard presentations that claim that you should scan photos at the "highest resolution supported by your scanner" and other such nonsense. The real issue, as I started to illustrate at the beginning of this post, is how are the images going to be displayed? In short, if you scan a photograph at 600 dpi and then print it out at 300 dpi what was the point of the higher resolution scan? What if you scan a photo as some huge level of dpi claimed by the scanner's manufacturer, what if you view the document on a lower resolution screen?

The Library of Congress document explains all this in detail. The summary of their comments on the subject are as follows: 

Higher spatial resolution provides more pixels, and generally will render more fine detail of the original in the digital image, but not always. The actual rendition of fine detail is more dependent on the spatial frequency response SFR) of the scanner or digital camera (see Quantifying Scanner/Digital Camera Performance below), the image processing applied, and the characteristics of the item being scanned. Also, depending on the intended usage of the master files, there may be a practical limit to how much fine detail is actually needed.

I absolutely agree with this statement. Did I mention that the document is 101 pages long and has four pages of links to additional documents. If you are at all serious about the subject of digitization, you should be familiar with the current standards. Here is the Library of Congress statement in summary on the issue of resolution:

Resolution[Resolution] Requires sufficient resolution to capture all the significant detail in originals. Currently the digital library community seems to be reaching a consensus on appropriate resolution levels for preservation digitization of text based originals – generally 400 ppi for grayscale and color digitization is considered sufficient as long as a QI of 8 is maintained for all significant text. This approach is based on typical legibility achieved on 35mm microfilm (the current standard for preservation reformatting of text-based originals), and studies of human perception indicate this is a reasonable threshold in regards to the level of detail perceived by the naked eye (without magnification). Certainly all originals have extremely fine detail that is not accurately rendered at 400 ppi. Also, for some reproduction requirements this resolution level may be too low, although the need for very large reproduction is infrequent. 

Unlike text-based originals, it is very difficult to determine appropriate resolution levels for preservation digitization of many types of photographic originals. For analog photographic preservation duplication, the common approach is to use photographic films that have finer grain and higher resolution than the majority of originals being duplicated. The analogous approach in the digital environment would be to digitize all photographic camera originals at a resolution of 3,000 ppi to 4,000 ppi regardless of size. Desired resolution levels may be difficult to achieve given limitations of current scanners. 

You might say that this doesn't help much, but in fact, it is exactly appropriate. While I was involved in digitizing documents for FamilySearch directly and with my recent digitization efforts of the photographic collection that ended up at the University of Arizona, I worked closely with both organizations to follow their own guidelines. In both cases, my digitized images were acceptable for archive purposes. 

Unfortunately, images usually uploaded online to various websites do not always comply with the standards of quality set forth by the Library of Congress. In addition, if you look around on the Web, you will see a number of entirely different standards. What I can say, is that any one selling you a device and claiming a particular resolution in dpi or ppi or lpi, is probably not telling the entire story and anyone telling you the same thing about doing your own digitization project is probably not telling you the whole story either. 

Of course, I have a lot more to say about these issues in future posts in this series. This will be a very long series. 

Here are the previous posts in this series:

http://genealogysstar.blogspot.com/2015/03/digitizing-genealogy-resolution-is.html
http://genealogysstar.blogspot.com/2015/02/digitizing-genealogy-scanners-vs-cameras.html
http://genealogysstar.blogspot.com/2015/02/digitizing-genealogy-what-is.html
http://genealogysstar.blogspot.com/2015/02/digitizing-your-genealogy-files.html

Digitizing Genealogy -- Resolution is always an issue

Roger Gilbertson [CC BY-SA 2.0 (http://creativecommons.org/licenses/by-sa/2.0)], via Wikimedia Commons

The manufacturers of both scanning devices and cameras have been locked in a pixel war for many years. As a result of this fixation, claims for high resolution are common. The results? The image shown above is an example of what is called a moire pattern. This is caused by the interference of the light rays and is very common in images where the pixel count exceeds the length of the waves of visible light. You can't ignore physics when you start to scan and take photographs. Before you run out and spend some hard-earned money on scanner or a camera, you should realize that resolution, as such, has its limits. At this point, we need some definitions:

• Resolution -- the amount of small detail you can see in an image
• Magnification -- how much an image can be enlarged
• Contrast -- the difference between the lightest and darkest parts of an image
• Sharpness or Definition -- the viewer's perception of the results of focus, depth of field, contrast, and the maximum potential detail inherent in the recording media
• Depth of Field -- the area of sharpness in front of and behind the main object in an image
• Grain -- the limit of the resolution of the media (from film photography; the size of the silver particles making up the developed film)
• Image -- the product of a scanning or photograph process
• Noise -- also known as artifacts, this is the amount of unwanted defects in the image
• Wave length -- the distance between the tops of the waves in angstroms or Nanometers
• DPI also LPI also PPI -- dots per inch, lines per inch and pixels per inch, all measurements of the resolution of a image making device
• Optical resolution -- the actual or physical resolution of a scanner or lens
• Sensor -- the light gathering mechanism for a scanner or camera
• Megapixels -- the total number of sensors in a sensor array
• Digitize -- using an electronic device such as a digital camera or scanner to create an image that can be manipulated and displayed with a computer-based device

For more definitions, see Lowel EDU, A Lighting Resource Center, Glossary of Lighting Terms.

OK, so let's start with a reality check. What is the absolute resolution of the human eye? This question means, what amount of detail can you see with your eyes assuming you have perfect vision?

You might be surprised to learn that if you put your eye's resolution into the terms used today for selling scanners and cameras that you can only resolve images at about 74 Megapixels. Interestingly, some high-end cameras are approaching that resolution level. But is resolution the end all and be all of making images? Not at all. The real issue, although the ads would have you believe otherwise, is the whether or not the image coveys the information intended. That is, can you read the document? That has more to do with contrast than resolution.

Here, I need to pause and sort out where I am going with this issue of resolution. There are really two completely different issues from a genealogical standpoint; the digitization of documents (i.e. paper etc.) and the digitization of photographs (i.e. images of ancestors etc.). Obtaining readable and acceptable images of documents is very different from digitizing photographs (unless they are photographs of documents such as microfilm etc.).

Now back to Megapixels and DPI. Digital cameras have, since they were first sold, have been touting the resolution of their sensors in Megapixels. Originally, the cameras were one or two Megapixels and lately, Canon has announced a new EOS 5DS and a companion camera, the EOS 5DSR, both with 50.6 Megapixel sensors. On the other hand, scanners are advertised with DPI ratings. For example, the Epson Perfection V600 Color Photo, Image, Film, Negative and Document Scanner claims a resolution of 6400 x 9600 dpi. So how do you compare the two?

It turns out that the numbers given by the companies that manufacture scanners are mostly exaggerated. The optical resolution of this Epson scanner is actually 6400 dpi. The claimed effective pixel count is 54,400 x 74,880 giving an incredible 4073 Megapixels. But there is a fatal flaw here in this reasoning. If that number were true, the file would be incredibly large. So how do you find out the answer to the question about comparison? You have to actually measure the resolution.

The way resolution is ultimately determined for lenses or for the produced image is to photograph or scan a standard resolution test document such as the 1951 USAF Resolution Test Chart or some other similar test. Here is an example of this type of chart:

EIA Resolution Chart 1956

This type of chart works with scanners, cameras and other optical equipment. But the catch is that buying the standard type of chart can cost from $300 on up.

After all that, for genealogists and archivists, there are standards for resolution. but that is the next post.

Digitizing Genealogy -- Scanners vs. Cameras

If the entire world is buying smartphones and cell phones with cameras, why are we worried about separate cameras or even flatbed scanners at all? Can't we just take a photograph of our documents with our phone and leave it at that. Why bother with a bulky, less-than-portable, scanner? If scanners are needed, why do the FamilySearch Document Acquisition people use cameras? Why have cameras been used for archiving since about 1938? Why don't they just scan the documents?

The answers to all these questions involve complex issues, some practical, some economical, and some chiefly political in nature. The answers also involve the evolution of technology and the rate at which technological changes are adopted by archivists and document conservators.

To start this particular discussion, I need to show three images. One, obtained by a camera and then developed as a microfilm image and subsequently digitized, the second, an image altered by modern image enhancement techniques, and the third taken by a modern digital camera directly from a document.

Before presenting the three examples, I need to explain what you are going to see. Document reproduction (and all photographic processes) depend on the quality of the physical document. Old documents are seldom in pristine condition and are subject to a variety of natural forces that may destroy the original documents and make them unreadable: fires, floods, mold, insects, chemical changes, rough handling, and many more. Original reflective light photography could do very little to improve the readability of the original document. Microfilm images were often unreadable. The advent of digital imagery and the technology involved has developed ways to restore unreadable images and even reveal images that are invisible to the naked eye. But much of this image enhancement technology depends on access to the original documents. There is only so much that can be done to enhance a poor microfilm image of an unreadable document. In addition, if the original document is unreadable, there are both temporal and economic issues the arise if document restoration techniques are to be used. More about this later on in this post and the series.

Here is the first image without any particular photographic enhancement. This is a copy of a U.S. Census record directly from the original documents as shown on Archive.org;

You may have to click on the image to see any detail. Here is a screenshot of the section of the document pertaining to my Great-grandfather, Henry Martin Tanner:

Again, you may need to click on the document to see the detail.

Here is the same page of the same U.S. Census record from FamilySearch.org:

Here is a screenshot of the two documents side by side:

You might not be able to tell, but the second image, from FamilySearch.org, has the contrast enhanced to show more detail. Both are fairly good images, but the second one probably shows more detail than than the first even though it appears darker.

Here is a current digitized image from FamilySearch.org. The format of the way the image was taken reveals that it was taken directly by a digital camera. This is a sample of the Tennessee, Freedmen's Bureau Field Office Records, 1865 - 1872 from Chattanooga, Tennessee.

Here is an enlarged section of this same Tennessee document:

You can see that the quality of the digital image is still very good even with some bleed through from the backside of the document.

The point here is that the limitations of the original often eclipse the sophistication of the technology and that there is little that can be done to enhance an image even with modern technology because of time and cost constraints. Would a modern scanner produce a better image? Yes, likely, but the problem is that the originals are not in a place nor do they have the format that would lend itself to using some type of scanner.

Most of the discussions about making a comparison between using a scanner versus using a camera revolve around the issue of resolution. Cheap scanners could make a higher resolution image than a cheap camera. Archive quality digital cameras were extremely expensive. The main issue was and still is, the resolution of the image. It was not until relatively recently that consumer or prosumer digital cameras achieved an acceptable resolution. I will have a lot more to say about the technical aspects of making digital images in successive posts.

Here are some the basic considerations, pro and con, between using a camera and a scanner:

Pros for using a camera:

• Easy to set up.
• Relatively fast imaging.
• Quick transfer of images to a computer or storage device.
• Can be used with very large documents.
• More likely to be allowed by a record repository.

Cons for a camera:

• The cost of a good quality camera is considerably more than the cost of a good quality scanner.
• Depending on the digitizing requirements, additional equipment, such as a camera stand and lights might be necessary. 
• Maintaining the proper focus across the entire image may be difficult, i.e. keeping the document flat with damaging the document. 

Pros for using a scanner:

• High quality images.
• Avoids much of the bleed-through on pages.
• Relatively inexpensive for good quality.

Cons for a scanner:

• Bulky, cannot be used at all in some document locations.
• Much slower than a camera.
• Original documents may be injured in the scanning process.
• Some documents cannot fit on the scanning bed and would have to be scanned in sections.
• Not as frequently allowed by record repositories.

One of the most common discussions about the use of scanners and/or cameras revolve around the distinction between scanning a photo and scanning text (i.e. documents without pictures). In this series, I hope to show that digital cameras have evolved to the point where they are more than just an alternative to scanners, but now have become the most effective tool for genealogists and archivists of all kinds.

Digitizing Genealogy -- What is digitization?

Reproduction of a bison of the cave of Altamira

Genealogical jargon can sometimes be difficult. This is especially true when overlaid with legal, scientific, DNA, or technological jargon. In the technological side of genealogy, you will frequently hear the word "digital" in all its forms (digitalize, digitalization, etc. also you may see it spelled with an "s" rather than a "z" in Great Britain).  What is all this?

It gets a little bit involved to understand the concepts and what is actually going on when we talk about digitizing something. Stay with me and I will walk you through how all this came about. 

Since ancient times, humans have tried to capture and preserve their impressions of the physical world. The cave painting depicted above is an example of what could be called an "analog" image. What we mean by an "analog image" is that the method of reproduction of the physical reality is also a physical reality. At the time this image was painted on the wall of the cave, there really was a bison out there in the world running around and eating grass. The technical definition of an analog image would be something like this: relating to or using signals or information represented by a continuously variable physical quantity such as spatial position or voltage. In the case of the cave painting, the "continuously variable physical quantity" is the paint used. 

When a genealogist looks at a document or other record of the past and copies out the information contained in the document, he or she is making an "analog" copy of the information using a pen or pencil and a piece of paper. Obviously, the cave painting above is not an exact replica of the original bison. Just as obviously, the pen or pencil copy of the information in a source document is also not an exact replica of the original. But for thousands of years, the only way to make a copy of a document at all was to copy it by hand. Printing was invented to speed up the process and enable the printer to make multiple copies of the same document. But each of those individual copies was still an "analog" of the original. Making any changes to the original analog copy essentially required remaking an original. If a painter painted a painting of a landscape, the only way I could acquire a copy of the painting was if someone copied the original in some format. 

In 1725, the limitation on making copies of an original began to change when Johann Heinrich Schulze made fleeting "photographs" of words by using stencils, sunlight, and a bottled solution of chalk and silver nitrate, simply as an interesting way to demonstrate that the mixture inside the bottle darkens where it is exposed to light. See Wikipedia, Timeline of photography technology.

If we fast forward through the history of the development of photography, we see that what was happening was that the inventors and developers of the photographic process were working on a new analog process of reproducing images. As photography developed, it became possible to use a camera to take a negative image (first negative invented in 1835 by Henry Fox Talbot) and then make as many positive image copies as desired of the "original" analog photograph. The media for the analog image was the glass plate or film. See Wikipedia, Timeline of photography technology.

Fast forwarding this whole process, for genealogists, the breakthrough for preserving documents came with the introduction of microfilm copies of the originals. The earliest microphotographs were made by John Benjamin Dancer in 1839, shortly after the introduction of the daguerreotype process. See Wikipedia: Microform. 

It is important to remember that all this fancy photographic stuff was still an analog of the physical reality as long as it involved some kind of physical film for capturing the image. What was important about film photography was the ability to make multiple copies rather cheaply and easily. Photography did for images what book printing did for books. 

So where do digital images come into all this? At the same time photography began to develop, the idea of manipulating information using mechanical and electronic devices also was beginning to emerge. A detailed history of computers is interesting, but beyond the scope of this post series. It is enough to say that the idea of a general-purpose computing device is usually attributed to Charles Babbage, who conceptualized the first mechanical computer beginning in 1833. See Wikipedia: Computer. It was necessary for a lot of other types of technology to develop before the first images could be transmitted electronically in 1920. Quoting from the Wikipedia article on Digital Imaging:

The first digital image was produced in 1920, by the Bartlane cable picture transmission system. British inventors, Harry G. Bartholomew and Maynard D. McFarlane, developed this method. The process consisted of “a series of negatives on zinc plates that were exposed for varying lengths of time, thus producing varying densities,”.[1] The Bartlane cable picture transmission system generated at both its transmitter and its receiver end a punched data card or tape that was recreated as an image.[2]

What happened here is that the "analog" representation of the physical reality had been transformed into a coded representation of the original in the form of electrical impulses. You could argue that this was still an "analog" image and that the medium of transmission had merely changed, but this development was significant to warrant a new category of "digital images." The word "digital" in this context focuses on the fact that the physical reality of the original is represented by a stream of electronic impulses. In the case of the original image transmission back in 1920, the punched data card or tape, was not recognizable as an image until it was processed by the receiver. 

The first digital photograph is attributed to Russell Kirsch in 1957. Here is a copy:

Pioneering digitally scanned image of Russell Kirsch's son Walden, 1957

The first digital camera is believed to be developed by Kodak in 1975. See "The World’s First Digital Camera by Kodak and Steve Sasson." Although the circuits and the devices have become smaller and smaller, the idea that an image can be made by discrete electronic sensors is at the heart of a digitalization. You could argue that the first movable type book was merely a small step from the original handwritten books, but this small step changed the world. Likewise, the first digital images began the same fundamental revolution in the way information was processed and transmitted. 

So, digitization is the process of taking a physical object (book, document, etc.) and using an electronic sensor, transforming the light rays from the object into a series of electronic impulses that can be transmitted, stored, manipulated and altered in an almost infinite number of ways. 

Digitizing Genealogy -- An Introduction to the Series

It has been a couple of years since I last wrote about the digitization process from a practical standpoint. It is time to review the products, methods and reasons for converting our paper copies to digital images. The overall process seems simple enough, the paper document is scanned/photographed and the image is then attached to the pertinent individuals in a genealogical database program and/or an online family tree. But it turns out that the details of such an operation can be overwhelming to many genealogists.

This is the first part of an ongoing series on the entire process of digitizing documents. I intend to include both the mechanics and theory of the process including most, if not all, of the options. I also intend to discuss the pros and cons of digital copies vs. original paper and the serious issues of digital preservation. The equipment for making very high quality digital images is very rapidly evolving. Just one example, Canon has announced a 50.6-Megapixel camera for introduction in June of 2015. Much of the previous discussion about making digital images centers around the issue of flatbed scanners vs. cameras for archive images. The resolution of the images is a controversial topic among archivists and others concerned with the quality of the images. As the equipment available for consumer use increases in quality, these quality issues become more and more esoteric.

My intention here is to meld my 62 years of photography experience with my 46 years of computer technology experience and explain the issues and details of the issues in terms of my now 33 years of genealogy experience. In the process, I may also discuss the legal/ethical issues based on 40 years of legal experience. I will also throw in, for good measure, my perspective of working and living in libraries and archives for the past 62 years.

The proliferation of cameras coupled with phones, makes estimates of the number of cameras in use today almost impossible. Some estimates run as high as 5.8 billion cameras in use around the world out of a total population of 7 billion and that includes a figure of 1.8 billion sold in 2014. As a result of this expansion, many genealogists are likely carrying around a camera that is perfectly adequate for document preservation and don't even know it. Recognizing this change in technology, there are a very few archivists who are seriously considering the use of consumer level cameras for archive purposes. See the University of Illinois at Urbana-Champaign Digital Historian Series.

At the same time cameras have evolved from the specialized purchase option into something everyone carries around all the time, the scanning technology has also rapidly evolved and the cost of a high quality multipage, double-sided, sheetfed scanner has dropped dramatically. The prices of all types of scanning equipment have plummeted and today the cost of setting up a scanning operation is minimal compared to the cost of the labor to do the scanning and catalog the results.

At the same time, the field of digital preservation is fraught with its own dangers and concerns. Access to existing paper collections is spotty throughout the world. Many genealogically significant records are still "locked up" by bureaucratic red tape and concerns about legal rights. The monetization of records is also a concern. Of course politics plays a huge part in the availability of records and repressive governments are not limited to developing countries. As technology makes taking the images less expensive, both public and private repositories and archives are seeing the revenue value of their collections and limiting reproduction rights to preserve cash flow.

The issues that accompany digitization do not end with the production of a digital image. What happens to that image after it is produced is probably as important as the creation of the image itself. Many large companies today make sizable incomes from the process of organizing and displaying digital images of documents.

Now, to summarize, this series will touch on as many issues as possible across the entire spectrum of using digital images for genealogical research and preservation. Rather than numbering the series, I will tag the posts with the words "Digitizing Genealogy" and all the posts in the series will appear in the sidebar list of topics I have created. You may have to scroll down a ways to the list, but the series will be there and available.