Handwriting Recognition is in Your Future

Probably the most sought after goal for assisting genealogical research is automated handwriting recognition, sometimes known as HTR or Handwritten Text Recognition. Computer recognition of printed text has already come a long way and many genealogists have been using OCR or Optical Character Recognition software for many years. In fact, nearly all of the searchable books and other documents online have been run through an OCR program. I once lost the computer file of some of my oldest journals but I had printed out a copy and was able to use OCR to reconstruct the computer file. However, because so much of what I do comes from the internet or in PDF format, I no longer need to rely on OCR programs per se. Currently, Adobe Acrobat DC or Adobe Acrobat Pro both automatically apply OCR to any PDF document and allow those PDF documents to be edited.

I have written about this subject a couple of times previously. See the following.

• Is Handwriting Recognition the Holy Grail of Genealogy?
• Handwriting Recognition, OCR and Genealogy

I am also aware, as I indicated previously, that the BYU Family History Technology Lab is working on handwriting recognition and has been for some time.

Handwriting recognition is a completely different matter. The individual variations in handwriting have, so far, defied almost all attempts to employ consistent handwriting recognition. There are a number of highly skilled computer programs used by the U.S. Post Office and other similar organizations for automating handwritten envelopes but in almost all cases, the handwritten portions are ultimately reviewed by a human worker.

Efforts to utilize handwriting recognition are split into two distinct usage areas: those programs that are aimed at personal and organizational uses where the writing is done on a tablet computer and those programs that are aimed at translating paper-based, already existing handwriting.

I can presently use my iPad Pro and the Apple Pencil and write on the screen and get a fairly good translation of my very sloppy handwriting but that is not much help to genealogical research. There are some major issues with developing an effective method of recognizing handwriting or cursive as follows:

• Variations in the individual writer's style and penmanship
• Different historical phases of handwriting style, with changes over time
• The always present changes in words, meanings and sound shifts
• The poor condition of the original manuscript or the microfilmed copy

Depending on the representational system used, i.e. Roman alphabet, Cyrillic, Chinese, Japanese etc., there is also a challenge in translating the characters from one system to another. Behind the language itself exist the variation in kinship patterns and genealogical relationships. 

Let me give a few simple examples. The first concern is the challenge of translation. If I ask the Google Translate program to translate the phrase "egg salad sandwich" into Spanish, I get "Ensalada de huevo," literally, "egg salad." I end up losing the "sandwich" part altogether. Using the SpanishDict.com website, I get "sándwich de Ensalada de huevo," which is a literal translation of the phrase from English. But it is likely that Spanish would properly say, "sándwich de Ensalada de huevos." with the word for "egg" in the plural. The English version of "egg salad sandwich" does not imply that only one egg is used in the recipe. The word "egg" becomes a preposition in English and is used in the singular form. Egg salad is a type of salad or filling used to make a sandwich. 

This is a rather simple example but illustrates the difficulty in translating one language to another. Can we expect our handwriting recognition software to be sophisticated enough to differentiate between these nuances in meaning and word usage?

Now, to have an adequate handwriting recognition program, the program would have to somehow incorporate the vast number of social and cultural nuances present in each of the target languages.

Here is an example of some current handwriting.

By Stevenson, A 2012 - Stevenson, A 2012. ‘We Seem to be Working in the Same Line’: A.H.L.F. Pitt-Rivers and W.M.F. Petrie. Bulletin of the History of Archaeology 22(1):4-13, doi:10.5334/bha.22112, CC BY 3.0, https://commons.wikimedia.org/w/index.php?curid=27324937

You can immediately see the challenge that anyone might have reading this handwriting much less the difficulty of programming a computer to read it. Here is an example of what a genealogist might be faced with deciphering.

By Niccolò de' Niccoli - Written by Niccolò de' Niccoli (1364–1437). Copied in The Origin and Development of Humanistic Script by Berthold Louis Ullman (Rome, 1960), Plate 37 Florence, Naz. Conv. Soppr. I. X. 44, folio 1v Lives of Alexander and Caesar translated by Guarino. Scanned by HDoug of The Fountain Pen Network. Posted on "italic" term in computers and writing. Uploaded to en.wikipedia by Wareh. Transferred to Commons by User:朝彦 using CommonsHelper., Public Domain, https://commons.wikimedia.org/w/index.php?curid=15389520

And this example is amazingly clear and readable.

Notwithstanding the difficulty of the challenge, there are quite a few institutions and other entities working generally on the problem. As is the case with many technological advances, the companies working on developing new, salable products are not interested in discussing their breakthroughs with other companies out of a fear of losing the patents to their products. One thing is certain, with more and more powerful computers with sophisticated artificial intelligence programs, there will be significant progress in the not-too-distant future in this area.