Discover the Depths of your Data with iarchives OWR The benefits of Optical Word Recognition

Transcription

1 Discover the Depths of your Data with iarchives OWR The benefits of Optical Word Recognition Through unique technological developments, iarchives is continually exceeding the quality and efficiency standards for the transfer of microfilm and other original documents into digital, searchable online databases. The iarchives process substantially reduces the cost and time required to archive documents while enhancing the user's experience in exploring those documents. iarchives provides rapid, accurate, and inexpensive document scanning, digitization, indexing and processing with custom search and retrieval tools for a variety of clients, with the ability to customize our process to meet our clients particular needs. Some of the industries to which we provide service and solutions include: Newspapers and Periodicals Insurance Companies Libraries Utility Organizations Universities Financial Institutions Law Firms Genealogy Service Companies Healthcare Organizations Government Agencies In the world of information technology, pure, untainted information and knowledge can be difficult to find; yet, it is expedient that this information be available at the touch of a button. It is through Optical Word Recognition (OWR), our patented technology at iarchives, that we are seeking to perfect the digitization of hard copy documentation to create easily accessible information for our clients and for the continuation of the growth of knowledge throughout the world. This article will focus on the superiority and breakthrough technology of OWR versus its counterpart, Optical Character Recognition; here, we will reveal the ways in which iarchives has furthered the digitization process through advancements on the development and accuracy of OCR. OCR vs. OWR In order to explain the unique nature of Optical Word Recognition (OWR), it is necessary to define and to contrast it to its predecessor, Optical Character Recognition (OCR). Optical Character Recognition (OCR) technology does exactly as the term suggests: it is an engine that recognizes, or reads, the visual characters of a document that has been scanned into a computer in picture form. In contrast, Optical Word Recognition (OWR) recognizes words rather than simply the characters that form words. OWR accomplishes this through the process of comparing and contrasting the results of several OCR engines, by which OWR evaluates and then identifies each word. Through our studies, which results we will review, OWR has proved more effective than OCR. Accuracy Defined In order to compare OCR and OWR, we must first review the ways in which accuracy is defined. For both technologies, accuracy is measured in an identical fashion: the percentage of characters that are accurately recognized, and also through two systems known as Recall and Precision. Word Character Accuracy When analyzing the percentage of characters that are correctly recognized, it is necessary to understand the role that word length plays. Depending on the dictionary, the average word length is about 9 characters. The average length of commonly used words, such as 'happy' and 'accuracy', is 7 characters; however, the average

2 word length in most English texts, i.e. magazine articles, newspaper articles, novels, etc., is typically much smaller. For example, the average word length in Moby Dick is only 4.67 characters. So, let us assume that an average page contains 200 words and that each word is 5 characters in length. This translates into 1000 characters per average page. These specifications are what we will refer to as the Average Page. When defining accuracy, it is necessary to understand that OCR recognizes characters and not whole words. So, in theory, if an OCR engine correctly identifies 990 of 1000 characters, it would naturally be assumed that the accuracy rate is 99%. This is not the case, however, when considering that each of the 10 misrecognized characters are most likely found within 10 different words. Therefore, it is necessary to judge the accuracy of OCR by applying it to words and not characters. Subsequently, on TAP we find that OCR has found only 190 of 200 correct words. This means that rather than a mere 1% failure rate, OCR faces a 5% word-failure rate, and it is by words that documents are searchable, not individual characters. Recall & Precision As stated previously, accuracy is also measured by Recall and Precision. Recall is the degree to which a search returns all the matches in a document collection. For example, there may be 100 matching documents, but due to OCR errors, a search may only yield 80 of them. This is an 80% Recall rate. Precision is related to recall in that it measures the degree to which a search is able to return relevant documents. For example, if a search returns 80 documents but only 20 of them are relevant to the search, then the Precision is only at 25%. How OWR Works Optical Character Recognition (OCR) delivers only a single result for each word in a text image. In contrast, through our patented process, Optical Word Recognition (OWR) delivers one or more results for each word by employing several OCR engines, each of which may give a unique result. This is especially important when considering the Recall accuracy rate when searching for co-related documents. The following example illustrates the ways in which OWR seeks to correct misrecognition issues of OCR engines: Figure 1 In Figure 1, we see that OWR seeks the word results for a single word: surname 'Black'. OCR Engine A misreads the word as Block, whereas OCR Engines B and C read the word correctly. By default, OWR preserves both results so the OWR text file will contain both 'Black' and 'Block'. For projects where multiple results are undesirable, however,

3 OWR can also eliminate, accordingly, the undesirable word by determining which result is most probable. When only a single result is desired, if all three engines make a different choice, the lead engine (Engine A) as illustrated in Figure 2, takes precedence over the other two. Figure 2 Thus, a search for the name 'Black' will not return this page unless two or more results are retained. Image Quality For OCR and OWR, image quality is the most important factor that affects text accuracy. Original documents which contain broken or blurred characters, complex formatting, or significant variation in font type or size can greatly affect accuracy. It is for this reason that iarchives scans images with proper attention to contrast, focus, skew, resolution, and brightness. We classify the quality of text images into three simple categories: Good, Medium, and Poor. Good Images. These images are usually first generation images scanned from original documents and printed with optimally functioning printers. They have no speckle noise or skew, and the characters are crisp and non-fragmented. Fifty pages from the O.K. Corral deposition are used as an example of Good images. Medium Images. These may include photocopies or faxes of an original document. The document may be printed on faded or yellow paper, contain some speckle noise, non-uniform background, and/or have low contrast. The characters may be slightly fragmented or blurred and the document may contain charts, graphs and other unique formatting. Several documents from the Microsoft Anti-trust trial serve as good examples of Medium images. Poor Images. These may include photocopies of copies or faxes. This type of document generally contains some form of speckling and/or a non-uniform background; also, the characters are not crisp and may be fragmented or blurred. Poor images may come from microfilm, have complex formatting, and several font types or sizes; these may also include second generation Medium images. Sixteen pages of different newspapers scanned from microfilm are used as examples of Poor images. Reference the Addendum for some samples of these. The importance of good quality images cannot be overemphasized as low quality images will give poor OWR and OCR results. Results Upon scanning the previously mentioned documents under each of the three categories: Good, Medium, and Poor, and

4 after assessing the results utilizing Recall, the following information was generated: 50 images of the O.K. Corral Deposition, classified as Good images: Good Images Recall OWR 99.36% OCR Engine A 98.52% OCR Engine B 98.47% OCR Engine C 97.52% The OWR has a.86 % higher accuracy than the highest OCR engine. In terms of an Average Page, 1.68 more words were identified by OWR than the best OCR engine. Note: keep in mind that these are highly readable documents. 39 Microsoft Anti-trust case images, classified as Medium images: Medium Images Recall OWR 93.78% OCR Engine A 88.74% OCR Engine B 90.58% OCR Engine C 81.95% In this example, the OWR has a 3.2% greater accuracy rate than the best OCR Engine. For an Average Page, OWR identifies 6.4 more words than the top OCR engine per page. 16 Newspaper images, classified as Poor images: Poor Images Recall OWR 67.43% OCR Engine A 37.75% OCR Engine B 38.53% OCR Engine C 34.87% Within this final example, we see that the OWR accuracy is 28.9% higher than that of the best OCR Engine. Again, in reference to an Average Page, the results show that the OWR identifies 57.8 more words, per page, than the best OCR engine. These results may better be illustrated as seen together in the following graph:

5 These numbers make a considerable difference when processing 100,000 pages, which iarchives manages 2-3 times that in a 24 hour period, even on the clearest documents where 1.68 more words are accurately discovered. It stands to reason that within a collection of 100,000 Average Pages OWR is discovering 168,000 more searchable words than the best OCR engine. OWR is clearly superior to any single shot of an OCR engine. OWR, iarchives & You Conclusion When it comes to transforming microfilm and other original documents into searchable, digitized, online databases, iarchives, Inc. has produced a technology which has created an unparalleled standard of quality and efficiency. Through the development of Optical Word Recognition (OWR), which effectively utilizes three OCR engines, iarchives is a leading producer of the highest caliber searchable information. Today, through this process, they are helping to create a better world for this and future generations through the digitization and preservation of delicate, educational, and historical documentation. Addendum Two examples of Good Images from the OK Corral Deposition:

6

7

8 Two Examples of Medium Images from the Microsoft Anti-trust Trial:

9

10 Two examples of Bad Images from Newspaper scanned from Microfilm:

11