The LAMP Batch OCR Handbook

Transcription

1 The LAMP Batch OCR Handbook by Daniel Ramsbrock, OCR Client OCR Client 1.0 with Asian Language Support...4 FineReader OCR Client...6 Sakhr OCR Client...8 ReadIRIS 9.0 Corporate Edition...10 How to Generate Summary Reports...12

2 OCR Client 2.1 User Documentation by Daniel Ramsbrock, LAMP, Written: April 7, 2004 Revised: June 30, 2004 Description This software allows you to upload any number of TIFF image files and have them remotely OCRed into text in any number of formats (including plain text, Word, and PDF). Requirements 1. Microsoft Windows (will NOT run on Mac, Linux, UNIX, etc.) 2. Java (might also work with Java 1.4 or better) 3. Network access to the pc.umiacs.umd.edu internal network (can be via VPN). Contact for help. Installation 1. Download the OCRClient2.1.zip file from 2. Unzip to a directory of your choice. 3. On Windows XP, you should just be able to double-click on the OCRClient2.jar file in order to star the program. 4. If that doesn't work, open a DOS window, cd to the directory where you extracted everything and type "java -jar OCRClient2.jar" Usage The top half of the main screen is used to select the TIFF input files to process. You can add individual files with the Add files button and whole folders with the Add folder button. When adding a whole folder, a small progress bar will appear at the bottom right corner of the file list indicating how many files are still left to be added. To delete a file from the list, select it and hit the Delete key on the keyboard. To remove all files from the list, click the Clear list button. If you have selected files from multiple input directories, you will most likely want to check the Use source directory as output option, which will put the output files into the same directory from which the input image came. If you would like all of the output files to be placed into the same directory, uncheck the box and use the Browse button to locate the directory. In the bottom half of the screen, you can select the OCR options, which are fairly selfexplanatory. Select the ones you want, and then click on "Perform OCR." Note that you can select multiple languages (with Shift/Ctrl), but only one spell-check language for the document. As each document is processed, it will give a result of either "SUCCESS: filename.tif" or "FAILED: filename.tif." Failures are usually due to either incorrectly formatted image

3 files (must be TIFF, no LZW compression) or such bad image quality that OCR is impossible. Also, double-check that you selected the right language for your document. If you have the Do not overwrite OCR files option checked, a third possibility is SKIPPED: filename.tif, which means that the output file already existed and OCR was not performed on that file. If you selected text as your output, you can click on the result to get a quick preview of the output files. Note that this will not work for Unicode text or any other output format. Multiple Datasets You can also perform batch OCR on multiple datasets, given that they all share the same parent directory (i.e. ENGLISH1 contains ORIGINAL, FAX, SCAN, etc.). To run on multiple datasets, execute the following: java -classpath OCRClient.jar BatchClient parentdir runname language parentdir - the top-level directory that contains all the datasets you want to process runname - the code name of the OCR software you are using (i.e. SCANSOFT12) language - the primary language of the documents Hint: You can set the language to check in order to verify that OCR has been completed for any given run. It will not actually perform the OCR but only compare the file counts in the IMAGES and OCR/runName_OCR. If they do not match, it will list the missing files individually. Problems? Contact Daniel Ramsbrock, spikes51@umiacs.umd.edu

4 OCR Java Client 1.0 with Asian Language Support by Daniel Ramsbrock, LAMP, Written: Dec. 13, 2003 Revised: June 25, 2004 Note: Even though the regular version of OCR Client 1.0 obsolete, this is still the most recent version available for Asian OCR with ScanSoft. Installation/Requirements The OCR client is being distributed as a Java.jar file, which means that the only "installation" necessary is to copy that file to your hard drive. It was coded under the release of the Java SDK, but may also run under 1.3.1; it will definitely not run under Most operating systems will simply let you double-click the.jar file, which will launch the program automatically. If your system does not allow this, the following command (in a command prompt) will launch it: java -jar OCRClient.jar. Note: The Asian OCR server must be running on Lamppc17 in order for this program to work properly. See the ScanSoft section in the Asian OCR Howto for instructions on how to start the server. Usage From the main window, follow these steps to perform an OCR operation: 1. Click the Browse... button next to "TIFF Input Files;" this will bring up a standard file selection dialog. You can select multiple files within this dialog using the Shift and Ctrl keys as usual. Click Open when you have selected all the desired files. Please note that the input files must be uncompressed TIFF files, with an extension of either.tif of.tiff. 2. The list of selected files will now appear in the list below the Browse... button. Also, the Output Directory has been automatically set to the same directory as the source directory of the TIFF files. This is where the resulting.txt files will be stored. You can pick a different directory by clicking the Browse... button next to the Output Directory input field. 3. At the bottom of the window are a few client-side OCR options, with the most common ones already selected. The labels are pretty self-explanatory. Note that only the last four languages on the list will work (Japanese, both types of Chinese, and Korean). The server should always be the given one: lamppc17.pc.umiacs.umd.edu. 4. Once all the files are selected and the options set, click Perform OCR. This will bring up a window which will show the status of each input file, along with a progress bar for the overall operation. If it says SUCCESS for any given file, the OCR operation completed successfully. INVALID INPUT FILE means that the input file could not be found and/or opened. FAILURE means that the input file was found, but the OCR engine could not parse it. In the latter case, ensure that the TIFF file is in the proper format (i.e. uncompressed). You might also try enlarging the picture and/or increasing the resolution.

5 5. To view the output, simply click on the file you'd like to view. The left and right arrows in the window that comes up can be used to view the previous and next files, respectively. Multiple Datasets You can also perform batch OCR on multiple datasets, given that they all share the same parent directory (i.e. ENGLISH1 contains ORIGINAL, FAX, SCAN, etc.). To run on multiple datasets, execute the following: java -classpath OCRClient.jar BatchClient parentdir runname language parentdir - the top-level directory that contains all the datasets you want to process runname - the code name of the OCR software you are using (i.e. SCANSOFT12) language - the primary language of the documents Hint: You can set the language to check in order to verify that OCR has been completed for any given run. It will not actually perform the OCR but only compare the file counts in the IMAGES and OCR/runName_OCR. If they do not match, it will list the missing files individually. Problems? Contact Daniel Ramsbrock, spikes51@umiacs.umd.edu

6 FineReader OCR Client User Documentation by Daniel Ramsbrock, LAMP, Written: June 29, 2004 Description This software allows you to perform batch OCR using the FineReader OCR engine. Requirements 4. Microsoft Windows 5. Local installation of the FineReader engine 6. FineReader USB license key in local computer Installation 5. Download the FineReaderOCRClient.zip file from 6. Unzip to a directory of your choice. Usage To perform OCR on a set of files where all input files are in the same directory and all output files will go into the same directory, use FineReaderClient.exe. Navigate to the directory where your input images are located. Notice that this directory must ONLY contain valid TIFF files; otherwise the program will crash when it tries to process them. The output path will follow the input path, so you can make small changes to it once you ve selected the input path. Note that if the output directory does not exist, it will be silently created. Also note that if any of the output files exist already, OCR will be skipped for them. Select the primary language of your OCR documents and click on Process. A window will come up informing you of the current file being processed and its status. Most likely, this window will appear to freeze at some point during the OCR process (I tried to fix this, but threads, Visual Basic, and myself don t get along too well ). At any rate, the program will continue to run properly even though the window appears to have crashed and may be not responding according to Windows. Just let it run, checking the output directory from time to time in Explorer to make sure that new files are still being created. Multiple Datasets You can also perform batch OCR on multiple datasets, given that they all share the same parent directory (i.e. ENGLISH1 contains ORIGINAL, FAX, SCAN, etc.) using BatchClient.exe. In the main window, fill in the following fields: Base Directory - the top-level directory that contains all the datasets Run Name - the code name of the OCR software you are using (i.e. SCANSOFT12)

7 Language - the primary language of the documents Click on Run Batch, and the program will enter each subdirectory of the base directory and attempt to process it as a dataset. The same disclaimer as above applies about the window appearing to hang, but the program continuing to run properly. If you are planning on an unattended run of this, it might be a good idea to use AutoEnterSender.exe. This program will simulate Enter being pressed on the keyboard every 20 seconds in order to clear any non-fatal error messages that may come up during the run. It is advisable to minimize any other applications that may be running and make sure that only BatchClient is in focus when you leave the workstation. Problems? Contact Daniel Ramsbrock, spikes51@umiacs.umd.edu

8 Sakhr OCR Client User Documentation by Daniel Ramsbrock, LAMP, Written: June 29, 2004 Description This software allows you to perform batch OCR using the Sakhr OCR engine. Requirements 7. Microsoft Windows 8. Local installation of the Sakhr engine Installation 7. Download the SakhrOCRClient.zip file from 8. Unzip to a directory of your choice. Usage To perform OCR on a set of files where all input files are in the same directory and all output files will go into the same directory, use SakhrClient.exe. Navigate to the directory where your input images are located. Notice that this directory must ONLY contain valid TIFF files; otherwise the program will crash when it tries to process them. The output path will follow the input path, so you can make small changes to it once you ve selected the input path. Note that if the output directory does not exist, it will be silently created. Also note that if any of the output files exist already, OCR will be skipped for them. Select the primary language of your OCR documents and click on Process. A window will come up informing you of the current file being processed and its status. Most likely, this window will appear to freeze at some point during the OCR process (I tried to fix this, but threads, Visual Basic, and myself don t get along too well ). At any rate, the program will continue to run properly even though the window appears to have crashed and may be not responding according to Windows. Just let it run, checking the output directory from time to time in Explorer to make sure that new files are still being created. Multiple Datasets You can also perform batch OCR on multiple datasets, given that they all share the same parent directory (i.e. ENGLISH1 contains ORIGINAL, FAX, SCAN, etc.) using BatchClient.exe. In the main window, fill in the following fields: Base Directory - the top-level directory that contains all the datasets Run Name - the code name of the OCR software you are using (i.e. SCANSOFT12) Language - the primary language of the documents (currently Russian and Arabic only)

9 Click on Run Batch, and the program will enter each subdirectory of the base directory and attempt to process it as a dataset. The same disclaimer as above applies about the window appearing to hang, but the program continuing to run properly. If you are planning on an unattended run of this, it might be a good idea to use AutoEnterSender.exe. This program will simulate Enter being pressed on the keyboard every 20 seconds in order to clear any non-fatal error messages that may come up during the run. It is advisable to minimize any other applications that may be running and make sure that only BatchClient is in focus when you leave the workstation. Problems? Contact Daniel Ramsbrock, spikes51@umiacs.umd.edu

10 Batch OCR with ReadIRIS 9.0 Corporate Edition by Daniel Ramsbrock, LAMP, Written: June 30, 2004 Unlike the other OCR packages, ReadIRIS does not provide an API that allows us to write our own batch processing program. The corporate version does, however, provide limited batch OCR capabilities from within the GUI. It allows you to specify an input folder and an output folder, but it is obviously not aware of our directory structure and can thus only process one dataset at a time. This is obviously very inconvenient since most datasets only contain a few hundred images and the OCR must be started manually each time. To address this issue, follow the instructions below. Note the following about the scripts: They are in /fs/lamp/users/doermann/unix/degredation/scripts/conversion, so be sure add that directory to your path. ds.txt refers to a list of all of the subdirectories that contain datasets (can usually be generated by ls -p1 grep "/") They are designed to be run under UNIX, so you will need to log into bo.cfar or some other UNIX machine with LAMP network file system access 1. In the parent directory that contains all the datasets you want to process, create a directory called ALL. Inside that directory, create directories IMAGES and OCR. Note: All further instructions assume you are in this parent directory. 2. The combine script creates symbolic links from all the images in each dataset s IMAGES directory to ALL/IMAGES. Usage combine.pl < ds.txt 3. In Windows, open the ReadIRIS program. Go to File -> Batch OCR. For the image folder, navigate to the ALL/IMAGES directory. For the text folder, choose the ALL/OCR folder you created above. You do not need to check any of the options just click OK and the OCR will start. 4. From time to time, a message about a DRS error will come up. You can just click OK and ignore it. Other times, the ReadIRIS engine will simply crash on certain images. There is no point in trying to run in on that image again it will crash again. In this case, you will need to use the crash recovery option of the extract script (see below). Once you have ran the extract script with the recovery option, you will need to go back to step When the OCR is finished or when a crash occurs, you will need to run the extract script, which redistributes the.txt files in ALL/OCR back to their respective OCR/run_OCR directories. It also deletes the corresponding images of the finished OCR files so that they will not be processed again by ReadIRIS. Usage: extract.pl < ds.txt 6. When all images have been processed (i.e. ALL/IMAGES is empty), you will need to delete the ALL directory and its (empty) subfolders. Before you do this, be sure to run extract.pl again if there are any files left in ALL/OCR. Note that sometimes, there

11 will be a few images left in ALL/IMAGES even after ReadIRIS has finished. These are most likely either blank images or of such bad quality that OCR was not possible. In this case, ReadIRIS does not create a corresponding OCR file, so you will manually have to create the corresponding blank OCR files (with the UNIX command touch, for example) and delete the remaining images. Crash recovery option (for extract.pl): in order to use this, you must have a list of ALL/IMAGES in "natural" (i.e. filesystem order) called "ocrorder.txt". This can be obtained by executing ls -1U under GNU Linux/UNIX (will NOT work under Solaris on bo). The script uses this file to figure out which file ReadIRIS last died on, creates an empty output file for it, and deletes the corresponding image. Also note that you must insert a line at the very beginning of your ds.txt file containing either "y" or "n" (without the quotes). Problems? Contact Daniel Ramsbrock, spikes51@umiacs.umd.edu

12 How to Generate Batch OCR Summary Reports by Daniel Ramsbrock, LAMP, Written: June 30, 2004 When managing collections of datasets containing several thousand documents, it is crucial to be able to quickly get an overview of the status of each dataset. What started out as a simple way for me to check how many files were in the IMAGES and OCR/run_OCR subdirectories turned into a conglomeration of scripts that now generate a complete report for all datasets in spreadsheet (CSV) format. Usage The scripts are in /fs/lamp/users/doermann/unix/degredation/scripts/reports, so be sure add that directory to your path. They are designed to be run under UNIX, so you will need to log into bo.cfar or some other UNIX machine with LAMP network file system access The files runs.txt and collections.txt determine which datasets will be included in the reporting. In runs.txt, list all the runs you d like to include, not including the _OCR suffix. In collections.txt, list the parent paths of the collections (i.e. the directory that contains all of the datasets for a given language/collection). These paths should be absolute UNIX paths from the machine where you will be running the scripts. 1. To generate reports for all existing datasets, simply run allreports. This will generate an individual report in /fs/lamp/users/doermann/unix/degredation/reports for each existing collection-run combination with the following naming convention: collection_run_report.csv and also one file called All Reports.csv containing all of the reports. Note: The scripts determine whether a dataset exists for a particular run based on the presence of the ORIGINIAL/OCR/run_OCR directory, assuming that every collection would at the very least have an ORIGINAL dataset. 2. To generate individual reports, use the following syntax: filecounts run Report.csv, where run is the name of the run (i.e. SCANSOFT12) and Report.csv is the name of the file where you d like the report to be generated. Note: You must be in the parent directory of the collection when running this command. 3. You can now open any of the.csv files in any spreadsheet program, such as OpenOffice or Microsoft Excel. Under the Hood Description of the Scripts The main script, allreports, calls allruns on each line of text in collections.txt. Allruns, in turn, calls fcwrapper on each line of text in runs.txt. Fcwrapper is simply a wrapper for the filecounts script to avoid a too-long-line error with xargs (which is used by allruns). Filecounts is called with the name of the run and the path of the report, just like in step 2 above. Now the real work begins: filecounts generates several scripts to go into each subdirectory of interest (IMAGES, OCR/run_OCR, etc.) and count the number of files in

13 each. It also calls timediff on the OCR/run_OCR directory, which averages the difference between sequential timestamps, discarding anything over one minute. This provides us with an estimate as to how long the OCR for each file took. Next, report.pl compiles the first five columns of the report: the name of the dataset, number of images, number of OCR files, number of failures, and the status, which is determined based on the number of image and OCR files. The rest of the columns are simply tacked on to the right using calls to mergecols. Finally, addtotals is called to add column totals, the failure rate in percent, and filesystem hyperlinks to the title and dataset names. Once all the reports have been generated in this fashion, allreports uses cat to combine all the.csv files into All Reports.csv. However, the cell addresses in the sum formulas added by addtotals are now no longer valid, so redototals is called to recalculate the cell addresses in the new.csv file. Problems? Contact Daniel Ramsbrock,