Fault Tolerance in JPEG Image Headers Using Backus-Naur Form

Transcription

1 Fault Tolerance in JPEG Image Headers Using Backus-Naur Form March 19, 2002 Presented by: Brandon Smith Advisor: G. Robert Redinbo, Ph.D. In Completion of Master s Degree Plan II Requirements University of California, Davis Department of Electrical & Computer Engineering

2 Abstract The JPEG standard, a popular color still image compression standard, is used in critical scientific, surveillance, medical, and remote-sensing applications. This lossy compression technique relies on the discrete cosine transform coupled with Huffman or arithmetic coding for significant file size savings. However, a decoder will not be able to properly decompress information from an incorrectly formatted JPEG image. Two-byte JPEG image headers delineate the encoding tables, parameters, and other data needed for image decompression. A Backus-Naur Form (BNF) description of the JPEG standard is utilized to create a JPEG image parser capable of syntactically verifying the integrity of a JPEG format image. This parser is part of a larger fault tolerant system that allows the integrity of JPEG images to be verified while detecting transient errors. 2

3 Project Motivation Many critical scientific, surveillance, medical, and remote-sensing applications use JPEG image encoding to compress still images. JPEG compression typically leads to drastic file size savings when compared to similar, uncompressed images. There are many benefits inherent in smaller files. One benefit is the ability to store a greater number of images given a finite amount of storage. This can be important in applications where the amount of storage is limited due to cost or power consumption constraints. In 1999, the United States Navy s USA satellite program faced this exact situation. The USA s mission was to capture galactic X-ray images. However, the number of image captures possible was limited by the storage space available aboard the satellite. It was not practical for the satellite to contain more storage due to power constraints. Orbiting the planet at a rate of one revolution approximately every 90 minutes, the photographs were downloaded as the satellite passed over a downlink station and the storage space released for reuse. If the images could be compressed aboard the craft, a dramatically larger number of images could be captured and transmitted with minimal effect on satellite power consumption. JPEG is also commonly used in medical imaging devices. Digital X-rays and other medical images are typically stored and transmitted as JPEG images. The ability for medical personnel to share detailed diagnostic images across large distances would not be practical without the use of JPEG compressed images. Should the image become unreadable due to an error during, the patient could be forced to endure further radiation exposure. However, the use of JPEG is a tradeoff. On one hand, are the many advantages associated with the use of image compression. On the other hand, as with all compressed data formats, there is an inherent risk of data loss associated with the use of JPEG images. Most notably, due to the use of instantaneous codes such as the Huffman code, a single bit error can result in all subsequent codewords being in error. More subtle, but just as detrimental, is the risk of an incorrectly formatted JPEG image. In this work, a fault tolerant system is proposed which allows the format of JPEG images to be verified. 3

4 Introduction to JPEG Images The JPEG standard is among the most popular color still image compression standards. Developed by and named after the Joint Photographic Experts Group, JPEG images are widely used because of their ability to accurately represent images using only a small percentage of the bytes found in similar uncompressed images. JPEG format images typically consume fewer bytes than uncompressed images for several reasons. JPEG images may be compressed using a lossy algorithm. This means that some data is approximated in a way that allows for more efficient compression of the image. Small details of an image may be lost if compressed using a lossy method, but will often result in a dramatically reduced image file size. However, at higher compression ratios, undesirable artifacts of compression become obvious when viewing the JPEG compressed image. After the image is transformed and quantized, either an arithmetic or Huffman coding scheme is used to compress most of the data found in JPEG images. Since the bulk of the data in an image is compressed, significant file size savings are attainable. Coding tables, quantization tables, and other necessary image parameters are stored within the image to facilitate decompression as dictated by the JPEG image standard. Two-byte JPEG image headers delineate these tables, parameters, and other image data, and are important items within the JPEG standard. In this work, a fault tolerant system is proposed which allows these headers and parameters to be syntactically verified before and after transmission and/or storage. JPEG Format Headers Each JPEG header segment begins with one or more byte-aligned 0xFF bytes. Typically, only a single 0xFF byte is seen, but the JPEG standard allows for multiple 0xFF bytes for use in JPEG systems where word alignment is desired. This is followed by a non-zero single-byte code used to determine the exact function of the data that follows. The segments fall into two categories: those with parameters and those without parameters. Should the marker code indicate that a particular data segment requires parameters, the subsequent two bytes will contain the byte length of the segment. Such lengths include the 4

5 two bytes in the length parameter, but exclude the bytes used in the marker that defines the function of the segment. If the length of a segment is known, it may contain accidental markers ; bytes that look like a marker, but are only data. These bytes will not be construed as data since the segment length is known for nearly all data segments. The exception is the entropy-coded data section, which is of unspecified length. This requires that any 0xFF bytes be stuffed with an additional 0x00 byte. Since 0xFF 00 is an illegal combination for a marker, the stuffing of 0xFF bytes with 0x00 bytes prevents inadvertent generation of markers within these segments. The following table lists the header codes, segment lengths and descriptions for all JPEG headers. All meaningful headers are contained within the table, including areas reserved for future expansion of the JPEG standard. Two byte patterns are not considered valid byte codes. All markers begin with one or more 0xFF bytes. For this reason, 0xFF is not included in the table since 0xFF will never be considered a valid header code. The byte immediately following the last 0xFF byte will be construed to be the intended marker. Similarly, the 0x00 byte is also not listed in the table because it is used for stuffing 0xFF bytes as explained above. Header Code 0xFE Segment Length Variable Segment Description Comment Marker Segment. May contain text comments. Prohibited from containing information that could affect decoding. 0xF0 0xFD Reserved for future expansion of JPEG standard. 0xE0 0xEF 0xDF 0xDE Variable 3 bytes Variable Application Data Marker Segment. Reserved for application use. Often contains image processing or coding hints. May contain JPEG stream for small thumbnail images. May also contain text such as copyright information and identification of application that compressed the image. Expand Marker Segment. Indicates expansion of the spatial resolution of the reference data needed by the hierarchical differential frame that follows. Define Hierarchical Progression Marker Segment. Indicates the parameters identifying the dimension, components, and relative sampling of the final image of the hierarchical progression. 5

6 0xDD 0xDC 0xDB 0xDA 0xD9 0xD8 0xD7 0xD0 0xCF, 0xCE, 0xCD, 0xC7, 0xC6, 0xC5 0xCB, 0xCA, 0xC9, 0xC3, 0xC2, 0xC1, 0xC0 0xCC 4 bytes 4 bytes Variable Variable Variable Variable Variable Define Restart Interval Marker Segment. Provides mechanism for setting and resetting restart interval. When the restart interval is zero, the restart function is disabled. Define Number of Line Marker Segment. Provides a mechanism for defining or redefining the number of lines in the image at the end of the first scan. Define Quantization Table Marker Segment. Defines one or more quantization tables. The DQT marker has meaning only for DCT based algorithm. Start of Scan Marker. Begins a scan header. Followed immediately by entropy-coded data for the scan. End of Image Marker. Terminates the JPEG compressed data stream. Start of Image Marker. Begins the JPEG compressed data stream. Restart Marker. Appended to the compressed data between restart intervals. Eight different marker codes allows for a three-bit, modulo-8 restart interval count. Differing marker codes allow decoding software to ensure sequential reception of data blocks. Analogous to a sliding window protocol. Differential Start of Frame Markers. Indicates the beginning of a frame header, which contains the frame parameters that apply to all scans within the frame. The particular value of the marker code identifies the mode of compression and the entropy coder used within the frame. Non-Differential Start of Frame Markers. Indicates the beginning of a frame header, which contains the frame parameters that apply to all scans within the frame. The particular value of the marker code identifies the mode of compression and the entropy coder used within the frame. Define Arithmetic Coding Conditioning Marker Segment. Defines one or more tables of parameters for the conditioning of the probability estimates used in the arithmetic coding procedures. 0xC8 Reserved for future expansion of JPEG standard. 0xC4 Variable Define Huffman Table Marker Segment. Defines one or more Huffman tables. 0x02 0xBF Reserved for future expansion of JPEG standard. 0x01 Parameterless Parameterless Parameterless Parameterless Temporary Marker. Temporarily used by arithmetic encoders. Figure 1. JPEG marker byte codes, segment lengths, and segment descriptions. 6

7 A walk-through of an actual JPEG example image is provided in Appendix B. JPEG header markers and their respective data segments are highlighted and an explanation is provided for each. Singly Occurring, Statistically Independent Faults Throughout this work, it is assumed that only a single fault occurs at any one time. Subsequent faults occur only after the system has completely processed and recovered from the previous fault. The probability of two statistically independent faults occurring simultaneously is the product of the probabilities of two independent faults occurring separately. Similarly, the probability of n statistically independent faults occurring simultaneously is equal to the product of the probabilities of n faults occurring individually. If the probability of a single, statistically independent fault is small enough, the probability of two or more simultaneously occurring faults becomes minute enough so as to become statistically insignificant. If faults are not independent, they may occur in larger numbers, which severely complicates achieving fault tolerance. Hence, it is assumed that there is a very small likelihood of recovering from multiple failures. We assume that if an event is severe enough to cause multiple errors, the likelihood that the system could flawlessly recover to a pre-failure state is very low. The handling of multiple failures is an area of study within fault tolerance known as Byzantine General problems. In some sense, Byzantine General problems are a slippery slope. If two faults occur within the system, it seems possible that three faults could just as easily occur. If three faults occur, could ten faults just as easily occur? In short, this work assumes that it is possible to recover from one concurrent fault, however, should more than one fault occur, the likelihood of recovery is very small. One fault does not necessarily translate into one error. In certain cases, a single bit error may result in many bits being incorrect. For example, if the incorrect memory address is read because of an error in the address, many bits will be wrong. In this case, a single fault is able to adversely affect many bits. 7

8 It should be clear that this work, like most works within the area of fault tolerance, is not meant to completely eliminate the possibility of failure. The objective is instead to dramatically reduce the probability that failure will go undetected. Backus-Naur Form The JPEG image format can be efficiently described using Backus-Naur Form (BNF). Originally created for the purposes of specifying and ensuring syntactic validity of programming languages, BNF can be extended slightly to include the description of JPEG format images. BNF definitions consist of terminal and non-terminal symbols. Terminal symbols are the most basic elements contained in the language or system. Non-terminal symbols are objects that may be built from terminal and non-terminal symbols, with recursion occurring often. Non-terminal symbols are expressed in lower case and surrounded by angle brackets ( < and > ). Decisions are handled by the operator, which indicates an or relationship. An example BNF description of a hexadecimal number text string is given in Figure 2. <hexnumber> 0 x <hex> <hex> <hdigit> <hex> <hdigit> <hdigit> A B C D E F Figure 2. Example BNF definition of a hexadecimal number text string. In the example shown in Figure 2, the non-terminal <hexnumber> contains the terminal symbols 0 and x followed by the non-terminal <hex>, in that respective order. <hex> consists of either a single hexadecimal digit (<hdigit>) or a single hexadecimal digit followed by another <hex> non-terminal. The result is that <hex> effectively contains one or more hexadecimal digits. <hdigit> contains a single digit in the range of 0, 1,, E, F. Using this BNF definition as an example, the parsing of the hexadecimal number 0xFFD8 would look as shown in Figure 3. The terminals are shown as triangles with their respective values inside. Non- 8

9 terminals are shown as rounded rectangles with the name of the non-terminal inside. The validated string can be read back by parsing the tree from left to right, reading only the terminals. <hexnumber> 0 x <hex> <hdigit> <hex> F <hdigit> <hex> F <hdigit> <hex> D <hdigit> Figure 3. Example BNF text parsing tree of hexadecimal number 0xFFD8. The BNF used in this example is shown in Figure 1. 8 In 1990, Thornton and Anderson created a BNF for describing the format of JPEG image headers and data segments. In this work, their BNF has been used to create a parser. The Hazards of using BNF The simple, recursive nature of BNF provides a means for constructing a syntactically correct parser to interpret a JPEG data stream. However, some important caveats exist in the use of BNF and must be resolved before a parser may be constructed from the given BNF. The example given in Figure 2 and traversed in Figure 3 has an important property. The recursion found within the <hex> non-terminal is right-handed. As shown in Figure 3, the tree tends to grow to the right as it progresses downwards. This is because within <hex>, <hdigit> comes before <hex>. If the 9

10 order of these non-terminals was reversed, the BNF would still be valid and the descriptions equivalent, but the recursion would be left-handed, and therefore the tree would grow to the left. The implications of this are seen when the BNF is exercised in the form of a function within a program. When writing code based on a BNF, non-terminals can be considered functions and terminal symbols considered constants. In the case of a left-handed version of the BNF similar to the right-handed version shown in Figure 2, within the <hex> non-terminal, <hex> would precede <hdigit>. This would cause every call to the hex() function to immediately initiate another call to hex(), which would in turn initiate another, resulting in an infinite loop. To prevent this from occurring in this work, any left-handed BNF has been coded in right-hand form to make implementation more straightforward. In the given example, the x terminal symbol is assumed to be lower case. Upper case X would not be allowed. Similarly, the alphabetic hexadecimal digits must be upper case. Lower case digits would be considered to be in error. Including both upper and lower case digits as valid terminals within <hdigit> causes the number of possible terminals to increase from 16 to 22. That is, there would be 22 possible valid hexadecimal digits, seemingly a paradox. In the existing example, preprocessing of the string of digits would be required to correctly parse lower case digits without increasing the number of terminals. Fortunately, throughout this work, we are working with hexadecimal digits made of 4 bits each (0000, 0001,, 1110, 1111), rather than text strings of hexadecimal digits. When constructing a JPEG header BNF, some non-standard modifications must be made. Most JPEG data segments (i.e. quantization and Huffman table definitions, etc.) begin with 2 bytes used to indicate the number of bytes contained in that particular data segment. A parser can read the length data and use this information to detect the location of the next JPEG header marker. Unfortunately, BNF does not include this kind of construct. There is no counting ability within BNF because of the larger issue of BNF being data independent. The BNF simply parses the data and does not have the ability to read or interpret data. To make up for this deficiency, function placeholders are shown in the BNF provided by Thornton and Anderson. These placeholders are s64chr(), s64int(), huffspec(), comspec(), appspec(), fspec(), and sspec(). Each placeholder function is distinct, but has a common purpose: to consume the 10

11 number of bytes needed to exactly arrive at the next data field while performing validation on parameters seen within the function. Another complication of the fact that Backus-Naur Form can only verify the syntax of the JPEG image headers is that the BNF is not able to perform any checking of the content of data segments within the image. The task of detecting faults in the JPEG data segments is currently under investigation by Professor Robert Redinbo and Cung Nguyen at the University of California at Davis. The goal of their work is to ensure that no errors exist within the JPEG data segments at the time these segments emerge from the compression unit. JPEG Image Header Parser The JPEG image header parser interprets a JPEG data stream and determines the validity of the stream. Valid streams will follow the assertions found within the BNF while invalid streams will not. When an error is encountered, the parser immediately ceases processing of the stream and generates an error message indicating where the error occurred. If the parser is able to parse to the End of Image marker (EOI) with no errors, it is highly probable that the image is formatted correctly. As a proof-of-concept, a JPEG image header parser has been created for this project. The C++ source code is given in Appendix E. As written, the parser accepts a JPEG image file name in the form of a command-line argument. This file is opened as a binary stream that is processed one byte at a time. The output of the parser is a text string indicating either a valid or invalid JPEG header structure. If the image is valid, the complete structure of the image is presented. If the image is not valid, a partial image structure is displayed along with a message indicating what the parser was anticipating but did not receive. This parser is considered a recursive descent parser. The stream is read from left to right, or beginning to end, and the next byte is not read until the current byte is found to fit legally into the assertions given in the BNF. If no legal fit can be found, the parser will backtrack to the last point of decision and attempt a new path. If all decisions are exhausted without legal placement of the byte, the parser will ascend to the next previous decision and again attempt to find an appropriate fit. This process 11

12 will continue until the byte is placed or all decisions are exhausted. When backtracking, all bytes read since this decision must be replaced to be read again later. In order to facilitate backtracking, several stack data structures have been implemented. Successful backtracking requires that bytes emitted from the input file stream and read by the parser getnextbyte() function be saved for later use. Bytes are saved into the Used Stack as shown in Figure 4 (part 1). When backtracking becomes necessary, the appropriate number of bytes are moved from the Used Stack to the Backtrack Reserve Stack Input Stream Used Stack Backtrack Reserve Stack (shown in part 2 of the figure). When the parser requires more bytes, the getnextbyte() function reads from the Backtrack Reserve Stack while placing the bytes back into the Used Stack (see 1 getnextbyte() function reads from Input Stream. Bytes also stored to Used Stack. getnextbyte() figure part 3). This double stack mechanism allows the left-to-right ordering of the bytes to be Input Stream Used Stack Backtrack Reserve Stack preserved while accommodating the need to backtrack. A pair of similar stacks is used to maintain and display the JPEG image structure. 2 getnextbyte() Backtracking. Move used bytes to Backtrack Reserve Stack. A quick glance at the JPEG BNF will reveal numerous examples of recursion. Some non-terminal functions are optional but may be Input Stream Used Stack Backtrack Reserve Stack executed many more times. Other functions may be executed an arbitrary number of times but are required to be executed at least once. To 3 getnextbyte() getnextbyte() reads from Backtrack Reserve Stack until empty. Bytes returned to Used Stack. accommodate this, the concepts of hard fails and soft fails are employed within the parser. Hard Figure 4. Explanation of the use of stacks to aid in backtracking within the parser. fails occur when the parser does not find a required marker. When a hard fail occurs, parsing of the image data ceases and the image is declared invalid. Soft fails occur in optional or recursive functions when a 12

13 match cannot be found for the current byte. When this occurs, the parsing of the image data will continue after a small amount of backtracking. Parsing begins with a call to the jpeg_data() function. As shown in its BNF equivalent, jpeg_data() allows for several possible forms of JPEG data, all beginning with the Start of Image (<soi>) non-terminal and ending with the End of Image (<eoi>) non-terminal. In this project, only the first form of JPEG data is supported. The remaining forms are primarily used to encapsulate JPEG segments outside of a formally created image. These forms are only of use within specialized software systems that may make use of auxiliary encoding tables, for example. Within jpeg_data(), function calls are immediately made to the soi(), nframe(), and eoi() functions. If all three of these functions return true, jpeg_data() will return true and the image will be considered valid. The first opportunity for a hard fail occurs within the soi() function. At least two calls to getnextbyte() are made from within soi(). The first bits returned must comprise be a string at least one 0xFF byte. The immediately following byte must be 0xD8, as 0xFF D8 is the marker code for a Start of Image marker. If these respective bytes are not found, the function will trigger a hard fail by initiating a call to the badimage() marker which will cease parsing of the image, indicate that no Start of Image marker was found, and exit the program. If the byte match, soi() will return true to jpeg_data(). After execution returns to jpeg_data(), a call is made to the nframe() function which calls tblsmisc(), nsof(), fparms(), and scanset(), as dictated by the BNF. A glance at the BNF confirms that the <nframe> non-terminal simply consists of non-terminals <tblsmisc>, <nsof>, <fparms>, and <scanset>. Again, the parser functions are constructed to duplicate the BNF. In this case, sequential calls are made to the above functions. Within tblsmisc(), the parser is presented with its first decision, and thus the first opportunity to soft fail. At the same time, tblsmisc() also represents the first recursive function. The <tblsmisc> nonterminal allows for the declaration of quantization tables, Huffman tables, arithmetic coding data, restart intervals, comments, and application data. Additionally, <tblsmisc> may also be empty. The empty set (denoted as ε ) serves dual purposes. First, <tblsmisc> may simply not contain any bytes, in 13

14 which case control flow will be returned to <nframe>. Secondarily, the empty set provides a means of exiting the recursion found in <tblsmisc>. If an instance of <tblsmisc> contains only a single data segment, two instances of <tblsmisc> will actually be required due to the non-terminals recursive structure. The second instance will be the empty set. In general, if <tblsmisc> contains n data segments, n+1 instances of <tblsmisc> will be required allowing for the empty set to exit the recursion. The only way the parser can detect the empty set is to receive a marker that is not valid. An invalid marker will be assumed to be part of a subsequent non-terminal and will represent the end of the tblsmisc() recursion tree. The mismatching bytes must be replaced. At this point, the backtracking function backup() is executed. When control is returned to nframe(), nsof() is called and will read the bytes out of the Backtrack Reserve stack. Parsing will continue in the manner described until either a hard failure is encountered or the end of the image is reached. If the end of the image is reached, the parser will display the entire structure of the image. If the parser were actually put into a live system, the code responsible for generating and displaying the image structure could be removed as it would not be necessary. Only a single lead output would be need to indicate valid or invalid JPEG formatting. Appendix C contains example output from the parser. Several correctly and incorrectly formatted images are parsed. The parser has indicated that valid images are correctly formed, while generating error messages for the invalid images. Parser Limitations Under certain circumstances, the proposed JPEG image format parser will not be able to detect certain formatting errors. As shown by the BNF given in Appendix A, within the JPEG standard there are multiple locations where any of several markers would be valid. If a Hamming distance of exactly one separates any two of these markers, special care must be taken. If the single bit that separates two markers is inverted, the parser may not be able to detect this error provided that the data segments that follow each marker use similar or compatible formatting. As an example, within the <tblsmisc> BNF non-terminal, the marker codes shown in Figure 5 could change from one to the other with only a single bit flip. 14

15 DHT[0xC4] DAC[0xCC] = ( ) b DAC[0xCC] APPn[0xEC] = ( ) b DHT[0xC4] APPn[0xE4] = ( ) b : COM[0xFE] : APPn[0xEE] COM[0xFE] APPn[0xEE] = ( ) b Figure 5. Markers subject to possible transformation with a single bit error. Expansion of exclusive-or operation between COM[0xFE] and APPn[0xEE] indicating Hamming distance of 1. Fortunately, the format of the data that succeeds each of these pairs of markers is different for all cases except the last. The differing data formats ensure with very high probability that should such a bit flip occur, the parser would be able catch the error and alert the system that an error has occurred. In the case of COM[0xFE] and APPn[0xEE], the last example in the list above, the data formats are not distinguishable; however, the data contained in those segments are not required for decompression, so the integrity of the actual viewable image will not be corrupted even if a comment segment is confused for an application data segment or vice-versa. Other points of contention may exist within the JPEG standard, but will not be investigated further in this project because of the very small probability that a random, transient error would corrupt a marker rather than a part of a parameter or data segment. The use of the empty set within certain JPEG non-terminals represents another shortcoming of using BNF as the basis for verification. As explained earlier, the empty set allows for empty non-terminals and also provides a mechanism for exiting a recursive non-terminal. Unfortunately, the empty set could also allow an incomplete JPEG image to escape detection. For example, within the <tblsmisc> nonterminal, in addition to non-terminals such as Huffman and quantization table definitions, the empty set is a valid terminal. A problem could arise if a necessary item such as a Huffman or quantization table was errantly excluded from the JPEG stream. Unfortunately, the parser would assume that instance of <tblsmisc> was empty and consider the empty contents valid. This problem could be addressed by creating global variables that could be initialized false but set true inside critical function. At the conclusion of parsing, these variables would be checked to ensure that all necessary functions were visited. 15

16 The JPEG Header Fault Tolerant System When putting together a system to verify and ensure error-free delivery of JPEG images, it is important to keep in mind some of the underlying project assumptions. First and most importantly, only one error will occur at any one time. In other words, faults will be singly occurring and statistically independent. Second, we assume an image may be either transmitted immediately or saved for later transmission. We can also assume that data segments are error-free immediately after being created because of Redinbo and Nguyen s ongoing work. Transient errors may subsequently occur, but data segments are correct at the time they exit the compression unit. This is an important distinction because as the image data is being collated, a cyclic redundancy check (CRC) may be calculated and saved to memory along with the image. Since the data is correct, the CRC will accurately reflect the contents of the JPEG image. However, there is no guarantee that the image is formatted correctly. The task of ensuring a valid JPEG image format is performed by the JPEG parser discussed in earlier sections. The proposed fault tolerant system can logically be divided into two parts. First is the front end of the system, which is responsible for moving images from the compression unit into memory and ensuring their validity. The back end of the system responsible for moving images from memory to the transmission unit while verifying the integrity of the image data. The front end of the JPEG fault tolerant system is shown in Figure 6. The system originates with JPEG image data and cyclic redundancy checks emerging from the compression unit. As stated earlier, the data within each data segment is assumed to be correct as it exits the compression unit because of the work of Redinbo and Nguyen. However, the image data may be incorrectly formatted. If it is, the parser will indicate that the format is invalid by placing a 0 on its output line. This will propagate to the output through the AND gate and the image will be declared invalid. 16

17 Compression Unit Image Data Image Data, CRCs CRCs CRC Cache Main Memory (Image Data, CRC) CRC Gen Image Data ( mem read) Parser (Software Format Verification) CRCs Comparator CRCs Valid Format? CRCs Match? & Valid Image? Figure 6. The front end of the proposed JPEG image fault tolerance system. Ensures that images are moved from the compression unit into memory with high assurance that the stored JPEG images are correct. The basic concept is that an image is written to memory, then read from memory. A CRC is generated from the resulting image. If this CRC matches the originally generated CRC, it can be assumed that the JPEG image stored in memory is correct, assuming it was correct when emerging from the compression unit. If any problems occur, such as mismatched CRCs or bad image data, the error will be propagated to the output and the appropriate action taken. Since it would be difficult to determine the exact location of the error, the image is simply recompressed should an error arise. The following table indicates the locations and descriptions of possible faults. Also included in the table is each fault s coverage or description of how the fault will be handled. Summary of Faults and Coverage for Front End of JPEG Fault Tolerant System Location of Fault Description/Effect of Fault Coverage for Fault Compression Unit Errors in image data segments Assuming will not occur due to Redinbo and Nguyen s work 17

18 Incorrect formatting of JPEG image data Parser will indicate invalid format. Recompress. Parser Main Memory Incorrect computation of CRC Error in parser incorrectly declares invalid format when format is actually valid Error in parser incorrectly declares image format valid when errors exist Image data not written correctly (bit errors, wrong address, write enable issues, etc.) Image data not read correctly CRC not written correctly CRC is regenerated after image data is placed in memory. Original CRC and regenerated CRC will not match. Recompress. Image will be declared invalid. Recompress. Assumption is that this will not occur because only a single error is allowed in system. Two errors would have to occur (incorrect formatting, parser error). Data read out of memory and CRC recomputed, compared with original CRC. If data is wrong, comparator will indicate. Recompress. Data read out of memory and CRC recomputed, compared with original CRC. If data is wrong, comparator will indicate. Recompress. See back end system description. (Under Figure 7.) CRC Cache CRC not written correctly Will not match regenerated CRC. Recompress. CRC not read correctly Will not match regenerated CRC. Recompress. CRC Generator Comparator Incorrect CRC generated (data is correct) Inadvertently correct CRC generated from bad image data CRCs match but comparator indicates mismatch CRCs mismatch but comparator indicates match CRCs match but are both incorrect Will not match cached CRC. Recompress. Will not happen. Only a single error is assumed. Would require two errors (bad image data from memory, incorrect generation). Image declared invalid. Recompress. Will not happen. Only a single error is assumed. Would require two errors (error to create bad CRC, incorrect comparison) Will not happen. Only a single error is assumed. 18

19 AND gate Both inputs 1, but output 0. Inputs not both 1, but output 1. Image declared invalid. Recompress. Will not happen. In order for an input to be 0, an error must have occurred. Cannot have a secondary error. The back end of the JPEG fault tolerant system is shown in Figure 7. The objective is now to transmit the image currently stored in memory. The communication protocol with which the images are transmitted will likely have error correction or detection capability, but this is not sufficient to guarantee that the image is successfully read from memory. Errors could occur between the image leaving memory and arriving under the jurisdiction of the communication protocol. For this reason, as the image is being transmitted, the communication system will be snooped and the serial image data returned to parallel form. A CRC is then calculated from this image data and compared to the CRC originally stored with the image. If an error occurs, such as mismatching CRCs or bad image data, the data could be reread from memory. If the error occurs again, recompression is necessary. This implies that the raw image data should not be released for reuse until the compressed image is successfully transmitted. Main Memory (Image Data, CRC) Image Data, CRCs Transmission Unit Serial Image Data CRCs CRC Cache CRCs Serial to Parallel CRC Gen CRCs Image Data Comparator Valid Image? Figure 7. The back end of the proposed JPEG image fault tolerance system. Ensures that images are moved from memory and transmitted with high assurance that the stored JPEG images remain intact. 19

20 The following table indicates the locations and descriptions of possible faults. Also included in the table is each fault s coverage or description of how the fault will be handled. Summary of Faults and Coverage for Back End of JPEG Fault Tolerant System Location of Fault Description/Effect of Fault Coverage for Fault Main Memory CRC Cache Transmission Unit Incorrect reading of image data Image data was written incorrectly Transient error causes single bit error in a memory word. More than one bit error in single word CRC written incorrectly CRC not read correctly CRC not written correctly CRC not read correctly Incorrect CRC transmitted Incorrect image data transmitted Regenerated CRC will not match stored CRC. Reread or recompress. See front-end description. (Under Figure 6.) ECC memory used and will correct single bit errors in any number of words. Assuming will not happen. Only a single error is assumed. Regenerated CRC will not match stored CRC. Recompress. Regenerated CRC will not match stored CRC. Reread or recompress. Will not match regenerated CRC Will not match regenerated CRC Assuming only single error, so image will be correct. Regenerated CRC will not match stored CRC. Reread or recompress. Serial to Parallel Error(s) in serial to parallel conversion Regenerated CRC will not match stored CRC. Reread or recompress. CRC Generator Incorrect CRC generated (data is correct) Will not match cached CRC. Reread or recompress. Inadvertently correct CRC generated from bad image data Will not happen. Only a single error is assumed. Would require two errors (bad image data from memory, incorrect generation). Comparator CRCs match but comparator indicates mismatch Image declared invalid. recompress. Reread or 20

21 CRCs mismatch but comparator indicates match CRCs match but are both incorrect Will not happen. Only a single error is assumed. Would require two errors (error to create bad CRC, incorrect comparison) Will not happen. Only a single error is assumed. Figures 6 and 7 both include separate output lines labeled Valid Image. The exact function of this output depends on the implementation of the system. However, one viable option would be to use the Valid Image line as an interrupt signal. In order to prevent meaningless interrupts causes by unsettled logic, the interrupt would likely be disabled the majority of the time. For the front-end system, at the same time the parser indicates a valid or invalid format, it could also signal the completion of parsing. This completion signal could be used as a mechanism for enabling the Valid Image interrupt. The interrupt service routine would simply recompress the previous image. For the back-end system, a similar scheme could be adopted. The CRC Generator could have a secondary output used to signal the completion of CRC calculation. Again, this signal could be used to enable the interrupt. 21

22 Appendix A: Backus-Naur Form of JPEG Image Data (Thornton and Anderson, 1990) <anyb> BY00 BYFF <non00b> BY01 BYFF <nonffb> BY00 BYFE <indexb> BY00 BY3F <kxindexb> BY01 BY3F <spechi0b> BY00 BY01 BY02 BY03 <spechi1b> BY10 BY11 BY12 BY13 <nsofb> BYC0 BYC1 BYC2 BYC3 BYC9 BYCA BYCB <dsofb> BYC5 BYC6 BYC7 BYCD BYCE BYCF <rstmb> BYD0 BYD7 <appnb> BYE0 BYEF <jpgnb> BYF0 BYFD <resb> BY02 BYBF Figure A-1. BNF for JPEG byte groupings. ( BY indicates a byte.) <fill> ε <fill> BYFF <prefix> <fill> BYFF <appn> <prefix> <appnb> <com> <prefix> BYFE <dac> <prefix> BYCC <dhp> <prefix> BYDE <dht> <prefix> BYC4 <dnl> <prefix> BYDC <dqt> <prefix> BYDB <dri> <prefix> BYDD <dsof> <prefix> <dsofb> <eoi> <prefix> BYD9 <exp> <prefix> BYDF <jpg> <prefix> BYC8 <jpgn> <prefix> <jpgnb> <nsof> <prefix> <nsofb> <rstm> <prefix> <rstmb> <soi> <prefix> BYD8 <sos> <prefix> BYDA <tem> <prefix> BY01 <forbid> <prefix> <resb> <tem> <jpg> <jpgn> Figure A-2. BNF for JPEG markers. 22

23 <intval> <anyb> <anyb> <len> <intval> <qbyt_spec> <spechi0b> <qint_spec> <spechi1b> <qt_tbl> <qbyt_spec> s64chr() <qbyt_spec> s64chr() <dqt_data> <qt_tbl> <dqt_data> <qt_tbl> <dc_spec> <spechi0b> <ac_spec> <spechi1b> <tabspec> <dc_spec> <ac_spec> <huff_tbl> <tabspec> huffspec() <dht_data> <huff_tbl> <dht_data> <huff_tbl> <ul> <anyb> <kx> <kxindexb> <dac_cond> <dc_spec> <ul> <ac_spec> <kx> <dac_data> <dac_cond> <dac_data> <dac_cond> <rstintv> <intval> <tblsmisc> ε <tblsmisc> <dqt> <len> <dqt_data> <tblsmisc> <dht> <len> <dht_data> <tblsmisc> <dac> <len> <dac_data> <tblsmisc> <dri> <len> <rstintv> <tblsmisc> <com> <len> comspec() <tblsmisc> <appn> <len> appspec() Figure A-3. BNF for JPEG markers. 23

24 <prec> <anyb> <nlines> <intval> <linlen> <intval> <nfc> <non00b> <fparms> <len> <prec> <nlines> <linlen> <nfc> fspec() <nsc> BY01 BY02 BY03 BY04 <ssstart> <indexb> <ssend> <indexb> <sapprox> <anyb> <prog_spec> <ssstart> <ssend> <sapprox> <sparms> <len> <nsc> sspec() <prog_spec> <ecdata> ε <ecdata> <nonffb> <ecdata> <rstm> <ecdata> BYFF BY00 <scan> <tblsmisc> <sos> <sparms> <ecdata> <ecdataf> <ecdata> <ecdata> <dnl> <len> <nlines> <scanf> <tblsmisc> <sos> <sparms> <ecdataf> <scanset> <scanf> <scanset> <scan> <nframe> <tblsmisc> <nsof> <fparms> <scanset> <exp_data> BY01 BY10 BY11 <dtblsmisc> <tblsmisc> <tblsmisc> <exp> BY00 BY03 <exp_data> <tblsmisc> <dframe> <tblsmisc> <dsof> <fparms> <scanset> Figure A-4. BNF for JPEG frames, scans, and entropy-coded data. <xframes> ε <xframes> <dframe> <xframes> <nframe> <fset> <nframe> <xframes> <jpeg_data> <soi> <nframe> <eoi> <soi> <tblsmisc> <eoi> <soi> <tblsmisc> <dhp> <fparms> <fset> <eoi> <soi> <forbid> <eoi> Figure A-5. BNF for JPEG compressed data streams. 24

25 5 pixels Appendix B: Sample JPEG Image The image shown enlarged to the right is 516 bytes. These bytes are shown below the image. The colors indicate the nature of the data the particular bytes describe (shown below). 4 pixels Start of Image / End of Image markers 0xFF D8 = Start of Image Marker 0xFF D9 = End of Image Marker Application Specific Data Marker + Data First 2 bytes comprise the marker. Next 2 bytes indicate the length of the field. Remainder of bytes are data. In this case, when displayed inside an ASCII text editor, the first 5 bytes of the second application data field (0x41 0x65) spell the word Adobe, evidence of the last application to have compressed this image. Define Quantization Table Marker + Data First 2 bytes comprise the marker. Next 2 bytes indicate the length of the field. Next byte indicates byte or integer specification. In this case, the fifth byte is 0x00, which corresponds to a byte specification. This is followed by 64 bytes describing the Huffman table. After those 64 bytes, we are not at the end of the field, but rather have a 0x01 byte. This indicates the presence of another Huffman table of byte specification. Another 64 bytes follow to describe the second Huffman table. Start of Frame Marker + Data First 2 bytes comprise the marker. Next 2 bytes indicate the length of the field. The next byte indicates ff d8 ff e a ff ee 00 0e f ff db a 12 1a c 1c 1c 1c c 0c 0c 0c 0c 0c 11 0c 0c 0c 0c 0c 0c 0c 0c 0c 0c 0c 0c 0c 0c 0c 0c 0c 0c 0c 0c 0c 0c 0c 0c 0c 0c 0c 0c a 1a 21 1d e 0e 0e e 0e 0e 0e c 0c 0c 0c 0c c 0c 0c 0c 0c 0c 11 0c 0c 0c 0c 0c 0c 0c 0c 0c 0c 0c 0c 0c 0c 0c 0c 0c 0c 0c 0c 0c 0c 0c 0c 0c 0c 0c 0c ff c ff dd ff c4 01 1b a 0b a 0b b1 42 a1 05 d1 c1 14 f e1 82 f a2 b2 15 d c e2 f2 a b3 84 c3 d3 75 e3 f a4 85 b4 95 c4 d4 e4 f4 a5 b5 c5 d5 e5 f a6 b6 c6 d6 e6 f f f0 a1 b1 04 c1 d1 e1 f a2 b c2 d a3 54 e2 f b3 84 c3 d3 75 e3 f a4 85 b4 95 c4 d4 e4 f4 a5 b5 c5 d5 e5 f ff da 00 0c f 00 fa 5f 3f a9 f2 57 f ff fe 8c e1 ff 00 3f fe 90 ff d9 the precision of the frame. This is followed by 2 bytes indicating the number of lines in the image (height) and 2 bytes indicating the line length. Together, these give the dimensions of the image. A single byte follows which indicates the number of frame components. For each frame component, a single byte identifier (counting from 1 to the number of components) is given. This is followed by a single byte that contains both the horizontal and vertical sampling factors. Last is a single byte giving the quantization table destination selector. Define Restart Interval First 2 bytes comprise the marker. Next 2 bytes indicate the length of the field. The last 2 bytes indicate the number of minimum coding units that will separate restart markers in the entropy-coded data Define Huffman Table Marker + Data First 2 bytes comprise the marker. Next 2 bytes indicate the length of the field. Next byte indicates the type of Huffman table (DC difference or AC coefficient). The next 16 bytes indicate the number of codes of that particular length (from 1 to 16). In other words, the first byte indicates the number of codes of length 1. The second byte indicates the number of codes of length 2, etc. Finally, the one-byte value associated with each Huffman code is given. This image contains several Huffman tables. Start of Scan Marker + Entropy-Coded Data First 2 bytes comprise the marker. Next 2 bytes indicate the length of the field. A single byte follows which indicates the number of frame components in the scan. For each component two selector bytes are given including a byte identifier. 3 spectral selection bytes follow. The remainder of the bytes are the entropy coded data. Note the stuffed 0x00 bytes that follow all 0xFF bytes. Also note that although the restart interval is specified, this image is too small to contain restart markers. 25

26 Appendix C: Output From JPEG Parser 5 pixels 1) The same 516 byte, 5x4 pixel image shown in Appendix B has been run through the JPEG parser. The output is shown below. This image is correctly formatted. 4 pixels [smithb@pc34 ~/jpegparser]$ jpegparser 5x4.jpg JPEG data is correctly formatted. Structure of Image: SOI marker NFRAME APP_DATA marker APP_DATA APP_DATA marker APP_DATA DQT marker DQT_DATA DQT_DATA NSOF FPARMS (lines: 4, line length: 5, precision: 8) SCANSET DRI marker RSTINTV (1 MCU's) DHT marker DHT_DATA SOS marker SPARMS ECDATA (18 bytes) EOI marker 2) Now, an error is injected into the image. The Start of Image marker is erroneously replaced with the End of Image marker. Since the parser needed the SOI marker, but did not receive it, an error is generated. [smithb@pc21 ~/jpegparser]$ jpegparser 5x4bad1.jpg JPEG data is NOT correctly formatted.!!!!! Needed SOI marker 26

27 3) If an invalid byte/word specifier is generated within the quantization table definition inside an image that is otherwise error-free, the following results: ~/jpegparser]$ jpegparser 5x4bad2.jpg JPEG data is NOT correctly formatted. SOI marker NFRAME APP_DATA marker APP_DATA APP_DATA marker APP_DATA DQT marker!!!!! Malformed DQT segment 4) In this case, the End of Image marker is missing from the image and the following error is generated: ~/jpegparser]$ jpegparser 5x4bad2.jpg JPEG data is NOT correctly formatted. SOI marker NFRAME APP_DATA marker APP_DATA APP_DATA marker APP_DATA DQT marker DQT_DATA DQT_DATA NSOF FPARMS (lines: 4, line length: 5, precision: 8) SCANSET DRI marker RSTINTV (1 MCU's) DHT marker DHT_DATA SOS marker SPARMS ECDATA (17 bytes)!!!!! Reached EOF 27

28 5) A different JPEG image is now parsed. Shown here in reduced size, the parsed image is 600x400 pixels and occupies 63,147 bytes. The image is correctly formatted: [smithb@pc21 ~/jpegparser]$ jpegparser gull.jpg JPEG data is correctly formatted. Structure of Image: SOI marker 600 pixels NFRAME APP_DATA marker APP_DATA APP_DATA marker 400 pixels APP_DATA APP_DATA marker APP_DATA APP_DATA marker APP_DATA DQT marker DQT_DATA DQT_DATA NSOF FPARMS (lines: 480, line length: 600, precision: 8) SCANSET DRI marker RSTINTV (75 MCU's) DHT marker DHT_DATA SOS marker SPARMS ECDATA (56174 bytes) RST marker RST marker... RST marker RST marker EOI marker 59 restart markers 28

29 6) Another JPEG image is parsed. Shown here in reduced size, the parsed image is 736x487 pixels and occupies 120,396 bytes. The image is correctly formatted: ~/jpegparser]$ jpegparser calcoast09.jpg JPEG data is correctly formatted. Structure of Image: 736 pixels SOI marker NFRAME APP_DATA marker APP_DATA 487 pixels APP_DATA marker APP_DATA APP_DATA marker APP_DATA DQT marker DQT_DATA DQT_DATA NSOF FPARMS (lines: 487, line length: 736, precision: 8) SCANSET DHT marker DHT_DATA DRI marker RSTINTV (46 MCU's) SOS marker SPARMS ECDATA ( bytes) RST marker RST marker... RST marker RST marker EOI marker 30 restart markers Along with the source code for the JPEG parser, all images shown in this appendix are available at: 29