HARDWARE IMPLEMENTATION OF POST-COMPRESSION RATE-DISTORTION OPTIMIZATION FOR EBCOT IN JPEG2000

Transcription

1 HARDWARE IMPLEMENTATION OF POST-COMPRESSION RATE-DISTORTION OPTIMIZATION FOR EBCOT IN JPEG2000 Thesis Submitted to the School of Engineering of the UNIVERSITY OF DAYTON In Partial Fulfillment of the Requirements for The Degree of Master of Science in Electrical Engineering By Andrew M. Kordik, B.S. UNIVERSITY OF DAYTON Dayton, Ohio August, 2011

2 HARDWARE IMPLEMENTATION OF POST-COMPRESSION RATE-DISTORTION OPTIMIZATION FOR EBCOT IN JPEG2000 Name: Kordik, Andrew Michael APPROVED BY: Eric Balster, Ph.D. Advisor Committee Chairman Electrical & Computer Engineering Assistant Professor Frank Scarpino, Ph.D. Committee Member Electrical & Computer Engineering Professor Keigo Hirakawa, Ph.D. Committee Member Electrical & Computer Engineering Assistant Professor John Weber, Ph.D. Associate Dean School of Engineering Tony Saliba, Ph.D. Dean, School of Engineering & Wilke Distinguished Professor ii

3 ABSTRACT HARDWARE IMPLEMENTATION OF POST-COMPRESSION RATE-DISTORTION OPTIMIZATION FOR EBCOT IN JPEG2000 Name: Kordik, Andrew Michael University of Dayton Advisor: Dr. Eric Balster As digital imaging sensors increase in size and capability, new ways to efficiently store/transmit the data they generate must be examined. JPEG2000 is the latest image compression standard from the Joint Photographic Experts Group which improves over earlier standards in its ability to compress images while maintaining image quality. However, with the compression gain advantage over other image compression standard comes an additional computational cost. The JPEG2000 compressor is, substantially computationally complex than its predecessor, JPEG [12]. There are 2 basic procedures for irreversible rate reduction of JPEG2000 compressed imagery: quantization, and post-compression rate-distortion optimization (PCRD- Opt). Quantization is the method of reducing the dynamic range of transformed image data prior to coding. Quantization is a computationally simple method for data reduction, but lacks in control of the compressed file size and is sub-optimal in terms of image quality. PCRD-Opt, however, gives the user precise control of the iii

4 output file size, and provides compressed imagery of the highest quality, per output bitrate [11]. This thesis is an embedded development of the PCRD-Opt algorithm, integrated into an FPGA-based JPEG2000 compression engine used for real-time compression of large-scale imagery. The embedded PCRD-Opt method provides imagery with a 2dB increase in quality over quantization on average, with a modest increase in complexity with an FPGA chip utilization increase of 11% in ALUTs and 15% increase in Memory ALUTs per Tier I encoder. iv

5 To those who have suffered me v

6 ACKNOWLEDGMENTS I would like to thank the following people for their support: Dr. Eric Balster: Without whom I would still be living in my car Kerry Hill, Al Scarpelli, and Frank Scarpino: For their continued support of the RC Lab which funded this thesis. Ben Fortner and David Walker: For their help with integration and previous work. Committee Members: For serving on my thesis committee. Bill Turri: For allowing me to continue this research with UDRI. vi

7 TABLE OF CONTENTS Page ABSTRACT iii DEDICATION v ACKNOWLEDGMENTS vi LIST OF TABLES ix LIST OF FIGURES x CHAPTER: 1. Introduction JPEG2000 Overview Level Offset Color Transform Image Tiling Discrete Wavelet Transform Quantization Data Partitioning Tier I Tier II Tier I Coding Passes MQ Coder Optimal Truncation vii

8 4. Optimal Truncation UD Encoder Hardware Overview Optimal Truncation Hardware Rate Design Distortion Design Results Usage Results Distortion Results and Comparison Remaining OT Processing OT vs Quantization Conclusions BIBLIOGRAPHY viii

9 LIST OF TABLES Table Page 3.1 Calculation of k sig Tier I Resource Usage Distortion Design Comparison ix

10 LIST OF FIGURES Figure Page 2.1 JPEG 2000 Encoder Overview Level Offset Block Diagram Demonstration of the Effect of the Color Transform Example of First Two DWT Levels Example of Data Partitioning on 1024x1024 tile Significance Neighborhood in σ table Visualization of code-stream with respect to possible truncation points Rate vs Distortion Curve Rate vs Distortion Curve UD-Encoder Segmentation of the JPEG2000 Encoding Process UD-Encoder Software Architecture UD-Encoder Hardware Top Level Block-Diagram Tier I Hardware Block Diagram Tier I Encoder with Rate Design Block Diagram of Rate Design Tier I Encoder with Distortion Block Diagram Tier I Encoder with Distortion Block Diagram Full LUT Distortion Design x

11 6.6 Smart LUT Distortion Design Final Tier I Encoder with Rate and Distortion Block Diagram GiDEL PROCeIV Development Board Pentagon Imagery Peppers Imagery Pentagon PSNR vs Bit-Rate Peppers PSNR vs Bit-Rate xi

12 CHAPTER 1 Introduction JPEG 2000 is the latest image compression standard developed by the Joint Photographic Experts Group (JPEG). The goal of this standard is to improve image quality beyond previous standards (JPEG 6.2) as well as advance the capabilities of the compressed filestream [7]. With respect to Peak-Signal-to-Noise-Ration (PSNR), JPEG2000 outperforms JPEG 6.2 by at least 2dB for all compression ratios when using non-reversible lossy methods [10]. Additionally, JPEG2000 adds capabilities to the compressed code stream such as progressive transmission by resolution and region of interest coding [3]. With increased capabilities and quality, the algorithms associated with the JPEG2000 standard are also computationally more complex when compared to previous standards [9]. Several methods to improve the speed of compression have been examined including: optimized software such as JasPer [1] and Kakadu [11], and hardware acceleration such as in the University of Dayton Encoder (UD-Encoder) [4]. Hardware acceleration can speed up JPEG2000 compression by relieving General Purpose Processors (GPPs) of the burden of bit-wise operations [8] as well as increasing potential parallelism beyond what is possible in Common Off The Shelf (COTS) processors [6]. Moreover, [6] finds that the hardware accelerated UD-Encoder improves 1

13 compression speed over the Intel Integrated Performance Primitives (IPP) JPEG2000 encoder implementation by up to 45%. The UD-Encoder is implemented using COTS FPGAs located on Peripheral Component Interconnect Express (PCIe) accelerator cards. FPGAs provide the advantage of being less expensive than Application-Specific Integrated Circuits (ASICs) in small quantities and can be easily reprogrammed to add additional functionalities. A key concern of lossy image compression is that of rate control. This is a result of when systems have limitations on digital storage space or transmission bandwidth. The JPEG2000 standard defines two primary methods for rate control: Quantization, and Post-Compression Rate-Distortion Optimization (PCRD-opt) also known as Optimal Truncation(OT)[11]. Quantization is performed on a subband level prior to encoding while OT is performed after encoding at the sub-bitplane level. The combination of the block coding structure with OT in JPEG2000 is often referred to as the Embedded Block Coder with Optimal Truncation (EBCOT). Because OT is performed after coding, the exact rate of the resulting file as well as the distortion to the image incurred by any such truncation is known. It is thereby possible to specify exact file sizes or maximum allowable distortions prior to encoding. Moreover, the use of distortion information at the sub-bitplane level allows for the optimization of rate versus distortion which much greater granularity than at the subband level. The majority of calculations performed by the OT process can be performed in parallel with the remainder of the EBCOT coder. It is, therefore, pragmatic to implement as much of the PCRD algorithm in hardware as possible. This is accomplished by utilizing the UD-Encoder hardware in conjunction with a software implementation provided by [5]. The JPEG2000 standard defines the OT process 2

14 in terms of floating point operations [7]. Floating point operations are costly to perform in hardware circuits in terms of logic utilization. It was found by [2] that PSNR equivalent calculations can be performed using integer only methods. This provides the basis for the OT hardware developed in this thesis which discusses the implementation of OT in the UD-Encoder hardware design in detail. After this introduction the remainder of this document is organized as follows: Chapter 2 provides an overview of the JPEG2000 process as presented by [7], Chapter 3 examines the Tier I, Chapter 4 provides a detailed discussion of the OT process as presented by [7] Chapter 5 provides an overview of the UD-Encoder, Chapter 6 develops the new research provided by this thesis in terms of the hardware designs developed to perform OT in the UD-Encoder which include a rate calculation design and several distortion calculation designs, Chapter 7 demonstrates the results of OT in the UD-Encoder in terms performance impact, PSNR comparison with Quantization and a comparison of the distortion calculation designs. Chapter 8 concludes this thesis and provides a discussion of future research. 3

15 CHAPTER 2 JPEG2000 Overview The JPEG2000 encoding algorithm can be described as a process with six primary steps as shown in Figure 2.1. These steps are: Level Offset, Color Transform, Wavelet Transform, Quantization, Tier I coding and Tier II. Given a raw input image, a JPEG2000 filestream is produced after performing these steps. Figure 2.1: JPEG 2000 Encoder Overview An overview of each of these these steps in sequence is provided in this chapter. 4

16 2.1 Level Offset The Level Offset process converts the unsigned input pixel data to signed data centered around zero. Thus, for a B-biti depth pixel sample data, there is an offset of 2 B 1 applied to the pixel sample. This operation is performed because it is a requirement of the Discrete Wavelet Transform (DWT) which is described further in Section 2.4. Equation 2.1 shows the calcuation for finding ˆx, the offset pixel value, and Figure 2.2 shows a block diagram of this process. ˆx = x 2 B 1 (2.1) Figure 2.2: Level Offset Block Diagram 2.2 Color Transform The JPEG Standard defines two different choices for performing an optional color transform: reversible and irreversible. The reversible transform is used by lossless encoders but degrades compression performace when compared with the irreversible transform in lossy compression. Although a color transform is not required it is recommended by the standard as it can improve compression ratio performance by removing potentially redundant data from a standard RGB image. The color transforms available convert images in the RGB color space to the YC r C b, or luminancechrominance color space. Figure 2.3 shows the effect of removing redundant data 5

17 from each color plane. Equations 2.2 and 2.3 show the reversible and irreversible transforms respctively, where x R [n], x G [n], x B [n], x Y [n], x Cb [n] and x Cr [n] are the red, green, blue, liminance and cromance values of the n th pixel respectively. x Y [n] x Cb [n] = x Cr [n] x Y [n] xr [n]+2x G [n]+x B [b] x Cb [n] x B [n] x G [n] x Cr [n] x R [n] x G [n] (2.2) x R [n] x G [n] (2.3) x B [n] Figure 2.3: Demonstration of the Effect of the Color Transform 2.3 Image Tiling Image tiling is a process that segments the original image content into rectangular pieces of set size. Tiling provides the first major potential for parallelism during the 6

18 image compression process. The remaining operations can be completed concurrently on each tile until the Tier II is reached and the tiles must be reorganized to form the final tile stream. As a general rule, the smaller the tile size, the more potential for concurrency and therefore improved speed performance. However as tile size gets smaller, reconstructed image quality suffers. 2.4 Discrete Wavelet Transform As with the color transform, JPEG2000 offers both lossy and lossless versions of the Discrete Wavelet Transform (DWT): the 9/7 and 5/3 transforms, respectively. The 9/7 form of the DWT is mathematically lossless. However it requires floating point operations which induce rounding errors in digital circuits. These errors incur quantization noise on the post transform coefficients. Like the 9/7 transform, the 5/3 transform is also mathematically lossless. The 5/3 transform, however, is based off of integer values and therefore does not cause quantization errors in the resulting coefficients. The purpose of the DWT, regardless of which form is used, is to reduce entropy in the tile data by separating data into different frequency subbands. This is accomplished by a set of row and column operations. The DWT can be preformed multiple times over the same image tile, Figure 2.4 shows an example of two levels being performed. First, a filter bank is applied to the rows, leaving the low-pass data on the left and removing the highpass data from the original image and placing on the right. Then, the same filter bank is applied column-wise leaving the low pass data on the top and removing the high pass data placing it on the bottom. This results in the first DWT 7

19 label 1DWT in Figure 2.4 which has 4 subbands: Low-Low in the top left, High- Low in the top right, Low-High in the bottom left and High High in the bottom right. This process leaves a thumbnail of the original image in the top left corner. The process can be repeated by applying the same method to this thumbnail, as shown by the 2DWT in Figure 2.4. Figure 2.4: Example of First Two DWT Levels 2.5 Quantization Quantization is one of two of the primary methods of lossy compression in the JPEG2000 standard. The quantization process is only performed when using lossy compression methods as it irreversibly removes data. The JPEG2000 standard allows each subband generated by the DWT to be quantized independently. By performing quantization the dynamic range of the coefficient values is reduced. This manifests in the coefficients as zero-valued bits in coefficients at the in the more significant bit planes. This increase in zeros improves coding speed of the Tier I coder. This increase in performance is due to the fact that by limiting the dynamic range of the data, the largest cofficient in a codeblock will have a smaller value and therefore the data that 8

20 needs to be coded will all be located in fewer bit planes, limiting the total number of bit-planes that need to be coded. Given a set of samples, y b [n], of position n and a quantization step, b, the quantized index, q b [n], is given by Equation 2.4 where q b [n] are the quantized wavelet coefficients. yb [n] q b [n] = sign(y b [n]) b (2.4) 2.6 Data Partitioning Once the coefficient data for the tile has been quantized, the data can be further partitioned into precincts and codeblocks. This partitioning allows for increased parallelism as well as localizing the data for coding. Both precincts and code blocks are rectangular regions in the wavelet domain. Code-blocks can be 2 m by 2 m samples in size where 32x32 or 62x64 codeblocks are typically used. Smaller codeblocks allows for greater parallelism and improved compression speed but increases the likelihood of artifacts and noise in the reconstructed image. Figure 2.5 demonstrates the results of this data partitioning. The red, green, purple and orange regions denote the various wavelet levels. The white lines show the precinct boundaries and the black lines show the codeblock size. 2.7 Tier I The Tier I performs the primary lossless coding portion of the JPEG2000 standard. It contains three coding pass modules which code the data based on their context as 9

21 Figure 2.5: Example of Data Partitioning on 1024x1024 tile it relates to the rest of the code block, an arithmetic coder and the logic required to perform the OT calculations. The Tier I is futher discussed in Chaper Tier II Tier II takes the compressed and truncated bit stream generated by the Tier I in conjuction with the information required for Optimal Truncation and organizes it into a compliant JPGE2000 filestream. Additional header information may be included that describes the code stream, such as which color transform was performed, how many DWTs were performed, and how much each sub-band is quantized. Once Tier II organizes the file, the JPEG2000 encoding process is complete. 10

22 CHAPTER 3 Tier I The Tier I coder operates on a single code block at a time. Each bit plane, starting with the most significant bit, is pre-processed by three coding passes: Significance Propagation Pass, Magnitude Refinement pass and the Clean-up pass. After the bitplanes are pre-processed by the coding passes, the results are processed by the MQ coder for the final coding phase in JPEG Coding Passes Each coding pass is performed on every bit plane except most significant bit plane where only the Clean-up pass is performed. As each pass works through a bit plane, the bits are selected to be operated on by the appropriate pass. Each bit is only operated on by a single code pass, that is once a coding pass operates on a bit no other coding pass operates on it. Selection of bits is performed by the following method: If the sample has not yet been declared significant, that is the most significant 1 in a sample has not been found yet, and if the current bits neighbor hood is significant, the Significance Propagation pass operates on that bit and the sample is declared significant. A bit s neighborhood is considered significant if anyone of the 11

23 8 neighboors is a significant bit. If the sample has already been declared significant but, it was not due to the previous coding pass, then the Magnitude Refinement pass operates on the bit. In all other cases the Clean-up pass operates on the bit. This information is maintained in two state tables, the σ table which keeps track of whether a sample has been declared significant and the π table which shows that the Significance Propagation Pass has operated on the bit. The σ table is used to determine the significance of the neighborhood, k sig. The σ table is reset after every code block and the π table is reset after each bitplane is completed. The significance of the neighborhood of a bit, j, is based on a combination of the significance of the vertical neighbors, k v, the significance of the horizontal neighbors, k h, and the significance of the diagonal neighbors, k d. A diagram of these relationships is shown in Figure 3.1 which is a view of the σ table for a given bit plane. Figure 3.1: Significance Neighborhood in σ table 12

24 Therefore, for a given bit, j, the horizontal, vertical and diagonal significance can be found using Equations 3.1, 3.2 and 3.3 respectively, where j 1 is the horizontal dimension of j and j 2 is the vertical dimension of j. k h [j] = σ[j 1, j 2 1] + σ[j 1, j 2 + 1] (3.1) k h [j] = σ[j 1 1, j 2 ] + σ[j 1 + 1, j 2 ] (3.2) k h [j] = σ[j 1 + 1, j 2 1] + σ[j 1 1, j 2 + 1] + σ[j 1 1, j 2 1] + σ[j 1 + 1, j 2 + 1] (3.3) k sig is dependant on which subband the code-block came from because if the code block belonges to a LL (all lowpass) or LH (column-wise highpass) codeblock, significance is most likely found in horizontal neighbors, the reverse is true for the HL (row-wise highpass), and for the HH( all highpass) significance is most likely found in the diagonal neighbors. Once k h [j], k v [j] and k d [j] are known, k sig [j] can be calculated using Table 3.1, where x means don t care. 13

25 k sig LL and LH blocks HL blocks HH blocks k h [j] k v [j] k d [j] k h [j] k v [j] k d [j] k d [j] k h [j] + k v [j] 8 2 x x x 2 x 3 x x 1 1 x x 2 0 x x 1 0 x Table 3.1: Calculation of k sig Using the state tables and the calculation for k sig the following rules determine which bit plane operates on a given bit j: If σ[j] == 0 and k sig > 0 then SPP is performed σ[j] = 1 π[j] = 1 Else if σ[j] == 1 and π[j] == 0 (Already declared significant but the SPP was not run on the current bit) then MRP is performed σ[j] = σ[j] Else if π[j] == 0 (No other pass has been performed on the bit) then CUP is performed 14

26 If v[j] == 1 then σ[j] = 1 π[j] = MQ Coder The MQ coder in the JPEG2000 compression standard is a state driven binary arithmetic coder, also known as an entropy coder. The MQ coder utilizes a lossless encoding mechanism that gives shorter codes to more frequently occurring patterns and longer codes to less frequent patterns. The MQ coder accepts one bit at a time from the coding passes and produces bytes as necessary until all coding passes have been completed and the bits remaining in the internal states of the MQ coder are flushed to the output. This flushing of bytes is always 4 bytes long. 3.3 Optimal Truncation Optimal Truncation calculations are performed at the same time as the MQ coder and code passes are operating. These calculations measure the rate of bytes coming out of the MQ Coder for each coding pass, and the distortion incurred by removing that code pass from the final code stream. Once this data is collected code-blocks can be truncated to meet rate requirements and minimize distortion. Once the codeblocks have been truncated, the remaining file organization performed by Tier II can be completed. Further discussion of Optimal Truncation is provided in Chapter 4. 15

27 CHAPTER 4 Optimal Truncation Optimal Truncation (OT) is one of the main methods that JPEG2000 can use to remove data from the compressed code stream in a lossy way, the other being Quantization. OT has several key advantages over quantization that make it a better choice for the desired compression gains. OT is performed post compression so the final rate of the output file is known at the time of processing. This allows for direct specification of rate of distortion requirements prior to encoding the data. OT differs from Quantization which is performed pre-encoding and has no guarantee on final rate or distortion after encoding is performed. Moreover, OT is performed at a subbitplane level of each code block, the end of each code-pass is considered a possible truncation point in the process. This provides several orders of magnitude of greater granularity than Quantization which is performed at the sub-band level of the tile. Finally, OT is designed to optimize rate vs distortion to ensure that any truncation results in the best possible quality and lowest distortion, for the desired rate. The rate calculation is the most straightforward in the OT process. Rate is a direct summation of the number of bytes produced by the MQ coder for a given code pass. Each rate is also summed with the rates of the code-passes performed on the current bit plane and previously encoded bit planes. That is, the rate for the i th code 16

28 block at the n th truncation point, R n i, is defined by Equation 4.1,, where r k i is the total number of bytes produced for the k th code-pass in the i th codeblock. n 1 Ri n = ri k (4.1) Distortion is calculated as a coding pass operates on its respective bits. This is possible because the coding passes and the MQ coders do not change the value of the coefficients but rather change their representation. Distortion is calculated by taking the normalized difference, between the sample at the current bit plane, p, and the largest possible quantization of the next, least significant bitplane, p 1. This quantization is based off of a mid-point reconstruction in the decoding process. The normalized difference, v p i [n], of the sample for the nth bit in the code block at the p th bit plane is defined by Equation 4.2. k=0 2 v p p i [n] = v i [n] 2 p v i [n] 2 2 (4.2) Equation 4.2 generates the error, used to calculate distortion in Equations 4.3 and 4.4, eqincurred by truncating at the p th bit plane for the n th sample. This value must be adjusted if the bit at the p th bit plane for the [n] th sample for the current code block makes the sample significant. That is, the current bit is the most significant 1 in the sample. Additionally, the distortion, δ p i [n] for the ith codeblock at the n th sample and p th bit plane needs to be un-normalized and scaled by the irreversiable DWT weight weight, ω i, and quantization, 2, applied to the DWT subband of the i th code-block. Therefore, if the bit being considered made the sample significant then the distortion is calculated by Equation

29 δ p i [n] = w2p ω i 2 [(v p i [n] 1)2 (v p i [m, n] 1, 5)2 ] (4.3) Otherwise, the distortion contributed by truncating at that bit is calculated by Equation 4.4, where v is the value of the bit being consdered which can take a value of either 1 or 0. δ p i [n] = w2p ω i 2 [(v p i [n] 1)2 (v p i [n] 0, 5)2 v] (4.4) As with the rate, the total distortion, Di n, incurred by truncating at a given code pass is given by the sum of the distortions incurred by the previous coding passes at the next less significant bit plane, which in turn is the sum of the distortions incurred at each bit processed by that code-pass. That is, the distortion, Di n, for the i th codeblock at the n th truncation point is given by Equation 4.5, where d k i is the sum of the distortions of each bit in the k th code-pass of the i th bit plane given by Equation 4.6. n 1 Di n = d k i (4.5) k=0 d k i = δ p i (4.6) p,n i Thus, we now have the set of possible truncation points for a given codeblock. Figure 4.1 shows a block diagram of the breakdown from the code-stream level to that of the truncation points. Although end of each code-pass is considered a possible truncation point, some truncation points are less optimal than others so optimization must be performed. 18

30 Figure 4.1: Visualization of code-stream with respect to possible truncation points Optimization be solved using the common Lagrangian Multipliers optmization method as shown in Equation 4.7 R k i λd k i (4.7) In practice, if it can be guaranteed that each successive truncation point in a code-block is less optimal than its predecessor, then we can simplify the Lagrangian optimization from a O(n 2 ) to a O(n) complexity problem. This can be accomplished by calculating the slope with respect to distortion vs rate of the current truncation point with respect to the next truncation point. This is shown in Equation 4.8, where the slope, Si n, for the i th codeblock at the n th truncation point is equal to the change in respective distortion vs the change in respective rate. Si n = Dn i D n+1 i Ri n Rn+1 i = Dn i R n i (4.8) The information provided by the rate, distortion, and slope can be visualized as a plot of a rate vs distortion curve, shown in Figure 4.2 Utilizing the rate-distortion curve, a convex hull analysis is performed which removes truncation points which are less optimal than their predecessors. Analytically 19

31 Figure 4.2: Rate vs Distortion Curve this is calculated by ensuring that the slope for the n th truncation point is greater than the slope for the (n + 1) th truncation point. If this is not satisfied then the truncation point is deemed infeasible and removed. The resulting rate-distortion curve is shown in Figure 4.3 Once the feasible truncation points have been selected, a slope is selected and all code blocks are truncated at the truncation point which most closely matches this slope. Then the total rate for the final file is calculated, if the desired rate is not reached, a new slope is selected and the process is repeated. When the rate requirement is satisfied, the code blocks are truncated at their respective truncation points and are passed to the remainder of Tier II for organization into the final file stream. 20

32 Figure 4.3: Rate vs Distortion Curve 21

33 CHAPTER 5 UD Encoder Hardware Overview Hardware acceleration is chosen for the UD-Encoder because it not only relieves the CPU of the processing burden required to perform JPEG2000 compression but also for it suitability for extreme parallelism beyond was is possible from COTS CPUs. FPGA technology is chosen over ASIC technology because of its rapid development and deployment time and the significantly lower cost of production in smaller quantities. Moreover, FPGA technology allows for complete repogramability of the integrated circuit (IC) which not only improves the ability to catch and fix bugs, but also allows for additional features to be added after delivery such as those added by this thesis. The JPEG2000 standard has many potential places for including parallel processing during the encoding process. Examples of possible parallelism include: Several Instances of the Entire Compressor (Image Level) Several Instances of the DWT and lower processes (Tile Level) Several Instances of Tier I encoders (Codeblock Level) The UD-Encoder takes advantage of all of the above possibilities. This is accomplished by breaking the process up between software and hardware processing as shown in 22

34 Figure 5.1, with Input/Output denoted by orange blocks, software processes denoted by blue blocks, hardware processes denoted by purple blocks and processes that are performed by both software and hardware denoted in red. Figure 5.1: UD-Encoder Segmentation of the JPEG2000 Encoding Process By allowing software to start several instances of the entire compressor along with associated Tier II s which run in software, image level parallelism is accomplished. This allows for a pipelining that improves performance by eliminating blocking while waiting on the hardware to complete. The software performs the any necessary level offset as well as color transforms before tiling the image and sending it to hardware. The software then waits for the hardware to finish all tiles for an image before performing Tier II. The software archetecture is shown in Figure 5.2 Tiles are sent to instances of DWTs located on the FPGA fabric implementing tile level parallelism. After thee DWT process is completed the image is partitioned into codeblocks and routed to multiple instances of Tier I encoders. This process is shown 23

35 Figure 5.2: UD-Encoder Software Architecture in Figure 5.3. Moreover, the UD-Encoder can utilize multiple FPGAs, replicating the entire structure shown in Figure 5.3. When the wavelet coefficients from the DWT are partitioned into code-blocks they are stored in each Tier I s own codeblock RAM. At this point, the Striper, which transforms the codeblock samples into coding pass bit-streams, begins to take the necessary data out of the code-block one bit plane at a time. This data is passed to the appropriate coding pass determined by Tier I State Tables. As the code passes operate they update the State Tables and pass their resulting data to a FIFO for storage until the MQ coder is ready to process them. This is shown in the Tier I block diagram in Figure 5.4 When the MQ has finished processing all of the data for a given code block the results are stored in an Output Buffer, shown in Figure 5.3 and the Tier I process is 24

36 Figure 5.3: UD-Encoder Hardware Top Level Block-Diagram repeated by requesting another code-block from the post DWT partioner. When all of the code blocks have been coded the Output Buffer is sent to Tier II in software to finish processing, at which point software sends the tiles for the next image back to hardware. 25

37 Figure 5.4: Tier I Hardware Block Diagram 26

38 CHAPTER 6 Optimal Truncation Hardware The design of Optimal Truncation hardware for the UD-Encoder presents several challenges, foremost the the desire to perform as much of the OT algorithm not only in hardware but also completely in parallel with Tier I encoder such that the calculations have little to no impact on the performance of the encoder and as FPGA fabric is limited. The design must be as small and efficient as possible. Moreover, because hardware design relies on concurrent-reactive algorithms, certain aspects of OT process must be compensated for as they are designed with iterative computation for CPUs in mind and are not available in hardware environment. 6.1 Rate Design The calculation for rate takes place solely after the MQ encoder performs. It requires information to determine when a byte is produced and when the code pass has changed. A top level block diagram of the Tier I with the addition of the rate hardware is shown in Figure 6.1 Unlike in software, where iteration is prevalent, it is impossible to predict with certainty when the MQ Coder will output a byte, with out additional controls. For this reason, the rate calculator takes advantage of the MQ coders Byte Out signal. 27

39 Figure 6.1: Tier I Encoder with Rate Design Every time the Byte Out signal goes high and a rising clock edge occurs the Rate Count is incremented. The next challenge to overcome is lack of knowledge of when a code-pass has been completely consumed by the MQ coder. This is solved by keeping a count of the bits produced by each code pass as they enter the MQ Fifo in the Code-pass Bit Count. The Code-pass Bit Count is reset every time a code-pass finishes and the next one starts. Then this value is temporarily stored in a register until the Count Down counter reaches the count total for the code pass, at which point a Last Bit Consumed signal stores the current rate count in the rate fifo and the register is reset. Figure 6.2 shows a block diagram of the rate design hardware, the blue blocks are pre-existing in the design and the red blocks are added to perform the OT rate calculation. 28

40 Figure 6.2: Block Diagram of Rate Design 6.2 Distortion Design The calculation for distortion is defined as a floating point operation by the standard, as shown by Equations 4.2, 4.3 and 4.4. It is found by [2] that these equations can be converted to integer operations with equivalent results to the operations defined by the standard. Moreover, these equations were simplified by [2] to require only the calculation of a Big Error, e p b, at the pth bit plane found by Equation 6.2, a Small Error, e p s, at the pth bit plane found by 6.1 to find the distortion, δ p i [n], for the n th sample in the p th bit plane, shown in Equation 6.3 for significant bits and Equation 6.4 for all other bits. e p s[n] = x i [n] &2 p 1 (6.1) e p b [n] = x i[n] &2 p (6.2) 29

41 δ p i [n] = x i[n] 2 (e p s[n]) 2 (6.3) δ p i [n] = (ep b [n])2 (e p s[n]) 2 (6.4) The logic required to complete the distortion calculations can be integrated with the pre-existing Tier I logic in the UD-Encoder according to the design shown in Figure 6.3. Figure 6.3: Tier I Encoder with Distortion Block Diagram The distortion design requires that significance is calculated because it can not be determined from the state tables as defined by the standard. The distortion design in Figure 6.4 shows this Is Significant calculation block. Additionally, e p b and ep s are only calculated once and routed to the calculations for each version of distortion, one for if the current bit is significant, and one for when the bit is not significant. The 30

42 appropriate distortion calculation is selected and accumulated until the code pass changes and the accumulator is reset. Figure 6.4: Tier I Encoder with Distortion Block Diagram The definition of the π table can be changed to made significant by the current code pass with out having any impact on the functionalty. This is because the magnitude refinement pass will never make a bit significant, as it only operates once significance has been found, and because the cleanup pass operates last right before the π table is reset. With this deviation from the standard, significance no longer needs to be calculated and can be examined directly from the state tables, saving logic. The distortion design shown in Figure 6.4 utilizes all combinatorial logic. This approach, henceforth reffered to as the Integer design, has two alternatives: a Full LUT design and a Smart LUT design. The Full LUT design is a custom design used for comparison and stores every possible distortion in a LUT and thereby reduces logic utilization and increases the maximum operating frequency at the cost of additional memory utilization. This design is shown in Figure

43 Figure 6.5: Full LUT Distortion Design Figure 6.6: Smart LUT Distortion Design The Smart LUT design is a balance between Intger and Full LUT designs. The Smart LUT stores only the possible big and small errors requiring a total of four times the sample-width in registers. This increases the maximum operating frequency over the Integer design and requires less memory than the Full LUT. This design is shown in Figure 6.6 The resuling Tier I logic including both rate and distortion calculations is shown in Figure

44 Figure 6.7: Final Tier I Encoder with Rate and Distortion Block Diagram The remaining Optmial Truncation processing including convex hull and truncation itself is performed in software as it is a highly iterative process which does not perform well in hardware logic. 33

45 CHAPTER 7 Results The UD-Encoder can be compiled to run on several FPGAs, however the discussion of results is focused on a single Altera Stratix IV 530 chip, installed on a GiDEL PROCeIV development board shown in Figure 7.1. Figure 7.1: GiDEL PROCeIV Development Board All of the added hardware for calculating the rate and distortion required to complete the OT process is added to the design according to the block diagrams in Chapter 6 and is capable of running completely in parallel with the pre-existing hardware. The Altera Statix IV 530 device has 424,960 combinatorial ALUTs, 212,480 memory ALUTs and 21,233,644 bits of block RAM. 34

46 7.1 Usage Results The primary method for parallelism in the UD-Encoder is at the codeblock level. It is, therefore, important to minimize the size of the Tier I as much as possible. Table 7.1 shows the usage of the entire Tier I as well as the the entire Optmial Truncation hardware design. The table also shows the usage of the Distortion Calculation and the Rate Calculation individually. It is found that the OT calculations account for just 15% of the entire Tier I ALUT usage and just 11% of the Block RAM usage. Part ALUTs Memory ALUTs Block RAM Tier I Top Optimal Truncation Top Distortion Calculation Rate Calculation Table 7.1: Tier I Resource Usage 7.2 Distortion Results and Comparison The distortion designs are compared in a stand alone project to examine the characteristics of each design without being affected by the other logic. Table 7.2 shows the results of analyzing the Logic Utilization and Maximum Frequency the device is able to run the logic successfully (as reported by the Altera Tools). It can be seen that the LUT based designs reduce the amount of logic required to complete the distortion calculation when compared with the Integer logic based design. However, these designs required a significant amount of memory to store the LUTs, especially for the Full LUT design which could not be fit in its entirety in the device fabric. 35

47 Moreover, as expected the Full LUT is the fastest design and there is an increase in speed for the Smart LUT when compared to the Integer design. Name ALUTs Memory ALUTs Total ALUTs Block RAM Max Frequency Integer MHz Full LUT ,457, MHz Smart LUT MHz Table 7.2: Distortion Design Comparison The UD-Encoder Tier I logic is limited to run at MHz, and therefore any potential beyond this would be unutilized by the distortion designs. Therefore, there is no performance gain by the LUT designs and the Integer design is sufficient. Moreover, FPGAs are generally memory limited when compared to logic. The Full LUT design will not fit in the Stratix IV 430 device as it requires 31,457,280 bits of block RAM and only 21,233,644 bits are available. The Smart LUTs fit on the device as they do not use block RAM because the LUTs are small enough to fit in the memory ALUTs. However, because FPGAs are limited in the amout of logic available in terms of ALUTs usage should be minimized. Therefore the Integer design, which usues less logic than the Smart LUT, is used in the final Design. All of the logic can be included with the rest of the UD-Encoder design and still fit 150 instances of the Tier I Encoder with 3 DWTs. Note that if the other components in the UD-Encoder are improved to operate at higher clock frequencies, it would be pragmatic to use the Smart LUT at the cost of 6 additional ALUTs. 36

48 7.3 Remaining OT Processing The remaining OT processing is included in software before Tier II and only incurres a 100ms throughput slowdown when compared to Quantization, which is performed in hardware. This slowdown accounts for 15% of the entire Tier II, which completes in about ms on 16k by 16k imagery, depeding on the image data. Moreover, because the software is threaded at the image level, this slowdown only affects the latency from the first input to the first compressed image output. 7.4 OT vs Quantization The comparison between the PSNR results of OT vs Quantization corroborates the results of [2]. Additionally, the Kakadu encoder is used to provide a comparison to an industry standard JPEG2000 encoder. Two representative images where used for comparison: Pentagon, shown in Figure 7.2 and Peppers shown in Figure 7.3. Pentagon provides an example of the behavior of the encoder on images that have alot of high frequency data which is traditionally difficult to compress. Conversely, Peppers offers an example of the behavior of the encoder on images that have a more common balance of high and low frequency data. The resulting PSNR measurements for several compression ratios are shown for each image in Figures 7.4 and 7.5. The figures show that as compression ratio increases, PSNR improvement of OT vs Quantizaion improves. At higher compression ratios, OT is shown to improve PSNR by 2dB, on average. 37

49 Figure 7.2: Pentagon Imagery 38

50 Figure 7.3: Peppers Imagery Figure 7.4: Pentagon PSNR vs Bit-Rate 39

51 Figure 7.5: Peppers PSNR vs Bit-Rate 40

52 CHAPTER 8 Conclusions JPEG2000 is the latest image compression standard developed by the Joint Photographic Experts Group designed to impove image quality and file steam capabilities over previous standards. These improvements cause an increase in compression algorithm complexity over the previous standards. The University of Dayton developed the UD- Encoder [6] which improves compression speed by taking advantage of the multitude of oportunities for parallelism in the compression standard such as: image level, tile level, and codeblock level. The UD-Encoder accomplishes this by leveraging hardware acceleration for the core encoding portions of the compression algorithm. The UD-Encoder previously used one of two lossy techniques for image compression, Quantization. Optimal Truncation provides an improved post compression quality by optimizing rate vs distortion of truncated data at the code block level. The Optimal Truncation algorithm is defined by the standard as a floating point process. This process is simplfied using integers with equivalent quality to the floating point methods. The integer based algorithm is ported to a COTS FPGA and incldued with the pre-existing UD-Encoder hardware. It is found that the additional calculations for rate and distortion, when included in hardware, operate completely in parallel with 41

53 the Tier I coder processing, without incurring any performance decrease. Moreover, several potential designs for the distortion calculation are presented: Full LUT, Smart LUT and Integer. It was found that the Full LUT design can not fit in the COTS FPGAs used by the UD-Encoder. The Smart LUT is found to require only 6 more ALUTs than the Integer design, and had a higher maximum operating frequency by 12 MHz. This additional speed can not be utilized by the UD-Encoder as it is well above the maximum operating frequency of the rest of design. It for this reason that the Integer design is selected for the UD-Encoder to save the additional logic. However, it is noted that this savings is minimal and if the remainder of the UD-Encoder is speed up, the Smart LUT may be used. The complete OT design is verifed to match expected quality improvements over Quantization providing atleast 2dB improvement in PSNR at high compression ratios. Moreover, the design is compared with industry standard JPEG2000 compressors and found to be within 0.5dB. 42

54 BIBLIOGRAPHY [1] Michael D. Adams and Faouzi Kossentini. Jasper: A software-based jpeg-2000 codec implementation. In Proceeding 2000 International Conference on Image Processing, volume 2, pages 53 56, September [2] Eric J. Balster. Integer-based, post-compression rate-distortion optimization computation in jpeg2000 compression. Optical Engineering, 49(7):6, July [3] Charilaos Christopoulos, Athanassios Smkdras, and Touradj Ebrahimi. The jpeg2000 still image coding system: An overview. IEEE Transactions on Consumer Electronics, 46(4): , November [4] Walker David, Luke Hogrebe, Ben Fortener, and Dave Lucking. Planning for a real-time jpeg2000 compression system. In Aerospace and Electronics Conference, NAECON 2008, pages , [5] Ben Fortener. Implementation of post-compression rate distortion optimizaion within ebcot in jpeg2000. Master s thesis, University of Dayton, Dayton, Ohio, December [6] Luke Hogrebe. A parallel architecture of jpeg2000 tier i encoding for fpga implementation. Master s thesis, University of Dayton, Dayton, Ohio, June

55 [7] Joint Photographic Experts Group. Information Technology - JPEG 2000 image coding system: Core coding system, second edition edition, September [8] Chung-Jr Lian, Kuan-Fu Chen, Hong-hui Chen, and Liang-Gee Chen. Analysis and architecture design of block-coding engine for ebcot in jpeg2000. IEEE Transactions on Circuits and Sysmtes for Video Technology, 13(3): , March [9] Diego Santa-Cruz, Raphael Grosbois, and Touradj Ebrahimi. Jpeg2000 performance evaluation and assessment. Signal Processing: Image Communication, 17(1): , [10] Athanassios Skodras, Charilaos Christopoulos, and Touradj Ebrahimi. Jpeg2000 still image compression standard. IEEE Signal Processing Magazine, pages 36 58, Septempber [11] David S. Taubman and Michael W. Marcellin. JPEG2000: image compression fundamentals, standards and practive. Kluwer Academic Publishers, [12] David S. Taubman, Erik Ordentlich, Marcelo Weinberger, and Gadiel Seroussi. Embded block codingh in jpeg2000. Signal Processing: Image Communication, 17:49 72,