1 2 Advanced Topics Multimedia Video (5LSH0), Module 01 B Introduction to JPEG2000: the next generation still image coding system Module 1B: JPEG2000 Part 1 Standardization issues, Requirements, Comparisons Peter H.N. de With ( p.h.n.de.with@tue.nl ) JPEG: Summary (1) 3 JPEG: Summary (2) 4 JPEG (Joint Photographic Experts Group) Digital Compression and Coding of Continuous-tone Still Images Joint ISO and ITU-T Published in 4 Parts: ISO/IEC 10918-1 ITU-T T.81 : Requirements and guidelines ISO/IEC 10918-2 ITU-T T.83 : Compliance testing ISO/IEC 10918-3 ITU-T T.84: Extensions ISO/IEC 10918-4 ITU-T T.86: Registration of JPEG Parameters, Profiles, Tags, Color Spaces, APPn Markers Compression Types, and Registration Authorities (REGAUT) JPEG derived industry standards JFIF (JPEG File Interchange Format, <xxxxxx.jpg>) JTIP (JPEG Tiled, Pyramid Format) TIFF (Tagged Image File Format) SPIFF (Still Picture Interchange File Format, JPEG Part 3) FlashPix Developed by Hewlett-Packard, Kodak, Microsoft, Live Picture (1996) Transferred to Digital Imaging Group (DIG), an industry consortium Motivation new still image st dard (1) Current standards fail to produce the best quality/performance, hence Should complement JPEG, not replace! Low bit-rate compression: For example below 0.25 bpp Lossless and lossy compression: No current standard exists providing superior lossy and lossless compression in a single codestream. Computer generated imagery: JPEG was optimized for natural imagery; does not perform well on computer generated imagery. Large images: JPEG does not go beyond 64Kx64K without tiling 5 Motivation new still image st dard (2) Random codestream access and processing Compound documents: Currently, JPEG is seldom used in the compression of compound documents because of its poor performance when applied to bi-level (text) imagery Transmission in noisy environments: The current JPEG standard has provision for restart intervals, but image quality suffers dramatically when bit errors are encountered. Open Architecture: Desirable to allow open architecture to optimise the system for different image types and applications. Also, JPEG2000 offers a single architecture, whereas JPEG has 44 modes of which most are not used. Progressive transmission by pixel accuracy and resolution 6 1
JPEG2000 Markets and Applications Internet Mobile Printing Scanning Digital Photography Remote Sensing Facsimile Medical Digital Libraries E-Commerce First real application: Digital Cinema! 7 JPEG2000 Requirements New still image coding standard for Bi-level images, like faxes and masks Gray-level images, like photography and newspapers Color images, like consumer & professional photography Multi-component images, like in professional printing Hyper-component images, like in art and prof. applications Characteristics/Applications natural, medical, scientific, remote sensing, professional Imaging models Client-server models, Real-time transmission Image library retrieval, Imaging limited resources (embedded, mobile) 8 JPEG2000 Features in Part I 9 J2K Application requirements (1) 10 High compression efficiency (superior to JPEG) Lossless colour transformations (continuous tone supp.) Lossy and lossless coding in one algorithm Embedded lossy to lossless coding (continuous vs. bi-level) Progressive by resolution, quality, position, Static and dynamic Region-of-Interest coding/decoding Error resilience (enable random access, robustness) Perceptual quality coding (facilitate regions of interest) Multiple component image coding Tiling (supports real-time coding, seq. build-up, limited space) Light file format (optional) (pursue open architecture, formats) Image type JPEG2000 Image width and height: 1 (2**32 1) Component depth: 1 32 bits Number of components: 1 255 (or more) Dissimilar comp. depths: each component has a different depth Dissimilar comp. spans: each component has its own coverage J2K Application requirements (2) Application profiles JPEG2000 Internet: image 32x32 up to 4Kx4K, with 3 (Y Cr Cb, RGB, ) comp s, or 4 comp s with 1-8 bit alpha channel Printing: compound images with typically 4800x6600 pixels (600 ppi, 8 x11 ) with 1, 3 or 4 components Scanning: compound images, typically 10Kx10K up to 20Kx20K or more, with 1, 3, or 4 components and 16-bit/comp. Digital Photography: natural, sizes up to 4K x 4K, with 1, or 3 components, and between 8-16 bits/comp. Remote sensing: infra-red, electro-optical, multi-spectral, SAR images, horiz. res. up to 24K pixels, 1-500 components, 8-20 b/comp. Mobile: compound, 32x32 till 4Kx4K pixels, with 1 or 3 components and 1-8 bits/component 11 J2K Application requirements (3) Uncompressed: image is stored in bit stream without compression Lossless compression: reconstructed image is identical, bit for bit, to the original, with a compression at least as good as JPEG-LS Visually lossless compression: reconstructed image differs numerically from original, the differences are not perceptible JPEG2000 Visually lossy compression: differences are perceptible Progressive Spatial Resolution: ability to extract lower resolution images from a stream without redundant coding Progressive Quality Resolution: ability to extract lower bit-rate images without redundant coding Complexity scalability: depends on applications, different levels complexity Strip processing: ability to compress and decompress images in a single sequential pass 12 2
13 JPEG at 0.125 bpp 14 Some examples JPEG2000 versus JPEG baseline JPEG2000 at 0.125 bpp 15 JPEG at 0.25 bpp 16 JPEG2000 at 0.25 bpp 17 JPEG at 0.5 bpp 18 3
JPEG2000 at 0.5 bpp 19 JPEG compound image 1.0 bpp 20 JPEG2000 compound im g 1.0 bpp 21 JPEG vs. JPEG2000: Major Differences 22 New functionalities Region-Of-Interest (ROI) Coding Better error resilience More flexible progressive coding (both in accuracy & resol.) Object-oriented functions (coding, information embedding) Lossy to lossless in one system Better compression at low bit-rates Better at compound images and graphics (palletized) JPEG2000 and other standards 23 Some lossless compress. results 24 46 44 42 8.00 7.00 PSNR (db) 40 38 36 34 32 30 28 26 24 0 0.5 1 1.5 2 bpp J2K R J2K NR JPEG VTC compression ratio 6.00 5.00 4.00 3.00 2.00 1.00 0.00 bike café cmpnd1 chart aerial2 target us average JPEG2000 JPEG-LS L-JPEG BZIP2 4
Comparison of various algorithms on functionality 25 More comparisons between JPEG2000 & other standards 26 JPEG 2000 JPEG-LS JPEG MPEG-4 VTC lossless compression performance +++ ++++ + - lossy compression performance +++++ + +++ ++++ progressive bitstreams ++++ - + ++ Region of Interest (ROI) coding +++ - - + arbitrary shaped objects - - - ++ random access ++ - - - low complexity ++ +++++ +++++ + error resilience +++ + + +++ non-iterative rate control +++ - - + genericity +++ +++ ++ ++ «JPEG 2000 still image coding versus other standards», D. Santa-Cruz, T. Ebrahimi, J. Askelöf, M. Larsson and Ch. Christopoulos, in Proc. of SPIE, Vol. 4115 «A study of JPEG 2000 still image coding versus other standards», D. Santa-Cruz, T. Ebrahimi, in Proc. of the X European Signal Processing Conference (EUSIPCO), Tampere, Finland, September 5-8, 2000 «An analytical study of JPEG 2000 functionalities», D. Santa-Cruz, T. Ebrahimi, in Proc. of the IEEE International Conference on Image Processing (ICIP), Vancouver, Canada, September 10-13, 2000 27 JPEG2000 / Basic encoding scheme 28 JPEG2000 Analysis coeff.data Joint RDO analysis Pre & post coding analysis! Part 2: Algorithm description Wavelet transform Codeblock partition Coefficient Quantization Entropy coding Rate allocation Multiple coeff. bands Codeblock partition Codeblock partition Coefficient Quantization Coefficient Quantization Entropy coding Entropy coding C Embedded Block Coding with Optimized Truncation (EBCOT) Each subband is partitioned into a set of blocks All blocks within a subband have the same size (possible exception for the blocks at the image boundaries) Blocks are encoded independently Post-processing operation determines the extent to which each block s bitstream should be truncated Final bitstream is composed of a collection of layers 29 Motivation block coding in bands? Exploit local variations in the statistics of the image from block to block Provide support for applications requiring random access to the image Reduce memory consumption in hardware implementations Both for compression or decompression engine Enable parallel implementation Typical blocks are 64x64 coefficients, later 4x256 to reduce memory 30 5
EBCOT coding operations Full-featured bit stream 31 EBCOT / Layered bitstream formation (1) 32 T2: Layered bitstream formation T1: Generation of embedded block bitstreams T2 Layer formation and block summary information coding engine Emb. block bit streams and summary information T1 Low-level embedded block coding engine Blocks subband samples Each bitstream is organized as a succession of layers Each layer contains additional contributions from each block (some contributions might be ) Block truncation points associated with each layer are optimal in the rate distortion sense Rate distortion optimization can be performed but it does not need to be standardized layer 5 layer 4 layer 3 layer 2 layer 1 EBCOT / Layered formation (2) block 1 bit-stream block 2 bit-stream block 3 bit-stream block 4 bit-stream block 5 bit-stream block 6 bit-stream block 7 bit-stream 33 JPEG2000 / Coding types (1) Zero Coding Run-Length Coding (RLC) Used in conjunction with ZC, in order to reduce the average binary symbols count that must be encoded with arithmetic coding Sign Coding (SC) Used at most once for each sample in the block immediately, a previously insignificant symbol is found to be significant during a ZC or RLC operation Magnitude refinement (MR) Used to encode an already significant sample 34 JPEG2000 / Coding types (2) 35 JPEG2000 / Zero Coding (ZC) 36 If the sample is not yet significant, a combination of Zero Coding and RLC primitives is used to encode whether or not a symbol is significant in the current bit-plane If significant, the Sign Coding (SC) primitive must also be invoked to send the sign If the sample is already significant, the Magnitude Refinement (MR) primitive is used to encode the new bit position Use 1 of 9 different context states to code the value of the symbol, depending on the significance state variables of: Immediate horizontal neighbors (h) Immediate vertical neighbors (v) Immediate diagonal neighbors (d) Non-immediate neighbors (f) h v v d h 6
JPEG2000 / Wavelet Transform 37 JPEG2000 / Filters supported 38 Two filters supported W9x7 (Floating point) for lossy coding W5x3 (Integer) for lossless coding Only dyadic decomposition supported Earlier experiments also with packet and spatial decompositions Dyadic decomposition Floating-point wavelets Daubechies (9 x 7), also (10 x 18) (9 x 7) proposed in Part 1 of the standard, coefficients in working p. Integer (13 x 7), CRF (13 x 7), (5 x 3), (2 x 10), etc. Default integer for lossy coding CRF (13 x 7) Default integer for lossless coding (5 x 3) User defined filters JPEG2000 / Quantization Scalar dead-zone quantization or Thresh. Coeff. Quant. Explicit Define a specific quantization step for each subband Smaller quantization steps for lower resolution subbands Implicit Quantization steps derived from LL subband quantization steps Smaller quantization steps for lower resolution subbands Reversible (Quantization and Coding) No quantization but pure bit-plane coding of transform coefficients Possibility of visual weighting Fixed visual weighting Visual progressive coding (VIP) 39 JPEG2000 Further capabilities (1) Advanced Rate Control Post-compression rate-distortion optimization Requires second pass algorithm Target one bit rate or an arbitrary set of specified bit rates Scalability Resolution scalability SNR scalability: select degree of SNR or Progr. Visual Weighting Random access into bit stream Arbitrary resolutions (up to 16 levels, up to 1x1 decom) Multi-component imaging (up to 256 components) 40 J2K Multiresolution decomposition (1) 41 Multiresolution decomposition (2) 42 LL 1 HL 1 Original Image LH 1 HH 1 7
Multiresolution decomposition (3) 43 Multiresolution decomposition (4) 44 LL 3 HL 3 LL 2 HL 2 HL 2 LH 3 HH 3 HL 1 HL 1 LH 2 HH 2 LH 2 HH 2 LH 1 HH 1 LH 1 HH 1 Multiresolution decomposition (5) 45 46 Example of dyadic decomposition into subbands JPEG2000 Part 3: Scalability and Progressive coding JPEG2000 / Scalabilityforms 47 Scalability / Progressive Resolution (1) 48 Different modes are realized depending on the way information is written into the codestream Code stream 8
Scalability / Progressive Resolution (2) 49 Scalability / Progressive Resolution (3) 50 Scalability / Progressive Resolution (4) 51 Scalability / Progressive Accuracy (1) 52 Scalability / Progressive Accuracy (2) 53 Scalability / Progressive Accuracy (3) 54 9
Example: Progressive by resolution 55 56 Image: Woman Resolution levels: 5 Decoded sizes: 1/16 1/8 1/4 1/2 1 57 58 59 60 10
Example: Progressive by quality 61 62 0.125 bpp Image: Bitrates: Woman 0.125 bpp 0.25 bpp 0.5 bpp 1.0 bpp 2.0 bpp 63 0.25 bpp 0.5 bpp 64 1.0 bpp 65 66 2.0 bpp 11
JPEG2000 Part 4: Specific features: Region Of Interest (ROI) coding and Conclusions 67 JPEG2000 / Region Of Interest coding Allows certain parts of an image to be coded or decoded in better quality Static: ROI is decided and coded once for all at the encoder side Dynamic: ROI can be decided and decoded on-the-fly from a same bitstream 68 JPEG2000 ROI: Some visual results 69 Regions Of Interest (1) 70 69:1 overall compression ratio No ROI Rectangular ROI Regions Of Interest (2) 71 Regions Of Interest (3) 72 12
J2K / ROI coding: mask computation Because of wavelet filtering, there are issues. Filter length over the boundaries! Mask adapted to filter characteristics 73 JPEG2000 / Region Of Interest coding Basic idea: Calculate wavelet transform of whole image/time calculate ROI mask == set of coefficients that are needed for up to lossless ROI coding Encoding is progressive by accuracy and resolution NOTE: ROI mask need NOT be transmitted to decoder (instead, location and shape of ROI is required) 74 JPEG2000 / Creation of ROI mask 75 JPEG2000 / Modes for ROI Coding 76 The ROI masks are acquired by looking at the inverse transform For each pixel (X) that is in the ROI, the low- and highfrequency coefficients (L:s and H:s) that are needed to reconstruct the pixel, are included in the ROI mask Inverse transform of the 5-3 filter Low High n n n+1 n-1 n+1 n-1 2n 2n+1 X:s Block-based mode Comes for free but not the optimal solution for static ROI Useful for dynamic ROI Arbitrary shape scaling based mode Scale ROI mask coefficients up (decoder down) During encoding, the ROI mask coefficients are found significant at early stages of the coding ROI always coded in better PQ than background MaxShift method JPEG2000 / ROI Scaling-based method Highest BG coeff value is found Coefficient values 77 JPEG2000 / ROI MaxShift method After shifting, all the ROI coefficients are larger than the largest BG coefficient Coefficient values If min. coefficient in ROI is larger than BG, then MaxShift method! 78 ROI Coefficients ROI Coefficients 13
79 80 JPEG2000 / Example ROI coding 0.125 bpp Image: Woman ROI: rectangular Scaling value: 6 Progressive type: SNR Bitrate: 4 bpp 81 82 0.25 bpp 0.5 bpp 1.0 bpp 83 84 2.0 bpp 14
4.0 bpp 85 ROI Maxshift mode / What is the gain? 86 Support for arbitrary shaped ROI s with minimal complexity No shape information to send No shape encoder and decoder is needed No ROI mask at decoder side is needed Decoder as simple as non-roi capable decoder Can decide in which subband the ROI will begin therefore it can give similar results to the general scaling method ROI coding: what do we pay? (1) Lossless image coding with ROI 87 ROI coding: what do we pay? (2) Lossless image coding with ROI 88 1,02 Gold: Rectangular ROI 1,1 Target - approx. 25% circular ROI - Relative sizes 1,08 1,015 1,06 1,01 1,005 1 No ROI S=2 S=4 Maxshift 1,04 1,02 1 0,98 No ROI S=2 S=4 Max shift 0,995 0,96 0,99 0,94 JPEG2000 / Conclusions 89 JPEG2000 / More background info 90 Advanced Still Image Coding System Better performance than JPEG and forms of scalability Offers many functionalities, yet more complex than JPEG RDO quantization, progressive resolution & accuracy, ROI, etc. No IPR associated to Part I of the standard (free licensing) Intended key standard for still image coding this decade Killer application upcoming: Digital Cinema JJ2000 http://jj2000.epfl.ch JPEG Web site: http://www.jpeg.org EUROSTILL http://ltswww.epfl.ch/~eurostill SPEAR http://spear.jpeg.org/ 15