Multimedia Signals and Systems Still Image Compression - JPEG Kunio Takaya Electrical and Computer Engineering University of Saskatchewan January 27, 2008 ** Go to full-screen mode now by hitting CTRL-L 1
Contents 1 Information Entropy 4 2 Huffman Coding 8 3 JPEG Image Compression 18 4 Application of DCT to 8 8 bolcks 27 5 Coding the reduced size DC image 41 6 Encoding DCT (AC) coefficients 46 7 Assignment JPEG 55 2
R µ ν 1 2 R δµ ν = 8πG c 4 T µ ν Here T µ ν is tensor of energy momentum. black red green yellow blue magenta cyan 3
1 Information Entropy The average amount of information is defined by the information entropy measured in bits. E = L 1 i=0 L 1 1 p i log 2 = p i i=0 p i log 2 p i The probability for the pixel value i to occur p i can be determined from the histogram of a picture by p i = h(i)/(n M). If a difference image, capable of reconstructing the original image (loss less), is produced by I (i, j) = I(i, j) 1 fi(i 1, j) + I(i, j 1)g 2 4
for 0 i N 1 and 0 j M 1, the information entropy for I is smaller than that of the image I. As long as the first row and the first column of the original image I are retained, the original image I is restored by I(i, j) = I (i, j) 1 fi(i 1, j) + I(i, j 1)g 2 The reconstruction must be done in the sequence of raster scanning by using the first row and column. Another simpler difference image define by I (i, j) = I(i, j) I(i 1, j) is usded in DPCM for the reduced size DC component image resulting from the 2D DCT. In this case, the first column must be retained for lossless reconstruction. 5
6
Information Entropy of the image I: 7.1757 Information Entropy of the image I : 4.2662 7
2 Huffman Coding Huffman coding is an entropy encoding algorithm used for lossless data compression. The term refers to the use of a variable-length code table for encoding a source symbol (such as a character in a file) where the variable-length code table has been derived in a particular way based on the estimated probability of occurrence for each possible value of the source symbol. It was developed by David A. Huffman while he was a Ph.D. student at MIT, and published in 1952. 8
Huffman coding is based on the frequency of occurance of a data item (pixel in images). The principle is to use a lower number of bits to encode the data that occurs more frequently. Codes are stored in a Code Book which may be constructed for each image or a set of images. In all cases the code book plus encoded data must be transmitted to enable decoding. The Huffman algorithm is a bottom-up approach. Consider a case of having five symbols (A, B, C, D, E). 9
Symbol Count Probability Entropy A 15 0.3846 1.3785 B 7 0.1795 2.4780 C 6 0.1538 2.7004 D 6 0.1538 2.7004 E 5 0.1282 2.9635 Total 39 1.0000 2.1858 (average) From this table, the theoretical total information is 2.1858 39 = 85.2467 bits. If we use a fixed length code of 3 bits, the total would be 4 39 = 156 bits. The procedure of the Huffman coding is as follows: 10
1. From the table, pick two nodes (symbols) having the lowest frequencies or probabilities. Assign 1 to the one with the lowest, then 0 to the second lowest. Or, simply assign 1 to one of the two, and 0 to the other. Create a parent node of these two symbols combined, DE for this case. Symbol Count Probability Code A 15 0.3846 - B 7 0.1795 - C 6 0.1538 - D 6 0.1538 0 E 5 0.1282 1 11
2. Update the table with the newly created parent node, then repeat the previous step. Symbol Count Probability Code A 15 0.3846 - B 7 0.1795 0 C 6 0.1538 1 DE 11 0.2820-0 (D), -1 (E) 12
3. Repeat until the table has only one node left. Symbol Count Probability Code A 15 0.3846 - BC 13 0.3333 00 (B), 01 (C) DE 11 0.2820 10 (D), 11 (E) 13
4. Repeat until the table has only one node left. Symbol Count Probability Code A 15 0.3846 0 (BC)(DE) 24 0.6154 100 (B), 101 (C) 110 (D), 111 (E) 14
15
Symbol Count Probability Entropy Code Subtotal A 15 0.3846 1.3785 0 15 B 7 0.1795 2.4780 100 21 C 6 0.1538 2.7004 101 18 D 6 0.1538 2.7004 110 18 E 5 0.1282 2.9635 111 15 Total 39 1.0000 2.1858 (avg) 87 Compare this with the theoretical total information of 2.1858 39 = 85.2467 bits. 16
problem: When the probabilities of occurrence for alphabets A to H are given, verify the following Huffman codes by drawing the code tree. Symbol Probability Code A 0.5 0 B 0.2 10 C 0.1 1100 D 0.08 1101 E 0.05 1110 F 0.04 11110 G 0.02 111110 H 0.01 111111 17
3 JPEG Image Compression A joint ISO/CCITT committee known as JPEG (Joint Photographic Experts Group) has established the international compression standard for continuoustone still images, both grayscale and color, early in 1990 s. JPEG now supports four modes of operation, sequential encoding, progressive encoding, lossless encoding, hierarchical encoding. The most fundamental sequential encoding that encodes a picture in a single left-to-right, top-to-bottom scan, is discussed here. This is a lossy compression. 18
The encoder consists of 3 major components, (1) Forward DCT (Discrete Cosine Transform, (2) Quantizer based on the quantization table, and (3) Entropy Encoder that employs Huffman coding and Run-Length coding. These are applied to each of three components in the YUV (YCbCr) color space, sequentially. The paper, Gregory K. Wallace, The JPEG Still Picture Compression Standard, is a good reference available at http://man.lupaworld.com/content/other/jpg.pdf 19
1. Forward 8 8 DCT An image is divided into a stream of 8 8 blocks of gray scale image samples. The image is scanned left-to-right, top-to-bottom. Source image samples grouped in 8 8 blocks are shifted from unsigned integers [0, 2 p 1] to signed integers [ 2 p 1, 2 p 1 1]. Each block of 8 8 pixels is then transformed by the forward DCT into the spectral domain. The forward DCT (FDCT) is given by F (u, v) = 1 4 C(u)C(v) 7 x=0 y=0 C(u), C(v) = 7 f(x, y) cos (2x + 1)uπ 16 1 2 for u, v = 0 1 otherwise cos (2y + 1)vπ 16 20
clear all; close all; I=imread( lenna-y.jpg ); imshow(i); hold on; [x0,y0]=ginput(1); x0=fix(x0); y0=fix(y0); x=x0; y=y0; x=[x-1,x+8,x+8,x-1,x-1] y=[y-1,y-1,y+8,y+8,y-1] plot(x,y, -r ); I88=double(I(x0:x0+7,y0:y0+7))-128 DCT88=fix(dct2(I88)) % FUJIFILM - FinePix F40fd () A=[6 5 6 6 7 10 20 29 4 5 5 7 9 14 26 37 4 6 6 9 15 22 31 38 6 8 10 12 22 26 35 39 10 10 16 20 27 32 41 45 16 23 23 35 44 42 48 40 20 24 28 32 41 45 48 41 24 22 22 25 31 37 40 40] DQ=fix(DCT88./A) 21
An area of DCT block set in the eye. Numerical values are sampled from this area. 22
A selected 8 8 image block: I88 = -41-54 -70-60 -6 68 82 78-58 -77-61 -19 47 82 83 80-62 -54-21 30 70 83 77 80-36 -3 28 65 69 75 67 78 7 36 62 64 75 72 81 80 30 47 61 61 70 65 73 72 32 39 47 48 55 63 70 61 18 26 34 40 43 37 54 48 23
The result of forward DCT: DCT88 = 286-263 -35 20-6 -3 0 3-133 -164 18 52 3-4 11-7 -98-17 66 41-8 5 0 1 28 31 46 4-20 -2 1 3-7 21 14-17 -11 0-2 -4 0 7 4-4 -6 0 1-10 -4 4 0-5 0-1 -2 0 5-3 4 2-5 0 0 3 24
A quantization matrix (FUJIFILM - FinePix F40fd): A = 6 5 6 6 7 10 20 29 4 5 5 7 9 14 26 37 4 6 6 9 15 22 31 38 6 8 10 12 22 26 35 39 10 10 16 20 27 32 41 45 16 23 23 35 44 42 48 40 20 24 28 32 41 45 48 41 24 22 22 25 31 37 40 40 25
The DCT matrix after quantizxation: DQ = 47-52 -5 3 0 0 0 0-33 -32 3 7 0 0 0 0-24 -2 11 4 0 0 0 0 4 3 4 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 26
4 Application of DCT to 8 8 bolcks DCT, Quantization, DC-component The detailed procedure of JPEG encoding takes the following steps: 1. Subdivide a given picture into blocks of a 8 8 pixel area. Then, process the blcoks sequentially from left-to-right then top-to-bottom. 2. Convert pixel values of a block from the unsigned integer [0, 2 N 1] to signed integer [ 2 N 1, +2 N 1 1]. 3. Apply 2D DCT to the block. For a 8 bit gray scale image [0, 255], the values of 2D DCT transform are in the range of 2048 I(i, j) 2048, 12 bits. The double summation makes the possible largest value be 255 (8 bits) times 64, 8 8, (6 bits). The constant 1 4 divides the result by 4 or (2 bits). Thus, 27
the DCT values are 12 bits. 4. Divide each of the 8 8 DCT elements by the corresponding value in the quantization table. 5. Repeating these steps until all blocks are processed. The encoding part of JPEG compression without Huffman and Run-length coding looks like the following: I=imread( lenna-y.jpg ); imshow(i); Img=double(I); % FUJIFILM - FinePix F40fd () A=[6 5 6 6 7 10 20 29 4 5 5 7 9 14 26 37 4 6 6 9 15 22 31 38 6 8 10 12 22 26 35 39 10 10 16 20 27 32 41 45 16 23 23 35 44 42 48 40 20 24 28 32 41 45 48 41 24 22 22 25 31 37 40 40] 28
[row,col]=size(i); DCTimg=zeros(row,col); blrow=row/8; blcol=col/8; DCcomp=zeros(bLrow,bLcol); % encoding JPEG for ii=1:blrow for jj=1:blcol r0=8*(ii-1)+1; r7=r0+7; c0=8*(jj-1)+1; c7=c0+7; I88=double(I(r0:r7,c0:c7))-128; DCT88=fix(dct2(I88)); DQ88=fix(DCT88./A); DCTimg(r0:r7,c0:c7)=DQ88; DCcomp(ii,jj)=DQ88(1,1); end end figure; imshow((dctimg+128)/256); figure; imshow((dccomp+128)/256); Application of 2D DCT produces an image of DCT coefficients, and an image of DC component of all blocks, which is the reduced size image of the original, reduced by 8 horizontally, and 8 vertically. 29
The original picture of Lena, and its quantized DCT image. 30
The image made of only the DC components of 2D DCT. The size is one eighth ( 1 8 ), horizontally and vertically. 31
Inverse DCT to reconstruct a compressed JPEG image In order to reconstruct the image from the quantized DCT coefficients, actually the image of quantized DCT shown above, the process of encoding with the 2D DCT was entirely reversed as shown in the following MATLAB codes. The 2D DCT was replaced by the 2D inverse DCT (idct2). 32
% decoding JPEG RCNimg=zeros(row,col); for ii=1:blrow for jj=1:blcol r0=8*(ii-1)+1; r7=r0+7; c0=8*(jj-1)+1; c7=c0+7; DQ88=DCTimg(r0:r7,c0:c7); DCT88=DQ88.*A; idct88=idct2(dct88)+128; RCNimg(r0:r7,c0:c7)=iDCT88; end end figure; imshow(rcnimg/256); 33
The original picture of Lena, and its reconstructed image with the inverse DCT and dequantization. 34
PSNR and Entropy Values Image reconstruction from the DCT image, quantized DCT coefficients to be exact, was successful in appearance. The peak signal to noise ratio PSNR was measured for the reconstructed image referenced to the original image. The value obtained was 37.8 db. The peak value of 255 was used. The mean squared error MSE was 10.78, giving an average error in magnitude be around 3.28, compared with the maximum pixel value of 255. The information entropy was calculated for the original and the reconstructed. They are very close, 7.4217 vs. 7.4080. The entropy of the DCT image was only 1.0266, which means that this image can be compressed down to about 1 bit per pixel. 35
MSE = 10.7792 PSNR = 37.8049 Entropy_Img = 7.4217 Entropy_Rcn = 7.4080 Entropy_dct = 1.0266 Finished 36
The histogram of the original picture of Lena, and the histogram of the DCT image. 37
MATLAB program to test JPEG encoding and decoding clear all; close all; I=imread( lenna-y.jpg ); imshow(i); Img=double(I); % FUJIFILM - FinePix F40fd () A=[6 5 6 6 7 10 20 29 4 5 5 7 9 14 26 37 4 6 6 9 15 22 31 38 6 8 10 12 22 26 35 39 10 10 16 20 27 32 41 45 16 23 23 35 44 42 48 40 20 24 28 32 41 45 48 41 24 22 22 25 31 37 40 40] [row,col]=size(i); DCTimg=zeros(row,col); blrow=row/8; blcol=col/8; DCcomp=zeros(bLrow,bLcol); % encoding JPEG for ii=1:blrow 38
for jj=1:blcol r0=8*(ii-1)+1; r7=r0+7; c0=8*(jj-1)+1; c7=c0+7; I88=double(I(r0:r7,c0:c7))-128; DCT88=fix(dct2(I88)); DQ88=fix(DCT88./A); DCTimg(r0:r7,c0:c7)=DQ88; DCcomp(ii,jj)=DQ88(1,1); end end figure; imshow((dctimg+128)/256); figure; imshow((dccomp+128)/256); % decoding JPEG RCNimg=zeros(row,col); for ii=1:blrow for jj=1:blcol r0=8*(ii-1)+1; r7=r0+7; c0=8*(jj-1)+1; c7=c0+7; DQ88=DCTimg(r0:r7,c0:c7); DCT88=DQ88.*A; idct88=idct2(dct88)+128; RCNimg(r0:r7,c0:c7)=iDCT88; end end 39
figure; imshow(rcnimg/256); % Calculate PSNR SSE=0; for i=1:row for j=1:col SSE=SSE+(Img(i,j)-RCNimg(i,j))^2; end end MSE=SSE/(row*col) PSNR=10*log10(255^2/MSE) Entropy_Img=entropy(Img/256) Entropy_Rcn=entropy(RCNimg/256) Entropy_dct=entropy((DCTimg+128)/256) figure; subplot(121); imhist(img/256); subplot(122); imhist((dctimg+128)/256); disp( Finished ); 40
5 Coding the reduced size DC image DPCM Losless Coding F (u, v) = 1 4 C(u)C(v) 7 x=0 y=0 C(u), C(v) = 7 f(x, y) cos (2x + 1)uπ 16 1 2 for u, v = 0 1 otherwise cos (2y + 1)vπ 16 In the DCT image F (u, v), F (0, 0) is DC component of an image block, located at the upper left corner of a 8 8 matrix. An image of the reduced size, down to 1/8 both horizontally and vertically is constructed from the DC components of all blocks. In JPEG, this reduced size image is coded with the lossless DPCM coding. The DC components that take a value -1023 F (u, v) 1023 because of 41
C(0) = 1/ 2, is subjected to the difference operation (DPCM), where I(i, j) is the reduced size DC image. D(i, j) = I(i, j) I(i 1, j) Possible values of D(i, j), -2047 D(i, j) 2047 are grouped into bins defined by [ 2 n 1, 2 n 1] and [2 n 1, 2 n 1] for each block of n = 1, 11. Then, these bins are Huffman coded for the group number. Elements in each block are coded with an additional bit length of n, which is the code group number. This coding scheme is dipicted in the following figure. 42
1 1(010) 2 0(00) 2 (011) Group code 3 3(100) 4(101) -7, -6, -5, -4 4, 5, 6, 7 Difference code 8, 9, 10,11,12,13,14,15 (000,001,010,011) (100,101,110,111) 5 31 6 63 7 8 127 255 9 512 10 11 1023 2047 43
Gr. Difference of DC values Group code Added bits 0 0 00 0 1-1,1 010 1 2-3,-2,2,3 011 2 3-7..-4,4..7 100 3 4-15...-8,8...15 101 4 5-31...-16,16...31 110 5 6-63...-32,32...63 1110 6 7-127...-64,64...127 11110 7 8-255...-128,128...255 111110 8 9-511...-256,256...511 1111110 9 10-1023...-512,512...1023 11111110 10 11-2047...-1024,1024...2047 111111110 11 44
Examples: difference=-5, Group code (100) + Added code (011) = (100011) difference=63, Group code (1110) + Added code (111111) = (1110111111) difference=1, Group code (101) + Added code (1) = (1011) difference=0, Group code (00) + no Added code 45
6 Encoding DCT (AC) coefficients Zigzag scanning DCT coefficients of a block other than the DC components are 46
scanned in a zigzag fashion as shown in the figure above. The zigzag scanning moves from lower frequencies to higher frequencies. As the DCT tends to concentrate AC coefficients (components) in the upper left area, the zigzag scan encounters more zeros as it goes to higher frequencies. Therefore, JPEG uses Huffmann coding for non-zero DCT coefficients, and Run-length coding to encode the length of repeated zeros (of the DCT coefficinets) in order to encode the DCT s AC components. Nonzero AC DCT coefficients Nonzero AC DCT coefficients, namely valid coefficients for Huffman coding, are grouped into 10 groups. the same number of bits as the group number are appended to the Huffman code for a group code. 47
Gr. Difference of DC values bits appended 0 0 0 1-1,1 1 2-3,-2,2,3 2 3-7..-4,4..7 3 4-15...-8,8...15 4 5-31...-16,16...31 5 6-63...-32,32...63 6 7-127...-64,64...127 7 8-255...-128,128...255 8 9-511...-256,256...511 9 10-1023...-512,512...1023 10 48
Huffman coding for Run-length, Group Number combined 49
Gr. Group No. Huffman Code 0 EOB 1010 0 1 00 0 2 01 0 3 100 0 4 1011 0 5 11010 0 6 1111000 0 7 11111000 0 8 1111110110 0 9 111111110000010 0 10 111111110000011 1 1 1100 1 2 11011 1 3 1111001 1 4 111110110... 50
Now consider to encode the following AC DCT coefficients. DQ = 7 60 0 7 0 0 0 0 15-10 0 0 0 0 0 0 6 0 0 0 0 0 0 0 4 3 4 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 51
Symbol Group Huffman Code bits Appended Run 0 value 60 6 1111000 6 111100 Run 0 value 15 4 1011 4 1111 Run 0 value 6 3 100 3 110 Run 0 value -10 4 1011 6 0101 Run 1 value 7 3 111001 3 111 EOB 0 1010 0 - The final code sequence is (1111000 111100)(1011 1111)(100 110)(1011 0101)(111001 111)(1010) 52
Chroma Subsampling Color images are transformed from RGB to YUV in JPEG. Luma component Y, and two chroma components U (Cb) and V (Cr) are independently quantized then entropy coded. However, chroma has less amount of informatio compared with luma. In JPEG, chroma subsampling can be specified. Chroma subsampling notation is shown in the figure below. Typically, one of 4:4:4, 4:2:2, or 4:2:0 is used. 53
4:4:4 4:2:2 4:2:0 x x x x x - x - x - x - x x x x x - x - - - - - x x x x x - x - x - x - x x x x x - x - - - - - 54
7 Assignment JPEG Free, portable C code for JPEG compression is available from the Independent JPEG Group. Source code, documentation, and test files are included. Version 6b is available from ftp.uu.net:/graphics/jpeg/jpegsrc.v6b.tar.gz. If you are on a PC you may prefer ZIP archive format jpegsr6b.zip, which you can find at http://www.sac.sk/files.php?d=5&l=j This assignment is to compile this JPEG source code on either PC or Linux platform then study jpeg files produced by this software with various control parameters. This free portable C code was tested with djgpp on Windows XP. Djgpp is a complete 32-bit C/C++ development system for Intel 80386 (and higher) PCs running DOS. If you are not familiar with Linux (Unix based PC operating system), you are advised to use 55
djgpp. Note that this source code jpegsr6b.zip provides Makefile and config.h for djgpp, but not specifically for Linux. The coding part of the compiled program cjpeg.exe has a command line switch -quality N which scales the quantization tables to adjust image qaulity. Quality is 0 (worst) to 100 (best); default is 75. For an image of your choice in ppm, bmp, gif, run cjpeg.exe with different setting of -quality N to obtain an output image of that specified quality. Calculate first the entropy of the input image to compare the degree of compression achieved for varied -quality N. Try N=100, N=75, N=50, N=25 and N=10. Measure the achieved entropy by dividing the total number of bits of the jpeg file by the image size. Also calculate MSE and PSNR of each image generated. Discuss how -quality N affects compression, and image quality. The wizard switches are intended for experimentation with 56
JPEG. These switches are documented in the file wizard.doc. One of the wizard switches is -qtables file. You can specify the quantization tables given in the specified text file to use it instead of default file. Quantization tables used in digital cameras are found at http://www.impulseadventure.com/photo/jpegquantization.html. Try the Quantization Table for FUJIFILM - FinePix F40fd () to encode your image. Discuss how this quantization table affects compression, and image quality. By default, cjpeg uses 2:1 horizontal and vertical downsampling when compressing YCbCr data. Other chroma subsampling can be experimented by -sample HxV[,...]. Try 4:4:4, 4:2:2 and 4:2:0. Discuss how this switch for setting subsampling factor affects compression, and image quality. 57