Quantization, conversion of values into bits

Transcription

1 Quantization, conversion of values into bits We have seen that for pure brightness values we need about 100 steps, for which 7 bits are sufcient. What is the situation with DCT coefcients? We saw with the Contrast Sensitivity Function that the eye has diferent sensitivity for diferent spatial frequencies. Since our DCT coefcients now represent diferent spatial frequencies, this also results in diferent sensitivities. How do we establish the connection between the DCT coefcients or sub-bands and the spatial frequencies (in brightness oscillations per degree)? To do this, we need to know how many pixels per degree appear to the eye at a given viewing distance or pixel size. Example: We have a 50cm wide screen with 1000 pixels over this width and a viewing distance of 50cm. 1 pixel has the width of 50/1000=5/100 cm. It appears under an angle of arctan ((5/100)/50)=1e-3 Rad=0.057 Degrees. So we get 1/0.057=17.45 pixel/degrees. Since each pixel represents a sampled value, this now corresponds to the sampling frequency. The Nyquist frequency is therefore 17.45/2=8.725

2 periods/degree. The 8x8 block based DCT divides the frequency range from 0 to Nyquist into 8 equal subbands. Each has a width of 8,725/8=1,0906 periods/degree. In this way, we have now obtained an assignment of the sub-bands or coefcients (the index k) of the DCT to the spatial frequency. The Contrast Sensitivity Function then shows us the relative sensitivity of the eye to the diferent sub-bands. This in turn shows us how best to quantize the DCT coefcients. The higher the contrast sensitivity in a subband, the more accurate we have to transfer the corresponding DCT coefcient. The coefcients of the higher spatial frequencies tend to require a lower accuracy and thus fewer bits. How do we now convert our DCT coefcients into bits with appropriate accuracy? We can divide our possible range of values into equal intervals. We can then assign an index to each of these intervals, which we can display and transfer with a binary code word. Depending on whether the value "0" appears in the middle or at the edge of a quantization interval, we distinguish between the so-called "Mid-Tread" and "Mid-Rise" quantizers. Example: The Mid-Tread quantizer (see:

3 (signal_processing). In the encoder and for the Mid-Tread case, the index I is calculated from a value x by means of a simple rounding, according to the formula I=round ( x Δ ) where Δ is the quantization stepsize. The reconstructed value x rek is calculated in the decoder by the so-called de-quantizer, using the formula x rek =I Δ If we want to use b bits to represent the indices of this range of values, index 0 represents the smallest possible value "min", and the largest index 2 b 1 represents the largest possible value "max" (if b is the number of bits we want to spend on the index), then we can divide the value range (max-min) into 2 b 1 equal intervals. Each interval then has the step height Δ of Δ= max min 2 b 1 Note: This formula also works for b=0 bits (no transfer) if we replace 1 with 0.99 to avoid division by 0! The diference between original and reconstruction is the quantization error e

4 e=x rek x Example: We have our normalized range with min=0 and max=1, b=2 bits and x=0.3. Then our step height Δ= max min 2 b 1 the quantization index is I=round ( x Δ )=round( = 1 3 = )=round (0.9)=1 the reconstructed value in the decoder is x rek =I Δ= = and the quantization error is e=x rek x= =

5 Example of the binary representation of the index for transmission from the encoder to the decoder: 2 bits or 4 indices: Index Binary representation (code word) The binary representation is also called "code word". Note that the example shows the so-called two's complement for negative numbers, see also or The code word for negative numbers is obtained by taking the binary representation of the positive number, inverting its bits and adding 1. Note: The rounded up logarithm to base 2 of the number of indexes ( MaxIndex MinIndex+1 ) indicates the required number of bits:

6 bit number=ceil(log 2 (MaxIndex MinIndex+1)) In the above example: bit number=ceil(log 2 (4))=ceil(log(4)/log (2))=2 Implementing in Python The conversion of indexes to and from code words works best in Python with the so called. Dictionaries. Example: Dictionary for converting an index (e. g. index=-1) into a code word: The codebook is defned as: codeword={0:'00',1:'01',-2:'10',-1:'11'} The conversion from index to code word (coding) is done by: codeword[-1] '11' This binary codeword is then transmitted to the decoder. The decoder needs to convert the codword back into an index frst. The conversion of a code word into an index in the decoder is done by defning the Dictionary decodeword={'00':0,'01':1,'10':-2,'11':-1} and then decode with

7 decodeword['11'] -1 Python example for direct quantization of the pixels of the Y-component of the video frames: We generate the quantization steps with: bits=2 #resulting quantization step size for 2^bits steps, with min=0 and max=1: #Dividing the value range by the number of #available quantization intervals, #to obtain the quantization step size: quantstufe=1.0/(2**bits-1) The quantization indices are calculated from the frame Y with: #Apply mid-tread for the encoder: indices=np.round(y/quantstufe) So all pixels get the same quantization step size. The de-quantization (in the decoder) is then the mid-tread formula for the decoding side: Yrek=indices*quantstufe Calling the program: python videorecprocyquant.py We see: The less bits per pixel we use, the less

8 brightness levels we have and the more visible the light jumps become. Python example for quantization of the 8x8 DCT coefcients: In this example we quantize all coefcients of the 8x8 DCT. Because of the Parseval theorem we have to adjust the range of values. It can happen that the total signal energy of all 64 pixels of a block is concentrated in the DC coefcient, e. g. when all pixels are equal. Then this coefcient contains 64 times the energy. The root of it is the factor for our signal, thus in extreme cases the factor 8. It is benefcial to observe the values with real pictures and then adjust them accoridngly. There, the value range from 0 to 5 can be obtained for the DC coefcients, and the value range for the other (AC) coefcients. In python: bits=2 #resulting quantization step size for 2^bits steps: #Steps for different spatial frequencies: #DC indices with range 0...5: quantstufedc=5.0/(2**bits-1) #Two AC coefficients, with range 0.5-(-0.5) bits=2 quantstufeac=1.0/(2**bits-1)

9 Calculation of the indices for the DC coefcients, every 8th coefcient horizontally and vertically: indices00=np.round(x[0::n,0::n]/quantstufedc) For 2 AC coefcients (the others are omitted for the sake of clarity): indices01=np.round(x[0::n,1::n]/quantstufeac) indices10=np.round(x[1::n,0::n]/quantstufeac) De-quantization: #de-quantization in the decoder: Xrek=np.zeros((r,c)); #DC values de-quantization: Xrek[0::N,0::N]=indices00*quantstufeDC #2 AC values de-quantization: Xrek[0::N,1::N]=indices01*quantstufeAC Xrek[1::N,0::N]=indices10*quantstufeAC Calling the Prorgam: python videorecdctblocksidctquant.py We see: Instead of long contours in the lightness jumps we now have block artifacts, similar to low pass fltering.

10 Writing bits to fles in Python The smallest unit we can write to a fle in Python is 1 byte (8 bits). Since we only have 2 bit per index in the example, we must frst form a bitstream, which we then divide into bytes. We generate the bit stream by connecting codewords with bitstring=bitstring+codeword[value] where "value" is our index to be coded. We implement this as a function 'data2codestring' in File writereadbits.py. Converting the code string into a sequence of 8- bit numbers is done with the command "eval", which executes a string (in the writebinaryfle function of the writereadbits. py library) which converts here into numbers: for m in range(numbytes): Bytes[m]=eval('0b'+bitstring[m*8:(m*8+8)]) Here partial strings of 8 characters each are evaluated, with the string' 0b' at the beginning they are interpreted as binary numbers, which are written into the array "Bytes". struct.pack, The conversion of the array of 8-bit numbers into a stream of bytes is done with the command s=struct.pack('b'*len(bytes), *Bytes) 'B' means: unsigned byte, because we already have

11 the sign in our code words. bytes means: Pointer to our array with 8-bit numbers, i. e. integers between 0 and 255. Example: import struct import numpy as np bitstring=' ' #Erzeugen unseres Arrays aus 8-bit Zahlen: Bytes=np.zeros(3,dtype=int) for m in range(3): Bytes Bytes[m]=eval('0b'+bitstring[m*8:(m*8+8)]) #array([ 1, 2, 255]) #Erzeugen des Byte-Stroms: s=struct.pack('b'*len(bytes), *Bytes) s '\x01\x02\xff' '\x..' means a "hexadecimal" number whose digits consist of 4 bits. 4 bits can accept 16 values, e. g. between 0 and to 15 values are represented by the letters 'a' to 'f'. Hence the hexadecimal number 'ff' for 255, which consists of two binary numbers '1111', ie hexadecimal 'f'. We can then write this stream to a fle, with file=open(filename, "w")

12 #'w' deletes exisiting file, 'a' appends file.write(s) file.close() We implement this as a function 'writebinaryfile' in the fle 'writereadbits.py'. In the decoder we have to reopen this fle and read the byte stream: file=open(filename, "r") #read the stream from file: readdata=file.read() #unpacks the stream into an array of Bytes: Bytesread=struct.unpack('B'*len(readdata),readdata) Afterwards we create a bitstream from it, by creating bits from the bytes and connecting them to each other: for byte in Bytesread: #create bit string from byte: bits=bin(byte) #remove leading '0b' and fill up to 8 bits with leading zeros: bits=bits[2:].zfill(8) #append to bits to bitstring: bitstring=bitstring+bits Our data values (or quantization indices) are then translated from the pairs of 2 in the bit string: decodeword={'00':0,'01':1,'10':-2,'11':-1} #convert sequence of Bytes into sequence of bits:

13 numdata=len(bitstring)/2; #converts groups of 2 bits into data: data=np.zeros(numdata,dtype=int) n=0; for i in range(numdata): data[n]=decodeword[bitstring[(i*2):(2+i*2)]] n+=1 The following functions serve as an example for translating a data array from our quantization indexes into strings from code words and back data2codestring and codestring2data in the fle writereadbits.py. The code string can be defned using the functions writebinaryfile and readbinaryfile written to a binary fle or read from it. We test these functions with: python writereadbits.py Note: The test routine is only activated when you call the function directly under if name == ' main ': not when importing the fle. Note: The test function shows: The functions data2codestring and writebinaryfile write a sequence of indexes with 2 bit each into the fle 'savebin. bin', and readbinaryfile and codestring2data decode the same sequence again.

14 Example of a frame encoder and decoder in Python: The following encoder takes the lowest 3 DCT coefcients (at the positions (0,0), (0,1), and (1,0), quantizes them with 2 bit each, and stores them in their own fle. For simplicity's sake, we defne the 8x8 pixel blockwise DCT and inverse DCT as function 'dct8x8' or 'invdct8x8' in the fle 'blockdct.py'. The size of the frame is additionally written in a text fle for the decoder. We start the encoder with python frameencfile.py We see the live video and save a frame when we press "c" (capture), in the fles y10enc. bin, y01enc. bin, y00enc. bin, framedim. txt. If we look at the total size of the fles, we get only 3.6 kb, an extreme compression for a 480x640 pixel image! The corresponding decoder is started with: python framedecfle.py This automatically reads the encoder's fles, decodes the image and displays it. As we can see, it is very blocky and has only a few shades of gray, a consequence of extreme compression. But we can still see the content of the

15 picture well. The Color Image Coder By including the color components U and V, our coder can be expanded to a color image encoder. Since the eye is sensitive to the color components at lower spatial frequencies than to the brightness in the luminance component Y, we limit ourselves to coding the DC coefcients (position (0,0)) of the block wise DCT of the color components U and V. The call of the encoder is most appropriate in a subdirectory for the resulting compressed image fles: mkdir Pic1; cd Pic1 python../picturecolorencoder.py The decoder is then started with python../picturecolordecoder.py Note: The resulting fles of the compressed color image have a total size of only about 6KB, i. e. only about 0.16 bit/pixel at 480x640 pixel image size! The resulting decoded image has clear artifacts (partly "artistic" in appearance), but the image content is still clearly visible.

16 Python example for the diferent quantization of the DCT coefcients: In order to use the diferent sensitivity of the eye for diferent orbital frequencies in order to improve the result, we quantize the respective DCT coefcients diferently. We use the Contrast Sensitivity Function here only qualitatively. Reminder: In slide set 4 we saw the Contrast Sensitivity Function (CSF): We assume that the spatial frequency range of the DC coefcient of the DCT is already in the maximum of the CSF (according to our example above, where

17 we found about 1 period/degrees per DCT subband and ftting to the curve for dark light). We therefore use the most accurate quantization for the DC value, so we use the characteristic of the eye that it becomes increasingly insensitive to high spatial frequencies (fne patterns). For the coefcients with a constant sum s=k+l (for k,l: horizontal and vertical DCT subband indices) we select the same quantization level, because they correspond to roughly edges of the same slope in diferent directions. This results in anti-diagonals with the same quantization step-size in our quantization matrix. At the top left is DC with the greatest accuracy, down to the right, towards high spatial frequencies, the accuracy becomes smaller and smaller. This will give us a more detailed quantization mask. In python we use the command np.diag, which creates a diagonal matrix with a given ofset from the main diagonal: B=np.diag([1,2,3]) array([[1, 0, 0], [0, 2, 0], [0, 0, 3]]) np.fliplr to make it an anti-diagonal: np.fliplr(b) array([[0, 0, 1], [0, 2, 0], [3, 0, 0]])

18 and np.tril, which generates a lower triangle matrix: np.tril(np.ones((4,4)),1) array([[ 1., 1., 0., 0.], [ 1., 1., 1., 0.], [ 1., 1., 1., 1.], [ 1., 1., 1., 1.]]) This is now our quantization matrix M: #Steps for different spatial frequencies: bits=4 quantstep1=5.0/(2**bits-1) bits=3 quantstep2=1.0/(2**bits-1) bits=2 quantstep3=0.6/(2**bits-1) bits=1 quantstep4=0.4/(2**bits-1) bits=0 quantstep5=8.0/(2**bits-0.99) #vermeide div. durch 0! #Zus.: 1*4 bits + 2* 3 bits + 3*2bits + 4*1 bits fuer 64 pixel, also bit pro pixel! #Quantization steps in "mask", anti-diagonals have the same quantization steps: M=np.zeros((8,8)) M[0,0]=quantstep1 M=M+ np.fliplr(np.diag([1,1],6))*quantstep2 M=M+ np.fliplr(np.diag([1,1,1],5))*quantstep3 M=M+ np.fliplr(np.diag([1,1,1,1],4))*quantstep4 M=M+ np.fliplr(np.tril(np.ones((8,8)),3))*quantstep5 print(m[0:3,0:3]) array([[ e-01, e+00, e+00], [ e+00, e+00, e+00], [ e+00, e+00, e+02]]) We then apply this matrix to each 8x8 DCT block. Each DCT coefcient is divided by the corresponding value in the quantization matrix M. Note that the coefcient in the upper left corner corresponds to the DC coefcient, and thus to the lowest spatial

19 frequency. To the right and down, the corresponding spatial frequencies are getting higher and higher, so we can quantize them more and more coarsely. We get * 4 bit for the DC coefcient, * 3 bit for the following anti-diagonal, which consists of 2 coefcients, then * 2 bit for the following 3 coefcients on the next antidiagonal, * 1 bit for the next 4 coefcients, and * 0 bit for the rest. In this way we need the following number of bits per 8x8 pixel block: 1*4 bits + 2* 3 bits + 3*2 bits + 4*1 bits=20 bits for 64 coefcients or pixels, thus only bits per pixel! For our frame, we only have to repeat this small 8x8 matrix-mask until it flls up the whole frame. We do this with the command np.kron. It implements the so-called Kronecker product. An example:

20 A= [ ] and M is our matrix mask. Then apply: np. kron(a, M)= [ M M M M ] So we can easily repeat our mask up to frame size. In our example: r8=r/8 c8=c/8 Mframe=np.kron(np.ones((r8,c8)),M); Then we quantize with the element-wise matrix division: index=np.round (X/ Mframe) and de-quantize with element-wise matrix multiplication: Xrek=indices*Mframe Calling the Python program: python videorecdctblocksidctquantmask.py Note: Despite the signifcantly lower bit rate of bits/pixel we have a signifcantly better quality than the examples with the same quantization height at 2 bits/pixel! But observe that we neglected the range clipping from using a fxed

21 number of bits in this example. Since we have used the Contrast Sensitivity Function here only qualitatively and very roughly, we can expect clear improvements with more precise application of the CSF! Example JPEG compression: JPEG stands for "Joint Photographic Expert Group", which has released the JPEG standard for image compression and encoding. This format is known from digital cameras, for example. It is also based on splitting the image into 8x8 pixel blocks, then a 2D DCT of these blocks, and then a quantization of the DCT coefcients. Our quantization mask M is called "Q" here. For a "Quality" setting of 50%, JPEG uses the following quantization mask: Note: Here we also see the smallest quantizationstep-heights at the DCT-coefcients of the small spatial frequencies, top left, with the highest accuracy. To the lower right, towards the coefcients of the higher spatial frequencies, the quantization

22 steps become larger and larger, i. e. the quantization becomes inaccurate. Note also that the number range for the quantization steps is diferent here, because an un-normalized image is assumed (pixels in the range 0 to 255), unlike in our example, where the image has been normalized, to pixel values in the range 0 to 1. (See also: Python example with JPEG quantization matrix: python videorecdctblocksidctquantjpgmask.py Note: The quality of the reconstructed video is not much better (but somewhat, in the fner details of the picture, by including more higher spatial frequencies), but now at a bit-rate that is not exactly known. The number of bits in JPEG is variable and depends on the image, because it uses Hufman coding, with variable lengths of codewords. Conversion into a sequence of indexes or bits In JPEG, after quantization of the DCT coefcients and conversion into bits, the so-called "zigzag scan" follows. The following fgure shows an 8x8 block of DCT coefcients, with the DC coefcient at the top left:

23 Thus, the coefcients are sorted by importance or bit length, and the transmission can be stoped in one place, depending on the "Quality" level (see Reimers:"DVB-Digital TV technology"). Encoder: -Y Cr Cb Color transform JPEG Processing steps: -Subsampling of Cr, Cb (4:2:0) - Apply 2D-DCT to 8x8 pixel blocks on the 3 components -Quantization using the quantization matrix -Conversion of the indexes from quantization to bits. Decoder: -Conversion of the bits into indexes -De-quantization of indices into reconstructed values

24 -Inverse 2D-DCT on the 8x8 blocks of the 3 components -Upsampling of Cr, Cb and conversion back to RGB. (See also: U. Reimers:"DVB-Digital television technology", Springer) Tips for debugging -Divide a program into small functional units (e. g. functions), which can be tested individually for themselves -Use "print" commands, starting from the front and back to the middle of a program, to "circle" in which area an error (bug) has to occur until it is found.