Lossless compression B 1 U 1 B 2 C R D! CSCI 470: Web Science Keith Vertanen

Transcription

1 Lossless compression B U B U B 2 U B A ϵ CSCI 47: Web Science Keith Vertanen C R D!

2 Lossless compression Mo7va7on Overview Rules and limits of the game Things to exploit Run- length encoding (RLE) Exploit runs of same character Huffman coding Variable- length codeword for each pajern (character) Transmit codewords plus compressed data Sec6on 5.5 2

3 Lossless compression Reduce size of a file Mo7va7on Save space while storing it Data always expands to fill available drive space Save space while transmipng it Bandwidth growing rapidly, but so are files! HD video: (92 * 8) pixels/frame * 3 frames/sec * 24 bits/pixel =.5Gbps! Lossless = get back exactly what you put in (e.g. zip) Lossly compression (stay tuned) Informa7on is lost (e.g. JPEG, MP3) 3

4 Name Value Million 6 megabyte Billion 9 gigabyte Trillion 2 terabyte Quadrillion 5 petabyte Quin7llion 8 exabyte hjp://www-.ibm.com/sobware/data/bigdata/ 4

5 Lossless compression: applica7ons Generic file compression compress, gzip, zip, bzip2, 7z, xz NTFS, HFS+, ZFS Image files GIF, PNG, TIFF Audio files Free Lossless Audio Codec (FLAC) Apple Lossless Audio Codec (ALAC) Data transmission HTTP, PPP, SSH, fax machines, v.92 modems 5

6 Compression and expansion bitstream B compressed bitstream C(B) original bitstream B Compress Expand Data we want to be smaller Smaller (hopefully) version of data Exact version of original data Compression ra8o: bits in C(B) / bits in B Example: 7 ASCII characters, 7 bits each = 9 bits Output 2 codewords, 8 bits/codeword = 96 bits Compression ra7o = 8% 6

7 "created_at":"mon Apr 4 3:54:57 + \u8ea\u5df\u96e3\u53d7 \u53ea\u6 79\u8ea\u5df\u624d\u6c2\u32","source":"\u3ca href=\" rel=\"nofollow\"\u3etwitter for iphone\u3c\/a\u3e","t runcated":false,"in_reply_to_status_id":null,"in_reply_to_status_id_str":null,"in_reply_to_user_id":null,"in_reply_to_user_id_str":null,"in_reply_to_screen_name ":null,"user":"id": ,"id_str":" ","name":"chyh Leng","screen_name":"chyhleng","location":"","url":null,"description":"272 :)Music\u266a Kpop \ ua7 Kdrama\uff2c\uff2f\uff36\uff25\uff32\u265 EverLastingFriend \u2665 Minoz \u2665","protected":false,"followers_count":96,"friends_count":39,"listed_count ":,"created_at":"wed Nov 2 3:4: ","favourites_count":82,"utc_offset":288,"time_zone":"Kuala Lumpur","geo_enabled":true,"verified":false,"sta tuses_count":4263,"lang":"en","contributors_enabled":false,"is_translator":false,"is_translation_enabled":false,"profile_background_color":"9ae4e8","profile_bac kground_image_url":" e6\/bg.gif","profile_background_tile":false,"profile_image_url":" ge_url_https":" \/ \/ ","profile_link_color":"84B4","profile_sidebar_border_color":"BDDCAD","profile_sidebar_fill_color":"DDFFCC","profile_text_color":" ","profile_use_background_image":true,"default_profile":false,"default_profile_image":false,"following":null,"follow_request_sent":null,"notifications":null," geo":null,"coordinates":null,"place":null,"contributors":null,"retweeted_status":"created_at":"mon Apr 4 3:47: + 24","id": ,"id_str" :" ","text":"\u8ea\u5df\u96e3\u53d7 \u53ea\u679\u8ea\u5df\u624d\u6c2\u32","source":"\u3ca href=\" ne\" rel=\"nofollow\"\u3etwitter for iphone\u3c\/a\u3e","truncated":false,"in_reply_to_status_id":null,"in_reply_to_status_id_str":null,"in_reply_to_user_ id":null,"in_reply_to_user_id_str":null,"in_reply_to_screen_name":null,"user":"id":7272,"id_str":"7272","name":"\u852\u7f8e\u5","screen_name":"fa nshu_choy","location":"","url":null,"description":"\u222 People live because of a dream \u222","protected":false,"followers_count":627,"friends_count":65,"li sted_count":4,"created_at":"tue Jul 7 4::4 + 22","favourites_count":46,"utc_offset":288,"time_zone":"Beijing","geo_enabled":true,"verified":false,"statuses_count":523,"lang":"zh- tw","contributors_enabled":false,"is_translator":false,"is_translation_enabled":false,"profile_background_color":"356","pr ofile_background_image_url":" 29,743,396 katie339.txt plaintext mage_url_https":" profile_image_url":" 52,583,76 katie339.txt.z compress LZW e_images\/ \/utpanlr_normal.jpeg","profile_banner_url":" "555","profile_sidebar_border_color":"FFFFFF","profile_sidebar_fill_color":"E6F6F9","profile_text_color":"333333","profile_use_background_image":true,"defaul 3,875,422 katie339.txt.zip zip DEFLATE t_profile":false,"default_profile_image":false,"following":null,"follow_request_sent":null,"notifications":null,"geo":null,"coordinates":null,"place":null,"con tributors":null,"retweet_count":,"favorite_count":5,"entities":"hashtags":[],"symbols":[],"urls":[],"user_mentions":[],"favorited":false,"retweeted":false," 23,862,52 katie339.txt.bz2 bzip2 Burrows- Wheeler lang":"zh","retweet_count":,"favorite_count":,"entities":"hashtags":[],"symbols":[],"urls":[],"user_mentions":["screen_name":"fanshu_choy","name":"\u852\u 7f8e\u5","id":7272,"id_str":"7272","indices":[3,5]],"favorited":false,"retweeted":false,"filter_level":"medium","lang":"zh" 2,84,286 katie339.txt.lzma2.7z 7- zip LZMA2 "created_at":"mon Apr 4 3:54: ","id": ,"id_str":" ","text":"hallo","source":"web","truncated":false,"in_reply_t o_status_id":null,"in_reply_to_status_id_str":null,"in_reply_to_user_id":null,"in_reply_to_user_id_str":null,"in_reply_to_screen_name":null,"user":"id": ,579,65 katie339.txt.7z 7- zip LZMA 73,"id_str":" ","name":"geilespuitfles","screen_name":"geilespuitfles","location":"","url":null,"description":null,"protected":false,"followers_count" :697,"friends_count":35,"listed_count":2,"created_at":"Thu Aug 9:8:8 + 23","favourites_count":,"utc_offset":null,"time_zone":null,"geo_enabled":f 2,32,68 katie339.txt.xz xz LZMA2 alse,"verified":false,"statuses_count":692,"lang":"nl","contributors_enabled":false,"is_translator":false,"is_translation_enabled":false,"profile_background_col or":"cdeed","profile_background_image_url":" 2,27,44 katie339.txt.ppmd.7z 7- zip PPMd om\/images\/themes\/theme\/bg.png","profile_background_tile":false,"profile_image_url":" e947fe4bf5d74f77_normal.jpeg","profile_image_url_https":" al.jpeg","profile_link_color":"84b4","profile_sidebar_border_color":"cdeed","profile_sidebar_fill_color":"ddeef6","profile_text_color":"333333","profile_use_ background_image":true,"default_profile":true,"default_profile_image":false,"following":null,"follow_request_sent":null,"notifications":null,"geo":null,"coordi nates":null,"place":null,"contributors":null,"retweet_count":,"favorite_count":,"entities":"hashtags":[],"symbols":[],"urls":[],"user_mentions":[],"favorite d":false,"retweeted":false,"filter_level":"medium","lang":"et" "delete":"status":"id": ,"user_id":337896,"id_str":" ","user_id_str":"337896" "created_at":"mon Apr 4 3:54: ","id": ,"id_str":" ","text":"@leeaoilove \u342\u34a\u344\u369\u3fc\u357\u 35f\uff","source":"\u3ca href=\" rel=\"nofollow\"\u3etwitter for iphone\u3c\/a\u3e","truncated":false,"in_rep ly_to_status_id": ,"in_reply_to_status_id_str":" ","in_reply_to_user_id": ,"in_reply_to_user_id_str":" ","in _reply_to_screen_name":"leeaoilove","user":"id": ,"id_str":" ","name":"\u35\u344\u3fc\u35f","screen_name":"keeita_423","location":""," url":null,"description":"\u9cf4\u6d7728 \u3c6\u3cb\u3b9\u9e8 \u9cf4\u6d77\u36e\u4eba\u3d5\u3a9\u3ed\u3fc\u357\u366\u3fc\uff","protected":false,"f ollowers_count":2,"friends_count":237,"listed_count":,"created_at":"wed Dec 4 4:4: ","favourites_count":78,"utc_offset":null,"time_zone":null,"geo_enabled":true,"verified":false,"statuses_count":336,"lang":"ja","contributors_enabled":false,"is_translator":false,"is_translation_enabled":false,"profil e_background_color":"cdeed","profile_background_image_url":" :\/\/abs.twimg.com\/images\/themes\/theme\/bg.png","profile_background_tile":false,"profile_image_url":" 626\/XG8Azfho_normal.jpeg","profile_image_url_https":" ttps:\/\/pbs.twimg.com\/profile_banners\/ \/ ","profile_link_color":"84b4","profile_sidebar_border_color":"cdeed","profile_sidebar_fill_col or":"ddeef6","profile_text_color":"333333","profile_use_background_image":true,"default_profile":true,"default_profile_image":false,"following":null,"follow_req uest_sent":null,"notifications":null,"geo":null,"coordinates":null,"place":null,"contributors":null,"retweet_count":,"favorite_count":,"entities":"hashtags" :[],"symbols":[],"urls":[],"user_mentions":["screen_name":"leeaoilove","name":"\u342\u34a\u374\u3c\u37e\u393","id": ,"id_str":" ","in dices":[,2]],"favorited":false,"retweeted":false,"filter_level":"medium","lang":"ja"

8 "A second aspect of the present inven7on which further enhances its ability to achieve high compression percentages, is its ability to be applied to data recursively. Specifically, the methods of the present inven7on are able to make mul7ple passes over a file, each 7me further compressing the file. Thus, a series of recursions are repeated un7l the desired compression level is achieved." "the direct bit encode method of the present inven7on is effec7ve for reducing an input string by one bit regardless of the bit pajern of the input string." 8

9 Universal data compression? Reality: No algorithm can compress every bitstream Proof (by contradic7on) Suppose you have a universal compressor U Given bitstream B, use U to compress to smaller B Compress B to get smaller B 2 Con7nue un7l bitstream is of size Thus all bitstream can be compressed to bits! Proof 2 (coun7ng) Suppose you can compress all - bit strings 2 possible bit strings with bits How many possible shorter encodings, 999 bits? (# of bit numbers) + (# of 2 bit numbers) + + (# of 999 bit numbers) = 2 - Thus fewer than the 2 we need for unique mapping B U B U B 2 U ϵ 9

10 24 x 768 = 786,432 bits java PictureDump < bits.bin Compressed with standard compression u7li7es My top- secret method! 98,34 bits.bin 98,346 bits.bin.gz 98,57 bits.bin.zip 99,8 bits.bin.bz2 98,368 bits.bin.xz 232 bits.bin.kdv 232 * 8 / 786,432.24% of original! public class RandomBits public static void main(string [] args) int x = ; for (int i = ; i < ; i++) x = x * ; BinaryStdOut.write(x > ); BinaryStdOut.close();

11 Another set of 24 x 768 = 786,432 bits What is the op7mal compressor for this image? Undecideable! In fact this image is completely random. java PictureDump < bin,48, bin,53, bz2,48, gz,49, zip

12 Redundancy in English How redundant is wrijen English? Yet aoccdrnig to a sudty at Cmabrigde Uinerv7sy, it deosn t mjaer in waht oredr the ljeers in a wrod are, the olny iprmoetnt 7hng is taht the frist and lsat ljeer be at the rghit pclae. The rset can be a Joal mses and you can sitll raed it wouthit a porbelm. Tihs is bcuseae the huamn mnid deos not raed ervey lteter by istlef, but the wrod as a wlohe. Answer: very! Shannon es7mate: bits per lejer 2

13 Approaches to compression Exploit + of the following: ) Small alphabets 2) Long sequences of iden7cal bits 3) Frequently used characters 4) Long reused bit sequences (next 7me) We'll look at an example of each Including Java implementa7on Using support classes for: Binary file input/output Data structures 3

14 Reading and wri7ng binary data public class BinaryStdIn boolean readboolean() // Read bit of data, return as a boolean value char readchar() // Read 8 bits of data, return as a char value char readchar(int r) // Read r bits of data, return as a char value // Read r bits for byte, short, int, long, double boolean isempty() // Is the bitstream empty? void close() // Close the bitstream public class BinaryStdOut void write(boolean b) // Write the specified bit void writechar(char c) // Write the specified 8- bit char void write(char c, int r) // Write r least significant bits of char c // Write r LSB of byte, short, int, long, double void close() // Close the bitstream 4

15 Visualizing a bitstream How to view a bitstream? % more abra.txt ABRACADABRA! Bitstream as characters % java BinaryDump 6 < abra.txt 4 bits Bitstream represented by and 's % java HexDump 4 < abra.txt a 4 bits Bitstream represented by 2- digit hex numbers % java PictureDump 6 6 < abra.txt Bitstream as pixels in a picture 5

16 Method : Small alphabets public class OutputDate public static void main(string [] args) int month = 2; int day = 3; int year = 999; int mode = Integer.parseInt(args[]); if (mode == ) System.out.print(month+"/"+day+"/"+year); else if (mode == ) BinaryStdOut.write(month); BinaryStdOut.write(day); BinaryStdOut.write(year); BinaryStdOut.close(); else BinaryStdOut.write(month, 4); BinaryStdOut.write(day, 5); BinaryStdOut.write(year, 2); BinaryStdOut.close(); 2 / 3 / % java OutputDate java BinaryDump 8 8 bits Mode : output as ASCII text Mode : output four 32- bit ints Mode 2: output variable length bit fields 6

17 public class OutputDate public static void main(string [] args) int month = 2; int day = 3; int year = 999; int mode = Integer.parseInt(args[]); if (mode == ) System.out.print(month+"/"+day+"/"+year); Method : Small alphabets else if (mode == ) BinaryStdOut.write(month); BinaryStdOut.write(day); BinaryStdOut.write(year); BinaryStdOut.close(); else BinaryStdOut.write(month, 4); BinaryStdOut.write(day, 5); BinaryStdOut.write(year, 2); BinaryStdOut.close(); % java OutputDate java BinaryDump bits Mode : output as ASCII text Mode : output four 32- bit ints Mode 2: output variable length bit fields 7

18 public class OutputDate public static void main(string [] args) int month = 2; int day = 3; int year = 999; int mode = Integer.parseInt(args[]); if (mode == ) System.out.print(month+"/"+day+"/"+year); Method : Small alphabets else if (mode == ) BinaryStdOut.write(month); BinaryStdOut.write(day); BinaryStdOut.write(year); BinaryStdOut.close(); else BinaryStdOut.write(month, 4); BinaryStdOut.write(day, 5); BinaryStdOut.write(year, 2); BinaryStdOut.close(); Compressing by using a small alphabet: e.g. use 2- bit code for nucleo8des A, C, T, G % java OutputDate 2 java BinaryDump bits Padding since we must end on a byte boundary. Mode : output as ASCII text Mode : output four 32- bit ints Mode 2: output variable length bit fields 8

19 Method 2: Long sequences Run Length Encoding (RLE) Exploits simple form of redundancy Long runs of same bit value Store the count using 8- bits (- 255) Alternate between and If >255, put run of length of other bit, con7nue 9

20 public class RunLength private static final int R = 256; // Maximum run- length count private static final int lgr = 8; // Number of bits per count public static void compress() char run = ; boolean old = false; while (!BinaryStdIn.isEmpty()) boolean b = BinaryStdIn.readBoolean(); if (b!= old) else BinaryStdOut.write(run, lgr); run = ; old =!old; if (run == R- ) run++; // Start out with - bit // Did the bit value change? // Write out the count for completed run // We have reached 255, time to output BinaryStdOut.write(run, lgr); // Write run of 255 run = ; BinaryStdOut.write(run, lgr); // Write a run of of the other bit BinaryStdOut.write(run, lgr); BinaryStdOut.close(); 2

21 public static void expand() boolean b = false; while (!BinaryStdIn.isEmpty()) int run = BinaryStdIn.readInt(lgR); // Read 8- bit count from stdin for (int i = ; i < run; i++) BinaryStdOut.write(b); // Write - bit to stdout b =!b; BinaryStdOut.close(); // Pads s to get to byte boundary public static void main(string[] args) if (args[].equals("- ")) compress(); else if (args[].equals("+")) expand(); else throw new RuntimeException("Illegal command line argument"); 2

22 % java BinaryDump 32 < q32x48.bin 536 bits bit- run = 79 () bit- run = 7 () bit- run = 22 () bit- run = 5 ()... % java RunLength - < q32x48.bin java BinaryDump bits Compression ratio: 44 / 536 = 74% 22

23 ABCDEFGHIJKLMNOPQRSTUVWXYZABABABABABABBABABBBABABABBA BBABABABABBBBBBABABABABABAABABABBABBABABABBBABABABABA Exploi8ng small alphabet: BABABABAABABABBABABABABABBABAABBABABABABAABABBBABABAB 55 columns x 8 rows = 99 characters ABABAAABABABABABAABABABABABBBABABABABAAABABABABABBABA 26 possible lejers, A- Z BABABABABABAAABABAABBABABAAABBBABABBABAAAAABABABABAAA 26 < 2 5 = 32 BABABABABABABABBBABABABABAAABAABABABABABBABABABABABAB 5 bits/character * 99 characters = 495 bits ABAAABABABABABABABABABABABABABABABABABABBABABBBABABAB ABABABABBBABABAAAAAAABAAAAAAABAABABABAAAAAABABBABABAB BABABAAABABBBBBBBBBAABABABBBBBBBBBBABABABABABABABBBAB Huffman coding: ABABABABABABABABABABBBABABABBABBBABBBABABABABBABABABB Key idea: Use shorter codewords for common characters ABABABABAAABAABABABABABBABBABBAABBBABABBBAABABABBAABA BABBABABAABABABABABABABAAABABABABABABABABABABABABABAB Different number of bits used to encode different characters ABABABABABABBABABBBABABABABABABABBBABBABAAAAAAABBAAAB ABABABABABABABABABABABABABABABBBABABABBABABBAABAABABA ABABABABABAABABBABABABABABABAABABABABABABABABABABBABA ABABBBBABABABBABBBABABBABABAAAABBAABBABABBABABABABABB ABBABBBBABAABABBABABABABAABBBABABABABAAABABABABABBABA ABAABABABABABABABAAABABABABABABBBABABAAABABAAAABBABBA 23

24 Method 3: Frequency of characters Variable- length prefix- free codes Map from characters to bit strings (codewords) Choose codewords so none is a prefix of another ABRACADABRA! Text to be compressed key value key value! A B C D R A B RA CA DA B RA! Scheme : 3 bits! A B C D R A B R A C A D A B R A! Scheme 2: 29 bits 24

25 Trie representa7on Represent prefix- free code as a binary trie Characters at leafs Codeword is path from root to leaf = leb, = right key value! A B C D B A R A B R A C A D A B R A! 29 bits C R D! 25

26 Using the trie Compression Start at leaf of target character Follow path to root, print bits in reverse order or create a symbol table Expansion Start at root Go leb if bit =, go right if bit = If leaf node, output character and return to root key value! A B C D B A R A B R A C A D A B R A! 29 bits C R D! 26

27 private static class Node implements Comparable<Node> private final char ch; private final int freq; private final Node left, right; // Initialize a new Node Node(char ch, int freq, Node left, Node right) this.ch = ch; this.freq = freq; this.left = left; this.right = right; // Is this node a leaf? private boolean isleaf() return (left == null && right == null); // Compare Nodes by frequency public int compareto(node that) return this.freq - that.freq; 27

28 Expansion public static void expand() // Read in the encoding trie Node root = readtrie(); // Number of bytes to write int length = BinaryStdIn.readInt(); // Decode using the Huffman trie for (int i = ; i < length; i++) Node x = root; // Expand codeword for the ith character while (!x.isleaf()) boolean bit = BinaryStdIn.readBoolean(); if (bit) x = x.right; else x = x.left; BinaryStdOut.write(x.ch); BinaryStdOut.flush(); Expansion Start at root Go leb if bit=, go right if bit= If leaf node, output character and return to root B C R D! A 28

29 TransmiPng the trie Trie needed in order to expand Must be sent along with the data Causes overhead, but small if message is long Write preorder traversal of trie Mark leaf and internal nodes with a bit B C R D! A private static void writetrie(node x) if (x.isleaf()) BinaryStdOut.write(true); BinaryStdOut.write(x.ch); return; BinaryStdOut.write(false); writetrie(x.left); writetrie(x.right); B C R D! A 29

30 Reading the trie Reconstruc7ng from preorder traversal Use / bits to decide if internal or leaf node B C R D! A private static Node readtrie() boolean isleaf = BinaryStdIn.readBoolean(); if (isleaf) return new Node(BinaryStdIn.readChar(), -, null, null); else return new Node('\', -, readtrie(), readtrie()); B C R D! A 3

31 Building the trie Can we always find op7mal prefix- free code? Yes! Discovered by David Huffman while a PhD student Huffman algorithm: Count frequency of each char in input Create forest of leaf nodes for each char Each leaf weighted according to its frequency Repeat un7l single trie: Select 2 tries with min sum of weights Merge into trie with sum of weights Provably op7mal This trie isn't the op7mal one for ABRACADBRA! 3

32 ABRACADABRA! 6) Join last two tries 2 5) Join two with sum = 7 4) Join two with sum = ) Join two with sum = 3 2) Join two with sum = 2 (in a 7e, exact choice arbitrary) 3 2 ) Create leafs with weight = frequency in en7re text Codewords: A 5 B 2 R 2 D C! 32

33 ABRACADABRA! % java Huffman - < abra.txt java BinaryDump 32 2 bits trie size of text 65 = A 68 = D 33 =! 67 = C 82 = R 66 = B int = 2 A B R D C! compressed data 33 A B R A C A D A B R A! padding 33

34 Lossless compression Summary Universal compression: impossible Op7mal data compression: undecidable Exploi7ng: Small alphabets Use only as many bits as needed to represent data Repeated symbols Run length encoding (RLE) Frequency of symbols Prefix- free codes Huffman coding 34