Information Retrieval 6. Index compression

Transcription

1 Ghislain Fourny Information Retrieval 6. Index compression Picture copyright: donest /123RF Stock Photo

2 What we have seen so far 2

3 Boolean retrieval lawyer AND Penang AND NOT silver query Input Set of documents Output Subset of documents 3

4 Standard inverted index ETH Zürich computer data CPU information retrieval

5 Search structures Hash tables Trees (B, B+) 5

6 Additional features comput* cmputer Wildcards Spell correction "Pfäffikon SZ" Phrase search 6

7 Bi-word indices (Phrase search feature) Help ETH Zurich to flexibly react to new challenges and to set new accents in the future. Index Help ETH ETH Zurich Zurich to to flexibly flexibly react react to 7

8 Positional index (phrase search feature) "ETH Zurich" Help C,1: 1 ETH C,1: 2 Zurich C,1: 3 to C,3: 4, 7, 11 flexibly C,1: 5 react C,1: 6 8

9 Trigram index (wildcard, spell correction) mpu com ran ter $co $te an$ com er$ err mpu omp put ran rra ter ute computer terran terran computer computer computer terran terran terran computer computer computer computer terran 9

10 TermIDs t1 t2 t3 t4 t5 t6 t7 1 t1 2 t1 3 3 t2 4 t2 7 1 t3 2 t3 4 1 t4 3 t4 5 2 t5 3 t5 4 1 t6 2 t6 4 3 t7 5 t

11 Blocked Sort-Based Indexing 11

12 Single-Pass In-Memory Indexing 12

13 Auxiliary Index ETH Computer Information Course Auxiliary index ETH Computer Information 5 4 Main index Course

14 Logarithmic Merging n postings 2n postings 4n postings Z 0 Z 1 I 0 I 2 14

15 Term Statistics 15

16 Number of terms 16

17 Number of terms 17

18 Notations used in the book N: number of documents T: number of tokens (non-positional postings) M: number of terms (or types if stemming/lemmatization) 18

19 Number of terms # Terms? # Tokens 19

20 Number of terms # Terms? # Tokens 20

21 Number of terms # Terms? # Tokens 21

22 Number of terms # Terms? # Tokens 22

23 Number of terms # Terms? max # Tokens 23

24 Number of terms # Terms We when it's linear # Tokens 24

25 Log-log scale (M) log # Terms We when it's linear log # Tokens (T) 25

26 Log-log scale (M) log # Terms log M = b log T + a We when it's linear log # Tokens (T) 26

27 "Exponential" growth (M) # Terms M = e a T b # Tokens (T) 27

28 Heaps' law (M) # Terms M = kt b # Tokens (T) 28

29 In practice (M) # Terms M = kt b b 1 2 # Tokens (T) 29

30 In practice (M) # Terms M = k p T # Tokens (T) 30

31 In practice (M) # Terms M = k p T 30 apple k apple 100 # Tokens (T) 31

32 Distribution of terms 32

33 Distribution of terms the: 56,271,872 were: 3,323,884 nearer: 51,456 moderate: 19,245 champion: 9400 stocks: 6,537 parallelogram: 503 pachyderm: 79 capacitance: 45 germanium: 12 sesquipedal: 7 33

34 Distribution of terms # Tokens the of and to in I was Rank 34

35 35 Distribution of terms # Tokens Rank

36 log-log scale log # Tokens log Rank 36

37 Zipf's law log Frequency = a log Rank + b 37

38 Zipf's law log Frequency = a log Rank + b log # Tokens log Rank 38

39 Zipf's law log # Tokens log Frequency = b log Rank log Rank 39

40 Zipf's law Frequency = k Rank

41 Compression techniques already covered 41

42 Compression techniques already covered Remove numbers 42

43 Compression techniques already covered Remove numbers Apple apple Case folding 43

44 Compression techniques already covered Remove numbers Apple apple Case folding and of the Remove stopwords 44

45 Compression techniques already covered Remove numbers Apple apple Case folding and of the Remove stopwords computing compute Stemming 45

46 Compression techniques already covered Remove numbers Apple apple and of the Case folding Remove stopwords This reduces the size of the dictionary! computing compute Stemming 46

47 Impact (number of terms/types) Remove numbers -2% Apple apple Case folding -17% -33% and of the Remove stopwords -0% computing compute Stemming -17% Source: Information Retrieval book 47

48 Impact (number of postings) Remove numbers -8% Apple apple Case folding -3% -42% and of the Remove stopwords -30% computing compute Stemming -4% Source: Information Retrieval book 48

49 Impact (number of tokens) Remove numbers -9% Apple apple Case folding -0% -52% and of the Remove stopwords -47% computing compute Stemming -0% Source: Information Retrieval book 49

50 Dictionary compression 50

51 Standard inverted index ETH Zürich computer data CPU information retrieval

52 Standard inverted index ETH Zürich computer data CPU information retrieval Let us start compressing the dictionary. 52

53 Status quo 53

54 Status quo: Dictionary stored as a B+ tree possess come is merely that thy upon almost be carefully is it Laertes possess should take thy time to come fair hour merely most my that thine this upon you your 54

55 Status quo: Dictionary stored as a B+ tree possess come is merely that thy upon almost be carefully is it Laertes possess should take thy time to come fair hour merely most my that thine this upon you your Pointers to postings lists 55

56 Status quo: Dictionary stored as a B+ tree possess come is merely that thy upon almost be carefully is it Laertes possess should take thy time to come fair hour merely most my that thine this upon you your Pointers to postings lists 56

57 Standard inverted index ETH Zürich computer data CPU information retrieval Let us start compressing the dictionary. 57

58 Standard inverted index ETH Zürich computer data CPU information retrieval We can then make it fit in RAM. 58

59 Approach 1: Array computer... CPU... data... ETH... information.. retrieval... Zürich

60 Approach 1: Array computer... CPU... data... ETH... information.. retrieval... Zürich bytes 4 bytes 4 bytes 60

61 Approach 1: Issue computer... CPU... data... ETH... information.. retrieval zupercalifragilisticexpialidocious 61

62 Approach 2: String computercpudataethinformationretrievalzürich 62

63 Approach 2: String computercpudataethinformationretrievalzürich bytes 4 bytes 63

64 Approach 2: String computercpudataethinformationretrievalzürich bytes 4 bytes 3 bytes 64

65 Approach 2: String computercpudataethinformationretrievalzürich bytes 4 bytes 3 bytes (+8 bytes) 65

66 Approach 3: Blocked storage 8computer3CPU4data3ETH11information9retrieval Only every k terms k 4 bytes 4 bytes bytes (+9 bytes) 66

67 No free lunch 67

68 No free lunch 68

69 No free lunch 69

70 Compromise between space and time 70

71 Binary search steps (no blocking) ETH CPU retrieval computer data information Zürich 71

72 Binary search steps (no blocking) ETH CPU One extra "memory seek" retrieval computer data information Zürich 72

73 Binary search steps (no blocking) ETH CPU retrieval computer data Two extra "memory seeks" information Zürich 73

74 Binary search steps (no blocking) ETH CPU retrieval computer data information Zürich Average: avg(0,1,2,2,1,2,2) =

75 Binary search steps (with blocking) ETH computer information CPU retrieval data Zürich 75

76 Binary search steps (with blocking) ETH computer information CPU Two extra "memory seeks" retrieval data Zürich 76

77 Binary search steps (with blocking) ETH computer information CPU retrieval data Three extra "memory seeks" Zürich 77

78 Binary search steps (with blocking) ETH computer information CPU retrieval data Zürich Average: avg(0,1,2,3,1,2,3) =

79 Approach 4: Front coding 8automata8automate9automatic10automation Only every k terms k 4 bytes 4 bytes bytes (+9 bytes) 79

80 Approach 4: Front coding 8automat*a8 e9 ic10 ion Only every k terms k 4 bytes 4 bytes bytes (less bytes) 80

81 How did we do? Collection: 960 MB Source: Information Retrieval book 81

82 How did we do? Fixed Width 11.2 MB Collection: 960 MB Source: Information Retrieval book 82

83 How did we do? Fixed Width 11.2 MB Unique string and pointers 7.6 MB Collection: 960 MB Source: Information Retrieval book 83

84 How did we do? Fixed Width 11.2 MB Unique string and pointers 7.6 MB Blocking (k=4) 7.1 MB Collection: 960 MB Source: Information Retrieval book 84

85 How did we do? Fixed Width 11.2 MB Unique string and pointers 7.6 MB Blocking (k=4) 7.1 MB Blocking and front coding 5.9 MB Collection: 960 MB Source: Information Retrieval book 85

86 Postings file compression 86

87 Standard inverted index ETH Zürich computer data CPU information retrieval

88 Standard inverted index ETH Zürich computer data CPU information retrieval We compressed this... 88

89 Standard inverted index ETH Zürich computer data CPU information retrieval Now, we want to compress this. 89

90 Standard inverted index In other words, we want to compress lists of integers 90

91 Standard storage bytes 4 bytes 4 bytes 4 bytes 4 bytes 4 bytes 91

92 Standard storage bytes 4 bytes 4 bytes 4 bytes 4 bytes 4 bytes (4 bytes = 32 bits) 92

93 Standard storage bytes 4 bytes 4 bytes 4 bytes 4 bytes 4 bytes (4 bytes = 32 bits) Numbers between 0 and 4,294,967,296 93

94 Encoding gaps bytes 4 bytes 4 bytes 4 bytes 4 bytes 4 bytes (4 bytes = 32 bits) Can we encode with less space? 94

95 Encoding gaps

96 Encoding gaps

97 Encoding gaps These are small gaps!

98 Encoding gaps

99 Encoding gaps

100 Encoding gaps

101 Encoding gaps But this only works for frequent terms! 101

102 Encoding gaps Can we have variable gap size? 102

103 Variable byte encoding 103

104 Fix-length encoding bytes 4 bytes 4 bytes 4 bytes 4 bytes 4 bytes We know exactly where the boundaries are. 104

105 Fix-length encoding bytes 4 bytes 4 bytes 4 bytes 4 bytes 4 bytes We know exactly where the boundaries are

106 Fix-length encoding bytes 4 bytes 4 bytes 4 bytes 4 bytes 4 bytes We know exactly where the boundaries are. 32 bits

107 Fix-length encoding bytes 4 bytes 4 bytes 4 bytes 4 bytes 4 bytes We know exactly where the boundaries are. Stop! 32 bits

108 Fix-length encoding bytes 4 bytes 4 bytes 4 bytes 4 bytes 4 bytes We know exactly where the boundaries are. Stop! 32 bits 32 bits

109 Variable length encodings bytes 2 bytes 4 bytes 3 bytes 5 bytes 4 bytes We do not know a priori where the boundaries are

110 Variable length codings bytes 2 bytes 4 bytes 3 bytes 5 bytes 4 bytes We do not know a priori where the boundaries are.? x bits

111 Prefix codes x bits we can deduce from the bits when to stop

112 Prefix codes: phone numbers Example Internally

113 Prefix codes: phone numbers Example Internally

114 Prefix codes: phone numbers Example Internally

115 Prefix codes: UTF-8 Character Codepoint Codepoint in binary UTF-8 (variable length) 115

116 Prefix codes: UTF-8 Character Codepoint Codepoint in binary UTF-8 (variable length) P U

117 Prefix codes: UTF-8 Character Codepoint Codepoint in binary UTF-8 (variable length) P U Less than 7 bits 117

118 Prefix codes: UTF-8 Character Codepoint Codepoint in binary UTF-8 (variable length) P U Less than 7 bits 118

119 Prefix codes: UTF-8 Character Codepoint Codepoint in binary UTF-8 (variable length) P U Less than 7 bits π U+03C

120 Prefix codes: UTF-8 Character Codepoint Codepoint in binary UTF-8 (variable length) P U Less than 7 bits π U+03C Less than 11 bits 120

121 Prefix codes: UTF-8 Character Codepoint Codepoint in binary UTF-8 (variable length) P U Less than 7 bits π U+03C Less than 11 bits 121

122 Prefix codes: UTF-8 Character Codepoint Codepoint in binary UTF-8 (variable length) P U Less than 7 bits π U+03C Less than 11 bits U+20AC

123 Prefix codes: UTF-8 Character Codepoint Codepoint in binary UTF-8 (variable length) P U Less than 7 bits π U+03C Less than 11 bits U+20AC Less than 16 bits 123

124 Prefix codes: UTF-8 Character Codepoint Codepoint in binary UTF-8 (variable length) P U Less than 7 bits π U+03C Less than 11 bits U+20AC Less than 16 bits 124

125 Variable byte encoding (here with 8 bit packets)

126 Variable byte encoding (here with 8 bit packets) continuation bit encoding on n-1 bits (here 7) 126

127 Case 1: less than 7 bits required 4 (100) 127

128 Case 1: less than 7 bits required 4 ( ) 128

129 Case 1: less than 7 bits required 4 ( )

130 Case 1: less than 7 bits required 4 ( ) = ends here 130

131 Case 2: Between 8 and 14 bits required 270 ( ) 131

132 Case 2: Between 8 and 14 bits required 270 ( ) 132

133 Case 2: Between 8 and 14 bits required 270 ( )

134 Case 2: Between 8 and 14 bits required 270 ( ) = doesn't end here = ends here 134

135 And so on and so forth

136 And so on and so forth

137 And so on and so forth = doesn't end here 0 = ends here 137

138 Variable byte encoding: example with 4 bit packets decimal 0 binary 0 variable byte encoding

139 Variable byte encoding: example with 4 bit packets decimal binary variable byte encoding

140 Variable byte encoding: example with 4 bit packets decimal binary variable byte encoding

141 Variable byte encoding: example with 4 bit packets decimal binary variable byte encoding

142 Variable byte encoding: example with 4 bit packets fits on 3 bits fits on 6 bits decimal binary variable byte encoding

143 Variable byte encoding: example with 4 bit packets fits on 3 bits fits on 6 bits decimal binary variable byte encoding

144 Variable byte encoding: example with 4 bit packets fits on 3 bits fits on 6 bits decimal binary variable byte encoding % less space

145 Variable byte encoding is a parameterized encoding xx xxxx xxxxxxxx xxxxxxxxxxxxxxxx n=2 n=4 n=8 n=16 145

146 Example (here, 8 bits)

147 Example (here, 8 bits)

148 Example (here, 8 bits)

149 Example (here, 8 bits)

150 Example (here, 8 bits)

151 Example (here, 8 bits) ,

152 No free lunch 152

153 Compromise for variable byte encoding Big packets Little compression Little overhead 153

154 Compromise for variable byte encoding Big packets Small packets Little compression Much compression Little overhead Lot of bits to manipulate 154

155 Can we compress even more? 155

156 Can we compress even more? bitwise? 156

157 Gamma encoding 157

158 Peter Elias

159 Unary code

160 Unary code ones 160

161 Unary code and a zero to mark the stop 161

162 First integers in unary code integer length (unary)

163 Example (here, 8 bits)

164 Example (here, 8 bits)

165 Example (here, 8 bits)

166 Example (here, 8 bits)

167 Gamma encoding: example

168 Gamma encoding: example 19 binary

169 Gamma encoding: example 19 binary

170 Gamma encoding: example 19 binary

171 Gamma encoding: example 19 binary Length in unary

172 Gamma encoding: example 19 binary Length in unary

173 Gamma encoding on the first integers decimal

174 Gamma encoding on the first integers decimal binary

175 Gamma encoding on the first integers decimal binary binary without leading

176 Gamma encoding on the first integers decimal binary binary without leading length

177 Gamma encoding on the first integers decimal binary binary without leading length length (unary)

178 Gamma encoding on the first integers decimal binary binary without leading length length (unary) gamma code

179 Gamma encoding on the first integers decimal binary binary without leading length length (unary) gamma code

180 180 Gamma encoding on the first integers length decimal binary length (unary) binary without leading gamma code

181 181 Gamma encoding on the first integers length decimal binary length (unary) binary without leading gamma code

182 Gamma encoding properties Variable length encoding 182

183 Gamma encoding properties Variable length encoding Prefix encoding 183

184 Gamma encoding properties Variable length encoding Prefix encoding Universal encoding 184

185 Shannon Entropy H(X) =E[I(X)] 185

186 Shannon Entropy H(X) =E[I(X)] "Amount of information" = number of bits 186

187 Shannon Entropy I(p) H(X) =E[I(X)] "Amount of information" = number of bits 0 1 p 187

188 Shannon Entropy I(p) H(X) =E[ log 2 (p X (X))] "Amount of information" = number of bits 0 1 p 188

189 Shannon Entropy H(X) =E[I(X)] = X x2x( ) p X (x) log 2 p X (x) 189

190 Shannon Entropy H(X) =E[I(X)] = X x2x( ) p X (x) log 2 p X (x) H(p) = 0 H(p) = log n 190

191 Expected length of gamma encoding E[L (X)] apple 3H(X) =3E[I(X)] one factor from optimal! 191

192 Expected length of gamma encoding E[L (X)] apple 2H(X)+1=2E[I(X)] + 1 one factor from optimal! 192

193 How much can we compress the inverted index? 193

194 Zipf's law Frequency = k Rank

195 Zipf's law (renormalized) Renormalized frequency = c Rank 195

196 Zipf's law (renormalized) Renormalized frequency = c Rank i=m X i=1 c Rank =1 196

197 Zipf's law Number of occurrences per document = Document length c Rank 197

198 Zipf's law Number of occurrences per document = Lc Rank 198

199 Zipf's law Number of postings = Number of documents Number of occurrences per documents 199

200 Zipf's law Number of postings = NLc Rank 200

201 Zipf's law Blocks with Lc terms Number of postings = NLc Rank 201

202 Zipf's law Rank = Lc Rank = 2Lc Rank = 3Lc Number of postings = NLc Rank 202

203 Zipf's law Approximations Rank = Lc Rank = 2Lc Rank = 3Lc Number of postings = NLc Rank 203

204 Zipf's law N postings Rank = Lc Rank = 2Lc Rank = 3Lc Number of postings = NLc Rank 204

205 Zipf's law N postings N 2 postings Rank = Lc Rank = 2Lc Rank = 3Lc Number of postings = NLc Rank 205

206 Zipf's law N postings N 2 postings N 3 postings Rank = Lc Rank = 2Lc Rank = 3Lc Number of postings = NLc Rank 206

207 Zipf's law Approximations N postings N 2 postings N 3 postings Rank = Lc Rank = 2Lc Rank = 3Lc Number of postings = NLc Rank 207

208 Zipf's law N/j postings 208

209 Zipf's law gap = j N/j postings 209

210 Zipf's law N postings gap =1 N 2 postings N 3 postings gap = 2 gap = 3 Rank = Lc Rank = 2Lc Rank = 3Lc Number of postings = Number of documents Lc Rank 210

211 Zipf's law N j postings gap = j Rank = (j-1)lc Rank = jlc 211

212 Zipf's law N j postings gap = j Rank = (j-1)lc Rank = jlc #bits per term N j (2 log 2(j) + 1) 212

213 Zipf's law N j postings gap = j Rank = (j-1)lc Rank = jlc #bits per term block NLc j (2 log 2 (j) + 1) 213

214 Zipf's law #bits j= X M Lc j=1 NLc j (2 log 2 (j) + 1) 214

215 Zipf's law #bits j= X M Lc j=1 2NLclog 2 (j) j 215

216 How did we do? Collection: 960 MB 216

217 How did we do? Uncompressed on 32 bits 400 MB Collection: 960 MB 217

218 How did we do? Uncompressed on 32 bits 400 MB Uncompressed on 20 bits 250 MB Collection: 960 MB 218

219 How did we do? Uncompressed on 32 bits 400 MB Uncompressed on 20 bits 250 MB Variable byte encoding (gaps) 116 MB Collection: 960 MB 219

220 How did we do? Uncompressed on 32 bits 400 MB Uncompressed on 20 bits 250 MB Variable byte encoding (gaps) 116 MB Elias γ encoding (gaps) 101 MB Collection: 960 MB 220

221 Credits This week: Chapter 5 221