Data Warehousing (Special Indexing Techniques)

Transcription

1 Data Warehousing (Special Indexing Techniques) Naveed Iqbal, Assistant Professor NUCES, Islamabad Campus (Lecture Slides Weeks # 13&14)

2 Special Index Structures Inverted index Bitmap index Cluster index Join indexes 2

3 Sample table Student Name Age Campus Tech s1 amir 20 Lahore Elect s2 javed 20 Islamabad CS s3 salim 21 Lahore CS s4 imran 20 Peshawar Elect s5 majid 20 Karachi Telecom s taslim 25 Karachi CS s7 tahir 21 Peshawar Telecom s8 sohaib 2 Peshawar CS s9 afridi 19 Lahore CS 3

4 Inverted Index: Concept Indexing used in search engine involves very infrequent updates, same being true for a DWH. Works like the index at the back of the book i.e. instead of scanning the records, the records are identified by a list (index). Instead of listing each record, the record ID are listed in the index that point to the records. An inverted index takes O(1) time for search as opposed to O(log n) for B-Tree, but the cost is very high update time. Storage space for inverted index can be very large, but the structure can be compressed. 4

5 Inverted Index: Example-1 D1: M. Aslam BS Computer Science Lahore Campus D2: Sana Aslam of Lahore MS Computer Engineering with GPA 3.4 Karachi Campus Inverted index for the documents D1 and D2 is as follows: 3.4 [D2] Aslam [D1, D2] BS [D1] Campus [D1, D2] Computer [D1, D2] Engineering [D2] GPA [D2] Karachi [D2] Lahore [D1, D2] M. [D1] MS [D2] of [D2] Sana [D2] Science [D1] with [D2] 5

6 ... Inverted Index: Example r4 r18 r34 r35 r5 r19 r37 r40 RID name age Campus r4 amir 20 Elect r18 javed 20 CS r19 salim 21 CS r34 imran 20 Elect r35 majid 20 Telecom r3 taslim 25 CS r5 tahir 21 Telecom r41 sohaib 2 CS B-tree Index inverted index r500 afridi 19 CS data records

7 Inverted Index: Query Query: Get students with age = 20 and tech = telecom List for age = 20: r4, r18, r34, r35 List for tech = telecom : r5, r35 Answer is intersection: r35 7

8 Bitmap Index: Concept Index on a particular column Index consists of a number of bit vectors or bitmaps Each value in the indexed column has a corresponding bit vector (bitmaps) The length of the bit vector is the number of records in the base table The i th bit is set to 1 if the i th row of the base table has the value for the indexed column 8

9 Bitmap Index: Example The index consists of bitmaps, with a column for each unique value: Index on City (larger table): SID Islamabad Lahore Karachi Peshawar Index on Tech (smaller table): SID CS Elect Telecom

10 Bitmap Index: Query Query: Get students with age = 20 and campus = Lahore List for age = 20: List for campus = Lahore : Answer is AND : Good if domain cardinality is small Bit vectors can be compressed Run length encoding 10

11 Bitmap Index: Compression Case-1 Case-2 Case-3 Basic Concept INPUT 14#04#14#0#15#0#15 OUTPUT INPUT 11#01#11#01#11#01#11#01# OUTPUT INPUT 117#017 OUTPUT 11

12 Bitmap Index: More Queries Which students from Lahore are enrolled in CS? How many students are enrolled in CS? 12

13 Bitmap Index: Advantage Very low storage space Reduction in I/O, just using index Counts & Joins Low level bit operations 13

14 Bitmap Index: Disadvantage Locking of many rows Low cardinality Keyword parsing Difficult to maintain Need re-organization when relation sizes change (new bitmaps) 14

15 Cluster Index: Concept A cluster index defines the sort order on the base table Ordering may be strictly enforced (guaranteed) or opportunistically maintained At most one cluster index defined per table Cluster index may include one or multiple columns Reduced I/O 15

16 Cluster Index: Example Student Name Age Campus Tech s9 afridi 19 Lahore CS s1 amir 20 Lahore Elect s2 javed 20 Islamabad CS s4 imran 20 Peshawar Elect s5 majid 20 Karachi Telecom s3 salim 21 Lahore CS s7 tahir 21 Peshawar Telecom s taslim 25 Karachi CS s8 sohaib 2 Peshawar CS Student Name Age Campus Tech s9 afridi 19 Lahore CS s2 javed 20 Islamabad CS s3 salim 21 Lahore CS s taslim 25 Karachi CS s8 sohaib 2 Peshawar CS s1 amir 20 Lahore Elect s4 imran 20 Peshawar Elect s5 majid 20 Karachi Telecom s7 tahir 21 Peshawar Telecom Cluster indexing on AGE One indexing column at a time Cluster indexing on TECH 1

17 Cluster Index: Issues Works well when a single index can be used for the majority of table accesses / queries. Selectivity requirements for making use of a cluster index are much less severe than for a non-clustered index. High maintenance cost to keep sorted order or frequent re-organizations to recover clustering factor. Optimizer must keep track of clustering factor (which can degrade over time) to determine optimal execution plan. 17

18 Join Index: Example The rows of the table consist entirely of such references, which are the RIDs of the relevant rows. PROGRAM id name NoS jindex p1 BS 10 r1,r3,r5,r p2 MS 5 r2,r4 join index CAMPUS rid progid CID date NoS r1 p1 c r2 p2 c r3 p1 c r4 p2 c2 1 8 r5 p1 c r p1 c

19 Materialized views Runtime views Pre-computed complex views but stored in physical tables Re-Materialization and Maintenance Issues 19

20 Joining Techniques

21 Background Used to retrieve data from multiple tables Joins used frequently, hence lot of work on improving or optimizing them is required Simplest join that works in most cases is nested loop join but result in quadratic time complexity Tables identified by FROM clause and condition by WHERE clause We will cover: Nested Loop Join Sort Merge Join Hash Based Join 21

22 Nested-Loop Join Typically used in OLTP environment Works: When small subset of data is to be used With efficient access path in the database (PK) In DSS environment, we deal with VLDB and large sets of data Typical queries do not involve PK and used multiple columns. Hence both conditions are NOT met We will still discuss so that you can appreciate the specialized techniques and 22

23 Nested-Loop Join: Code FOR i = 1 to N DO BEGIN /* N rows in T1 */ IF i th row of T1 qualifies THEN BEGIN For j = 1 to M DO BEGIN /* M rows in T2 */ IF the i th row of T1 matches to j th row of T2 on join key THEN BEGIN IF the j th row of T2 qualifies THEN BEGIN produce output row END END END END END 23

24 Nested-Loop Join: Working Example What is the average GPA of undergraduate male students? Student Personal Table Student Academic Table Search Search Search Results Results Results For each qualifying row of Personal table, Academic table is examined for matching rows. 24

25 Nested-Loop Join: Order of Tables If the outer loop executes R times and the inner loop executes S times, then the time complexity will be O(RS). The time complexity should be independent of the order of tables i.e. O(RS) is same as O(SR). However, in the context of I/Os, the order of tables does matter. Along-with this, the relationship between the number of qualifying rows/blocks between the two tables ALSO does matter. 25

26 Nested-Loop Join: Cost Formula Join cost = Cost of accessing Table_A + # of qualifying rows in Table_A Blocks of Table_B to be scanned for each qualifying row OR Join cost = Blocks accessed for Table_A + (Blocks accessed for Table_A Blocks accessed for Table_B) 2

27 Nested-Loop Join: Cost of reorder Table_A = 500 blocks and Table_B = 700 blocks. Qualifying blocks for Table_A QB(A) = 50 Qualifying blocks for Table_B QB(B) = 100 Join cost A&B = = 35,500 I/Os Join cost B&A = = 50,700 I/Os i.e. an increase in I/O of about 43%. 27

28 Nested-Loop Join: Variants Naïve Nested Loop Join No indexing, entire table scanner Index Nested Loop Join Index is there and is exploited Temporary Index Nested Loop Join Index is built as part of the query plan and subsequently dropped Query Optimizer Before selecting the most appropriate join algorithm / technique 28

29 Sort-Merge Join Requires joined tables to be sorted on columns that are identified by the equality in the WHERE clause of the join predicate. The query optimizer scans for (cluster) index on the columns which are part of the join, if one exists it is used. Sorted tables are merged based on the join columns. In the absence of index, tables are sorted on the columns to be joined, resulting in what is called a cluster index. In rare cases, there are multiple equalities in the WHERE clause, in such a case, merge columns are taken from only some of the given equality clauses. 29

30 Sort-Merge Join: Process Sort Table_A and Table_B on the join column in ascending order, then scan them to do a merge (on join column), and output result tuple/rows. Proceed with scanning of Table_A until current A_tuple <= current B_tuple, then proceed scanning of Table-B until current B_tuple <= current A_tuple; do this until current A_tuple = current B_tuple At this point, all A_tuples with same value in Ai (current A_group) and all B_tuples with same value in Bj (current B-group) match; output <a, b> for all pairs of such tuple/records. Update pointers, resume scanning Table-A and Table_B. Table_A is scanned once; each B_group is scanned once per matching Table_A tuple. (Multiple scans of a B group are likely to find needed pages in buffer) Cost: M log M + N log N + (M+N) The cost of scanning is M+N, could be M*N (very unlikely!) 30

31 Table_A Table_B Table_A Table_B Table_A Table_B Sort-Merge Join Example

32 Sort-Merge Join: Note Sort Merge Join itself is very fast. Can be expensive for frequent sort operations i.e. the contents of the table change often resulting in deterioration of the sort order. However, even for large data volume, presorted data can be obtained from existing B- Tree. For such a case, sort merge join is often the fastest available join algorithm. 32

33 Hash-Based Join Suitable for the VLDB environment as they are useful for joining large data sets or tables. The choice which table first gets hashed plays a pivotal role in the overall performance of the join operation. The optimizer decides by using the smaller of the two tables (say) Table_A (build table) or data sources to build a hash table in the main memory on the join key used in the WHERE clause. It then scans the larger table (say) Table_B (probe table) and probes the hashed table to find the joined rows. The joined rows are identified by collisions i.e. collisions are good in case of hash join. 33

34 Hash-Based Join: Example Original Relation MAIN MEMORY 1 Join Result... Table_A Table_B hash function h N M N Disk Table_A in main memory Table_B on disk Disk 34

35 Hash-Based Join: Large small Table Smaller of the two tables may grow too big to fit into the main memory. Optimizer breaks it up by partitioning, such that a partition can fit in the main memory. However, not that simple, as qualifying rows of both tables have to fall in the corresponding partition pairs. In such a case, hash join proceeds in several steps. Each step has a build phase and probe phase. 35

36 Hash-Based Join: Large small Table Initially, two tables are entirely consumed and partitioned (using a hash function on the hash keys) into multiple partitions. Using the hash function on the hash keys (based on the predicates in the WHERE clause) guarantees that any two joining records fall in smaller pair of partitions. Hence the task of joining two large tables gets reduced to multiple, but smaller, instances of the same tasks. 3

37 Hash-Based Join: Partition Skew Partition skew can become a problem. Hashing works under the assumption of uniformity of data distribution. However, an attribute may not be uniformly distributed within the relation. Some buckets may then contain more records than other buckets. In extreme case, the corresponding bucket may not fit in the main memory. Consequently hash based join degrades into nested loop join. Solution: Make available other hash functions to be chosen by the optimizer; that better distribute the input. 37