Has Indexes: Cap. CS64 Lecture 6 HW Benc Table Table of M rows, Columns of different cardinalities CREATE TABLE BENCH ( KSEQ integer primary key, K5K integer not null, K5K integer not null, KK integer not null, K4K integer not null, KK integer not null, KK integer not null, K integer not null, K5 integer not null, K integer not null, K5 integer not null, K4 integer not null, K integer not null, S car(8) not null, S car() not null, S car() not null, S4 car() not null, S5 car() not null, S6 car() not null, S7 car() not null, S8 car() not null) tablespace setq storage(initial M next M ); Column K5K as 5K different values,,, 5, Column K as different values, (cardinality ) Slides based on Database Management Systems rd ed, Ramakrisnan and Gerke Table Benc is in tablespace setq create tablespace setq datafile '/ome/oracle/app/oracle/oradata/dbs/eoneil_setq.dbf' size G default storage ( initial M next M); Sows ow a disk file becomes part of te database. Oracle makes te file based on tis spec. MySQL v. 5.7 is te first version to allow tis simple way of adding a file to an Innodb database, CREATE TABLESPACE tablespace_name ADD DATAFILE 'file_name No size spec, so presumably auto-extend. Loading table benc First a C program creates a datafile benc.dat: % ead - benc.dat 688 55 574 659 89 7 45 4 4 5 45678 456789987654 456789987654 456789987654 456789987654 456789987654 456789987654 456789987654 48449 44 7988 54 8 7 4 4 5 45678 456789987654 456789987654 456789987654 456789987654 456789987654 456789987654 456789987654 956 794 9 79 87 6 98 45678 456789987654 456789987654 456789987654 456789987654 456789987654 456789987654 456789987654 Ten a bulk load topcat$ more benc.ctl load data replace into table benc fields terminated by " " (KSEQ, K5K, K5K, KK, K4K, KK, KK, K, K5, K, K5, K4, K, S, S, S, S4, S5, S6, S7, S8) Note tis builds te PK index, not clustered. Would be slower if te data file was on networked disk. Te load of benc took about one minute. Tat s MB data read in about 6 s, or about MB/s read rate. Load of benc5: 5 GB in 6 min, about 5 MB/s Vs. only 6 min to create input file wit C program. Ten add secondary indexes on some columns CREATE INDEX k5kin ON benc (k5k) storage (initial M next M) pctfree 5 tablespace setq; CREATE INDEX kkin on benc (kk) storage (initial M next M) pctfree 5 tablespace setq CREATE INDEX kkin on benc (kk) storage (initial M next M) pctfree 5 tablespace setq; CREATE INDEX kin on benc (k) storage (initial M next M) pctfree 5 tablespace setq; CREATE INDEX kin on benc (k) storage (initial M next M) pctfree 5 tablespace setq; CREATE INDEX k4in on benc (k4) storage (initial M next M) pctfree 5 tablespace setq; We could make a tablespace for tese indexes and get better performance for some queries, if we were using two disks, say. But we are using RAID over many disks.
Final Steps for Benc Table Analyze te table to get stats for te query processor SQL>exec dbms_stats.gater_table_stats( SETQ_DB, BENCH,cascade=>true); Here cascade means analyze its indexes too. Make it publicly readable: grant select on benc to public; Try it out from anoter (non-priv) account dbs()% sqlplus cs64test/ SQL> select count(*) from setq_db.benc; COUNT(*) ---------- SQL> select tablespace_name from all_tables were table_name = 'BENCH'; TABLESPACE_NAME ------------------------------ SETQ SQL> select index_name,index_type, uniqueness from all_indexes were table_name='bench'; INDEX_NAME INDEX_TYPE UNIQUENES ------------------------------ --------------------------- --------- SYS_C49 NORMAL UNIQUE K5KIN NORMAL NONUNIQUE KKIN NORMAL NONUNIQUE Overview Has-based indexes are best for equality selections Cannot support range searces, except by generating all values Static and dynamic asing tecniques exist Has indexes not as widespread as B+-Trees Some DBMS do not provide as indexes But asing still useful in query optimizers (DB Internals) E.g., in case of equality joins As for tree indexes, alternatives for data entries k* Coice ortogonal to te indexing tecnique Hasing in Memory and on Disk Te as table may be located in memory, supporting fast lookup to records on disk, or even on disk, supporting fast access to furter disk. In fact, a disk-resident as table tat is in frequent use ends up being in memory because of te memory "cacing" of disk pages in te file system. keys as table Data records Example memory memory memory typical HasMap apps memory memory disk use HasMap to old disk record locations as values memory disk disk ased files, some database tables Static Hasing Number of buckets N fixed, eac wit primary, overflow pages primary pages are allocated sequentially overflow pages may be needed wen file grows Buckets contain data entries Has value: (k) mod N = bucket for data entry wit key k (key) mod N key N- Primary bucket pages Overflow pages Static Hasing Has function is applied on searc key field Must distribute values over range... N-. (key) = (a * key + b) is a typical coice (for numerical keys) a and b are constants, cosen to tune te asing, and prime Example: (key) = 7*key + Has function for string keys? A tricky subject, easy to go wrong See Wikipedia article ttps://en.wikipedia.org/wiki/has_function Algoritm used by Perl: ttps://en.wikipedia.org/wiki/jenkins_as_function
Data entries can be full rows (Alt ()) Data entries can be (key, rid(s)) (Alt (,)) (key) mod N key (key) mod N key N- Primary bucket pages Overflow pages CLUSTERED N- UNCLUSTERED Primary pages are sequential on disk, so full table scan is fast if not too many overflow pages, or overflow pages are also sequential Is a clustered index (data records in as key order), but te clustering is not useful. pages of Data entries (from above) (Index File) (Data file) Requires data sorted in as-value order, but NOT USEFUL, so only teoretical Static Hasing Works well if we know ow many keys tere can be Ten we can size te primary page sequence properly: keep it under about alf full Can ave collisions : two keys wit same as value But wen file grows considerably tere are problems Long overflow cains develop and degrade performance Example: loader took over an our to load a big program Found it was asing using -spot as table for global symbols! One line edit solved te problem. General Solution: Dynamic Hasing, contenders described: Extendible Hasing Linear Hasing Extendible Hasing Main Idea: wen primary page becomes full, double te number of buckets But reading and writing all pages is expensive Use directory of pointers to buckets Double te directory size, and only split te bucket tat just overflowed! Directory muc smaller tan file, so doubling it is ceap Tere are no overflow pages (unless te same key appears a lot of times, i.e., very skewed distribution many duplicates) Extendible Hasing Example Insert (k)= (Causes Doubling) Directory is array of size 4 Directory entry corresponds to last two bits of as value If (k) = 5 = binary, it is in bucket pointed to by Insertion into non-full buckets is trivial Insertion into full buckets requires split and directory doubling E.g., insert (k)= LOCAL DEPTH 4* * * * 5* * * * 4* 5* 7* 9* Bucket A Bucket B Insert (k)= = But bucket for (Bucket A) is full. Need to split Bucket A Look at its as values: all are, i.e. ave last two binary digits =. But now we ll use last binary digits: 4 = new bucket = new bucket = gets left in old bucket 6 = gets left in old bucket New value: = new bucket LOCAL DEPTH 4* * * * 5* * * * 4* 5* 7* 9* Bucket A Bucket B DATA DATA
Insert (k)= (Causes Doubling) LOCAL DEPTH 4* * * Bucket A * * 5* ** Bucket B * 5* 7* 9* Bucket A (`split image' of Bucket A) LOCAL DEPTH 5* 7* 9* * Bucket A * 5* * * Bucket B * 4* * * Use last bits in split bucket! Bucket A (`split image' of Bucket A) Global vs Local Dept Global dept of directory: Max # of bits needed to tell wic bucket an entry belongs to Local dept of a bucket: # of bits used to determine if an entry belongs to tis bucket Wen does bucket split cause directory doubling? Before insert, local dept of bucket = global dept Insert causes local dept to become > global dept Directory is doubled by copying it over Use of least significant bits enables efficient doubling via copying of directory Delete: if bucket becomes empty, merge wit `split image If eac directory element points to same bucket as its split image, can alve directory Directory Doubling 6 = Wy use least significant bits in directory? It allows for doubling via copying! 6 = Extendible Hasing Properties If directory fits in memory, equality searc answered wit one I/O; oterwise wit two I/Os MB file, bytes/rec, 4K pages contains,, records (Tat s MB/( bytes/rec) = M recs) 5, directory elements will fit in memory (Tat s assuming a bucket is one page, 4KB = 496 bytes, so can old 4 bytes/( bytes/rec)= 4 recs, plus 96 bytes of eader, so Mrecs/(4 recs/bucket) = 5, buckets, so 5, directory elements) Multiple entries wit same as value cause problems! Tese are called collisions Cause possibly long overflow cains Least Significant vs. Most Significant Linear Hasing Dynamic asing sceme Handles te problem of long overflow cains But does not require a directory! Deals well wit collisions! Linear Hasing Main Idea: use a family of as functions,,,... i (key) = (key) mod( i N) N = initial number of buckets If N = d, for some d, i consists of applying and looking at te last d i bits, were d i = d + i i+ doubles te range of i (similar to directory doubling) Example: N=4, conveniently a power of i (key) = (key)mod( i N)=(key), last +i bits of key (key) = last bits of key (key) = last bits of key
Linear Hasing: Rounds During round, use and During round, use and Start a round wen some bucket overflows (or possibly oter criteria, but we consider only tis) Let te overflow entry itself be eld in an overflow cain During a round, split buckets, in order from te first Do one bucket-split per overflow, to spread out overead So some buckets are split, oters not yet, during round. Need to track division point: Next = bucket to split next Overview of Linear Hasing Bucket to be split Next Buckets tat existed at te beginning of tis round: tis is te range of Level Buckets split in tis round: If Level (searc key value) is in tis range, must use Level+ (searc key value) to decide if entry is in `split image' bucket. `split image' buckets: Note tis is a file, i.e., contiguous in memory or in a real file. Linear Hasing Properties Buckets are split round-robin Splitting proceeds in `rounds Round ends wen all N R initial buckets are split (for round R) Buckets to Next- ave been split; Next to N R yet to be split. Current round number referred to as Level Searc for data entry r : If Level (r) in range `Next to N R, searc bucket Level (r) Oterwise, apply Level+ (r) to find bucket Linear Hasing Properties Insert: Find bucket by applying Level or Level+ (based on Next value) If bucket to insert into is full: Add overflow page and insert data entry. Split Next bucket and increment Next Can coose oter criterion to trigger split E.g., occupancy tresold Split round-robin prevents long overflow cains Example of Linear Hasing End of a Round On split, Level+ is used to re-distribute entries. Level=, N=4 Next= *44* 9* 5* 5* 4*8*** *5* 7* * (Te actual contents of te linear ased file) Data entry r wit (r)=5 Primary bucket page Level= * Next= 9* 5* 5* 4*8*** *5* 7* * 44* After inserting 4* OVERFLOW 4* Insert (x) = 5 =, overflows bucket, bucket splits Level= * 9* 5* 6 8* * 4* Next= * 5* 7* * 4* 44* 5* 7*9* 4* * * OVERFLOW Level= Next= * 9* 5* 6 8* * 4* 4* 5* * 44* 5* 7* 9* 4* * * *7* OVERFLOW 5*
Advantages of Linear Hasing Linear Hasing avoids directory by: splitting buckets round-robin using overflow pages in a way, it is te same as aving directory doubling gradually Primary bucket pages are created in order Easy in a disk file, toug may not be really contiguous But ard to allocate uge areas of memory Summary Has-based indexes: best for equality searces, (almost) cannot support range searces. Static Hasing can lead to long overflow cains. Extendible Hasing avoids overflow pages by splitting a full bucket wen a new data entry is to be added to it. (Duplicates may require overflow pages.) Directory to keep track of buckets, doubles periodically. Can get large wit skewed data; additional I/O if tis does not fit in main memory. Summary (Contd.) Linear Hasing avoids directory by splitting buckets roundrobin, and using overflow pages. Overflow pages not likely to be long. Duplicates andled easily. Space utilization could be lower tan Extendible Hasing, since splits not concentrated on `dense data areas in te early part of a round. For as-based indexes, a skewed data distribution is one in wic te as values of data entries are not uniformly distributed Need a good as function! Indexes in Standards SQL9/99/ does not standardize use of indexes (BNF for SQL) But all DBMS providers support it X/OPEN actually standardized CREATE INDEX clause CREATE [UNIQUE] INDEX indexname ON tablename (colname [ASC DESC] [,colname [ASC DESC],...]); ASC DESC are just tere for compatibility, ave no effect in any DB I know of. Index as as key te concatenation of column names In te order specified Indexes in Oracle Oracle supports mainly B+-Tree Indexes Tese are te default, so just use create index No way to ask for clustered directly Clustering on PK is available via index-organized tables (IOTs) In tis case, te RID is different, affecting secondary index performance Also table cluster for co-locating data of tables often joined Hasing: via as cluster Also a form of as partitioning supported Also supports bitmap indexes Has cluster example Example Oracle Has Cluster CREATE CLUSTER trial_cluster (trialno DECIMAL(5,)) SIZE HASH IS trialno HASHKEYS ; CREATE TABLE trial ( trialno DECIMAL(5,) KEY,...) CLUSTER trial_cluster (trialno); SIZE sould estimate te max storage in bytes of te rows needed for one as key Here HASHKEYS <value> specifies a limit on te number of unique keys in use, for as table sizing. Oracle rounds up to a prime, ere. Tis is static asing.
Oracle Has Index, continued For static asing in general: rule of tumb Estimate te max possible number of keys and double it. Tis way, about alf te as cells are in use at most. Te as cluster is a good coice if queries usually specify an exact trialno value. Oracle will also create a B-tree index on trialno because it is te PK. But it will use te as index for equality searces. MySQL Indexes, for InnoDB Engine CREATE [UNIQUE] INDEX index_name [index_type] ON tbl_name (index_col_name,...) index_col_name: col_name [(lengt)] [ASC DESC] index_type: USING {BTREE HASH} Syntax allows for as index, but not supported by InnoDB. For InnoDB, index on primary key is clustered. Clustered index on PK: coose your PK wisely Available in Oracle and MySQL, as only kind of clustered B-tree index. Common PKs are ids, arbitrary, not commonly used in range queries, so not getting te good from te clustered B-tree. However, a PK is wat we say it is for a table, and doesn t need to be minimalistic, just a unique identifier. So (zipcode, custid) works as a PK and clusters te data by zipcode. Custid is a uniquifier ere. Ten useful range queries on zipcode run fast. Indexes in Practice Typically, data is inserted first, ten index is created Exception: alternative () indexes (of course!) Ten best to sort first, ten load How to sort? Use database: load, sort, dump, load for real Index bulk-loading is a good idea recall it is muc faster Delete an index DROP INDEX indexname; Guidelines: Create index if you frequently retrieve less tan 5% of te table To improve join performance, index columns used for joins Small tables do not require indexes, except ones for PKs. Compare B-Tree and Has Indexes Dynamic Has tables ave variable insert times Worst-case access time & best average access time But only useful for equality key lookups Note tere are bitmap indexes too