Citation for published version (APA): Ydraios, E. (2010). Database cracking: towards auto-tunning database kernels

Transcription

1 UvA-DARE (Digital Academic Repository) Database cracking: towards auto-tunning database kernels Ydraios, E. Link to publication Citation for published version (APA): Ydraios, E. (2010). Database cracking: towards auto-tunning database kernels General rights It is not permitted to download or to forward/distribute the text or part of it without the consent of the author(s) and/or copyright holder(s), other than for strictly personal, individual use, unless the work is under an open content license (like Creative Commons). Disclaimer/Complaints regulations If you believe that digital publication of certain material infringes any of your rights or (privacy) interests, please let the Library know, stating your reasons. In case of a legitimate complaint, the Library will make the material inaccessible and/or remove it from the website. Please Ask the Library: or a letter to: Library of the University of Amsterdam, Secretariat, Singel 425, 1012 WP Amsterdam, The Netherlands. You will be contacted as soon as possible. UvA-DARE is a service provided by the library of the University of Amsterdam ( Download date: 17 Jan 2019

2 Contents 1 Introduction The Information Era Database Management Systems Query Optimization Physical Design: Something is not Right! Self-organization DB Cracking: Towards DBA-free Systems The Basics Thinking Outside the Box Contributions Published Papers Thesis Outline (How to read) Related Work and Background Row-oriented Storage and Data Access Column-stores Early Tuple Reconstruction Late Tuple Reconstruction Updates Cracking on Column-stores Future Column-store Research C-Store Cracking vs C-store Indices Cracking vs Indices Auto-tuning Tools What-if Analysis

3 6 CONTENTS Cracking vs Auto-tuning Tools Materialized Views Cracking vs Views Partial Indexing Join Processing Crack Joins vs Traditional Joins Distributed and Peer-to-peer DBMSs The MonetDB System Summary Selection Cracking Introduction Contributions Outline Selection Cracking A Simple Example The Cracker Index Data Properties Self-organization Challenges How to Crack: The Cracking Algorithms Cracking Columns The Algorithms Cracking Only the Boundary Pieces Self-organization When to Crack: The Operators and Plans Crack While Processing Crack Operators The crackers.select Operator The crackers.rel select Operator Complexity and Expected Behavior Tuple Reconstruction Experimentation Select Operator Benchmark Crack vs Sort Scalability Selectivity Full Query Evaluation Summary

4 CONTENTS 7 4 Updates Introduction Contributions Outline When to Update: Self-organizing Updates Updating On Demand Update-aware Select Operator Insertions General Discussion Pending Insertions Column Discarding the Cracker Index Cracker Index Maintenance Shuffling a Cracker Column Shuffling From The Top of a Column Shuffling From The Bottom of a Column Merge-like Algorithms MCI MGI MRI Deletions Updates Experimental Analysis Basic Insights Effect of the Number of Pending Insertions Selectivity effect Long Query Sequences Performance under Deletions Full Updates Performance Summary Sideways Cracking Introduction The Ultimate Access Pattern Contributions Outline The Tuple Reconstruction Problem Example Queries Experimental Analysis Exp1: Basic Performance

5 8 CONTENTS Exp2: Multiple Tuple Reconstructions Exp3: Reordering intermediate results Exp4 & Exp5: Multiple Selections Summary Sideways Cracking Basic Definitions Multi-projection Queries The Problem: Non-aligned Cracker Maps The Solution: Adaptive Alignment Multi-selection Queries The Problem: Non-aligned Map Sets The Solution: Use a Single Aligned Set Map Set Choice: Self-organizing Histograms Disjunctive Queries Complex Queries Updates Experimental Analysis Exp1: Varying Tuple Reconstructions Exp2: Varying Selectivity Exp3: Join Queries Exp4: Skewed Workload Exp5: Updates Partial Sideways Cracking Partial Maps Basic Definitions Creating Chunks Storage Management Dropping the Head Column Chunk-wise Processing Partial Alignment Updates Experimental Analysis Handling Storage Restrictions Adaptation No Overhead in Query Sequence Cost Adapting to Frequently Changing Workloads Alignment Improvements TPC-H experiments Summary

6 CONTENTS 9 6 Crack Joins Introduction Contributions Outline The Basic Crack Joins The Algorithms The Simple Join The Cutter Join The Smart Cutter Join Updates Experimental Analysis Experimental Set-up Basic Observations Simple Vs. the Cutters Selection Improvements Varying Column Sizes Various Scenarios The Cache Conscious Crack Join Long Query Sequences Cache Conscious Crack Join Balancing the Costs Avoid Restricting Physical Pieces Using Super Pieces Tuning the Super Piece Size The Active Crack Join Basic Observations Passive Cracking Exploit Alignment Problem Generalization More Crack Operators Active Cracking Candidate Pieces Splitting Strategies Multi-cracking and Radix-partitioning Multi-Crack Sorting Crack Pieces Foreign key Vs. Arbitrary Joins Experimental Analysis

7 10 CONTENTS Basic Performance Crack Join Vs Radix and Merge Join Varying Column Sizes Various Scenarios Long Sequences Varying Skew Updates Sorting and Multi-cracking Mixed Sequences Mixed Join Pairs Beyond the Memory Bounds Complete Queries Summary Adaptive Indexing Hybrids Introduction Contributions Outline Adaptive Merging Motivation for Hybrid Designs The Hybrid Algorithm Data Structures Algorithm The First Query Rest of the Query Sequence Hybrid First Query Cracking Insights Updates and Multi-column Indexes Updates Multi-column Indexes Experimental Analysis Implementation Details Experimental set-up Random Workloads Scan and Sort Adaptive Merging Cracking Hybrid Focused Workloads

8 CONTENTS 11 Jump Patterns Zoom Patterns Exploration Patterns Discussion Summary The Big Picture 225 What we Did A New Challenging Research Area Cracking Operators Hardware and Data Sensitive Cracking Cache-conscious and Opportunistic Cracking Updates Alignment External Cracking External Algorithms Multi-pass Cracking Forgetting Flash-based Cracking Divide and Conquer Alternative Hybrid Designs Alternative Implementations Hybrid Cracking Idle time Cracking Forgetting Administration Costs Updates Reverse Cracking Multi-query Processing and Transactions Compression on Cracked Columns Cracking Compressed Columns Crack for Compression Adaptive Denormalization via Cracking Cracking Row-stores Adaptive Indexing in Auto-tuning Tools A Histogram for Free Distributed Cracking Beyond the Horizon

9 12 CONTENTS Bibliography 241 List of Figures 247 Summary 251 Samenvatting 253 Acknowledgments 255 CURRICULUM VITAE 257 Education Employment & Academic Experience Publications Reviewing SIKS Dissertation Series 263