High-Performance Parallel Database Processing and Grid Databases

Transcription

1 High-Performance Parallel Database Processing and Grid Databases David Taniar Monash University, Australia Clement H.C. Leung Hong Kong Baptist University and Victoria University, Australia Wenny Rahayu La Trobe University, Australia Sushant Goel RM1T University, Australia WILEY A John Wiley & Sons, Inc., Publication

2 Contents Preface xv Pari I Introduction 1. Introduction 1.1. A Brief Overview: Parallel Databases and Grid Databases 1.2. Parallel Query Processing: Motivations Parallel Query Processing: Objectives Speed Up Scale Up Parallel Obstacles Forms of Parallelism Interquery Parallelism Intraquery Parallelism Intraoperation Parallelism Interoperation Parallelism Mixed Parallelism A More Practical Solution Parallel Database Architectures Shared-Memory and Shared-Disk Architectures Shared-Nothing Architecture Shared-Something Architecture Interconnection Networks Grid Database Architecture Structure of this Book Summary Bibliographical Notes Exercises 31

3 vi CONTENTS 2. Analytical Models Cost Models Cost Notations Data Parameters Systems Parameters Query Parameters Time Unit Costs Communication Costs Skew Model Basic Operations in Parallel Databases Disk Operations Main Memory Operations Data Computation and Data Distribution Summary Bibliographical Notes Exercises 47 Part II Basic Query Parallelism 3. Parallel Search Search Queries Exact-Match Search Range Search Query Multiattribute Search Query Data Partitioning Basic Data Partitioning Complex Data Partitioning Search Algorithms Serial Search Algorithms Parallel Search Algorithms Summary Bibliographical Notes Exercises Parallel Sort and GroupBy Sorting, Duplicate Removal, and Aggregate Queries Sorting and Duplicate Removal Scalar Aggregate GroupBy Serial External Sorting Method 80

4 CONTENTS vii 4.3. Algorithms for Parallel External Sort Parallel Merge-All Sort Parallel Binary-Merge Sort Parallel Redistribution Binary-Merge Sort Parallel Redistribution Merge-All Sort Parallel Partitioned Sort Parallel Algorithms for GroupBy Queries Traditional Methods (Merge-All and Hierarchical Merging) Two-Phase Method Redistribution Method Cost Models for Parallel Sort Cost Models for Serial External Merge-Sort Cost Models for Parallel Merge-All Sort Cost Models for Parallel Binary-Merge Sort Cost Models for Parallel Redistribution Binary-Merge Sort Cost Models for Parallel Redistribution Merge-All Sort Cost Models for Parallel Partitioned Sort Cost Models for Parallel GroupBy Cost Models for Parallel Two-Phase Method Cost Models for Parallel Redistribution Method Summary Bibliographical Notes Exercises Parallel Join Join Operations Serial Join Algorithms Nested-Loop Join Algorithm Sort-Merge Join Algorithm Hash-Based Join Algorithm Comparison Parallel Join Algorithms Divide and Broadcast-Based Parallel Join Algorithms Disjoint Partitioning-Based Parallel Join Algorithms Cost Models Cost Models for Divide and Broadcast Cost Models for Disjoint Partitioning Cost Models for Local Join 130

5 viii CONTENTS 5.5. Parallel Join Optimization Optimizing Main Memory Load Balancing Summary Bibliographical Notes Exercises 136 Part HI Advanced Parallel Query Processing 6. Parallel GroupBy-Join Groupby-Join Queries Groupby BeforeJoin Groupby After Join Parallel Algorithms for Groupby-Before-Join Query Processing Early Distribution Scheme Early GroupBy with Partitioning Scheme Early GroupBy with Replication Scheme Parallel Algorithms for Groupby-After-Join Query Processing Join Partitioning Scheme GroupBy Partitioning Scheme Cost Model Notations Cost Model for Groupby-Before-Join Query Processing Cost Models for the Early Distribution Scheme Cost Models for the Early GroupBy with Partitioning Scheme Cost Models for the Early GroupBy with Replication Scheme Cost Model for "Groupby-After-Join" Query Processing Cost Models for the Join Partitioning Scheme Cost Models for the GroupBy Partitioning Scheme Summary Bibliographical Notes Exercises 164

6 CONTENTS ix 7. Parallel Indexing Parallel Indexing-an Internal Perspective on Parallel Indexing Structures Parallel Indexing Structures Nonreplicated Indexing (NRI) Structures Partially Replicated Indexing (PRI) Structures Fully Replicated Indexing (FRI) Structures Index Maintenance Maintaining a Parallel Nonreplicated Index Maintaining a Parallel Partially Replicated Index Maintaining a Parallel Fully Replicated Index Complexity Degree of Index Maintenance Index Storage Analysis Storage Cost Models for Uniprocessors Storage Cost Models for Parallel Processors Parallel Processing of Search Queries using Index Parallel One-Index Search Query Processing Parallel Multi-Index Search Query Processing Parallel Index Join Algorithms Parallel One-Index Join Parallel Two-Index Join Comparative Analysis Comparative Analysis of Parallel Search Index Comparative Analysis of Parallel Index Join Summary Bibliographical Notes Exercises Parallel Universal Qualification Collection Join Queries Universal Quantification and Collection Join Collection Types and Collection Join Queries Collection-Equi Join Queries Collection-Intersect Join Queries Subcollection Join Queries Parallel Algorithms for Collection Join Queries Parallel Collection-Equi Join Algorithms Disjoint Data Partitioning 226

7 x CONTENTS Parallel Double Sort-Merge Collection-Equi Join Algorithm Parallel Sort-Hash Collection-Equi Join Algorithm Parallel Hash Collection-Equi Join Algorithm Parallel Collection-Intersect Join Algorithms Non-Disjoint Data Partitioning Parallel Sort-Merge Nested-Loop Collection-Intersect Join Algorithm Parallel Sort-Hash Collection-Intersect Join Algorithm Parallel Hash Collection-Intersect Join Algorithm Parallel Subcollection Join Algorithms Data Partitioning Parallel Sort-Merge Nested-Loop Subcollection Join Algorithm Parallel Sort-Hash Subcollection Join Algorithm Parallel Hash Subcollection Join Algorithm Summary Bibliographical Notes Exercises Parallel Query Scheduling and Optimization Query Execution Plan Subqueries Execution Scheduling Strategies Serial Execution Among Subqueries Parallel Execution Among Subqueries Serial vs. Parallel Execution Scheduling Nonskewed Subqueries Skewed Subqueries Skewed and Nonskewed Subqueries Scheduling Rules Cluster Query Processing Model Overview of Dynamic Query Processing A Cluster Query Processing Architecture Load Information Exchange Dynamic Cluster Query Optimization Correction Migration Partition Other Approaches to Dynamic Query Optimization Summary 285

8 CONTENTS xi 9.9. Bibliographical Notes Exercises 286 Part IV Grid Databases 10. Transactions in Distributed and Grid Databases Grid Database Challenges Distributed Database Systems and Multidatabase Systems Distributed Database Systems Multidatabase Systems Basic Definitions on Transaction Management Acid Properties of Transactions Transaction Management in Various Database Systems Transaction Management in Centralized and Homogeneous Distributed Database Systems Transaction Management in Heterogeneous Distributed Database Systems Requirements in Grid Database Systems Concurrency Control Protocols Atomic Commit Protocols Homogeneous Distributed Database Systems Heterogeneous Distributed Database Systems Replica Synchronization Protocols Network Partitioning Replica Synchronization Protocols Summary Bibliographical Notes Exercises Grid Concurrency Control A Grid Database Environment An Example Grid Concurrency Control Basic Functions Required by GCC Grid Serializability Theorem Grid Concurrency Control Protocol Revisiting the Earlier Example Comparison with Traditional Concurrency Control Protocols 334

9 xii CONTENTS Correctness of GCC Protocol Features of GCC Protocol Summary Bibliographical Notes Exercises Grid Transaction Atomicity and Durability Motivation Grid Atomic Commit Protocol (Grid-ACP) State Diagram of Grid-ACP Grid-ACP Algorithm Early-Abort Grid-ACP Discussion Message and Time Complexity Comparison Analysis Correctness of Grid-ACP Handling Failure of Sites with Grid-ACP Model for Storing Log Files at the Originator and Participating Sites Logs Required at the Originator Site Logs Required at the Participant Site Failure Recovery Algorithm for Grid-ACP Comparison of Recovery Protocols Correctness of Recovery Algorithm Summary Bibliographical Notes Exercises Replica Management in Grids Motivation Replica Architecture High-Level Replica Management Architecture Some Problems Grid Replica Access Protocol (GRAP) Read Transaction Operation for GRAP Write Transaction Operation for GRAP Revisiting the Example Problem Correctness of GRAP Handling Multiple Partitioning Contingency GRAP Comparison of Replica Management Protocols Correctness of Contingency GRAP 383

10 CONTENTS xiii Summary Bibliographical Notes Exercises Grid Atomic Commitment in Replicated Data Motivation Architectural Reasons Motivating Example Modified Grid Atomic Commitment Protocol Modified Grid-ACP Correctness of Modified Grid-ACP Transaction Properties in Replicated Environment Summary Bibliographical Notes Exercises 398 Part V Other Data-Intensive Applications 15. Parallel Online Analytic Processing (OLAP) and Business Intelligence Parallel Multidimensional Analysis Parallelization of ROLLUP Queries Analysis of Basic Single ROLLUP Queries Analysis of Multiple ROLLUP Queries Analysis of Partial ROLLUP Queries Parallelization Without Using ROLLUP Parallelization of CUBE Queries Analysis of Basic CUBE Queries Analysis of Partial CUBE Queries Parallelization Without Using CUBE Parallelization of Top-/V and Ranking Queries Parallelization of CumeJDist Queries Parallelization of NTILE and Histogram Queries Parallelization of Moving Average and Windowing Queries Summary Bibliographical Notes Exercises 425

11 xiv CONTENTS 16. Parallel Data Mining Association Rules and Sequential Patterns From Databases To Data Warehousing To Data Mining: A Journey Data Mining: A Brief Overview Data Mining Tasks Querying vs. Mining Parallelism in Data Mining Parallel Association Rules Association Rules: Concepts Association Rules: Processes Association Rules: Parallel Processing Parallel Sequential Patterns Sequential Patterns: Concepts Sequential Patterns: Processes Sequential Patterns: Parallel Processing Summary Bibliographical Notes Exercises Parallel Clustering and Classification Clustering and Classification Clustering Classification Parallel Clustering Clustering: Concepts it-means Algorithm Parallel fc-means Clustering Parallel Classification Decision Tree Classification: Structures Decision Tree Classification: Processes Decision Tree Classification: Parallel Processing Summary Bibliographical Notes Exercises 498 Permissions 501 List of Conferences and Journals 507 Bibliography 511 Index 541