Session 14: Data Warehousing

Size: px

Start display at page:

Download "Session 14: Data Warehousing"

Mervin Riley
5 years ago
Views:

1 ICDE2014 勉強会 Session 14: Data Warehousing 担当 : 若森拓馬 (NTT)

2 紹介する論文 1. Distributed Interactive Cube Exploration N. Kamat, P. Jayachandran, K. Tunga, A. Nandi (Ohio State University) 2. A Tunable Compression Framework for Bitmap Indices G. Guzun, G. Canahuate (The University of Iowa), D. Chiu (Washington State University), J. Sawin (University of St. Thomas) 3. Pagrol: Parallel Graph OLAP over Large-scale Attributed Graphs Z. Wang, Q. Fan, H. Wang, K-L Tan (National University of Singapore), D. Agrawal, A. E. Abbadi (University of California) 2

3 Distributed Interactive Cube Exploration } 目的 rack hour } 大規模データ ( キューブ ) のインタラクティブなアドホック探索 } 貢献 iops SELECT rack, AVG(iops) FROM events WHERE datacenter = EU AND hour = 6 GROUP BY rack; <*, *, *> <month, *, *> <*, *, zone> <hour, iops, rack> } DICE: 10 億タプルを数秒で検索する分散システムを構築 } 分散環境でのキューブ探索の応答性能を向上するクエリの投機実行とオンラインデータサンプリングを組み合わせたコストベースのフレームワーク } 投機クエリの探索空間を制限するため, 連続するクエリをクエリセッションとみなす多面キューブ探索モデルを提案 <*, *, zone> キューブのマテリアライズはコストが高い 3

4 the following schema: Database table events catalogs all the system events across the cluster and has three dimensions, two of which are hierarchical: location[zone:datacenter:rack], time[month:week:hour], iops Distributed Interactive Cube Exploration } Facet Ses f1...n that from one We now d Challenges in Exploration: As a user exploring a data cube, inspired b the number of possible parts of the cube to explore (i.e. cube to move f groups) is very large, and thus, exploration can be unwieldy. To of the des this end, we introduce the faceted model of cube exploration, Parent Fa which simplifies cube exploration into a set of facet traversals, by genera } as described below. As we will see in the following section, fp (dpg, dp the faceted model drastically reduces the space of possible cube exploration and simplifies speculative query execution, dpg = dg in the cu which is essential to the DICE architecture. generalize We introduce the term facet as parent Child Fa the basic state of exploration of SELECT rack, AVG(iops) FROM events WHERE by specia a data cube, drawing from the pivot b$ datacenter = EU /* datacenter =use * of*/ category /* rackcounts = 12 in*/ the a$ fc (dcg, dc exploratory search paradigm of dg and dc AND hour = 6 child faceted search [53]. Empirically, a,b$ lattice. Th GROUP BY rack; /* GROUP BY datacenter; */ such as map most visualizations w1, hour sibling views and bar charts found in visibling F sual analytics tools can be con-fig. Fig Faceted Faceted Cube ExploCubetained by ration Traversals structed from aggregations along スライド(*) P11 Traversals a single dimension. Facets are Exploration meant to be perused in an dimension interactive fashion a user is expected to fluidly explore the facet f (d differ by e entire data cube by successively perusing multiple facets. m1, week Intuitively, a user explores a cube by inspecting a facet of More'Dimensions' Master a particular region in the data cube a histogram view of a Pivot Fac Sampling Rate! Query!! Sample! Specula1ve!Query! subset of groups from one region, along a specific dimension. switching UI actions ry Federa1on! Aggrega1on! Execu1on! e u! The user then explores the cube by traversing from that facet Thus, a fa et q fac to another facet. This successive facet can be a parent facet in fv (dvg, dv Network the case of a rollup, a child facet in the case of a drilldown, all but on ult res a sibling facet in the case of change of a dimension value f (week, z in the group and a pivot facet in the case of a change in facet exam slave1! theslave2! inspected slave3! dimension.result Thus, Cache the user isslaven! effectively moving facet analytics around the cube lattice Workers to either a parent region, or a child E XPLORA frontends traversals region or remaining in the same region using sibling and pivot the user i traversals to look at the data differently. A session comprises groups ca (result = measure & std dev) any facet DB! The formal definitions DB! of multiple DB! traversals. DB! are as follows. be obtaine! Facet: For a region r in cube C, a facet f is a set of groups d dg. Fig. 1. DICE Approach: Allow tunable sampling rates on low-latency frontends, with UI actions translated to facet traversal queries (Section II-A) over g 2 approach r(d1...n )(Section such that the master groupmanages labels session differ on the data cube. Queries are executed by the master over distributed slaves (Section II-B). In the DICE III), the state,exactly one from anot query speculation and result aggregation, while the slaves manage execution and caching. For each query, the master dimension di, i.e.distributes 8ga, gb the 2 query f, di to (gaeach ) 6=slave, di (gwhich traversals b ) ^ dj (ga ) = may have some results speculatively executed and cached. Results from each slave are then aggregated, bounds calculated, andgrouping returned todimension, the user. dj (gb ) error where i 6= jareand di is the and the switch be remaining dimensions are the bound dimensions. In its SQL 隣接ディメンションに優先度を付け投機的にマテリアライズキューブの探索を4種類(parent, child, sibling, pivot)に制限し投機クエリの実行数を制限 *$ } (rollup) ユーザ指定のサンプリングレート下で投機クエリを分散実行 Fig. 1. DICE Approach 4 Session 14: Data Warehousing 担当若森拓馬 NTT

Distributed Interactive Cube Exploration } ユーザ指定のサンプリングレート (R) で確度の高いものから優先実行 } 並列実行クエリ数を減らすための工夫 : 複数クエリをセッションに統合 } カーディナリティやや高複数 WHERE 句を GROUP BY に統合 } カーディナリティ高ディメンションのドメインを複数のレンジに分割確度 (

5 Distributed Interactive Cube Exploration } ユーザ指定のサンプリングレート (R) で確度の高いものから優先実行 } 並列実行クエリ数を減らすための工夫 : 複数クエリをセッションに統合 } カーディナリティやや高複数 WHERE 句を GROUP BY に統合 } カーディナリティ高ディメンションのドメインを複数のレンジに分割確度 ( サンプリングレート R に依存 ) 1 1 AccuracyGain(R curr )=c ( p p Rcurr Rcurr +1 ) 整数計画法で優先度計算 : クエリqが呼ばれる確立 ( クエリログから算出 ) スライド (*) P14 MAXIMIZE: P q2q Prob(q) AccuracyGain(SR) x q SUBJECT TO: P q2q T ime(q) x q <= totalspect ime WHERE: x q 2 {0, 1}. 5

6 Distributed Interactive Cube Exploration } 結果 } Small クラスタ :Xeon 15 slave nodes (4GiB RAM) } Cloud クラスタ :Amazon EC2 50 slave nodes (7GiB Mem, 8C) Time%(ms)% 1600$ 1400$ 1200$ 1000$ 800$ 600$ 400$ 200$ 0$ Fig. 4. Cluster:)Cloud) 投機実行しない場合と比べて大幅に高速化 NOSPEC$ DICE$ Perfect$ 250$ 500$ 750$ 1000$ Sample%Size%(millions%of%rows)% Varying Size of Dataset: CLUSTER CLOUD Time%(ms)% 3500" 3000" 2500" 2000" 1500" 1000" 500" 0" Cluster: Small 分散が小さい 0.75 NOSPEC" DICE" Perfect" Fig. 9. Individual Latencies for Anecdotal Query Session } (*) 発表スライド : } 6

A Tunable Compression Framework for Bitmap Indices } } } 背景 } 従来のビットマップインデックスの圧縮は, 伸長のオーバーヘッドを最小化するために,bit vectorを固定エンコード長 ( ワード長 ) に圧縮 : 伸長せずに処理可能 } エンコード長が短いほど圧縮率が高く, 伸長時間が長い e.g.

7 A Tunable Compression Framework for Bitmap Indices } } } 背景 } 従来のビットマップインデックスの圧縮は, 伸長のオーバーヘッドを最小化するために,bit vectorを固定エンコード長 ( ワード長 ) に圧縮 : 伸長せずに処理可能 } エンコード長が短いほど圧縮率が高く, 伸長時間が長い e.g. ) Word-Aligned Hybrid (WAH) encoding literal bitmap index row b 0 b 1 b 2 b 3 ID X =0 =1 =2 = Fig. 1. bitmap index 図は [K. Wu et al, Optimizing Bitmap Indices With Efficient Compression, 2006] より引用 128 bits 1*1,20*0,3*1,79*0,25*1 31-bit groups 1,20*0,3*1,7*0 62*0 10*0,21*1 4*1 literal (hex) FFFFF F WAH (hex) FFFFF F Fig. 2. A WAH compressed bitmap. 目的 } 可変エンコード長を用いることによる高圧縮率かつ短伸長時間のビットマップインデックス圧縮貢献 } 複数のエンコーディング手法と可変セグメント長を用いてbit vectorを圧縮する Variable Aligned Length (VAL) というフレームワークを提案 } VAL 上でWAHを実装 (VAL-WAH) } 空間 / 時間コストのチューニングパラメータλを指定可能.VAL-WAHは64bit WAHと比較して圧縮率が3.4 倍, オーバーヘッドが3%.32bit WAHと比較して 1.8 倍,30% 高速化 fill 7

A Tunable Compression Framework for Bitmap Indices Data Set tuning (lambda) word size (w), alignment factor (b) Bitmap Characterization Segment Length Selector Legal Segments (LS) Encoding Selector

8 A Tunable Compression Framework for Bitmap Indices Data Set tuning (lambda) word size (w), alignment factor (b) Bitmap Characterization Segment Length Selector Legal Segments (LS) Encoding Selector Compress WAH PLWAH... EWAH Query Processor Fetch, Decode Compressed Bitmap Disk Queries Results 異なるセグメント長でエンコードされた 2 つの bit vector をクエリ可能なアルゴリズムを設計 VAL Encoder VAL Query Engine Fig. 1. The VAL System Framework: Encoder and Query Engine } WAH の拡張 (Enhanced WAH(EWAH), Concide, PLWAH) や可変セグメント長 (VLC, Partitioned WAH(PWAH)) のアルゴリズムを共存させ, 最もパフォーマンスの良いものを選択 } 可変セグメント長のビットマップを用いた効率的なクエリ処理アルゴリズム ( セグメントのアライメントを保持 ) を開発 8

9 A Tunable Compression Framework for Bitmap Indices } } ビットマップ処理の共通化 } WAHのコアのアルゴリズムに小さな修正を加える事でPLWAH, Concise, Compax, EWAHに対して拡張可能 9 typedef struct {! /* holds encoded word value */! word_t val;! /* fill-specific vars */! bool fillbit;! int runlength;! } activeword;! val の最上位ビット 0: fill,1: literal activeword が fill の場合のみ代入されるセグメント長の変換アルゴリズム 64-bit VAL-WAH(s=30) new Block s= bit VAL-WAH(s=15) activeblock s=30 左シフト new Block s= new Block s=15 } セグメント長の選択 Legal Segment Lengths: LS LS = {2 i (b 1) 0 apple i apple (log 2 w log 2 b)} 64bit(w=64),b=16のときLS={15,30,60} でアライメント (s s = c+i, if size(b,s c) (1+ ) 1+i+ i+1 size(b,s c+i ) s c, otherwise s c = arg min s i 2LS {size(b,s i )} activeblock s= apple (size(b, s) は bit vector B のサイズを表す ) Fig bit VAL (s=30) を 64-bit VAL (s=15) に変換する例左シフト

10 A Tunable Compression Framework for Bitmap Indices } 結果 } チューニングパラメータ λ で圧縮率とクエリ時間を設定可能! Fig Fig. 8. Compression Ratio Query Time (ms) Lambda 2M-rows 4M-rows 6M-rows 8M-rows 10M-rows (a) Synthetic Data (zipf2) - Sorted: Compression Ratio Lambda 2M-rows 4M-rows 6M-rows 8M-rows 10M-rows (b) Synthetic Data (zipf2) - Sorted: Query Time } VAL-WAR (λ=0.2) がリアルデータに対して高い圧縮率と短いクエリ時間を達成 Compression Ratio Query Time Ratio kddcup berkeley earth PLWAH32 EWAH32 EWAH64 WAH 32 WAH64 VAL0.2 (a) Compression Ratio: Real Data kddcup berkeley earth PLWAH32 EWAH32 EWAH 64 WAH 32 WAH64 VAL 0.2 (b) Query Time Ratio: Real Data 10

11 Pagrol: Parallel Graph OLAP over Large-scale Attributed Graphs } 目的 } 属性グラフ (Attributed Graphs) に対する OLAP 分析 } 貢献 } Pagrol: 並列グラフOLAPシステムを構築し, 属性グラフを意思決定 An attributed graph is able to model various information サービスに拡張する networks by adding attributes tohyper each vertex andgraph edge. We firstcubeモデルを設計 provide a formal definition of an attributed graph. } MapReduce Definition 1 (Attributed 上のグラフキューブ計算を最適化する手法 Graph): An attributed graph, G, is (selfcontained a graph denoted join, as G=(V, cuboid E, A v,a e ), batching, where V is a set cost-based of batch processing) vertices, E V V is a set of edges, and A v = (A v1, を提案 A v2,..., A vn ) is a set of n vertex-specific attributes, i.e. u V, there is a multidimensional tuple A v (u) denoted } 効率的なキューブマテリアライゼーション手法 as A v (u) = (A v1 (u), A v2 (u),..., A vn (u)), and A e =(A e1,a e2, (MRGraph-Cubing):..., A 大規模属性グラフに対する最初のグラフキューブ手法 em ) is a set of m edge-specific attributes, i.e. e E, there is a multidimensional tuple A e (e) denoted as A e (e)= 11 Fig. 1. A running example of an attributed graph (A e1 (e),a e2 (e),...,a em (e)). To develop graph OLAP and warehousing, we formally define two types of dimensions in attributed graphs as follows: 属性グラフ (Attributed graph) 頂点 : オブジェクト辺 : 関係頂点は頂点の属性 (vertex attributes) をもつ辺は辺の属性 (edge attributes) をもつ Fig. 1. A running example of an attributed graph

頂点や辺のdimensionが階層的になる位置 : 都市州国全部時間 : 日週月年全部 Vertex Dimensions (V-Dims) The Hyper Graph Cube lattice and aggregate cuboids analysis.

12 Fig. 2. Pagrol: Parallel Graph OLAP over Location: Large-scale City State Country Attributed all Graphs } Hyper Graph Cubeモデル } Hyper Graph Cube 上のOLAP 処理 } Roll-Up/Drill-Down 12 } } } 頂点や辺のdimensionが階層的になる位置 : 都市州国全部時間 : 日週月年全部 Vertex Dimensions (V-Dims) The Hyper Graph Cube lattice and aggregate cuboids analysis. For instance, the vertex degree can be extracted as one V-Dim for aggregation. This may extend the utility of the OLAP over Hyper Graph Cube Roll-Up/Drill-Down Vertex or edge dimensions could be hieratical Time: Day Week Month Year all <country, month> <state, month> <state, year> Edge Dimensions (E-Dims) <country, year> <state, month> Fig. 2. Hyper Graph Cube の格子 (lattice) と集約された cuboid <city, year> <city, month> <state, week> <city, week> <country, week> Motivation Graph Aggregation Graph Cube Model Cube Materialization Experiments A e(e) =A e(e ) and e G e (e ). The weight of e, w(e )=Γ (G (e )), where Γ ( ) is an edge aggregate 15 プレゼン (*) P.15

13 Pagrol: Parallel Graph OLAP over Large-scale Attributed Graphs } } } } Self-Contained Join } DFSへの書き込み / 再読み込みが発生しない } Blk-Gen job } } 頂点と辺のテーブルをMapフェーズで読み込む Reduceフェーズでself-containedなデータブロックを生成して一つのファイルに書き出す Cuboid Batching } バッチのない場合 : ABC, AB, A ABC Mapper <ABC> <AB> <A> Partition by ABC Partition by AB Partition by A ABC Batch Processing } Prefix orderにしたがって実行 } ABCàA,AB,ABC } EFGàE,EF,EFG } <ABC, EFG> à { <A, E>, <A, EF>, <A, EFG>, <AB, E>, <AB, EF>, <AB, EFG>, <ABC, E>, <ABC, EF>, <ABC, EFG>} Cost-Based Execution Plan Optimization AB A Reducer ABC Mapper <ABC> Fig. 3. self-contained file format } Computing バッチのある場合 cuboids into : prefix batch: order according にしたがって実行 to prefix order Partition by A Project via descendant cuboid A 1 BC Partition via ancestor cuboid A 2 BC A 3 BC Reducer A 1 BC A 1 B A 1 プレゼン (*) P.21 Small intermediate data Share sorting Share shuffling Fig. 4. 生成されたバッチ 13

14 Pagrol: Parallel Graph OLAP over Large-scale Attributed Graphs } MRGraph-Cubing } ナイーブな手法 : 各 cuboid を独立して計算 } Cuboid を Batch に統合 } Batch の実行プランを生成 } 複数の Batch を Bag に置く Cuboids } 1 つの Bag を 1 M/R Job で実行 Batches Bags プレゼン (*) P.20 14

15 Pagrol: Parallel Graph OLAP over Large-scale Attributed Graphs } 結果 map-join blk-gen VE-join VE-parse 1200 map-join blk-gen VE-join VE-parse single-c-per-job single-b-per-job all-cs-one-job all-bs-one-job Execution time (s) Execution time (s) Running Time(s) MRGraph-Cubing により 2x~4.5x の高速化 } (*) 発表スライド : Figure 8 (c) provides the comparison results based on the } datasets with 600M edges and with degree 60 when we vary the average combine ratio of the graph. The results indicate Pagrol_ICDE14_Slides.pdf k=10 k=20 k=40 k=60 k=80 k=100 (a) Impact of graph degree Running Time(s) single-b-per-job Plan all-bs-one-job Plan optimized Plan 400 Size=10 Size=20 Size=30 Size=40 Size=50 (b) Impact of V-Dim size Combine Ratio (c) Impact of batch processing Fig. 8. self-containing join と batch processing の評価 Running Time(s) cube materialization For the following experiments, 4000 we use the synthetic dataset. Each graph contains 3 V-Dims and 3 E-Dims. By default, the 3500 average size of each V-Dim or E-Dim is 7 bytes. Meanwhile, 3000 we choose COUNT( ) as the measure function for both the 2500 vertex and edge aggregation. Note that if a graph contains N edges with degree K, there will be 2000 N*2/K vertices in the graph. B. Self-Contained Join Optimization In this experiment, we compare the proposed self-contained M 180M 270M 405M 600M 750M join optimization with the naive approach. For our proposed Combine Ratio Number of Edges Number of Cluster Nodes (a) Batch execution plans (b) self-contained Impact of graph join size optimization, we(c) have Impact two MR of parallelism jobs: blkgen job which reads the vertex and edge attribute tables and Fig. 9. Evaluation of the plan optimizer and scalability Fig. 9. plan optimizer generates とスケーラビリティの評価 a series of self-contained files, and map-join job which reads the self-contained files, conducts the map side join and ends time after becomes the joinalmost operation 1.5X in the slower. map phase. This indicates For the that the naive approach, algorithm we also scales havewell twoinmr terms jobs: ofve-join dataset job size. which joins the vertex with edge attribute tables, and VE-parse job which reads the The joined second data set and of experiments ends when itaims that the algorithms with batching optimization are 2.5X and 4X finishes at studying parsing the impact faster than single-c-per-job and all-cs-one-job respectively. each joinedofdata parallelism. in the map Fig. phase. 9 (c) provides the execution times of cube materialization on a dataset with 750M vertices with degree Meanwhile, we observe that the larger the intermediate data Figure 8 (a) shows the performance comparison on datasets (with a bigger combine ratio) are, the higher the performance results indicate that when the computation power is doubled, gain by the batching optimization. from 10 tothe100. execution The results time almost indicate reduces that our to half optimization from 16 to 64 nodes. Running Time(s) cube materialization 60 when we vary the cluster size from 16 to 128 nodes. The Session with 14: 200 Data million(m) Warehousing vertices when we vary the 担当 graph : degree 若森拓馬 (NTT)

Googleの強みはささえるのは世界一のインフラ. Google File System 2008年度後期情報システム構成論2 第10回クラウドと協調フィルタリング. 初期(1999年)の Googleクラスタ. 最近のデータセンタ Google Chrome Comicより

Googleの強みはささえるのは世界一のインフラ. Google File System 2008年度後期情報システム構成論2 第10回クラウドと協調フィルタリング. 初期(1999年)の Googleクラスタ. 最近のデータセンタ Google Chrome Comicより Googleの強みは 2008年度後期情報システム構成論2 第10回クラウドと協調フィルタリング西尾信彦 nishio@cs.ritsumei.ac.jp 立命館大学情報理工学部 Cloud Computing 全地球規模で構成された圧倒的なPCクラスタ部分的な機能不全を補う機能あらゆる種類の情報へのサービスの提供 Web上の全情報地図情報 (実世界情報) どのように利用されているかを機械学習