Mapper An Efficient Data Transformation Operator

Transcription

1 Mapper An Efficient Data Transformation Operator Paulo Carreira University of Lisbon

2 2 Context Source: IMPCRED CREDTID MERCH FT RT PCS LARGE CONTAINER MODULE X PCS COTTON CORDUROY MTS XS PCS POLYESTER TUBE W PCS ARTIFICIAL FIBRE 3 PCS SPANDEX MTS W PCS POLYESTER TUBE W PCS ARTIFICIAL FIBRE MTS NEW CHECK DESIGN 4800 PCS IC CAN TRANSCEIVER 3.3V 8-SOIC SN65HVD232D 2400 PCS PCA9532BS-T 16-bit I2C LED DIMMER Target: MERCHDETL ORDRID QTY DESCR LARGE CONTAINER MODULE X067 COTTON CORDUROY MTS XS POLYESTER TUBE W ARTIFICIAL FIBRE SPANDEX MTS W15719 POLYESTER TUBE W989 ARTIFICIAL FIBRE MTS NEW CHECK DESIGN IC CAN TRANSCEIVER 3.3V 8-SOIC SN65HVD232D PCA9532BS-T 16-bit I2C LED DIMMER

3 3 State of the art General Purpose Language ETL Tool Union Unpivot RDBMS Recursive Queries PSMs? Declarativeness - +/ /- + Optimizability /- - + Expressiveness + +/

4 4 Main Contributions 1. Extending of Relational Algebra Data Mapper: An operator for expressing One-to-many data transformations 2. Logical optimization rules for the mapper operator, e.g., Pushdown of projections Pushdown of selections 3. Physical execution algorithms for the mapper operator Naïve algorithm Shortcircuiting algorithm Cache-based algorithm

5 5 The mapper operator S = Source schema IMPCRED(CREDID, MERCHANDISE) CREDID MERCHANDISE 4800 PCS IC CAN TRANSCEIVER 3.3V 8- SOIC SN65HVD232D RT PCS PCA9532BS-T 16-bit I2C LED DIMMER u Target schema MERCHDETL(ORDERID, QTY, DESCR) ORDRID QTY DESCR IC CAN TRANSCEIVER 3.3V 8- SOIC SN65HVD232D PCA9532BS-T 16-bit I2C LED DIMMER cidordrid(u) extrqty,descr(u) t[ordrid] X IC CAN TRANSCEIVER 3.3V 8- SOIC SN65HVD232D PCA9532BS-T 16-bit I2C LED DIMMER t[qty, DESCR]

6 6 Algebraic Optimization Pushdown of projections 13 2 Introduction of projections Pushdown of selections Pushing selections to mapper functions Distributing mappers over unions Distributing mappers over Cartesian products

7 7 Physical Algorithms Replacement Policies Input relation Cheap Functions Expensive Functions Cache of Function Results LRU Functions LUR returning empty Utility list sets LRU stack Most Recently Used Naïve Least Shortcircuiting Recently Used Recency Most Useful Least Useful Utility XLUR Multiple LRU Stacks MF MRU ME LF LE LRU Duplicate input values Approximate Utility Output relation Naïve Cache-based Next victim?

8 8 Experimental Validation 1. Comparison of RDBMSs with mappers Mappers are 1.7x 3.6x faster than the best RDBMS solution 2. Gain of logical optimizations Gains of 1.3x 5.5x for pushing selections 3. Performance of physical algorithms Shortcircuiting algorithm: 50x faster than the Naïve (at ~1ms/call) Cache-based algorithm (XLUR) Interesting savings: 50% duplicates = 1.4x faster than the Naïve Lightweight: virtually the same overhead as LRU Successful at performing cost-based decisions: outperforms LRU for low CHRs

9 9 Conclusions n New unary operator extension to Relational Algebra: Data Mapper n Addresses: One-to-many data transformations Declarative Optimizable Expressive n Consequences: Broadens the span of application of RDBMSs Significant improvement for ETL tools (Data Fusion tool)

10 n Selected Publications [Carreira et al. 2007] Carreira, P., Galhardas, H., Lopes, A. & Pereira, J., One-to-many Data Transformations through Data Mappers, Data & Knowledge Engineering Journal (DKE), 62(3), , Elsevier-Science, n [Carreira et al. 2005] Carreira, P., Galhardas, H., Pereira, J., & Lopes, A., Data Mapper: An operator for expressing one-to-many data transformations. 7th Int l Conference on Data Warehousing and Knowledge Discovery, DaWaK '05, Vol of LNCS, Springer-Verlag, n [Carreira et al. 2005b] Carreira, P., Galhardas, H., Lopes, A. & Pereira, J. (2005). Extending relational algebra to express one-to-many data transformations. In 20th Brazillian Symposium on Databases SBBD '05, n n [Carreira & Galhardas 2004] Carreira, P., & Galhardas, "Efficient development of data migration transformations", Demo paper, Proc. of the ACM Conference on the Management of Data, SIGMOD'04, [Carreira et al. 2007b] Carreira, P., Galhardas, H., Pereira, J., Martins F. & Silva, Mário J., On the performance of One-to-Many Data Transformations, Venkatesh Ganti, Felix Naumann (Eds.): Proceedings of the 5th International Workshop on Quality in Databases, QDB 2007 at VLDB Conference,

11 Reserve Slides 11

12 12 Future work n Study mapper functions Study the properties of mapper functions Develop a taxonomy of mapper functions Propose a library of useful mapper functions n Enhance the physical algorithms n Incorporate the mapper operator on an Open Source RDBMS

13 13 Extracting rows from unstructured data Source: CITEDATA AUTHORS TITLE Periklis Andritsos, Ronald Fagin, Ariel Fuxman, Laura M. Haas, Mauricio A. Hernández, C. T. Howard Ho, Anastasios Kementsietsidis, Renée J. Schema Management Miller, Felix Naumann, Lucian Popa, Yannis Velegrakis, Charlotte Vilarem, Ling-Ling Yan Jorge Vieira, Jorge Bernardino, Henrique Madeira Efficient Compression of Text Attributes of Data Warehouse Dimensions Target: EVENTS NAME TITLE Periklis Andritsos Schema Management Ronald Fagin Schema Management Ariel Fuxman Schema Management Laura M. Haas Schema Management Mauricio A. Hernández Schema Management C. T. Howard Ho Schema Management Anastasios Kementsietsidis Schema Management Renée J. Miller Schema Management Felix Naumann Schema Management Lucian Popa Schema Management Yannis Velegrakis Schema Management Charlotte Vilarem Ling-Ling Yan Schema Management Schema Management

14 Example: Converting lines into columns Target: EVENTS LOANO EVTYP AMTYP AMT Source: LOANEVT LOANO EVTYP CAPTL TAX EXPNS BONUS 1234 OPEN PAY PAY EARLY FULL CLOSED OPEN TAX OPEN EXPNS OPEN BONUS PAY CAPTL PAY TAX PAY CAPTL PAY TAX EARLY CAPTL FULL CAPTL FULL TAX FULL EXPNS FULL BONUS CLOSED EXPNS 0.1

15 Implementations of one-to-many data transformations 15 Bounded Unbounded RA/SQL PSM Recursive Queries Unpivot RA/SQL PSM Recursive Queries Unpivot DBX YES YES YES N/A NO YES YES NO OEX YES YES N/A N/A NO YES N/A NO

16 16 Extensions to Relational Algebra Type of data transformation Many-to-one One-to-one One-to-many Unary Extension to RA Aggregation Extended Projection Mapper Type of function Aggregate Function Set Value Scalar Function Value Value Mapper Function Value Set

17 Mapper example Extra slide 4

18 18 Simplified SELECT syntax for mappers Syntactic details for mapper functions

19 Example: Query with the mapper operator 19

20 Union 20

21 Recursive Query 21

22 22 Table function n An iterator that scans the input relation once n With a nested while loop that inserts pipes multiple several output tuples

23 23 Algebraic rewriting rules Pushdown of projections Introduction of projections Pushdown of selections Pushing selections to mapper functions Distributing mappers over unions Distributing mappers over Cartesian products

24 24 Statistics for computing the cost of One-to-many data transformations n Given a mapper 1. Average selectivity of the mapper ( ) Ratio of input tuples that are transformed 2. Average fanout factor of the mapper ( ) Ratio of output tuples produced for each input tuple 3. Average mapper function cost ( ) Cost of evaluating a mapper function

25 25 Estimates for cost-based plan selection n Estimating the cardinality of a mapper expression Fanout factor number of input tuples Selectivity factor input tuples that are not filtered total number of output tuples n Estimating the CPU cost of a mapper expression number of input tuples CPU cost of the cartesian product operation CPU cost of evaluating all the mapper functions per-tuple CPU cost total CPU cost

26 26 Cache-based evaluation 1/2 Cache of Function Results LRU stack Utility list Input relation Most Recently Used Cost Most Useful O(m.log(m)) Output relation Utility Recency Least Freq Recently Used Least Useful Next victim LRU LUR

27 27 Cache-based evaluation 2/2 Cache of Function Results LRU stacks Input relation Most Recent High Frequency Low Frequency Least Recent Aproximate Utility Output relation Selection of the Next Victim XLRU

28 28 An utility metric based on statistical inference Time of arrival (t a ) time of replacement (t 0 ) Time of departue (t d )... Ref string (signal) Time of last ref (t l ) time... n Which entry to replace? n Evolution of (1-θ) k? n Utility u t 0(e)based on: Ref prob for θ = 5/2 Ref prob for θ = 7/2 Number of past references n h Cost c Avg frequency θ Time to last k = (t 0 t l )

29 Experimental Setup OEX and DBX RDBMS logs temp Extra slide 2 source Java target XXL framework n Workload LOANEVT and LOANS synthetic input relations (on raw devices) Relations sizes: 100K, 500K, 1M, 2.5M, 5M (in tuples)

30 Settings checklist n n Same configuration for SGBDs Same block size Same multi-block read count Same size for Buffer pools Same concurrency settings (DBRD and DBWR processes) Align the configuration of the SGBDS with Java Same physical conditions for I/O Same record-size Same number of records per page Logging disabled Asynchronous I/O disabled Parallel query execution disabled

31 31 Resource consumption Union Unpivot RQ TF > fanout more input more output bigger query input once more output input once more output more temp input once more output > select same input more output same input more output same input more temp more output same input more output

32 32 Throughput of one-to-many transformations ~50K ~33K UN/SQL PV/SQL

33 33 Influence of selectivity similar degradation fastest degradation

34 34 Original v.s. optimized expression TF Mapper 3x faster 3x faster