Michalis Petropoulos Alin Deutsch Yannis Papakonstantinou Athens University of Economics and Business, May 2006

Transcription

1 Interactive Query Formulation over Web Service-Accessed Sources Michalis Petropoulos Alin Deutsch Yannis Papakonstantinou SIGMOD 2006 Best Paper Runner-Up Athens University of Economics and Business, May 2006

2 Large-Scale Data Integration Systems End User Developer Web Domain Application Domain Web Forms & Reports Application Application CNET s Top Combinations CNET s Search Desktops PCWorld s Product Finder CNET Computer PCWorld Portals Integration Engineer Integration Domain Mediator Integrated Schema Compatible Combinations of Computers, Routers and Printers Source Owner Source Domain Source Schema Web Service Data Source Web Service Data Source Web Service Source Schema Dell Computers by CPU Cisco Routers by Rate HP Printers by Speed Dell Computers Cisco Routers HP Printers 2

3 Large-Scale Data Integration Systems End User Web Domain Web Forms & Reports Developer Application Domain Application Application What queries can the mediator answer for me? CLIDE Integration Engineer Integration Domain Mediator Integrated Schema Source Owner Source Domain Web Service Web Service Web Service Source Schema Data Source Data Source Source Schema 3

4 Running Example Parameterized Views Schema Dell Schema Cisco Views V1 V2 Computers(cid, cpu, ram, price) NetCards(cid, rate, standard, interface) ComByCpu(cpu) (Computer)* SELECT DISTINCT Com1.* FROM Computers Com1 WHERE Com1.cpu=cpu ComNetByCpuRate(cpu, rate) (Computer, NetCard)* SELECT DISTINCT Com1.*, Net1.* FROM Computers Com1, Network Net1 WHERE Com1.cid=Net1.cid AND Com1.cpu=cpu AND Net1.rate=rate Views Routers(rate, standard, price, type) Computers V3 RouWired() (Router)* for a given cpu SELECT DISTINCT Rou1.* FROM Routers Rou1 WHERE Rou1.type='Wired' Computers & NetCards for a givenv4 cpu RouWireless() & rate (Router)* SELECT DISTINCT Rou1.* FROM Routers Rou1 WHERE Rou1.type='Wireless' Conjunctive Queries CQ Wired Routers Wireless Routers Equality & Comparison Conditions Parameters 4

5 Running Example Integrated Schema Application Developer Mediator Integrated Schema Integrated schema puts together the Dell and Cisco schemas Attribute Associations V1 V2 V3 V4 (Computers.cid, NetCards.cid) (NetCards.rate, Routers.rate) (NetCards.standard, Routers.standard) Dell Cisco 5

6 Sophisticated Mediators Make Feasible Queries Hard to Predict Feasible Queries FQ Equivalent CQ query rewritings using the views Might involve more than one views Order might matter Query: Infeasible Feasible Get all P4 Computers Computers, together with their NetCards and their compatible Wireless Routers Computers.* A123 P B123 P NetCards.* A b USB B g USB Routers.* 10.11b 50 Wireless 54.11g 120 Wireless E B Routers.* 10.11b 50 Wireless 54.11g 120 Wireless Mediator Computers.* A123 P B123 P NetCards.* A b USB B g USB D Mediator A RouWireless() C ComNetByCpuRate( P4, 10 ) V1 ComNetByCpuRate( P4, 54 ) V4 V2 6

7 Problem 1. Large number of sources 2. Large number of views (web-services) 3. Mediator capabilities Developer formulates an application query Is an application query feasible? If not, how do I know which ones are feasible? Previous options: The developer had to browse the view definitions and somehow formulate a feasible query Or formulate queries until a feasible one is found (trial-and-error) No system-provided guidance 7

8 The CLIDE Solution Application Developer Mediator Integrated Schema CLIDE A query formulation interface, which interactively guides the developer toward feasible queries by employing a coloring scheme V1 V2 V3 V4 Dell Cisco 8

9 QBE-Like Interfaces Microsoft SQL-Server 9

10 CLIDE Interface Table Alias Feasibility Flag Selection Boxes Table Boxes Projection Boxes Table, selection, projection and join actions Feasibility Flag Color-based suggestions 10

11 Example Interaction Snapshot 1 Yellow required action All feasible queries require this action White optional action Feasible queries can be formulated w/ or w/o these actions 11

12 Example Interaction Snapshot 2 Blue required choice of action At least one feasible query cannot C ram price Mediator be 512 formulated 400 unless this action is performed A ComByCpu( P4 ) cid A123 B123 cpu P4 P4 ram price B V1 12

13 Example Interaction Snapshot 3 Join Lines: Only yellow and blue are displayed Must appear in Attribute Associations 13

14 Example Interaction Snapshot 4 * any other constant Red prohibited action Does not appear in any feasible query Lead to Dead End state 14

15 Example Interaction Snapshot 5 ram price rate interface price USB 50 USB 120 F Mediator A RouWireless() Routers.* 10.11b 512 Wireless 54.11g 1024Wireless D ComNetByCpuRate( P4, rate) Computers.* A123 P B123 P NetCards.* A b 50 B g 120 V4 B V2 E 15

16 CLIDE Properties Completeness of Suggestions Every feasible query can be formulated by performing yellow and blue actions at every step Summarization of Suggestions At every step, only a minimal number of actions is suggested, i.e., the ones that are needed to preserve completeness Rapid Convergence By Following Suggestions The shortest sequence of actions from a query to any feasible query consists of suggested actions 16

17 Interaction Graph s Selection Action Table Action Join Action Com1 Com1.ram Com1.price Com1.cpu= P4 Net1 Com1.cid=Net1.cid Rou1 Nodes are queries One for each q CQ Edges are actions Table, selection, projection and join actions Green nodes are feasible queries Infinitely big structure All CQ queries All possible combinations of actions formulating them 17

18 Interaction Graph: Colorable Actions Current Node Com1.cid Com1.cpu Com1.cid=* Com1.cpu=* Com1.ram=* Colorable actions A C label outgoing edges of the current node Com1.price=* Net1 Rou1 Com2 18

19 Interaction Graph: Colors Yellow action α Every path from current node n to a feasible node contains α Blue action α At least one feasible query cannot be formulated unless this action is performed (summarization) Current Node Com1.cid Com1.cpu Com1.cid=* Com1.cpu=* Net1 Red action α No path to a feasible node contains α Current Node Com1.cid=Net1.cid Net1.rate= 54Mbps Com1.ram=* Com1.price=* Net1 Rou1 Com2 Net1.rate= 54Mbps Com2 Com2.cid=Net1.cid Com1.cid=Net1.cid Com1.cpu=* Rou1 Com1.cid=Net1.cid Net1.rate=Rou1.rate Com2.cpu= P4 Net1.rate= 54Mbps 19

20 CLIDE Architecture Front-End Back-End Current Query Colored Actions + Feasibility Flag Color Algorithm Seed Queries SQ Parameters Algorithm Actions User Closest Feasible Queries FQ C Closest Feasible Queries Algorithm Aliases Collapse Rule Maximally-Contained Rewriter Minimal Feasible Extension Queries Schemas Views Column Associations Back-End invoked every time the user performs an action i.e., the user arrives at a new node in the interactions graph 20

21 Color Determined By a Finite Set of Feasible Queries Challenge: Infinitely Many Feasible Queries Radius? Closest Feasible Queries FQ C n Solution: Closest Feasible Queries FQ C FQ C is sufficient to color actions in A C Theorem: Set of Closest Feasible Queries is Finite Challenge: How far can the Closest Feasible Queries FQ C be? Solution: Based on Maximally Contained Queries FQ MC 21

22 Maximally Contained Queries FQ MC Query: Q1 Get all Computers Maximally Contained Query Query: Q2 Get all Computers with a given cpu Not Maximally Maximally Contained Contained Query Query: Q3 Q4 Get all Computers with a given cpu ram & ram Assuming fixed SELECT clause (projection list) Covered extensively in literature MiniCon, Bucket, InverseRules Algorithms FQ MC is finite 22

23 Closest Feasible Queries FQ C Algorithm Challenge: How far can the Closest Feasible Queries FQ C be? Solution: Maximally Contained Queries FQ MC Closest Feasible Queries FQ C p L Radius n Maximally Contained Queries FQ MC Compute maximally contained queries FQ MC Theorem: All FQ C queries are reachable via a path of length p p L The radius p L is the longest path to a maximally contained query 23

24 Closest Feasible Queries FQ C Algorithm Challenge: Find the Closest Feasible Queries Closest Feasible Queries FQ C n Maximally Contained Feasible Queries FQ MC More feasible nodes Theorem: All queries in FQ MC are in FQ C But not all queries in FQ C are in FQ MC 24

25 Closest Feasible Queries FQ C Algorithm Solution: Collapse Aliases Closest Feasible Queries FQ C n Maximally Contained Feasible Queries FQ MC Collapse Aliases to compute FQ C \FQ MC Check satisfiability 25

26 Color Algorithm Yellow and Blue An action α is colored based on which closest feasible queries it appear in Yellow, if α appears in all queries in FQ C Blue, if α appears in at least one (but not all) query in FQ C White and Red Attach Maximum Projection Lists to Closest Feasible Queries Projections that can be added to a feasible query, without compromising feasibility Projection α is white if in the maximum projection list Color selections based on projections 26

27 CLIDE Implementation & Optimizations Front-End Maximally-Contained Rewriter Current Query Minimal Feasible Extension Colored Queries Actions FQ + Feasibility Flag ME + Maximum Projection Lists Back-End Redundant Actions Removal Color Algorithm Maximally-Contained Feasible Extension Queries + Seed Maximum Queries Projection SQ Lists Redundant Queries Removal Parameters Algorithm Feasible Extension Queries + Closest Maximum Feasible Projection Queries Lists FQ C Closest Feasible Views Expansion Queries Algorithm Maximally-Contained Aliases Collapse Rule Feasible Queries over Views Minimal Feasible + Containment Mappings Extension Queries Maximally-Contained Rewriter MiniCon Containment Mappings Logging Column Schemas Views Associations Views expansion introduce redundancy Affects CLIDE s rapid convergence and summarization Efficient containment test crucial to redundancy removal 27

28 CLIDE Performance Chains of Stars Time (sec) (2,1) (2,2) 112 Views 168 Views 224 Views 280 Views (4,3) (4,4) (6,5) (6,6) (8,7) (8,8) (10,9) (10,10) (12,11) (12,12) (14,13) (14,14) (# of Joins, # of Selections) in Current Query Queries Schema Views A B 1 B 2 C 1 B i1 B i B im C 1 C C i1 1 C i C im C L A-span = 7 B-span = 3 Selections = 4,6,8,10 B K 28

29 CLIDE Summary First interactive query formulation interface based on source and mediator capabilities Applicability Service-Oriented Architectures Privacy-Preserving Services Contributions Interaction Guarantees: Rapid Convergence, Completeness, Summarization of Suggestions Interaction Graph Back-End Algorithms Closest Feasible Queries, Colors, Parameters Modular, Customizable Architecture 29