A Collective, Probabilistic Approach to Schema Mapping

Transcription

1 A Collective, Probabilistic Approach to Schema Mapping Angelika Kimmig, Alex Memory, Renée Miller, Lise Getoor ILP 2017 (published at ICDE 2017) 1

2 Context: Data Exchange & source emp id company 1 Alice SAP 2 Bob IBM 3 Pat MS 4 Carl X 5 Tom Y7 proj topic mgr lead BigData 1 2 ML 1 1 egov 4 5 DM 5 5 Data Integration 2 BigData Alice 111 ML Alice 111 MT Justin 444 Deep Ann 333 leader Alice Bob Jim Ann Igor org target oid 111 SAP 222 MS 333 Z 444 HC

3 Context: Data Exchange & source emp id company 1 Alice SAP 2 Bob IBM 3 Pat MS 4 Carl X 5 Tom Y7 proj topic mgr lead BigData 1 2 ML 1 1 egov 4 5 DM 5 5 Data Integration transfer data 2 BigData Alice 111 ML Alice 111 MT Justin 444 Deep Ann 333 leader Alice Bob Jim Ann Igor org target oid 111 SAP 222 MS 333 Z 444 HC

4 Context: Data Exchange & source emp id company 1 Alice SAP 2 Bob IBM 3 Pat MS 4 Carl X 5 Tom Y7 proj topic mgr lead BigData 1 2 ML 1 1 egov 4 5 DM 5 5 Data Integration transfer data rewrite queries 2 BigData Alice 111 ML Alice 111 MT Justin 444 Deep Ann 333 leader Alice Bob Jim Ann Igor org target oid 111 SAP 222 MS 333 Z 444 HC

5 Context: Data Exchange & source emp id company 1 Alice SAP 2 Bob IBM 3 Pat MS 4 Carl X 5 Tom Y7 proj topic mgr lead BigData 1 2 ML 1 1 egov 4 5 DM 5 5 Data Integration transfer data need schema mapping rewrite queries 2 BigData Alice 111 ML Alice 111 MT Justin 444 Deep Ann 333 leader Alice Bob Jim Ann Igor org oid target 111 SAP 222 MS 333 Z 444 HC

6 source emp proj Context: Data Exchange & id company 1 Alice SAP 2 Bob IBM 3 Pat MS 4 Carl X Data Integration transfer data emp(i,n,c) leader(n) need schema proj(t,m,l) S. O. (T,S,O) BigData Alice 111 ML Alice 111 MT Justin 444 Deep Ann 333 leader org target 5 Tom Y7 mapping proj(t,m,l) & emp(l,n,c) O. (T,N,O) & org(o,c) oid Alice 111 SAP topic mgr lead Bob 222 MS BigData 1 2 Jim 333 Z ML 1 1 rewrite queries egov 4 5 Ann 444 HC DM 5 5 Igor 2

7 Schema Mapping st tgd = source-target tuple generating dependency [Fagin et al, 05; ten Cate & Kolaitis, 10] first order rule 8x (x)!9y (x, y) conjunctive query on source schema conjunctive query on target schema a schema mapping is a set of st tgds 3

8 Goal: learn schema mapping proj topic mgr lead BigData 1 2 ML 1 1 emp id company 1 Alice SAP 2 Bob IBM 3 Pat MS from data BigData Alice 111 ML Alice 111 leader org Alice Bob oid SAP MS proj(t,m,l) & emp(l,n,c) leader(n) emp(i,n,c) O. org(o,c) proj(t,m,l) & emp(m,n,c) O. (T,N,O) & org(o,c) 4

9 Goal: learn schema mapping proj topic mgr lead BigData 1 2 ML 1 1 emp id company 1 Alice SAP 2 Bob IBM 3 Pat MS from data BigData Alice 111 ML Alice 111 leader org Alice Bob oid SAP MS Challenges: ambiguous metadata imperfect data existentials / nulls proj(t,m,l) & emp(l,n,c) leader(n) emp(i,n,c) O. org(o,c) proj(t,m,l) & emp(m,n,c) O. (T,N,O) & org(o,c) 4

10 Here: Given data example (I,J), candidate set C Find optimal M C for (I,J) proj I J C topic mgr lead BigData 1 2 BigData Alice 111 ML 1 1 ML Alice 111 emp leader org id company oid 1 Alice SAP Alice 111 SAP 2 Bob IBM Bob 222 MS 3 Pat MS emp(i,n,c) leader(n) proj(t,m,l) S. O. (T,S,O) proj(t,m,l) & emp(l,n,c) leader(n) proj(t,m,l) & emp(m,n,c) O. (T,N,O) & org(o,c) proj(t,m,l) & emp(l,n,c) O. (T,N,O) & org(o,c) emp(i,n,c) O. org(o,c) emp(i,n,c) O. org(o,n)... M proj(t,m,l) & emp(l,n,c) leader(n) emp(i,n,c) O. org(o,c) proj(t,m,l) & emp(m,n,c) O. (T,N,O) & org(o,c) 5

11 Full st tgds (no ) errors of M explained by M J Universal(M,I) not explained by M goal: find small M that maximizes intersection 6

12 Arbitrary st tgds: replace containment with homomorphism checks errors of M explained by M Universal(M,I) J 7

13 Arbitrary st tgds: replace containment with homomorphism checks errors of M explained by M Universal(M,I) J tuples where we can replace nulls by constants such that we get tuples in J = no errors 7

14 Arbitrary st tgds: replace containment with homomorphism checks errors of M explained by M Universal(M,I) J tuples where we can replace nulls by constants such that we get tuples in J = no errors tuples which can be obtained from tuples in Universal(M,I) replacing nulls by constants = (partially) explained 7

15 Example BigData Bob null1 ML Alice null2 BigData Alice 111 ML Alice 111 leader org oid leader org oid null1 IBM Alice 111 SAP null2 SAP Bob 222 MS 8

16 Example BigData Bob null1 ML Alice null2 BigData Alice 111 ML Alice 111 leader org oid leader org oid null1 IBM Alice 111 SAP null2 SAP Bob 222 MS no errors: replace null2 by

17 Example BigData Bob null1 ML Alice null2 BigData Alice 111 ML Alice 111 leader org oid null1 null2 IBM SAP leader Alice Bob org oid 111 SAP 222 MS no errors: replace null2 by to explain these

18 Example errors: can t replace null1 to get tuples in J BigData Bob null1 ML Alice null2 BigData Alice 111 ML Alice 111 leader org oid null1 null2 IBM SAP leader Alice Bob org oid 111 SAP 222 MS no errors: replace null2 by to explain these

19 Example errors: can t replace null1 to get tuples in J not explained by any tuple in Universal(M,I) BigData Bob null1 ML Alice null2 BigData Alice 111 ML Alice 111 leader org oid null1 null2 IBM SAP leader Alice Bob org oid 111 SAP 222 MS no errors: replace null2 by to explain these

20 Task Universal(M,I) J Given source schema S, target schema T errors of M data example (I,J) explained by M set C of candidate st tgds Find an optimal mapping M, i.e., arg min M C 2 4size(M)+ X t2j(1 explains(m,t)) + X t2universal(c,i) J 3 error(m,t) 5 9

21 Task Universal(M,I) J Given source schema S, target schema T errors of M data example (I,J) set C of candidate st tgds explained by M NP-hard even Find an optimal mapping M, i.e., arg min M C 2 4size(M)+ X t2j(1 explains(m,t)) + X t2universal(c,i) for full st tgds J 3 error(m,t) 5 9

22 Probabilistic Soft Logic (PSL) declarative language to specify probabilistic models over logical atoms / relational domains PSL program = set of weighted first order rules w : b 1 ( ~ X) ^...^ b n ( ~ X)! h 1 ( ~ X) _..._ h m ( ~ X) MPE inference = finding most likely model efficient approximate solver with guarantee on solution quality 10

23 Our PSL program J(T ) given J(T ) covers(f, T) creates(f, T) creates(f, T) covers(f, T) 11

24 Our PSL program J(T ) given J(T ) covers(f, T) covers(f, T) creates(f, T) creates(f, T) find optimal M: in(f ), F 2 M 11

25 Our PSL program J(T ) given J(T ) covers(f, T) covers(f, T) creates(f, T) creates(f, T) find optimal M: in(f ), F 2 M minimize #errors: 1:in(F ) ^ creates(f, T)! J(T ) 11

26 Our PSL program J(T ) given J(T ) covers(f, T) covers(f, T) creates(f, T) creates(f, T) find optimal M: in(f ), F 2 M minimize #errors: minimize #unexplained: 1:in(F ) ^ creates(f, T)! J(T ) 1:J(T )!9F.covers(F, T) ^ in(f ) 11

27 Our PSL program J(T ) given J(T ) covers(f, T) covers(f, T) creates(f, T) creates(f, T) find optimal M: in(f ), F 2 M minimize #errors: minimize #unexplained: minimize size of M: 1:in(F ) ^ creates(f, T)! J(T ) 1:J(T )!9F.covers(F, T) ^ in(f ) size(f ):in(f )!? 11

28 New PSL construct: prioritized disjunction rules 1:J(T )!9F.covers(F, T) ^ in(f ) observed priority between 0 (low) and 1 (high) to be inferred automatically transformed into set of standard PSL rules expressing a preference for inferred atoms with higher priority 12

29 Experimental evaluation scenarios generated using ibench [Arocena et al, 15] candidate st tgds generated using Clio [Fagin et al, 09] E1: increasingly noisy candidates, perfect data metadata-only baseline suffers, we get perfect mappings E2: ambiguous set of candidates, increasingly noisy data high quality mappings found for up to 25% unexpected and 10% missing target tuples in J 13

30 Given - metadata - data example - candidate st tgds Our Contributions Find - small set of st tgds - minimally invalid - maximally explaining 14

31 Given - metadata - data example - candidate st tgds Our Contributions Collective Mapping Discovery (CMD) declarative probabilistic model + MPE inference using Probabilistic Soft Logic (PSL) Find - small set of st tgds - minimally invalid - maximally explaining 14

32 Given - metadata - data example - candidate st tgds Find Collective Mapping - small set of st tgds - minimally invalid Discovery (CMD) declarative probabilistic model + MPE inference using Probabilistic Soft Logic (PSL) - maximally explaining Our Contributions supports arbitrary st tgds jointly reasons about metadata and data handles noisy input efficient solver with quality guarantee declarative, extensible definition of optimization 14

33 Given - metadata - data example - candidate st tgds Find Collective Mapping - small set of st tgds - minimally invalid Discovery (CMD) declarative probabilistic model + MPE inference using Probabilistic Soft Logic (PSL) - maximally explaining Our Contributions supports arbitrary st tgds jointly reasons about metadata and data handles noisy input efficient solver with quality guarantee declarative, extensible definition of optimization Thanks! 14