Do we really understand SQL?

Transcription

1 Do we really understand SQL? Leonid Libkin University of Edinburgh Joint work with Paolo Guagliardo, also from Edinburgh

2 Basic questions We are taught that the core of SQL is essentially syntax for relational calculus (first-order logic). Is it true? We are taught that core SQL can be translated into relational algebra. Is it true? We are taught that SQL needs 3-valued logic to deal with missing information (nulls). Is it true?

3 Motivation Why even ask such questions? It s the stuff from the 1980s (or earlier). It s all in database textbooks and taught in all database courses. This is exactly what we thought until we got into some specific problems related to real-life SQL So we start with a bit of history

4 Old days (before 1969) Various ad-hoc database modes: network hierarchical writing queries: a very elaborate task ll changed in 1969: Codd s relational model; now dominates the world

5 Relational Model Orders Pay Customer ORDER_ID TITLE PRICE Ord1 Big Data 30 Ord2 SQL 35 Ord3 Logic 50 CUST_ID c1 c2 ORDER Ord1 Ord2 CUST_ID c1 c2 NME John Mary

6 Relational Model Orders Pay Customer ORDER_ID TITLE PRICE Ord1 Big Data 30 Ord2 SQL 35 Ord3 Logic 50 CUST_ID c1 c2 ORDER Ord1 Ord2 CUST_ID c1 c2 NME John Mary Language: Relational lgebra (R) projection π (find book titles) selection σ (find books that cost at least 40) Cartesian product union difference -

7 Queries Find ids of customers who buy all books: πcust_id (Pay) - πcust_id ((π cust_id(pay) πtitle(order)) - πcust_id,title (σorder_id=order (Order Pay)))

8 Queries Find ids of customers who buy all books: πcust_id (Pay) - πcust_id ((π cust_id(pay) πtitle(order)) - πcust_id,title (σorder_id=order (Order Pay))) That s not pretty. But here is a better idea (1971): express queries in logic

9 Queries Find ids of customers who buy all books: πcust_id (Pay) - πcust_id ((π cust_id(pay) πtitle(order)) - πcust_id,title (σorder_id=order (Order Pay))) That s not pretty. But here is a better idea (1971): express queries in logic {c (o,t,p) Order (o,t,p ) Order: (c,o ) Pay}

10 Queries Find ids of customers who buy all books: πcust_id (Pay) - πcust_id ((π cust_id(pay) πtitle(order)) - πcust_id,title (σorder_id=order (Order Pay))) That s not pretty. But here is a better idea (1971): express queries in logic {c (o,t,p) Order (o,t,p ) Order: (c,o ) Pay} This is first-order logic (FO). Codd 1971: R = FO.

11 History continued Of course programmers don t write logical sentences, they need a programming syntax. Enters SQL: SELECT P.cust_id FROM P WHERE NOT EXISTS (SELECT * FROM Order O WHERE NOT EXISTS (SELECT * FROM Order O1 WHERE O1.title=O.title ND O1.order_id=P.order))

12 History continued Of course programmers don t write logical sentences, they need a programming syntax. Enters SQL: SELECT P.cust_id FROM P WHERE NOT EXISTS xf(x) = (SELECT * FROM Order O x F(x) WHERE NOT EXISTS (SELECT * FROM Order O1 WHERE O1.title=O.title ND O1.order_id=P.order))

13 History continued Of course programmers don t write logical sentences, they need a programming syntax. Enters SQL: SELECT P.cust_id FROM P WHERE NOT EXISTS (SELECT * FROM Order O WHERE NOT EXISTS (SELECT * FROM Order O1 WHERE O1.title=O.title ND O1.order_id=P.order)) xf(x) = x F(x) Idea: Take FO and turn into into programming syntax. Then use R to implement queries.

14 SQL development SQL has since become the dominant language for relational databases Standards: SQL-86, SQL-89, SQL-92, SQL:1999, SQL:2003, SQL:2008, SQL:2011, SQL:2016 The latest standard is in 9 parts, will make you $1000 poorer if you buy them all. But the core remains the same, essentially FO. nd it is the main big data tool!

15 Data scientists favorite tools 70% TOOLS LNGUGES, DT PLTFORMS, NLYTICS 60% Share of Respondents 50% 40% 30% 20% 10% 0% SQL Excel Python R MySQL Python: numpy, scipy, scikit-learn ggplot Microsoft SQL Server Tableau JavaScript Matplotlib (Python) Java PostgreSQL Oracle D3 Homegrown analysis tools Hive Spark Cloudera Visual Basic/VB MongoDB pache Hadoop SS C++ PowerPivot Scala Tool: language, data platform, analytics SQLite C Pig mazon RedShift Weka Hbase mazon Elastic MapReduce (EMR) Perl SPSS Teradata

16 Future data scientists favorite tools

17 But do we understand it? Even the basic fragment, that stays the same in all the Standards: does it have the power of R? Does it have the power of FO? Is there a formal semantics of it? Let s do a little quiz and see how well we know the basics.

18 TSK: Relations R(), S() Compute R - S.

19 TSK: Relations R(), S() Compute R - S. Every student will write: select R. from R where R. not in (select S. from S)

20 TSK: Relations R(), S() Compute R - S. Every student will write: select R. from R where R. not in (select S. from S) nd they are taught it is equivalent to : select R. from R where not exists (select S. from S where S.=R.)

21 TSK: Relations R(), S() Compute R - S. Every student will write: select R. from R where R. not in (select S. from S) nd they are taught it is equivalent to : select R. from R where not exists (select S. from S where S.=R.) and that they can do it directly in SQL: select * from r except select * from s

22 TSK: Relations R(), S() Compute R - S. R 1 null S null Every student will write: select R. from R where R. not in (select S. from S) nd they are taught it is equivalent to : select R. from R where not exists (select S. from S where S.=R.) and that they can do it directly in SQL: select * from r except select * from s

23 TSK: Relations R(), S() Compute R - S. R 1 null S null Outputs: Every student will write: select R. from R where R. not in (select S. from S) nd they are taught it is equivalent to : select R. from R where not exists (select S. from S where S.=R.) and that they can do it directly in SQL: select * from r except select * from s

24 TSK: Relations R(), S() Compute R - S. R 1 null S null Outputs: Every student will write: select R. from R where R. not in (select S. from S) nd they are taught it is equivalent to : select R. from R where not exists (select S. from S where S.=R.) and that they can do it directly in SQL: select * from r except select * from s

25 TSK: Relations R(), S() Compute R - S. R 1 null S null Outputs: Every student will write: select R. from R where R. not in (select S. from S) nd they are taught it is equivalent to : 1 null select R. from R where not exists (select S. from S where S.=R.) and that they can do it directly in SQL: select * from r except select * from s

26 TSK: Relations R(), S() Compute R - S. R 1 null S null Outputs: Every student will write: select R. from R where R. not in (select S. from S) nd they are taught it is equivalent to : 1 null select R. from R where not exists (select S. from S where S.=R.) and that they can do it directly in SQL: select * from r except select * from s 1

27 n exam question that nicely brings down the average grade What is the output of these queries? SELECT 1 FROM S WHERE (null = ((null = ((null = ((null = null) is null)) is null)) is null)) is null SELECT 1 FROM S WHERE (null = ((null = ((null = ((null = null) is null)) is null)) is null))

28 n exam question that nicely brings down the average grade What is the output of these queries? SELECT 1 FROM S WHERE (null = ((null = ((null = ((null = null) is null)) is null)) is null)) is null 1 SELECT 1 FROM S WHERE (null = ((null = ((null = ((null = null) is null)) is null)) is null))

29 SQL vs Relational lgebra: attributes may repeat Q = SELECT R., R. FROM R on 1 gives 1 1 null null null

30 SQL vs Relational lgebra: attributes may repeat Q = SELECT R., R. FROM R on 1 gives 1 1 null null null Let s use it as a subquery: Q = SELECT * FROM (Q) S T

31 SQL vs Relational lgebra: attributes may repeat Q = SELECT R., R. FROM R on 1 gives 1 1 null null null Let s use it as a subquery: Q = SELECT * FROM (Q) S T Output: Postgres: as above Oracle, MS SQL Server: compile-time error

32 SQL vs Relational lgebra: attributes may repeat Q = SELECT R., R. FROM R on 1 gives 1 1 null null null Let s use it as a subquery: Q = SELECT * FROM (Q) S T Output: Postgres: as above Oracle, MS SQL Server: compile-time error SELECT R. FROM R WHERE EXISTS (Q )

33 SQL vs Relational lgebra: attributes may repeat Q = SELECT R., R. FROM R on 1 gives 1 1 null null null Let s use it as a subquery: Q = SELECT * FROM (Q) S T Output: Postgres: as above Oracle, MS SQL Server: compile-time error SELECT R. FROM R WHERE EXISTS (Q ) nswer: 1 null

34 Why do we find these questions difficult? Reason 1: there is no formal semantics of SQL. The Standard is rather vague, not written formally, and different vendors interpret it differently. Reason 2: theory works with a simplified model, no nulls, no duplicates. Under these assumptions several semantics exist ( ) but they do not model the real language.

35 It is much harder to deal with the real thing than with theoretical abstractions

36 It is much harder to deal with the real thing than with theoretical abstractions

37 nother example: Query equivalences Q1(x) :- T(x,y) Q2(x) :- T(x,y), T(u,v)

38 nother example: Query equivalences Q1(x) :- T(x,y) Q2(x) :- T(x,y), T(u,v) In theory: equivalent; on B return 1 3

39 nother example: Query equivalences Q1(x) :- T(x,y) Q2(x) :- T(x,y), T(u,v) In theory: equivalent; on B return 1 3 Now the same in SQL:

40 nother example: Query equivalences Q1(x) :- T(x,y) Q2(x) :- T(x,y), T(u,v) In theory: equivalent; on B return 1 3 Now the same in SQL: Q1 = SELECT R. FROM R returns 1 3

41 nother example: Query equivalences Q1(x) :- T(x,y) Q2(x) :- T(x,y), T(u,v) In theory: equivalent; on B return 1 3 Now the same in SQL: Q1 = SELECT R. FROM R returns 1 3 Q2 = SELECT R1. FROM R R1, R R2 returns

42 The infamous NULL Comparisons with nulls, like 2 = NULL, result in truth value unknown It then propagates: true unknown = unknown, true unknown = true rules of propositional 3-valued logic of Kleene When condition is evaluated, only tuples for which it is true are returned false and unknown are treated the same It s a weird logic and it is not the 3-valued predicate calculus!

43 The bottom line Many spherical cows out there but no real one. There are lots and lots of issues to address to give proper semantics of SQL None of the simplified semantics came even close. We do it for the basic fragment of SQL: SELECT-FROM-WHERE without aggregation but with pretty much everything else

44 Syntax : := T 1 S N 1,..., T k S N k for =(T 1,...,T k ), =(N 1,...,N k ), k > 0 : 0 := t 1 S N1, 0..., t m S Nk 0 for =(t 1,...,t m ), 0 =(N1,...,N 0 m), 0 m>0 Queries: Q := SELECT [ DISTINCT ] : 0 FROM : WHERE SELECT [ DISTINCT ] FROM : WHERE Q (UNION INTERSECT EXCEPT) [LL ] Q Conditions: := TRUE FLSE P (t 1,...,t k ),P2P t IS [ NOT ] NULL t [ NOT ] IN Q EXISTS Q ND OR NOT P Names: either simple (R, ) or composite (R.) Terms t: constants, nulls, or composite names Predicates: anything you want on constants

45 Semantics: labels `(R) = tuple of names provided by the schema `( ) =`(T 1 ) `(T k ) for =(T 1,...,T k ) SELECT [ DISTINCT ] : 0 ` FROM : WHERE = 0 ` SELECT [ DISTINCT ] FROM : WHERE = `( ) ` Q 1 (UNION INTERSECT EXCEPT) [LL ] Q 2 = `(Q 1 )

46 Semantics Q D,η,x Q: query D: database η: environment (values for composite names) x: Boolean switch to account for non-compositional nature of SELECT * (to show where we are in the query)

47 Semantics of terms JtK = J(t 1,...,t n )K = 8 >< () c >: NULL if t = if t = c 2 C if t = NULL Jt 1 K,...,Jt n K

48 Semantics: queries u s FROM : WHERE v SELECT : 0 FROM : WHERE u v SELECT FROM : WHERE u v SELECT FROM : WHERE s SELECT DISTINCT : 0 FROM : WHERE JRK D,,x = R D J : K D,,x = JT 1 K D,,0 JT k K D,,0 for =(T 1,...,T k ) 8 { < } ~ } ~ } ~ { D,,x D,,x D,,0 D,,1 D,,x = : 8 < = r,..., r {z } k times : J K 0,...,J K 0 {z } k times u } SELECT `( : ):`( ) = v FROM : ~ WHERE u = v SELECT c S N } FROM : ~ WHERE D,,1 s { SELECT : = " 0 FROM : WHERE r 2 k J : K D,,0, J K D, 0 = t, 0 = r `( : ) 0 = r `( : ), r 2 k s FROM : WHERE D,,0 for arbitrary c 2 C and N 2 N D,,x! { 9 = ; D,,x 9 = ;

49 Semantics: conditions JP (t 1,...,t k )K D, = Jt IS NULLK D, = 8 >< t if P Jt 1 K,...,Jt k K holds and Jt i K 6= NULL for all i 2{1,...,k} f if P Jt 1 K,...,Jt k K does not hold and Jt i K 6= NULL for all i 2{1,...,k} >: u if Jt i K = NULL for some i 2{1,...,k} ( t if JtK = NULL f if JtK 6= NULL Jt IS NOT NULLK D, = Jt IS NULLK D, n^ J(t 1,...t n )=(t 0 1,...,t 0 n)k D, = Jt i = t 0 ik D, J(t 1,...t n ) 6= (t 0 1,...,t 0 n)k D, = J t IN QK D, = i=1 8 >< t f >: u J t NOT IN QK D, = J t IN QK D, ( t if JQK D,,1 6=? JEXISTS QK D, = f if JQK D,,1 =? n_ Jt i 6= t 0 ik D, if 9 r 2 JQK D,,0 s.t. J t = rk D, = t if 8 r 2 JQK D,,0 s.t. J t = rk D, = f r 2 JQK D,,0 s.t. J t = rk D, = t and 9 r 2 JQK D,,0 s.t. J t = rk D, 6= f i=1 JTRUEK D, = t J 1 ND 2 K D, = J 1 K D, ^ J 2 K D, JNOT K D, = J K D, JFLSEK D, = f J 1 OR 2 K D, = J 1 K D, _ J 2 K D, Truth Tables: ^ t f u t t f u f f f f u u f u _ t f u t t t t f t f u u t u u t f u f t u

50 Semantics: operations JQ 1 UNION LL Q 2 K D,,x = JQ 1 K D,,0 [ JQ 2 K D,,0 JQ 1 INTERSECT LL Q 2 K D,,x = JQ 1 K D,,0 \ JQ 2 K D,,0 JQ 1 EXCEPT LL Q 2 K D,,x = JQ 1 K D,,0 JQ 2 K D,,0 JQ 1 UNION Q 2 K D,,x = " JQ 1 UNION LL Q 2 K D,,x JQ 1 INTERSECT Q 2 K D,,x = " JQ 1 INTERSECT LL Q 2 K D,,x JQ 1 EXCEPT Q 2 K D,,x = " JQ 1 K D,,0 JQ 2 K D,,0 Bag interpretation of operations; is duplicate elimination

51 Looks simple, no? It does not. Such basic things as variable binding changed several times till we got them right. The meaning of the new environment: FROM τ : β WHERE θ D,η,x = r,..., r }{{} k times r k τ : β D,η,0, θ D,η = t, η = η r l(τ : β) in η, unbind every name that occurs among labels of the FROM clause then bind non-repeated names among those to values taken from record r

52 How do we know we got it right? Since the Standard is rather vague, there is only one way experiments. But what kind of benchmark can we use? For performance studies there are standard benchmarks like TPC-H. But they won t work for us: not enough queries.

53 Experimental Validation Benchmarks have rather few queries (22 in TPC-H). Validating on 22 queries is not a good evidence. But we can look at benchmarks, and then generate lots of queries that look the same. In TPC-H: 8 tables, maximum nesting depth = 3, average number of tables per query = 3.2, at most 8 conditions in WHERE (except two queries)

54 Validation: results Small adjustments of the Standard semantics (for Postgres and Oracle) Random query generator Naive implementation of the semantics Finally: experiments on 100,000 random queries

55 Validation: results Small adjustments of the Standard semantics (for Postgres and Oracle) Random query generator Naive implementation of the semantics Finally: experiments on 100,000 random queries Yes, we got it right!

56 What can we do with this? Equivalence of basic SQL and Relational lgebra: formally proved for the first time. 3-valued logic of SQL vs the usual Boolean logic: is there any difference?

57 Basic SQL = Relational lgebra We formally prove SQL = Relational lgebra (R) with nulls, subqueries, bags, all there is. nd R has to be defined properly too, to use bags and SQL s 3-valued logic. a small caveat: in R, attributes cannot repeat. So the equality is wrt queries that do not return repeated attributes.

58 3-valued logic of nulls From the early SQL days and database textbooks: if you have nulls, you need 3-valued logic. But 3-valued logic is not the first thing you think of as a logician. nd it makes sense to think as a logician: after all, the core of SQL is claimed to be first-order logic in a different syntax.

59 What would a logician do?

60 What would a logician do? First Order Logic (FO) domain has usual values and NULL Syntactic equality: NULL = NULL but NULL 5 etc Boolean logic rules for,, Quantifiers: is conjunction, is disjunction

61 What did SQL do?

62 What did SQL do? 3-valued FO (a textbook version) domain has usual values and NULL comparisons with NULL result in unknown Kleene logic rules for,, Quantifiers: is conjunction, is disjunction

63 What did SQL do? 3-valued FO (a textbook version) domain has usual values and NULL comparisons with NULL result in unknown Kleene logic rules for,, Quantifiers: is conjunction, is disjunction Seemingly more expressive.

64 What did SQL do? 3-valued FO (a textbook version) domain has usual values and NULL comparisons with NULL result in unknown Kleene logic rules for,, Quantifiers: is conjunction, is disjunction Seemingly more expressive. But does it correspond to reality?

65 SQL logic is NOT 2-valued or 3-valued: it s a mix Conditions in WHERE are evaluated under 3-valued logic. But then only those evaluated to true matter. Studied before only at the level of propositional logic. In 1939, Russian logician Bochvar wanted to give a formal treatment of logical paradoxes. He needed to assert that something is true, and introduced a new connective: p means that p is true. mazingly, 40 years later SQL adopted the same idea.

66 What did SQL really do? 3-valued FO with : domain has usual values and NULL comparisons with NULL result in unknown Kleene logic rules for,, Quantifiers: is conjunction, is disjunction dd with the semantics true, if φ is true φ = { false, if φ is false or unknown

67 What IS the logic of SQL?

68 What IS the logic of SQL? We have: logician s 2-valued FO 3-valued FO (Kleene logic) 3-valued FO + Bochvar s assertion (SQL logic)

69 What IS the logic of SQL? We have: logician s 2-valued FO 3-valued FO (Kleene logic) 3-valued FO + Bochvar s assertion (SQL logic) ND THEY RE LL THE SME!

70 THEOREM: can be expressed in 3-valued FO. 3-valued FO = 3-valued FO with THEOREM: For every formula φ of 3-valued FO, there is a formula ψ of the usual 2-valued FO such that φ is true ψ is true

71 THEOREM: can be expressed in 3-valued FO. 3-valued FO = 3-valued FO with THEOREM: For every formula φ of 3-valued FO, there is a formula ψ of the usual 2-valued FO such that φ is true ψ is true Translations work at the level of SQL too!

72 2-valued SQL Idea 3 simultaneous translations: conditions P P t and P f Queries Q Q P t and P f are Boolean conditions: P t / P f is true iff P under 3-valued logic is true / false. In Q we simply replace P by P t

73 2-valued SQL: translation P ( t) t = P ( t) P (t 1,...,t k ) f = NOT P (t 1,...,t k ) ND t IS NOT NULL (EXISTS Q) t = EXISTS Q 0 (EXISTS Q) f = NOT EXISTS Q 0 ( 1 ^ 2 ) t = t 1 ^ t 2 ( 1 ^ 2 ) f = f 1 _ f 2 ( 1 _ 2 ) t = t 1 _ t 2 ( 1 _ 2 ) f = f 1 ^ f 2 ( ) t = f ( ) f = t (t IS NULL) t = t IS NULL (t IS NULL) f = t IS NOT NULL ( t IN Q) t = t IN Q 0 (t 1,...,t n ) IN Q f = NOT EXISTS SELECT * FROM Q 0 S N( 1,..., n ) WHERE (t 1 IS NULL OR 1 IS NULL OR t 1 = N. 1 ) ND ND (t n IS NULL OR n IS NULL OR t n = N. n ) 7! 0 Note: a lot of disjunctions with IS NULL conditions

74 Shall we switch to 2-valued SQL? Not so fast perhaps. Two reasons: all the legacy code that uses 3-values using 2 truth values introduces many new disjunctions. nd DBMSs don t like disjunctions!

75 Shall we switch to 2-valued SQL? Not so fast perhaps. Two reasons: all the legacy code that uses 3-values using 2 truth values introduces many new disjunctions. nd DBMSs don t like disjunctions! s to why, this comment line in Postgres optimizer code sheds some light:

76 Shall we switch to 2-valued SQL? Not so fast perhaps. Two reasons: all the legacy code that uses 3-values using 2 truth values introduces many new disjunctions. nd DBMSs don t like disjunctions! s to why, this comment line in Postgres optimizer code sheds some light: /* we stop as soon as we hit a non-nd item */

77 Questions?