MySQL 8.0 Common Table Expressions & Windowing Functions

Transcription

1 1 / 66

2 2 / 66 MySQL 8.0 Common Table Expressions & Windowing Functions Dave Stokes - MySQL Community Manager - Oracle

3 3 / 66 Safe Harbor Statement The following is intended to outline our general product direction. It is intended for information purpose only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied up in making purchasing decisions. The development, release and timing of any features or functionality described for Oracle s product remains at the sole discretion of Oracle.

4 4 / 66 about me Who am I?

5 5 / 66 Dave MySQL Evangelist New Book: MySQL & JSON Pre Order Now on Amazon

6 6 / 66 RDBMS & MySQL Beyond Descriptive Structured Query Language

7 7 / 66 Structured Query Language (SQL)

8 8 / 66 Structured Query Language (SQL) Realtional Model

9 9 / 66 Structured Query Language (SQL) Realtional Model 1970s

10 10 / 66 Structured Query Language (SQL) Realtional Model 1970s First way to return multiple records with one command

11 11 / 66 Structured Query Language (SQL) Realtional Model 1970s First way to return multiple records with one command Descriptive

12 12 / 66 Structured Query Language (SQL) Realtional Model 1970s First way to return multiple records with one command Descriptive Common Table Expresions or CTEs

13 13 / 66 Structured Query Language (SQL) Realtional Model 1970s First way to return multiple records with one command Descriptive Common Table Expresions or CTEs Easier to write, easier to comprehend than sub queries

14 14 / 66 Structured Query Language (SQL) Realtional Model 1970s First way to return multiple records with one command Descriptive Common Table Expresions or CTEs Easier to write, easier to comprehend than sub queries Turns SQL from Descriptive to Imperitive for more powerful programming

15 15 / 66 Structured Query Language (SQL) Realtional Model 1970s First way to return multiple records with one command Descriptive Common Table Expresions or CTEs Easier to write, easier to comprehend than sub queries Turns SQL from Descriptive to Imperitive for more powerful programming Windowing Functions

16 16 / 66 Structured Query Language (SQL) Realtional Model 1970s First way to return multiple records with one command Descriptive Common Table Expresions or CTEs Easier to write, easier to comprehend than sub queries Turns SQL from Descriptive to Imperitive for more powerful programming Windowing Functions Data Analytics

17 17 / 66 Structured Query Language (SQL) Realtional Model 1970s First way to return multiple records with one command Descriptive Common Table Expresions or CTEs Easier to write, easier to comprehend than sub queries Turns SQL from Descriptive to Imperitive for more powerful programming Windowing Functions Data Analytics

18 18 / 66 CTEs & MySQL Common Table Expressions

19 19 / 66 Syntax for Sub Query SELECT * FROM t1 WHERE column1 = (SELECT column1 FROM t2);

20 20 / 66 Syntax for Sub Query SELECT * FROM t1 WHERE column1 = (SELECT column1 FROM t2); Sub queries are hard to write, often harder to understand! Correlated and uncorrelated mess

21 21 / 66 Syntax for Sub Query SELECT * FROM t1 WHERE column1 = (SELECT column1 FROM t2); Sub queries are hard to write, often harder to understand! Correlated and uncorrelated mess A Very poor example SELECT * FROM City WHERE CountryCode = (SELECT Code FROM Country WHERE Code='USA');

22 22 / 66 A Very poor example SELECT * FROM City WHERE CountryCode = (SELECT Code FROM Country WHERE Code='USA'); The MySQL takes the above query and turns it into a Query Plan Note (code 1003): /* select#1 */ select `world`.`city`.`id` AS `ID`, `world`.`city`.`name`as`name`, `world`.`city`.`countrycode` AS CountryCode`, `world`.`city`.`district` AS `District`, `world`.`city`.`population` AS `Population` from `world`.`city` where (`world`.`city`.`countrycode` = (/* select#2 */ select 'USA' from `world`.`country` where 1))

23 23 / 66 Common Table Expression Syntax with_clause: WITH [RECURSIVE] cte_name [(col_name [, col_name]...)] AS (subquery) [, cte_name [(col_name [, col_name]...)] AS (subquery)]...

24 24 / 66 Common Table Expression Syntax with_clause: WITH [RECURSIVE] cte_name [(col_name [, col_name]...)] AS (subquery) [, cte_name [(col_name [, col_name]...)] AS (subquery)]... Example > WITH ->cte AS (SELECT Name, CountryCode FROM city) -> SELECT Name FROM cte limit 1; Name Kabul Note: One colum is SELECTed from the CTE on purpose

25 25 / 66 Naming parameters WITH ccc (x,y) AS (SELECT Name, CountryCode FROM city) SELECT x FROM ccc; Helps when columns names reused

26 CTEs can be joined 26 / 66 WITH cte1 AS (SELECT Name, CountryCode FROM City), cte2 AS (Select Name, Code FROM Country) SELECT cte1.name, Cte2.Name FROM cte1 JOIN cte2 WHERE cte1.countrycode = cte2.code limit 2; Name Name Oranjestad Aruba Kabul Afghanistan

27 CTEs can be joined 27 / 66 WITH cte1 AS (SELECT Name, CountryCode FROM City), cte2 AS (Select Name, Code FROM Country) SELECT cte1.name, Cte2.Name FROM cte1 JOIN cte2 WHERE cte1.countrycode = cte2.code limit 2; Name Name Oranjestad Aruba Kabul Afghanistan Note (code 1003): /* select#1 */ select `world`.`city`.`name` AS `Name`, `world`.`country`.`name` AS `Name` from `world`.`city` join `world`.`country` where (`world`.`city`.`countrycode`= `world`.`country`.`code`) limit 2 Query Plan

28 28 / 66 CTEs can be.. called multiple times can be used with JOINs recursive

29 29 / 66 Recursive Limit WITH RECURSIVE source AS ( SELECT 1 UNION ALL SELECT 1 FROM source ) SELECT * FROM source; ERROR: 3636: Recursive query aborted after 1001 iterations. Try to a larger value.

30 30 / 66 Recursive Limit WITH RECURSIVE source AS ( SELECT 1 UNION ALL SELECT 1 FROM source ) SELECT * FROM source; ERROR: 3636: Recursive query aborted after 1001 iterations. Try to a larger value. Yes, you can run youurself out of resources with recursion.

31 31 / 66 CTE with Recursion WITH RECURSIVE source (count) AS ( SELECT 5 UNION ALL SELECT count - 1 FROM source WHERE count > 0 ) SELECT * FROM source; count rows in set ( sec)

32 32 / 66 A more complex example With CTEs you can compute values recursively WITH RECURSIVE myfunct ( i, j) AS ( SELECT 1, 1 UNION ALL SELECT i + 1, j*(i+1) FROM myfunct WHERE i < 5 ) SELECT i, j FROM myfunct; i j

33 33 / 66 CTE Bene ts Provides imperative processing in SQL Merges multiple SQL queries and their connecting application logic into a single, uni ed SQL query Improves performance by issuing fewer queries reduces transmission overhead, unless server-side functions are being used reduces parsing/optimizing overhead, unless prepared statements are being used Uses the same row visibility snapshot for the entire query, rather than requiring serializable isolation mode

34 34 / 66 Window Functions & MySQL Window Functions

35 35 / 66 What is a Window Function A window function performs a calculation across a set of table rows that are somehow related to the current row. This is comparable to the type of calculation that can be done with an aggregate function. However, window functions do not cause rows to become grouped into a single output row like non-window aggregate calls would. Instead, the rows retain their separate identities. Behind the scenes, the window function is able to access more than just the current row of the query result.

36 36 / 66 Window Function Syntax window_spec: [window_name] [partition_clause] [order_clause] [frame_clause] partition_clause: PARTITION BY expr [, expr]... frame_clause: frame_units frame_extent frame_units: {ROWS RANGE} frame_between: BETWEEN frame_start AND frame_end frame_start, frame_end: { CURRENT ROW UNBOUNDED PRECEDING UNBOUNDED FOLLOWING expr PRECEDING expr FOLLOWING } (gets bumpier from here -- please put on your seat belts)

37 37 / 66 Non Aggragate Functions for Windows CUME_DIST() -Cumulative distribution value DENSE_RANK() -Rank of current row within its partition, without gaps FIRST_VALUE() -Value of argument from rst row of window frame LAG() -Value of argument from row lagging current row within partition LAST_VALUE() -Value of argument from row lagging current row within partition

38 38 / 66 Non Aggregate Functions for Windows continued LEAD() -Value of argument from row leading current row within partition NTH_VALUE() -Value of argument from N-th row of window frame NTILE() -Bucket number of current row within its partition PERCENT_RANK() -Percentage rank value RANK() -Rank of current row within its partition, with gaps ROW_NUMBER() -Number of current row within its partition

39 39 / 66 And Aggregate Functions AVG() BIT_AND() BIT_OR() BIT_XOR() COUNT() MAX() MIN() STDDEV_POP(), STDDEV(), STD() STDDEV_SAMP() SUM() VAR_POP(), VARIANCE() VAR_SAMP()

40 40 / 66 How Does One Recognize a Window Function??

41 41 / 66 How Does One Recognize a Window Function?? Look for the OVER() keyword CREATE TABLE x (y INT unsigned); INSERT INTO x VALUES (1),(2),(3),(4),(5); SELECT y, SUM(y) OVER() FROM x; y SUM(y) OVER()

42 42 / 66 A window function call always contains an OVER clause directly following the window function's name and argument(s). This is what syntactically distinguishes it from a regular function or aggregate function. The OVER clause determines exactly how the rows of the query are split up for processing by the window function. The PARTITION BY list within OVER speci es dividing the rows into groups, or partitions, that share the same values of the PARTITION BY expression(s). For each row, the window function is computed across the rows that fall into the same partition as the current row.

43 43 / 66 An Example of a Window Function in action SELECT y, COUNT(y) OVER(), SUM(y) OVER() FROM x; y COUNT(y) OVER() SUM(y) OVER() rows in set ( sec) In this case the OVER() is all the data in the table.

44 44 / 66 Calulating Average Salary by Department SELECT depname, empno, salary, avg(salary) OVER (PARTITION BY depname) FROM empsalary; depname empno salary avg(salary) OVER (PARTITION BY depname) develop develop develop develop develop personnel personnel sales sales sales rows in set ( sec)

45 45 / 66 Calulating Average Salary by Department SELECT depname, empno, salary, avg(salary) OVER (PARTITION BY depname) FROM empsalary; depname empno salary avg(salary) OVER (PARTITION BY depname) develop develop develop develop develop personnel personnel sales sales sales rows in set ( sec)

46 46 / 66 De ning a Window SELECT y, COUNT(y) OVER w, SUM(y) OVER w FROM x WINDOW w AS (); y COUNT(y) OVER w SUM(y) OVER w rows in set ( sec) It is easy to de ne one or more windows over the data and reference more than once

47 47 / 66 SELECT year, country, product, pro t, SUM(pro t) OVER() AS total_pro t, SUM(pro t) OVER(PARTITION BY country) AS country_pro t FROM sales ORDER BY country, year, product, pro t; year country product pro t total_pro t country_pro t Finland Computer Finland Phone India Calculator India Computer USA Calculator The rst OVER clause is empty, which treats the entire set of query rows as a single partition. The window function thus produces a global sum, but does so for each row. The second OVER clause partitions rows by country, producing a sum per partition (per country). The function produces this sum for each partition row.

48 48 / 66 Window 'Frames' Frames are determined with respect to the current row, which enables a frame to move within a partition depending on the location of the current row within its partition. Examples: By de ning a frame to be all rows from the partition start to the current row, you can compute running totals for each row. By de ning a frame as extending N rows on either side of the current row, you can compute rolling averages.

49 49 / 66 Framing Your Frames frame_start and frame_end can be: UNBOUNDED PRECEDING value PRECEDING CURRENT ROW value FOLLOWING UNBOUNDED FOLLOWING

50 Running totals within each group of time-ordered level values and rolling averages computed from the current row and the rows that immediately precede and follow it 50 / 66 SELECT time, subject, val, SUM(val) OVER (PARTITION BY subject ORDER BY time ROWS UNBOUNDED PRECEDING) AS running_total, AVG(val) OVER (PARTITION BY subject ORDER BY time ROWS BETWEEN 1 PRECEDING AND 1 FOLLOWING) AS running_average FROM observations; time subject val running_total running_average :00:00 st :15:00 st :30:00 st :45:00 st :00:00 xh :15:00 xh :30:00 xh :45:00 xh :00:00 xh

51 51 / 66 LEAD and LAG SELECT y, LAG(y, 2) OVER w as 'Lag', LEAD(y,2) x window w AS (order by y); y Lag Lead NULL 3 2 NULL NULL 5 3 NULL rows in set ( sec)

52 52 / 66 select * from emp; id name department salary Andy Shipping Betty Marketing Tracy Shipping Mike Marketing Sandy Sales James Shipping Carol Sales Company Example -- Data For the next few examples we will use this data

53 53 / 66 SELECT department, COUNT(*), SUM(salary) round(avg(salary), 2) AS avg FROM emp GROUP BY department ORDER BY department; department COUNT(*) SUM(salary) avg Marketing Sales Shipping rows in set ( sec) Company Example -- Simple calculation Calculation without Window Function!!

54 54 / 66 Window Average SELECT name, salary, round(avg(salary) OVER (), 2) AS Avg from emp order by salary desc; name salary Avg Mike James Betty Andy Sandy Tracy Carol rows in set ( sec)

55 55 / 66 Calulate Di erence to Average SELECT name, salary, round(avg(salary) OVER (), 2) AS Avg, round(salary - AVG(salary) over (), 2) AS Diff FROM emp order by salary desc; name salary Avg Diff Mike James Betty Andy Sandy Tracy Carol

56 56 / 66 SELECT name, salary, salary - LEAD(salary, 1) OVER (ORDER BY salary DESC) as diff_next FROM emp ORDER BY salary DESC; name salary diff_next Mike James Betty Andy Sandy Tracy Carol NULL rows in set ( sec) Di erence in Salary Compared to Next Employee

57 57 / 66 SELECT name, salary, salary - LAST_VALUE(salary) OVER w AS more, round((salary - LAST_VALUE(salary) OVER w) / LAST_VALUE(salary) OVER w * 100) AS pct_more FROM emp WINDOW w AS (ORDER BY salary DESC ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING) ORDER BY salary DESC; name salary more pct_more Mike James Betty Andy Sandy Tracy Carol Compared to Lowest Paid Employee

58 58 / 66 Rank and Dense Rank SELECT name, salary, RANK() OVER s, DENSE_RANK() OVER s FROM emp WINDOW s AS (ORDER BY salary DESC) ORDER BY salary DESC; name salary RANK() OVER s DENSE_RANK() OVER s Mike James Betty Andy Sandy Tracy Carol Rank and Dense Rank both show ranking but dense rank has no gaps while rank may

59 59 / 66 Deparmental averages SELECT name, salary, RANK() OVER s, DENSE_RANK() OVER s FROM emp WINDOW s AS (ORDER BY salary DESC) ORDER BY salary DESC; name salary RANK() OVER s DENSE_RANK() OVER s Mike James Betty Andy Sandy Tracy Carol

60 60 / 66 SELECT name, department, salary, round(avg(salary) OVER d, 2) AS avg, round(salary - AVG(salary) OVER d, 2) AS diff_avg FROM emp WINDOW d AS (PARTITION BY department) ORDER BY department, salary DESC; name department salary avg diff_avg Mike Marketing Betty Marketing Sandy Sales Carol Sales James Shipping Andy Shipping Tracy Shipping A Little Cleaner Window de nition

61 61 / 66 SELECT name, department, salary, salary - LEAD(salary, 1) OVER (PARTITION BY department ORDER BY salary DESC) AS diff_next FROM emp ORDER BY department, salary DESC; name department salary diff_next Mike Marketing Betty Marketing NULL Sandy Sales Carol Sales NULL James Shipping Andy Shipping Tracy Shipping NULL Departmental Di erences

62 62 / 66 Department and Company Rankings SELECT name, department, salary, RANK() OVER s AS dept_rank RANK() OVER (ORDER BY salary DESC) AS global_rank FROM emp WINDOW s AS (PARTITION BY department ORDER BY salary DESC) ORDER BY department, salary DESC; name department salary dept_rank global_rank Mike Marketing Betty Marketing Sandy Sales Carol Sales James Shipping Andy Shipping Tracy Shipping

63 63 / 66 Tips Do you want to split the set? (PARTITION BY creates multiple partitions) Do you want an order in the partition? (use ORDER BY) How do you want to handle rows with the same ORDER BY values? RANGE vs ROW RANK vs DENSE_RANK

64 64 / 66 Tips continued Do you need to de ne a window frame? Window functions can de ne their own partitions, ordering, and window frames. Multiple window names can be de ned in the WINDOW clause. Pay attention to whether window functions operate on frames or partitions. These tips and most of the examples are from Bruce Momjian from EnterpriseDB and his presemtations on CTEs and Windowing Functions at

65 65 / 66 Conclusion Common Table Expressions and Windowing Functions are power additions to traditional Structured Query Langauge class: middle, center CTEs Goodbye messy sub queries Easy to write Easy to comprehend Windowing Functions Powerful analytics Extremely exible

66 66 / 66 Thank you! Any Questions?