Copyright 2017, Oracle and/or its aff iliates. All rights reserved.

Size: px

Start display at page:

Martha Lucas
6 years ago
Views:

2 Safe Harbor Statement The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle s products remains at the sole discretion of Oracle.

3 Window functions in MySQL 8 Dag H. Wanvik Senior database engineer MySQL optimizer team Sep. 2017

4 Program Agenda Introduction: what & why What's supported Ranking and analytical wfs Implementation & performance

5 PART I Gentle intro in which we meet the SUM aggregate used as a window function and get introduced to window partitions and window frames

6 Why window functions? Part of SQL standard since 2003, with later additions Frequently requested feature(s) for data analysis Improves readability and often performance Most vendors support it, but to varying degrees (YMMV)

7 Why window functions? SELECT name o_name, department_id, salary AS o_salary, (SELECT SUM(salary) AS sum FROM employee WHERE salary <= o_salary AND NOT (salary = o_salary AND o_name > name)) AS sum FROM employee ORDER BY sum, name; SELECT name, department_id, salary, SUM(salary) OVER w AS sum FROM employee WINDOW w AS (ORDER BY salary, name ROWS UNBOUNDED PRECEDING) ORDER BY sum, name; name department_id salary sum Dag 10 NULL NULL Frederik Jon Lena Paula Michael William Nils NULL Nils Erik Rose Readability Performance my laptop: 50,000 rows: 16m vs 0.14s Or use self join, but tricky

8 What's a SQL window function? Short answer: a function that gets its arguments from a set of rows; a window defined by a partition and a frame. OK, but what is partitioned data? what is a frame? what does a window function look like? Hint: OVER keyword

9 Running ex: employees SELECT name, department_id, salary FROM employee ORDER BY department_id, name; name department_id salary Nils NULL Dag 10 NULL Erik Frederik Jon Michael Lena Nils Paula Rose William

10 Salaries per dept Query: find sums of salaries per department SELECT department_id, SUM(salary) FROM employee GROUP BY department_id ORDER BY department_id; name department_id salary Nils NULL Dag 10 NULL Erik Frederik Jon Michael Lena Nils Paula Rose William department_id SUM(salary) NULL

11 Grouping loss SELECT department_id, SUM(salary) FROM employee GROUP BY department_id; Identity of names and salaries lost. Lena Nils Paula

12 Windowing SELECT name, department_id, salary FROM employee ORDER BY department_id, name; name department_id salary Nils NULL Dag 10 NULL Erik Frederik Jon Michael Lena Nils Paula Rose William

13 Windowing, rows kept SELECT name, department_id, salary, SUM(salary) OVER () sum FROM employee ORDER BY department_id, name; name department_id salary sum Nils NULL Dag 10 NULL Erik Frederik Jon Michael Lena Nils Paula Rose William

14 Windowing, «grouped» SELECT name, department_id, salary, SUM(salary) OVER (PARTITION BY department_id) sum FROM employee ORDER BY department_id, name; name department_id salary sum Nils NULL Dag 10 NULL Erik Frederik Jon Michael Lena Nils Paula Rose William

15 Partition == frame SELECT name, department_id, salary, SUM(salary) OVER (PARTITION BY department_id) sum FROM employee ORDER BY department_id, name; name department_id salary sum Nils NULL Dag 10 NULL Erik Frederik Jon Michael Lena Nils Paula Rose William All salaries in partition added: the window frame is the entire partition

16 Windowing, rows kept SELECT name, salary, department_id, SUM(salary) OVER (PARTITION BY department_id) sum FROM employee ORDER BY department_id; Identity of department names and salaries kept: no rows are lost => A window function is similar to a scalar function: adds a result column => BUT: can read data from other rows than its own: within its WINDOW partition or frame Lena Nils Paula

17 Default partition SELECT name, department_id, salary, SUM(salary) OVER () sum FROM employee ORDER BY department_id, name; name department_id salary sum Nils NULL Dag 10 NULL Erik Frederik Jon Michael Lena Nils Paula Rose William No partition specified: the window frame is the entire result set

18 Cumulative sum SELECT name, department_id, salary, SUM(salary) OVER (ORDER BY department_id, name ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) sum FROM employee ORDER BY department_id, name; name department_id salary sum Nils NULL Dag 10 NULL Erik Frederik Jon Michael Lena Nils Paula Rose William No partition specified: the window frame grows

19 Cumulative sum SELECT name, department_id, salary, SUM(salary) OVER (ORDER BY department_id, name ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) sum FROM employee ORDER BY department_id, name; name department_id salary sum Nils NULL Dag 10 NULL Erik Frederik Jon Michael Lena Nils Paula Rose William No partition specified: the window frame grows

20 Cumulative sum SELECT name, department_id, salary, SUM(salary) OVER (ORDER BY department_id, name ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) sum FROM employee ORDER BY department_id, name; name department_id salary sum Nils NULL Dag 10 NULL Erik Frederik Jon Michael Lena Nils Paula Rose William No partition specified: the window frame grows

21 Cumulative sum SELECT name, department_id, salary, SUM(salary) OVER (ORDER BY department_id, name ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) sum FROM employee ORDER BY department_id, name; name department_id salary sum Nils NULL Dag 10 NULL Erik Frederik Jon Michael Lena Nils Paula Rose William No partition specified: the window frame grows

22 Cumulative sum SELECT name, department_id, salary, SUM(salary) OVER (ORDER BY department_id, name ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) sum FROM employee ORDER BY department_id, name; name department_id salary sum Nils NULL Dag 10 NULL Erik Frederik Jon Michael Lena Nils Paula Rose William No partition specified: the window frame grows

23 Cumulative sum, partitioned SELECT name, department_id, salary, SUM(salary) OVER (PARTITION BY department_id ORDER BY name ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) sum FROM employee ORDER BY department_id, name; name department_id salary sum Nils NULL Dag 10 NULL NULL Erik Frederik Jon Michael Lena Nils Paula Rose William No partition specified: the window frame grows New partition: reset sum

24 Cumulative sum, partitioned SELECT name, department_id, salary, SUM(salary) OVER (PARTITION BY department_id ORDER BY name ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) sum FROM employee ORDER BY department_id, name; name department_id salary sum Nils NULL Dag 10 NULL NULL Erik Frederik Jon Michael Lena Nils Paula Rose William No partition specified: the window frame grows

25 Cumulative sum, partitioned SELECT name, department_id, salary, SUM(salary) OVER (PARTITION BY department_id ORDER BY name ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) sum FROM employee ORDER BY department_id, name; name department_id salary sum Nils NULL Dag 10 NULL NULL Erik Frederik Jon Michael Lena Nils Paula Rose William No partition specified: the window frame grows

26 Parts of window function SELECT name, department_id, salary, SUM(salary) OVER (PARTITION BY department_id ORDER BY name ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) sum FROM employee ORDER BY department_id, name; Function call + OVER keyword signals a window function

27 Parts of window function SELECT name, department_id, salary, SUM(salary) OVER (PARTITION BY department_id ORDER BY name ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) sum FROM employee ORDER BY department_id, name; An optional partition specification: PARTITION BY <expression> {, <expression}* A partition divides up the result set in disjoint sets A window function does not see rows in partitions other than that of the current row for which it is being evaluated

28 Parts of window function SELECT name, department_id, salary, SUM(salary) OVER (PARTITION BY department_id ORDER BY name ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) sum FROM employee ORDER BY department_id, name; An optional ordering specification: ORDER BY <expression> {, <expression}* Orders the row within the partition Not the same as a final query ORDER BY and makes no guarantees of final query row ordering.

29 ORDER BY: growing frame Some window functions need row ordering to be useful, e.g. RANK Peers: same value for ORDER BY expression(s) SELECT name, department_id, salary, SUM(salary) OVER (ORDER BY salary DESC) AS sum FROM employee; name department_id salary sum Rose Erik Nils Nils NULL Michael William Lena Paula Frederik Jon Dag 10 NULL

30 ORDER BY: growing frame Some window functions need row ordering to be useful, e.g. RANK Peers: same value for ORDER BY expression(s) SELECT name, department_id, salary, SUM(salary) OVER (ORDER BY salary DESC) AS sum FROM employee; name department_id salary sum Rose Erik Nils Nils NULL Michael William Lena Paula Frederik Jon Dag 10 NULL

31 ORDER BY: growing frame Some window functions need row ordering to be useful, e.g. RANK Peers: same value for ORDER BY expression(s) SELECT name, department_id, salary, SUM(salary) OVER (ORDER BY salary DESC) AS sum FROM employee; name department_id salary sum Rose Erik Nils Nils NULL Michael William Lena Paula Frederik Jon Dag 10 NULL

32 ORDER BY: growing frame Some window functions need row ordering to be useful, e.g. RANK Peers: same value for ORDER BY expression(s) SELECT name, department_id, salary, SUM(salary) OVER (ORDER BY salary DESC) AS sum FROM employee; name department_id salary sum Rose Erik Nils Nils NULL Michael William Lena Paula Frederik Jon Dag 10 NULL

33 ORDER BY: growing frame PEERS Some window functions need row ordering to be useful, e.g. RANK Peers: same value for ORDER BY expression(s) SELECT name, department_id, salary, SUM(salary) OVER (ORDER BY salary DESC) AS sum FROM employee; name department_id salary sum Rose Erik Nils Nils NULL Michael William Lena Paula Frederik Jon Dag 10 NULL What happened here? Answer: Two rows are peers w.r.t. salary This is an example of a RANGE frame (implicit)

34 Parts of window function SELECT name, department_id, salary, SUM(salary) OVER (PARTITION BY department_id ORDER BY name ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) sum FROM employee ORDER BY department_id, name; An optional frame specification A subset of rows within a partition Extent can depend on the current row Default frame: partition Not all window functions heed frame

35 Frame anatomy partition n PRECEDING m FOLLOWING UNBOUNDED PRECEDING CURRENT ROW UNBOUNDED FOLLOWING Examples: ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW RANGE CURRENT ROW ROWS BETWEEN CURRENT ROW AND 3 FOLLOWING ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING n: numeric or temporal RANGE INTERVAL 6 DAY PRECEDING

36 Frame anatomy partition n PRECEDING m FOLLOWING UNBOUNDED PRECEDING CURRENT ROW UNBOUNDED FOLLOWING Examples: ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW RANGE CURRENT ROW ROWS BETWEEN CURRENT ROW AND 3 FOLLOWING ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING n: numeric or temporal RANGE INTERVAL 6 DAY PRECEDING

37 Frame anatomy partition UNBOUNDED PRECEDING CURRENT ROW Examples: ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW RANGE CURRENT ROW ROWS BETWEEN CURRENT ROW AND 3 FOLLOWING ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING n: numeric or temporal RANGE INTERVAL 6 DAY PRECEDING

38 Frame anatomy partition Examples: ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW CURRENT ROW and peers RANGE CURRENT ROW ROWS BETWEEN CURRENT ROW AND 3 FOLLOWING ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING RANGE INTERVAL 6 DAY PRECEDING

39 Frame anatomy partition Examples: 3 FOLLOWING CURRENT ROW ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW RANGE CURRENT ROW ROWS BETWEEN CURRENT ROW AND 3 FOLLOWING ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING RANGE INTERVAL 6 DAY PRECEDING

40 Frame anatomy partition 3 PRECEDING 2 FOLLOWING CURRENT ROW Examples: ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW RANGE CURRENT ROW ROWS BETWEEN CURRENT ROW AND 3 FOLLOWING ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING RANGE INTERVAL 6 DAY PRECEDING

41 ROWS vs. RANGE Frame boundaries: physical (ROWS) or logical (RANGE) ROWS: bound N: # rows. Peers are ignored. RANGE requires ORDER BY on a single numeric or temporal expression RANGE: bound N: rows with value for ascending ORDER BY expression within N lower (PRECEDING) and M higher (FOLLOWING) of value of the current row. Peers are always included in frame. Ex: ORDER BY date RANGE BETWEEN INTERVAL 6 DAY PRECEDING AND CURRENT ROW specifies all rows within last week. Default (std): OVER (ORDER BY n) == OVER (ORDER BY n RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW)

42 Determinacy SELECT name, department_id, salary, SUM(salary) OVER (PARTITION BY department_id ORDER BY name ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) sum FROM employee; In general, window queries are not deterministic unless one orders on enough expressions to designate the row uniquely. Minimum guarantee by SQL std: several equivalent non-deterministic orderings in same query give the same order (within query).

43 Determinacy SELECT name, department_id, salary, SUM(salary) OVER (PARTITION BY department_id ORDER BY name ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) sum FROM employee ORDER BY department_id, name; A final ORDER BY is still required if ordering is desired: no guarantees from window. In general, window queries are not deterministic unless one orders on enough expressions to designate the row uniquely. Minimum guarantee by SQL std: several equivalent non-deterministic orderings in same query give the same order (within query).

44 Example: salary analysis Question: find the employees with the largest difference between their wage and that of the department average SELECT name, department_id, salary, AVG(salary) OVER (PARTITION BY department_id) AS avg, salary - AVG(salary) OVER (PARTITION BY department_id) AS diff FROM employee ORDER BY diff desc; name department_id salary avg diff Rose Erik Nils Nils NULL Michael Lena Paula Frederik Jon William Dag 10 NULL NULL

45 Example: salary analysis Question: find the employees with the largest difference between their wage and that of the department average SELECT name, department_id, salary, AVG(salary) OVER (PARTITION BY department_id) AS avg, salary - AVG(salary) OVER (PARTITION BY department_id) AS diff FROM employee ORDER BY diff desc; Here: two distinct windows A query can use have any number of windows Logically evaluated in multiple phases

46 Example: named window Question: find the employees with the largest difference between their wage and that of the department average SELECT name, department_id, salary, AVG(salary) OVER w AS avg, salary - AVG(salary) OVER w AS diff FROM employee WINDOW w as (PARTITION BY department_id) ORDER BY diff desc; Named window w References to w Multiple window functions per window Will be evaluated in same phase (efficiency) Better readability

47 Ex: moving AVG, smoothing CREATE TABLE sales(id INT AUTO_INCREMENT PRIMARY KEY, date DATE, sale INT);...; SELECT * FROM sales; id date sale

48 Ex: moving AVG, smoothing Sum up sales per month SELECT MONTH(date), SUM(sale) FROM sales GROUP BY MONTH(date); MONTH(date) SUM(sale)

49 Ex: moving AVG, smoothing Moving AVG over 3 months SELECT MONTH(date), SUM(sale), AVG(SUM(sale)) OVER w AS sliding_avg FROM sales GROUP BY MONTH(date) WINDOW w AS (ORDER BY MONTH(date) RANGE BETWEEN 1 PRECEDING AND 1 FOLLOWING); MONTH(date) SUM(sale) sliding_avg

50 Ex: moving AVG, smoothing Moving AVG over 3 months SELECT MONTH(date), SUM(sale), AVG(SUM(sale)) OVER w AS sliding_avg FROM sales GROUP BY MONTH(date) WINDOW w AS (ORDER BY MONTH(date) RANGE BETWEEN 1 PRECEDING AND 1 FOLLOWING); MONTH(date) SUM(sale) sliding_avg

51 Ex: moving AVG, smoothing Moving AVG over 3 months SELECT MONTH(date), SUM(sale), AVG(SUM(sale)) OVER w AS sliding_avg FROM sales GROUP BY MONTH(date) WINDOW w AS (ORDER BY MONTH(date) RANGE BETWEEN 1 PRECEDING AND 1 FOLLOWING); MONTH(date) SUM(sale) sliding_avg moving frame

52 Ex: moving AVG, smoothing Moving AVG over 3 months SELECT MONTH(date), SUM(sale), AVG(SUM(sale)) OVER w AS sliding_avg FROM sales GROUP BY MONTH(date) WINDOW w AS (ORDER BY MONTH(date) RANGE BETWEEN 1 PRECEDING AND 1 FOLLOWING); MONTH(date) SUM(sale) sliding_avg

53 Ex: moving AVG, smoothing Moving AVG over 3 months SELECT MONTH(date), SUM(sale), AVG(SUM(sale)) OVER w AS sliding_avg FROM sales GROUP BY MONTH(date) WINDOW w AS (ORDER BY MONTH(date) RANGE BETWEEN 1 PRECEDING AND 1 FOLLOWING); MONTH(date) SUM(sale) sliding_avg

54 Ex: moving AVG, smoothing Moving AVG over 3 months SELECT MONTH(date), SUM(sale), AVG(SUM(sale)) OVER w AS sliding_avg FROM sales GROUP BY MONTH(date) WINDOW w AS (ORDER BY MONTH(date) RANGE BETWEEN 1 PRECEDING AND 1 FOLLOWING); MONTH(date) SUM(sale) sliding_avg

55 Ex: moving AVG, smoothing Moving AVG over 3 months SELECT MONTH(date), SUM(sale), AVG(SUM(sale)) OVER w AS sliding_avg FROM sales GROUP BY MONTH(date) WINDOW w AS (ORDER BY MONTH(date) RANGE BETWEEN 1 PRECEDING AND 1 FOLLOWING); MONTH(date) SUM(sale) sliding_avg

56 Windowing in an SQL query JOIN GROUP BY, HAVING WINDOW 1 WINDOW n ORDER BY/ DISTINCT/ LIMIT Window functions see query result set after grouping/having - filtering on wf results requires subquery Ordering not semantically significant Window functions can't use window functions in same query (without using subqueries) In practice, ordering matters. The optimizer can is allowed to - reorder to minimize sorting required - merge window phases if equivalent

57 Program Agenda Introduction: what & why What's supported Ranking and analytical wfs Implementation & performance

58 PART II in which we learn which window functions are supported by MySQL

59 MySQL 8.0 Most aggregate functions in MySQL can be used as window functions: COUNT, SUM, AVG, MAX, MIN, STDDEV_POP (& synonyms), STDDEV_SAMP, VAR_POP (& synonym), VAR_SAMP Limitation: No DISTINCT in aggregates yet All SQL standard specialized window functions ROW_NUMBER, RANK, DENSE_RANK, PERCENT_RANK, CUME_DIST, NTILE, LEAD, LAG, FIRST_VALUE, LAST_VALUE, NTH_VALUE Next phase (probably post-ga), more aggregates: BIT_OR, BIT_XOR, BIT_AND, JSON_ARRAYAGG, JSON_OBJECTAGG [ GROUP_CONCAT ]

60 Std compliance: extensions Target SQL standard semantics, but Expression (not only column) allowed in PARTITION BY Benefit: more flexible Missing ORDER BY tolerated even if useless (all rows are peers) except for RANGE <value>: requires single ORDER BY expression Tolerate frame clause even for window functions that operate on entire partition (std is stricter) Benefit: Many wfs can use same window definition

61 Std compliance: restrictions Target SQL standard semantics, but Valued frame bounds must be static in query No GROUPS in frame clause (Feature T620) No EXCLUDE in frame clause No DISTINCT in aggregates with windowing IGNORE NULLS not supported FROM LAST not supported (NTH_VALUE) No nested window functions (Feature T619) No row pattern recognition in window clause (Feature R020) Operator subqueries with windowing only if materializable (WL#10431)

62 Program Agenda Introduction: what & why What's supported Ranking and analytical wfs Implementation & performance

63 PART III in which we learn about the specialized window functions Ranking: ROW_NUMBER, RANK, DENSE_RANK, PERCENT_RANK, CUME_DIST, NTILE Analytical: LEAD, LAG, NTH_VALUE, FIRST_VALUE, LAST_VALUE Blue ones use frames, the others work on the entire partition.

64 ROW_NUMBER Assign number to row in ascending order Example: give employees a number according to their salary SELECT name, department_id AS dept, salary, ROW_NUMBER() OVER w AS `row#` FROM employee WINDOW w AS (PARTITION BY department_id ORDER BY salary DESC, name ASC) ORDER BY department_id, `row#`; name dept salary row# Nils NULL Erik Michael Frederik Jon Dag 10 NULL 5 Nils Lena Paula Rose William

65 RANK Rows that are the same w.r.t any ordering have the same rank Example: rank employees within each department according to their salary SELECT name, department_id AS dept, salary,.., RANK() OVER w AS `rank` FROM employee WINDOW w AS (PARTITION BY department_id ORDER BY salary DESC) ORDER BY department_id, `row#`; name dept salary row# rank Nils NULL Erik Michael Frederik Jon Dag 10 NULL 5 5 Nils Lena Paula Rose William Peer rows w.r.t ordering, skip rank 4.

66 DENSE_RANK Rows that are the same wrt any ordering have the same rank Example: rank employees within each department according to their salary SELECT name, department_id AS dept, salary,.., DENSE_RANK() OVER w AS dense FROM employee WINDOW w AS (PARTITION BY department_id ORDER BY salary DESC) ORDER BY department_id, `row#`; name dept salary row# rank dense Nils NULL Erik Michael Frederik Jon Dag 10 NULL Nils Lena Paula Rose William Peer rows w.r.t ordering, do not skip rank 4.

67 PERCENT_RANK Relative rank: (rank - 1) / (total p.rows - 1), or 0 if one row in partition Example: rank employees within each department according to their salary SELECT name, department_id AS dept, salary,.., PERCENT_RANK() OVER w AS `%rank` FROM employee WINDOW w AS (PARTITION BY department_id ORDER BY salary DESC) ORDER BY department_id, `row#`; name dept salary row# rank dense %rank Nils NULL Erik Michael Frederik Jon Dag 10 NULL Nils Lena Paula Rose William (3-1)/(5-1)=0.5

68 CUME_DIST Cumulative relative rank: preceding rows incl. peers / total p.rows Example: cumulative rank of employees within each department according to their salary SELECT name, department_id AS dept, salary,.., CUME_DIST() OVER w AS cume FROM employee WINDOW w AS (PARTITION BY department_id ORDER BY salary DESC) ORDER BY department_id, `row#`; name dept salary row# rank dense %rank cume Nils NULL Erik Michael Frederik Jon Dag 10 NULL Nils Lena Paula Rose William /5=0.8

69 NTILE Divides an ordered partition into a specified number of groups aka buckets as evenly as possible and assigns a bucket number to each row in the partition. In spite of name, not the same as percentile! SELECT name, department_id AS dept, salary,.., NTILE(3) OVER w AS `3-tile` FROM employee WINDOW w AS (PARTITION BY department_id ORDER BY salary DESC) ORDER BY department_id, `row#`; name dept salary row# rank dense %rank cume 3-tile Nils NULL Erik Michael Frederik Jon Dag 10 NULL Nils Lena Paula Rose William

70 LEAD, LAG Returns value evaluated at the row that is offset rows after/before the current row within the partition; if there is no such row, instead return an optional default expression (which must be of the same type as value). Both offset and default expr are evaluated with respect to the current row. If omitted, offset defaults to 1 and default expr to null Syntax: LEAD( <expr> [, <offset> [, <default expr> ] ] ) [ <RESPECT NULLS> ] Example: LEAD(date) OVER (..) Note: IGNORE NULLS not supported, RESPECT NULLS is default but can be specified. Any window frame is ignored

71 LEAD SELECT name, department_id AS dept, salary, LEAD(salary, 1) OVER w AS `lead` FROM employee WINDOW w AS (ORDER BY salary DESC); name dept salary lead Rose Erik Nils Nils NULL Michael William : Lena Paula Frederik Jon Dag 10 NULL

72 LEAD SELECT name, department_id AS dept, salary, LEAD(salary, 1) OVER w AS `lead` FROM employee WINDOW w AS (ORDER BY salary DESC); name dept salary lead Rose Erik Nils Nils NULL Michael William Lena Paula Frederik Jon Dag 10 NULL

73 LEAD SELECT name, department_id AS dept, salary, LEAD(salary, 1) OVER w AS `lead` FROM employee WINDOW w AS (ORDER BY salary DESC); name dept salary lead Rose Erik Nils Nils NULL Michael William Lena Paula Frederik Jon Dag 10 NULL

74 LEAD SELECT name, department_id AS dept, salary, LEAD(salary, 1) OVER w AS `lead` FROM employee WINDOW w AS (ORDER BY salary DESC); name dept salary lead Rose Erik Nils Nils NULL Michael William Lena Paula Frederik Jon NULL Dag 10 NULL NULL

75 LEAD SELECT name, department_id AS dept, salary, LEAD(salary, 1, 77000) OVER w AS `lead` FROM employee WINDOW w AS (ORDER BY salary DESC); name dept salary lead Rose Erik Nils Nils NULL Michael William Lena Paula Frederik Jon NULL Dag 10 NULL default

76 LEAD - gap detection Classic example: CREATE TABLE t(i INT); INSERT INTO t VALUES (1), (2), (4), (5), (6), (8), (9), (10); SELECT i, l FROM (SELECT i, LEAD(i) OVER (ORDER BY i) AS l FROM t) d WHERE i + 1 <> l; i l

77 FIRST_VALUE, LAST_VALUE Returns value evaluated at the first, last in the frame of the current row within the partition; if there is no nth row (frame is too small), the NTH_VALUE returns NULL. Note: IGNORE NULLS is not supported, RESPECT NULLS is default but can be specified.

78 FIRST_VALUE Difference between employee wages and best paid in department SELECT name, department_id AS dept, salary, FIRST_VALUE(salary) OVER w - salary AS diff FROM employee WINDOW w AS (PARTITION BY department_id ORDER BY salary DESC) name dept salary diff Nils NULL Erik Michael Frederik Jon Dag 10 NULL NULL Nils Lena Paula Rose William

79 NTH_VALUE Returns value evaluated at the nth in the frame of the current row within the partition; if there is no nth row (frame is too small), the NTH_VALUE returns NULL. Note: IGNORE NULLS is not supported, RESPECT NULLS is used but can be specified. Note: For NTH_VALUE, FROM LAST is not supported, FROM FIRST is used but can be specified

80 Program Agenda Introduction: what & why What's supported Ranking and analytical wfs Implementation & performance

81 PART IV is where we look at implementation and performance aspects

82 Windowing in an SQL query JOIN GROUP BY, HAVING WINDOW 1 WINDOW n ORDER BY/ DISTINCT/ LIMIT Window functions see query result set after grouping/having - filtering on wf results requires subquery Ordering not semantically significant Window functions can't use window functions in same query (without using subqueries) In practice, ordering matters. The optimizer is allowed to - reorder to minimize sorting required - merge window phases if equivalent

83 Processing window functions JOIN GROUP BY WINDOW 1 WINDOW n ORDER BY/ DISTINCT/ LIMIT Sort by concatenation of PARTITION BY and ORDER BY Tmp table between each windowing step Optimization: re-order windows to eliminate sorting steps: when equal PARTITION BY/ORDER BY expressions SELECT name, SUM(salary) OVER () FROM employee LIMIT name SUM(salary) OVER () Dag Erik Frederik Need to read all rows to get SUM before we can output row 1: need buffering to merge original row with result of window function

84 Processing window functions JOIN GROUP BY WINDOW 1 WINDOW n ORDER BY/ DISTINCT/ LIMIT Row addressable buffer aka frame buffer in-memory tmp table; overflows automatically to disk if needed Permits re-reading rows when frame moves

85 Streaming window functions Frame buffer not needed for streaming window functions: ROW_NUMBER, RANK, DENSE_RANK Aggregates with ROW frame and dynamically growing upper bound, e.g. SELECT SUM(salary) OVER (ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) FROM employee; Non-streaming (need buffer), e.g. CUME_DIST, SUM() OVER () Row addressable buffer aka frame buffer => a function on what window function and which frame specification If in doubt, check with EXPLAIN FORMAT=JSON

86 EXPLAIN: streaming EXPLAIN format=json SELECT SUM(salary) OVER (ROWS UNBOUNDED PRECEDING) FROM employee; "query_block": { "select_id": 1, "cost_info": { "query_cost": "1.35" }, "windowing": { "windows": [ { "name": "<unnamed window>", "functions": [ "sum" ] } ], "table": { "table_name": "employee", "access_type": "ALL", "rows_examined_per_scan": 11, "rows_produced_per_join": 11, "filtered": "100.00", "cost_info": { "read_cost": "0.25", "eval_cost": "1.10", "prefix_cost": "1.35", "data_read_per_join": "1K" }, "used_columns": [ "salary" ] } } } }

87 EXPLAIN: frame buffer EXPLAIN format=json SELECT COUNT(salary) OVER () FROM employee; "query_block": { "select_id": 1, "cost_info": { "query_cost": "1.35" }, "windowing": { "windows": [ { "name": "<unnamed window>", "frame_buffer": { "using_temporary_table": true, "optimized_frame_evaluation": true }, "functions": [ "count" ] } ], "table": { "table_name": "employee", "access_type": "ALL", "rows_examined_per_scan": 11, "rows_produced_per_join": 11, "filtered": "100.00", "cost_info": { "read_cost": "0.25", "eval_cost": "1.10", "prefix_cost": "1.35", "data_read_per_join": "1K" }, "used_columns": [ "salary" ] } } }

88 Frame buffer processing JOIN GROUP BY WINDOW 1 WINDOW n ORDER BY/ DISTINCT/ LIMIT Row addressable buffer aka frame buffer in-memory; overflows automatically to disk if needed Permits re-reading rows when frame moves SELECT name, SUM(salary) OVER () FROM employee LIMIT name SUM(salary) OVER () Dag Erik Frederik Optimization 1: Compute SUM only once (static frame) But what if frame changes?

89 Frame buffer processing Case: expanding frame SELECT name, salary, SUM(salary) OVER (ORDER BY name) AS `sum` FROM employee LIMIT name salary sum Dag NULL NULL Erik Frederik Optimization 2: Remember SUM and adjust: here add next row's contribution: (NULL+100k)+60k=160k But what if we have a moving frame?

90 Inversion Case: moving frame: Sales over this month and last SELECT MONTH(date) AS month, SUM(sale) AS monthly, SUM(SUM(sale)) OVER (ORDER BY MONTH(date) RANGE 1 PRECEDING) AS `this&last` FROM sales GROUP BY MONTH(date); month monthly this&last Optimization 3: Remember SUM and adjust: here remove last contribution from 2 PRECEDING, then add current row's contribution: inversion NOTE: Ok only if: a + b - a + c = b + c This isn't always the case for aggregates Penalty if we can't do this for large window frames: O(#partition size) vs. O(#partition size * #frame size)

91 Conditional inversion When: a + b - a + c b + c Floating aggregates, standard deviation and variance aggregates mysql> show variables like '%high%'; Variable_name Value windowing_use_high_precision ON recomputes: slow, but guaranteed same results as grouped aggregates SET windowing_use_high_precision= off; linear performance For variance, the differences are only in the last significant few digits to the the incremental algorithm yielding slightly different results (usually insignificant) For floats, this can matter of if summing very large and small numbers: mysql> select E E+307; E E

92 Performance hints Use named windows to eliminate several windowing steps whenever possible Possibility: analyze and collapse windows where semantics allow it. We might want to add this capability to the optimizer if large demand. Streaming window functions faster than those that need buffering MAX/MIN do not support inversion, so can slow with large frames unless the expression is a prefix of the ORDER BY expressions JSON_OBJECTAGG is not invertible, so can slow with large frames

93 Q&A Thank you for using MySQL! Blog:

94 Safe Harbor Statement The preceding is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle s products remains at the sole discretion of Oracle.

Chapter 6 Windowed Tables and Window Functions in SQL

Prof. Dr.-Ing. Stefan Deßloch AG Heterogene Informationssysteme Geb. 36, Raum 329 Tel. 0631/205 3275 dessloch@informatik.uni-kl.de Chapter 6 Windowed Tables and Window Functions in SQL Recent Developments