Understanding Data Queries and Logging

Transcription

1

2 Welcome

3 # T C 1 8 Understanding Data Queries and Logging Priyatham Pamu Engineering Manager Tableau Software Luis Enciso Staff Software Engineer Tableau Software

4 Agenda Query Ecosystem Performance 101 Behind the scenes: Queries in different Scenarios Query Optimizations and Logging Q&A

5 Query Ecosystem Motto: Generate fewer and simpler queries Support for more than 60 data sources And counting Live and Extract connections Option to Publish Multi-connection data sources Federated connections

6 Query Ecosystem Hyper hyper! What is Hyper? Tableau s Data Engine TL;DR: a high performance database When is it used? Extracts Connecting to files Federation a,b { ; } PDF CSV JSON XLS *.hyper *.tde

7 Performance 101

8 Performance 101 Measure! Measurement tools ideally: Are easy to use Are precise Keep you in the flow Don t affect the performance

9 Repeated Marks Example scenario Overlapped marks in Map Viz credit: Luisa Bez Viz of the Day on Tableau Public June 11th 2018

10 When Do We See Performance Issues? Complexity A lot of worksheets Many quick filters Long calculations Nested calculations String manipulation Complexity Data size Data size Data IN or OUT A lot of columns A lot of rows Use of expensive types

11 Behind the scenes: Queries in different scenarios

12 Basic Scenario Simple viz Classic example: Sum of sales example 1 dimension, 1 measure Demo

13 Tableau Log Viewer (TLV) TLV: Tool to view and monitor tableau logs Allows you to see detailed information about performance Download:

14 TLV usage Features: Live capture. Look at events as they are happening Highlight + filters Highlight relevant events Hide irrelevant ones Find Useful events (for this session) begin-query / end-query qp-batch-summary abstract-query

15 Basic Scenario Viz with a filter

16 Basic Scenario Viz with filter Run time The connection Query Category Data dimensions Dimension Measure Filter Group by the 1 st thing in the SELECT list

17 Viz with large filters Small filter: inline SQL predicate Large filter: temporary table Heuristic driven Example: Sales for selected customers Large filter

18 Viz with large filters ID: 3198 Key: data-inserter-summary bytes: 9861 elapsed: 0.17 protocol-id: 6 rows: 549 table-name: #Tableau_4_0436A1D8-013F-4112-AAB3-8481D7BBD6F2_3_Filter

19 Viz with quick filters Quick Filters Extra query to populate domain Relevant value Quick filters (avoid these!!) May involve joins with other dimensions Acceleration views Speeds up later interactions Async queries

20 Viz with quick filters + Acceleration View Query

21 Viz on multi-connection data source Data Integration / Data federation Combine data from multiple data sources Federation Engine: Hyper

22 Federated Scenario (example) Example: Combine sales data from SQL Server and customer data from Oracle Query Filter on Customer JOIN SALES SQL Server CUSTOMER Oracle

23 Federated Scenario (example) Example: Combine sales data from SQL Server and customer data from Oracle Query Query Hyper Filter on Customer JOIN TEMP 1 TEMP 2 JOIN Move to Hyper Move to Hyper Filter on Customer SALES SQL Server CUSTOMER Oracle SALES SQL Server CUSTOMER Oracle

24 Nested calculations Example scenario Find the #hashtags in tweet Viz credit: Amit Naik Viz of the Day on Tableau Public December 13th 2017

25 Nested calculations: Original query SELECT (CASE WHEN ("t0"."$temp0_cse" = '#') THEN '#No Hashtag' WHEN ("t0"."$temp0_cse" = '#t=23') THEN '#No Hashtag' WHEN TABLEAU.CONTAINS("t0"."$temp4_cse",'Mann') THEN '#MaanKiBaat' WHEN TABL EAU.CONTAINS("t0"."$temp4_cse",'2CrUjjwala') THEN '#2CrUjjwala' WHEN TABLEAU.CONTAINS("t0"."$temp4_cse",'350thPrakashParv') THEN '#350thPrakashParv' WHEN TABLEAU.CONTAINS("t0"."$temp4_cse",'Dig idhanmela') THEN '#DigiDhanMela' WHEN TABLEAU.CONTAINS("t0"."$temp4_cse",'DigitalDialogue') THEN '#DigitalDialogue' WHEN TABLEAU.CONTAINS("t0"."$temp4_cse",'YogaDay') THEN '#YogaDay' WHEN TABLE AU.CONTAINS("t0"."$temp4_cse",'ThankYouPM') THEN '#ThankYouPM' WHEN TABLEAU.CONTAINS("t0"."$temp4_cse",'Tiranga') THEN '#TirangaYatra' WHEN TABLEAU.CONTAINS("t0"."$temp4_cse",'Tirangya') THEN ' #TirangaYatra' ELSE RTRIM(LTRIM("t0"."$temp0_cse",E' \t\n\x0b\f\r'),e' \t\n\x0b\f\r') END) AS "Calculation_ ", SUM(1) AS "sum:number of Records:ok" FROM ( SELECT "modi.csv"."pathorder" AS "PathOrder", "modi.csv"."ctid" AS "ctid", "modi.csv"."replies" AS "Replies", "modi.csv"."favorites" AS "Favorites", "modi.csv"."retweets" AS "Retweets", "modi.csv"."is Retweet" AS "Is Retweet", "modi.csv"."tweet" AS "Tweet", "modi.csv"."link" AS "Link", "modi.csv"."tweet Date" AS "Tweet Date", REPLACE((CASE WHEN TABLEAU.CONTAINS("modi.csv"."Tweet",'#') THEN TABLEAU.LEFT(TABLEAU.MID("modi.csv"."Tweet",TABLEAU.FIND("modi.csv"."Tweet", '#')),LENGTH("modi.csv"."Tweet")) ELSE 'No Ha shtag' END),'pic',' ') AS "$temp2_cse", REPLACE(REPLACE((CASE WHEN TABLEAU.CONTAINS("modi.csv"."Tweet",'#') THEN TABLEAU.LEFT(TABLEAU.MID("modi.csv"."Tweet",TABLEAU.FIND("modi.csv"."Tweet", '#')),LENGTH("modi.csv"."Tweet")) ELS E 'No Hashtag' END),'pic',' '),'http',' ') AS "$temp3_cse", (CASE WHEN TABLEAU.CONTAINS(REPLACE((CASE WHEN TABLEAU.CONTAINS("modi.csv"."Tweet",'#') THEN TABLEAU.LEFT(TABLEAU.MID("modi.csv"."Tweet",TABLEAU.FIND("modi.csv"."Tweet", '#')),LENGTH("mod i.csv"."tweet")) ELSE 'No Hashtag' END),'pic',' '),'http') THEN REPLACE(REPLACE((CASE WHEN TABLEAU.CONTAINS("modi.csv"."Tweet",'#') THEN TABLEAU.LEFT(TABLEAU.MID("modi.csv"."Tweet",TABLEAU.FIND ("modi.csv"."tweet", '#')),LENGTH("modi.csv"."Tweet")) ELSE 'No Hashtag' END),'pic',' '),'http',' ') WHEN TABLEAU.CONTAINS(REPLACE((CASE WHEN TABLEAU.CONTAINS("modi.csv"."Tweet",'#') THEN TABLE AU.LEFT(TABLEAU.MID("modi.csv"."Tweet",TABLEAU.FIND("modi.csv"."Tweet", '#')),LENGTH("modi.csv"."Tweet")) ELSE 'No Hashtag' END),'pic',' '),'.') THEN REPLACE(REPLACE((CASE WHEN TABLEAU.CONTAINS ("modi.csv"."tweet",'#') THEN TABLEAU.LEFT(TABLEAU.MID("modi.csv"."Tweet",TABLEAU.FIND("modi.csv"."Tweet", '#')),LENGTH("modi.csv"."Tweet")) ELSE 'No Hashtag' END),'pic',' '),'http',' ') WHEN T ABLEAU.CONTAINS(REPLACE((CASE WHEN TABLEAU.CONTAINS("modi.csv"."Tweet",'#') THEN TABLEAU.LEFT(TABLEAU.MID("modi.csv"."Tweet",TABLEAU.FIND("modi.csv"."Tweet", '#')),LENGTH("modi.csv"."Tweet")) E LSE 'No Hashtag' END),'pic',' '),'#MaanKiBaat') THEN '#MaanKiBaat' WHEN TABLEAU.CONTAINS(REPLACE((CASE WHEN TABLEAU.CONTAINS("modi.csv"."Tweet",'#') THEN TABLEAU.LEFT(TABLEAU.MID("modi.csv"."Tw eet",tableau.find("modi.csv"."tweet", '#')),LENGTH("modi.csv"."Tweet")) ELSE 'No Hashtag' END),'pic',' '),'#1CrPplGaveUpLPGSubsidy..') THEN '#1CrPplGaveUpLPGSubsidy' WHEN TABLEAU.CONTAINS(REPLA CE((CASE WHEN TABLEAU.CONTAINS("modi.csv"."Tweet",'#') THEN TABLEAU.LEFT(TABLEAU.MID("modi.csv"."Tweet",TABLEAU.FIND("modi.csv"."Tweet", '#')),LENGTH("modi.csv"."Tweet")) ELSE 'No Hashtag' END),'pic',' '),'#MyCleanIndia') THEN '#MyCleanIndia' WHEN TABLEAU.CONTAINS(REPLACE((CASE WHEN TABLEAU.CONTAINS("modi.csv"."Tweet",'#') THEN TABLEAU.LEFT(TABLEAU.MID("modi.csv"."Tweet",TABLEAU.FIND ("modi.csv"."tweet", '#')),LENGTH("modi.csv"."Tweet")) ELSE 'No Hashtag' END),'pic',' '),'#YogaDay') THEN '#YogaDay' WHEN TABLEAU.CONTAINS(REPLACE((CASE WHEN TABLEAU.CONTAINS("modi.csv"."Tweet",'#') THEN TABLEAU.LEFT(TABLEAU.MID("modi.csv"."Tweet",TABLEAU.FIND("modi.csv"."Tweet", '#')),LENGTH("modi.csv"."Tweet")) ELSE 'No Hashtag' END),'pic',' '),'IncredibleIndia') THEN '#IncredibleI ndia' ELSE REPLACE((CASE WHEN TABLEAU.CONTAINS("modi.csv"."Tweet",'#') THEN TABLEAU.LEFT(TABLEAU.MID("modi.csv"."Tweet",TABLEAU.FIND("modi.csv"."Tweet", '#')),LENGTH("modi.csv"."Tweet")) ELSE ' No Hashtag' END),'pic',' ') END) AS "$temp1_cse", REPLACE(REPLACE((CASE WHEN ((CASE WHEN TABLEAU.CONTAINS(REPLACE((CASE WHEN TABLEAU.CONTAINS("modi.csv"."Tweet",'#') THEN TABLEAU.LEFT(TABLEAU.MID("modi.csv"."Tweet",TABLEAU.FIND("modi.csv "."Tweet", '#')),LENGTH("modi.csv"."Tweet")) ELSE 'No Hashtag' END),'pic',' '),'http') THEN REPLACE(REPLACE((CASE WHEN TABLEAU.CONTAINS("modi.csv"."Tweet",'#') THEN TABLEAU.LEFT(TABLEAU.MID("mo di.csv"."tweet",tableau.find("modi.csv"."tweet", '#')),LENGTH("modi.csv"."Tweet")) ELSE 'No Hashtag' END),'pic',' '),'http',' ') WHEN TABLEAU.CONTAINS(REPLACE((CASE WHEN TABLEAU.CONTAINS("modi. csv"."tweet",'#') THEN TABLEAU.LEFT(TABLEAU.MID("modi.csv"."Tweet",TABLEAU.FIND("modi.csv"."Tweet", '#')),LENGTH("modi.csv"."Tweet")) ELSE 'No Hashtag' END),'pic',' '),'.') THEN REPLACE(REPLACE ((CASE WHEN TABLEAU.CONTAINS("modi.csv"."Tweet",'#') THEN TABLEAU.LEFT(TABLEAU.MID("modi.csv"."Tweet",TABLEAU.FIND("modi.csv"."Tweet", '#')),LENGTH("modi.csv"."Tweet")) ELSE 'No Hashtag' END),' pic',' '),'http',' ') WHEN TABLEAU.CONTAINS(REPLACE((CASE WHEN TABLEAU.CONTAINS("modi.csv"."Tweet",'#') THEN TABLEAU.LEFT(TABLEAU.MID("modi.csv"."Tweet",TABLEAU.FIND("modi.csv"."Tweet", '#')),L ENGTH("modi.csv"."Tweet")) ELSE 'No Hashtag' END),'pic',' '),'#MaanKiBaat') THEN '#MaanKiBaat' WHEN TABLEAU.CONTAINS(REPLACE((CASE WHEN TABLEAU.CONTAINS("modi.csv"."Tweet",'#') THEN TABLEAU.LEF T(TABLEAU.MID("modi.csv"."Tweet",TABLEAU.FIND("modi.csv"."Tweet", '#')),LENGTH("modi.csv"."Tweet")) ELSE 'No Hashtag' END),'pic',' '),'#1CrPplGaveUpLPGSubsidy..') THEN '#1CrPplGaveUpLPGSubsidy' WHEN TABLEAU.CONTAINS(REPLACE((CASE WHEN TABLEAU.CONTAINS("modi.csv"."Tweet",'#') THEN TABLEAU.LEFT(TABLEAU.MID("modi.csv"."Tweet",TABLEAU.FIND("modi.csv"."Tweet", '#')),LENGTH("modi.csv"."Twe et")) ELSE 'No Hashtag' END),'pic',' '),'#MyCleanIndia') THEN '#MyCleanIndia' WHEN TABLEAU.CONTAINS(REPLACE((CASE WHEN TABLEAU.CONTAINS("modi.csv"."Tweet",'#') THEN TABLEAU.LEFT(TABLEAU.MID("mo di.csv"."tweet",tableau.find("modi.csv"."tweet", '#')),LENGTH("modi.csv"."Tweet")) ELSE 'No Hashtag' END),'pic',' '),'#YogaDay') THEN '#YogaDay' WHEN TABLEAU.CONTAINS(REPLACE((CASE WHEN TABLEAU.CONTAINS("modi.csv"."Tweet",'#') THEN TABLEAU.LEFT(TABLEAU.MID("modi.csv"."Tweet",TABLEAU.FIND("modi.csv"."Tweet", '#')),LENGTH("modi.csv"."Tweet")) ELSE 'No Hashtag' END),'pic',' '),'Incredib leindia') THEN '#IncredibleIndia' ELSE REPLACE((CASE WHEN TABLEAU.CONTAINS("modi.csv"."Tweet",'#') THEN TABLEAU.LEFT(TABLEAU.MID("modi.csv"."Tweet",TABLEAU.FIND("modi.csv"."Tweet", '#')),LENGTH ("modi.csv"."tweet")) ELSE 'No Hashtag' END),'pic',' ') END) = 'No Hashtag') THEN 'No Hashtag' WHEN TABLEAU.CONTAINS((CASE WHEN TABLEAU.CONTAINS(REPLACE((CASE WHEN TABLEAU.CONTAINS("modi.csv"." Tweet",'#') THEN TABLEAU.LEFT(TABLEAU.MID("modi.csv"."Tweet",TABLEAU.FIND("modi.csv"."Tweet", '#')),LENGTH("modi.csv"."Tweet")) ELSE 'No Hashtag' END),'pic',' '),'http') THEN REPLACE(REPLACE((C ASE WHEN TABLEAU.CONTAINS("modi.csv"."Tweet",'#') THEN TABLEAU.LEFT(TABLEAU.MID("modi.csv"."Tweet",TABLEAU.FIND("modi.csv"."Tweet", '#')),LENGTH("modi.csv"."Tweet")) ELSE 'No Hashtag' END),'pic

26 String manipulation Original calculation* I congratulate each & every person who practiced Yoga today & made the 1st #YogaDay a success. IF CONTAINS([Tweet],"#") THEN MID([Tweet],FIND([Tweet],"#"), LEN([Tweet]) ) ELSE 'No Hashtag END 1. Checks if there s a # in the tweet 2. Finds the position of # 3. Uses MID to get the text section after # Progress so far: #YogaDay a success. Target: #YogaDay *simplified and redacted for illustrative purposes

27 [Nested!] String manipulation Original calculation (part 2) #YogaDay a success. IF [Hashtag - Part1] = 'No Hashtag THEN 'No Hashtag ELSEIF CONTAINS([Hashtag - Part1]," ") THEN LEFT([Hashtag - Part1],FIND([Hashtag - Part1]," ")) ELSE [Hashtag - Part1] END 1. Check if the text has a space 2. Find the position of the space 3. Use LEFT to get the text section before the space Output: #YogaDay

28 String manipulation REGEXP calculation REGEXP_EXTRACT([Tweet],"(#\w+)") I congratulate each & every person who practiced Yoga today & made the 1st #YogaDay a success. 1. Finds the # 2. Followed by one or more (+) word characters (\w). 3. Captures it (parenthesis) Done! No IFs No nested calculations No multiple functions

29 Tips: String manipulation If you are doing string manipulation, use Regexp Don t get intimidated by it! Plenty of help online Tip: copy sample data to a tester site and build your expressions there!

30 Tips: Long and Nested calculations If you are using an Extract, optimize it This will pre-compute the calculations and store them in the extract

31 Viz on LOD Calc Example: Cohort Analysis How much revenue is contributed annually by each customer acquisition year?

32 Viz on LOD Calc Customer Acquisition Date {FIXED [Customer ID]: MIN([Order Date])} SQL SELECT SUM("Extract"."Sales") AS "sum:sales:ok", CAST(EXTRACT(YEAR FROM "t0"." measure 0") AS BIGINT OR NULL) AS "yr:customeracquisitiondate:ok", CAST(EXTRACT(YEAR FROM "Extract"."Order Date") AS BIGINT OR NULL) AS "yr:order Date:ok" FROM "Extract"."Extract" "Extract" INNER JOIN ( SELECT "Extract"."Customer ID" AS "Customer ID", MIN("Extract"."Order Date") AS " measure 0" FROM "Extract"."Extract" "Extract" GROUP BY 1 ) "t0" ON ("Extract"."Customer ID" = "t0"."customer ID") WHERE ("Extract"."Market" = 'EMEA') GROUP BY 2, 3

33 Tips: Viz on LOD Calc LOD calcs are awesome, but Avoid using too many in a single view Avoid using them in nested calculations (!!)

34 Queries during Extract creation/refresh Extract Creation, Full/Incremental Refresh Compute Calculations Now Tableau Desktop Tableau Server Tableau Desktop Query Database Tableau Server Backgrounder Query Database

35 Query Optimizations and Logging

36 Query Caching Why do caching? Caching always helps, ON by default Refresh (F5) expires cache entries Cache served first from In-memory then by external cache Two levels of Query Caching

37 Two levels of Query Caching Q1 Q2 Q3 Abstract Query Cache SQL 2 Native Query Cache SQL 2 Database Q4 SQL 4 External Cache External Cache

38 Query Cost & Result Size

39 Example: Partial matching Viz Query Cache SQL Queries executed Sales for all cities in 'US' No entries SELECT country, city, sum(sales) FROM salestable WHERE country = 'US GROUP BY 1, 2 Sales for 'US' Sales for all cities in US None Sales for city 'San Jose' in 'US' Sales for all cities in US None Sales for all regions in 'US' Sales for all cities in US SELECT country, region, sum(sales) FROM salestable WHERE country = 'US GROUP BY 1, 2

40 Query Batching Query Batching Does dependency analysis Executes queries in parallel Query Fusion Fuses queries with same level of detail Example: sales for all cities, costs for all cities Fused to one query: sales, costs for all cities

41 Query Batching Queries to be executed Queries that are dependent Q1 Q2 Q3 Q4 Query Cache Queries to be executed in parallel Q5 Q6 Database Do not run duplicates and subsumed queries remotely Leverage abstract query cache Local restriction, filtering, aggregation

42 Sheet 3 Sheet 2 Sheet 1 Optimization Across Queries Category Data QuickFilter QuickFilter MAP QuickFilter Data QuickFilter Query SELECT SUM([Profit]), SUM([Costs]) FROM [Sales] GROUP BY [Product] WHERE [Region]= West SELECT [Region] FROM [Sales] GROUP BY [Region] SELECT YEAR([Date]) FROM [Sales] GROUP BY YEAR([Date]) SELECT [Product], [Region], SUM([Profit]), SUM([Costs]) FROM [Sales] GROUP BY [Product], [Region] SELECT [City] FROM [Sales] GROUP BY [City] SELECT SUM([Profit]), YEAR([Date]) FROM [Sales] GROUP BY YEAR([Date]) SELECT [Product] FROM [Sales] GROUP BY [Product] Query SELECT [Product], [Region], SUM([Profit]), SUM([Costs]) FROM [Sales] GROUP BY [Product], [Region] SELECT SUM([Profit]), YEAR([Date]) FROM [Sales] GROUP BY YEAR([Date]) SELECT [City] FROM [Sales] GROUP BY [City]

43 Query Batching Event Total Elapsed time for query batch Number of queries Source of query Query Category SQL query Query id

44 Query Batching Limitations Queries in viz-post processing Queries during compilation Dashboard dependencies Metadata queries

45 Query Categories Data Metadata Domain Now Nullcheck Geocoding SecondaryDSInterpreter Sort QuickFilter TempTable

46 Query Categories Category Data Metadata Domain Now NullCheck Geocoding SecondaryDSInterpreter Sort QuickFilter TempTable Description Gets the data that will be displayed in the Viz Gets data about the data: what tables are there, what fields do they have, etc. Gets the possible values of a field. Used in the filter dialog Get the current time Checks if a field has NULL values Gets data from the internal geocoding database. Used in maps. Blending Used when selecting to Sort by -> Field in the Sort Dialog To get data to populate the quick filters Related to creating or filling up temporary tables

47 Query Logging Tableau Desktop Log Location Mac: /Users/<username>/Documents/My Tableau Repository/Logs Windows: C:\Users\<username>\Documents\My Tableau Repository\Logs Tableau Desktop Query Database Processes Process Log File Name Log Description tableau.exe log.txt (or log_*.txt) Captures all queries issued hyperd.exe Hyperd.txt (or hyperd_*.txt) Processes queries against extracts

48 Query Logging Tableau Server Log Location Linux: /var/opt/tableau/tableau_server/data/tabsvc Windows: C:\ProgramData\Tableau\Tableau Server\data\tabsvc Processes VizQL Server Data Server Database Process Log File Name Log Description vizqlserver.exe vizqlserver_*.txt Captures all queries issued dataserver.exe dataserver_*.txt Processes published data sources queries hyperd.exe hyperd.txt (or hyperd_*.txt) Processes queries against extracts Backgrounder.exe backgrounder_*.txt Captures logs for extract refreshes

49 R E L AT E D S E S S I O N S Show me the queries! Thursday 1:45 2:45 MCCNO L3 346 Designing efficient workbooks Live on stage Thursday 10:45 11:45 MCCNO L2 La Nouvelle Ballroom C

50 #TC18 Thank you!

51 Please complete the session survey from the Session Details screen in your TC18 app

52