Things I Learned The Hard Way About Azure Data Platform Services So You Don t Have To -Meagan Longoria

Transcription

1 Things I Learned The Hard Way About Azure Data Platform Services So You Don t Have To -Meagan Longoria

2 2

3 University of Nebraska at Omaha Special thanks to UNO and the College of Business Administration for this awesome facility! 3

4 4 Sponsors

5 HELLO! I am Meagan Longoria I am a Solution Architect with BlueGranite in Denver, CO. I m a Microsoft Data Platform MVP. You can find me on Twitter or on my blog at DataSavvy.me. 5 Catch me at IT/Dev Connections and PASS Summit in October!

6 My Azure Experience Migration to Azure SQL DB Multiple Azure SQL DB and Azure SQL DW POCs Data Factory + Data Lake + Azure SQL DW project Data Mart with SSIS + SQL Server on a VM Lots of Power BI A bit of Power Apps + Common Data Service 6

7 So How Did It Go? It was great! And frustrating! And interesting! 7

8 Disclaimer Azure is constantly being updated. If you download these slides for reference, be sure to check the Azure blogs for the latest updates. 8 ie_song)#/media/file:bowiechanges2.jpg

9 Example Project Goals Their goals: Enable scalability to handle future growth Use Azure to optimize cost effectiveness and performance Create a design that can be used with multiple reporting tools Provide a reporting platform with data that users trust that arrives in a timely manner We added: Simple and resilient enough for them to manage with limited staff Note: They asked for a DW and we added a data lake for staging and to enable future possibilities. 9

10 10

11 11 My ADF Data Flow

12 12 Azure SQL Data Warehouse

13 Azure SQL DW Differences 13 Database projects aren t supported for DW. You can use it, but you can t publish or schema compare & no DW-specific syntax. Try VS Code! SSMS connects but no intellisense & scripting objects is slow. Try VS Code! You must manually update statistics there is no auto-update Different load patterns for MPP CTAS is your friend but you can t put it in a transaction

14 14 Demo Database Projects

15 15 Demo: Scripting Objects in SSMS

16 Azure SQL DW Differences 16 T-SQL limitations: Stored procs don t support default parameters No USE statement to switch DBs No MERGE statements No ANSI joins with updates or deletes No sequences No computed columns No primary keys, foreign keys, unique and check tables constraints SQL DW can only be accessed via public internet, cannot be part of a VNET with reserved IP, can t use with Express Route to make it like an internal resource

17 Example Dimension Load Compare to SSIS Pattern 17

18 PolyBase Is A Picky Eater Don t bother with text. Just use ORC too many issues with characters that can t be escaped in delimited text files (new lines, quotes, etc.). But ORC files aren t human readable. No predicate pushdown in PolyBase it reads the whole file every time. Handling schema changes isn t easy. What happens when you add a new column to a table that is being loaded to ADLS and read through PolyBase? This is mostly why we have our staging area in the Data Lake. We just need the current load to match the external table. 18

19 Azure SQL DW Thoughts Learn the Load Patterns and SQL Differences Get ready to write a bunch of T-SQL stored procedures SSMS is not really great here get used to VS You don t need DW if you have < 1 TB of data and a slow growth pattern Understand data distribution and movement Don t forget to pause 19

20 20 Simple Logging For Stored Procedures

21 21 Azure SQL Database

22 Azure SQL Database Advice No good way to migrate AND drop security move schema first, then load data Check out new Azure SQL Managed Instance Can t pause like Azure SQL DW Consider elastic pools Elastic queries only allow select, not insert/update/delete 22

23 Elastic Pools Large differences between peak and avg utilization Peak utilization for each DB occurs at different times edtus are shared between many DBs 23

24 Elastic Query (Preview) Horizontal vs Vertical Partitioning External table syntax similar (not same) as PolyBase 24

25 25 Azure Data Lake Store

26 ADLS Gaps Happy Note: You can now use Service Principal authentication with ADF to load data from ADLS to Azure SQL DW with PolyBase No dynamic partition creation available (must know your partitions beforehand) No georedundant storage for disaster recovery 26

27 27 Azure Data Factory

28 Note on Azure Data Factory ADF is not an ETL/ELT too. There is no T. You have to add in other technologies to get the T. It s an orchestration and data movement tool. 28

29 Azure Data Factory Gaps Non-timesliced data is awkward (But timesliced data is great!) ADF executes in UTC time. The source database was on Eastern time. We had to use Dateadd to get the appropriate timeslices. What happens during Daylight Savings time changes? ADF cannot move a file, only copy. If you want to delete the file from its original location, you must use a custom C# activity. You can overwrite a file of the same name. 29

30 Azure Data Factory Gaps One-time pipelines don t show up in the Monitor & Manage app with slices you can execute, and they aren t automatically executed upon deployment. The Monitor & Manage app needs a lot of work. Lots of freezing and slow response times with interactivity datasets causes the visual portion to be mostly useless. Refreshing the page removes any filters on activities/slices. 30

31 Azure Data Factory Gaps ADF executes in UTC time. The source database was on Eastern time. We had to use Dateadd to get the appropriate timeslices. What happens during Daylight Savings time changes? Option: convert to custom activity. The Monitor & Manage app can be slow.100+ datasets causes the visual portion to be mostly useless. Refreshing the page removes any filters on activities/slices. ADF has no error handling Can t execute ADF natively more than every 15 minute 31

32 Azure Data Factory Issues 32 When you build a Data Factory project in VS, it revalidates all the files in the project. 175 SQL tables 575 data sets and a handful of pipelines: several minutes of waiting. When you build the VS project and are using config files, the debug folder will include items at its root AND a folder for each configuration (dev/test/prod) containing each ADF object built to that configuration. Deploying can take 1 2 hours depending on resources available in Azure and network connectivity. On-premises gateway troubleshooting fun: missing JRE = missing Visual C++ redistributable package.

33 Azure Data Factory Strengths PaaS requires no infrastructure and is easy to scale out ADF JSON is easier to source control than the GUIcreated XML of SSIS Native support for zip/unzip Dynamic partitioning for folder and file name Data lineage 33

34 Azure Data Factory Advice 34 Biml! Deploy with PowerShell Make your peace with timeslices Take advantage of data lineage/dependencies Decide whether to use ADF scheduling or Azure Automation Sharpen your C# skills you ll need them for custom activities Gerhard s Power BI ADF monitor (

35 Final Thoughts Azure SQL DW is great but it s not just a big SQL Server. Azure data solutions may use 5 10 languages. Are we ready to become full stack developers? Get to know the Azure pricing calculator, but you ll probably need a baseline first. Slim down before you go to the cloud to avoid unnecessary charges. 35

36 THANKS! Any questions? You can find me DataSavvy.me 36