Migrating Enterprise BI to Azure

Transcription

1 Migrating Enterprise BI to Azure Best Practices Wlodek Bielski

2 SQLSat Kyiv Team Yevhen Nedashkivskyi Mykola Pobyivovk Denis Reznik Eugene Polonichko Oksana Borysenko Oksana Tkach

3 Sponsors

4 Session will begin very soon :) Please complete the evaluation form from your pocket after the session. Your feedback will help us to improve future conferences and speakers will appreciate your feedback! Enjoy the conference!

5 About me MCSD: Azure Solutions Architect MCSE: Cloud Platform and Infrastructure MCSE: Data Management and Analytics AWS Certified Solution Architect Google Professional Data Architect Google Professional Data Engineer

6 Agenda Setting up Azure environment Networking, VPN and on-prem connectivity Migrating Analysis Migrating Staging / DWH Migrating ETL/ELT Enabling Big Data scenarios

7 SETTING UP AZURE ENVIRONMENT

8 Azure purchasing options Free Trial Pay-as-you-Go Open Licensing Enterprise Agreement Cloud Solution Provider (CSP)

9 Dev/Test offering

10 Setting up Azure environment Dev/Test + Prod subscriptions Design multiple Resource Groups bi-prod bi-test bi-dev bi-common Networking (VNet, VPN) Might be costly to change later

11 NETWORKING AND CONNECTIVITY

12 Networking part Point-to-Site VPN quick start Site-to-Site VPN will take time ADF v2 Self-Hosted IR On-premises Gateway (Analysis and Power BI)

13 VPN: Hub and Spoke topology

14 MIGRATION JOURNEY

15 Migration Roadmap Starting point Phase 1 Phase 2 Phase 3 Database ETL/ELT OLAP Reporting SQL 2008 R2 SQL Server SQL Server Azure SQL DB / DWH Integration Integration Azure Data Factory v2 with SSIS Azure Data Factory v2 Analysis Analysis Azure Analysis Azure Analysis Reporting Reporting Power BI Power BI

16 Phase 1: Lift & Shift Starting point Phase 1 Phase 2 Phase 3 Database ETL/ELT OLAP Reporting SQL 2008 R2 SQL Server SQL Server Azure SQL DB / DWH Integration Integration Azure Data Factory v2 with SSIS Azure Data Factory v2 Analysis Analysis Azure Analysis Azure Analysis Reporting Reporting Power BI Power BI

17 Main objectives Migrate solution to VM(s) Take advantage of Azure services Setup secure and manageable environment Gain confidence

18 Lift & Shift options Backup-restore Data Migration Assistant Windows template SQL Server templates SQL Database Managed Instances License considerations

19 Data Migration Tools Data Migration Assistant Supports both scenarios On-premises -> -> PaaS Azure Database Migration Service Managed service for migrations at scale

20 PaaS services coverage Covered Database Engine (SMP / MPP) Analysis Integration (via ADF) Not covered SQL Server Agent Reporting Master Data Data Quality Managed Instance?

21 SQL Database Managed Instance

22 Infrastructure recommendations Plan for PaaS from the beginning Decouple services to separate VMs Use Premium managed disks Less burden with storage accounts (leftovers) Consider 2/4 TB disks (P40/P50) Azure Backup supports up to 16 data disks Azure Backup + Recovery Vault Use both (VM + disks/data)

23 Licensing considerations Pay-as-you-Go vs BYOL BYOL: 2008 R2+ are supported Consider SQL Server Developer Edition Consider memory-optimized VMs (G- / M-) for Analysis Tabular Dev/Test subscriptions

24 Cost control recommendations Start small Cloudyn Azure Cost Management Consider Auto-Shutdown Reserved instance? Commitment vs flexibility DEMO Azure Portal

25 Phase 2: Transition Starting point Phase 1 Phase 2 Phase 3 Database ETL/ELT OLAP Reporting SQL 2008 R2 SQL Server SQL Server Azure SQL DB / DWH Integration Integration Azure Data Factory v2 with SSIS Azure Data Factory v2 Analysis Analysis Azure Analysis Azure Analysis Reporting Reporting Power BI Power BI

26 Main objectives Start using PaaS services Optimize infrastructure Understand and control costs

27 PaaS recommendations Start with Analysis quick win Consider Azure SQL DB for DWHs < 1 TB Redesign ETL/ELT in phases Start with SSIS runtime Rewrite to pure ELT afterwards Orchestration instead of linear pipelines

28 Analysis deployment options Multidimensional On-premises (BYOL) (SQL Server template) Tabular On-premises (BYOL) (SQL Server template) Azure Analysis Power BI Premium (de facto)

29 Azure Analysis Scaling up / out finally! Pricing tiers Replicas Managed backups Expensive service use with care REST API Can be automated via Logic Apps / Functions

30 SSAS: On-premises Data Gateway

31 Azure Data Factory v2 Great orchestration tool with simple GUI Integrated Git support easy versioning Runs SSIS (lots of other engines as well!) Supports ARM templates Not an ETL tool actually!

32 Power BI vs SSRS Power BI has limited support for pinning SSRS reports Paginated reports likely to be supported in future Power BI releases Plan for leveraging Power BI from moment of migration Keep SSRS on small separate VM if needed

33 Phase 3: Cloud Native Starting point Phase 1 Phase 2 Phase 3 Database ETL/ELT OLAP Reporting SQL 2008 R2 SQL Server SQL Server Azure SQL DB / DWH Integration Integration Azure Data Factory v2 with SSIS Azure Data Factory v2 Analysis Analysis Azure Analysis Azure Analysis Reporting Reporting Power BI Power BI

34 Main objectives Operate pure PaaS solution Get ready for Azure DWH with ELT Enable Big Data / real-time scenarios

35 Main activities Migrating to Azure SQL Database Getting rid of SSIS / ETL Rewriting new ELT orchestrations Minimizing VM workloads

36 Beyond classical DWH/BI HDInsight Data Lake Storage / Analytics Databricks.. Data Factory to orchestrate them all!

37 Wrap-up Phased approach Organization needs to learn new ways Not too fast....but it s just the right time to start!

38 Thank you!

39 Sponsors