Data Warehouse Design Decisions

Transcription

1 Data Warehouse Design Decisions August 2015 Colleen Barnitz Director, IT Development MVT Services

2 Colleen Barnitz over 20 Years in IT worked with SQL Server since version 6.5 developer and an architect on data warehouse projects during that time leads a team of developers in providing applications and business intelligence solutions for the enterprise co-founded and is active in the Las Cruces & El Paso SQL Server User Group Colleen.Barnitz@gmail.com 2

3 It s An Exciting Time to be in the Data Business 3

4 Agenda IT and Data Services Traditional Data Warehouse Schema ETL OLAP Big Data Hadoop Streaming Data Event Processing Integration one big happy data family 4

5 Data Services Businesses need to adapt quickly to changing conditions IT role is to make the data resources available for decision making Data for both analysis and applications Make it pain-free for users Be prepared to be Agile 5

6 What is a Data Warehouse? Central repository for integrated data from one or more sources Concept originated in the late 1980 s Stores current and historical data Typically built in relational database Schema-driven Optimized for query and analysis 6

7 Built for the Business A source of approved data for your users and applications User friendly renamed cryptic columns and table names Eliminates user research into what is relevant data / what isn t Contains approved business rule logic Prevents querying against the line of business systems 7

8 Still valid after all these years sub-terabyte data Setup a RDBMS on a server or in a cloud environment Implement your data model Populate your data Hook up your front-end BI tools 8

9 The Data Warehouse Project Project Management basics Get senior management sponsorship Define and prioritize specific business needs to be addressed Define data architecture to be used Define the technology stack Iterate to add increasing value Sustain the data warehouse with governance Promote and train data warehouse users 9

10 Data Modeling Choices: Kimball Dimensional Modeling Star Schemas Inmon model 3 rd Normal Form first then build data warehouse Data Vault Hub, Links, and Satellites Anchor Modelling variation upon the Data Vault 1 0

11 Kimball Methodology Enterprise Data Warehouse Bus Architecture 1 1

12 Dimensional Modeling Gather business requirements and source data realities Select the process Declare the grain Identify the dimensions Identify the facts Get Business buy-in 1 2

13 Star Schema Fact table(s) linked to their associated dimension tables Linked by Primary / Foreign key relationships OLAP cubes can be derived from the star schema 1 3

14 1 4

15 Dimension Tables Provide WHO, WHAT, WHERE, WHEN, WHY, and HOW context Descriptive attributes used for filtering, grouping Take time to design critical for the end user s BI experience Different flavors: Slowly Changing Dimensions Types 0 thru 7 Junk Dimensions Bridge Tables 1 5

16 Conformed Dimensions Design common dimensions to be used for multiple fact tables Support drill across and integrate data Reuse shortens development time 1 6

17 Fact Tables Measurements from the business process Almost always numeric Only facts consistent with the grain A physical observable event - e.g. a sales transaction Kinds: Periodic Accumulating Temporal 1 7

18 Data Vault Modeling Unified Decomposition Decompose the data into: Business Keys Relationships Attributes Strengths: Agility Auditability History Tracking Easy to Automate 1 8

19 Data Vault Cont d Every row has to have source and load date No distinction between good and bad data (not cleansed) Parallel loading can scale without major redesign Easy going in but challenging getting data out More tables and MORE joins Not intended for direct access Will need to be translated to a dimensional or other data mart model for distribution (lost me here) 1 9

20 2 0

21 Anchor Modeling Open source database modeling technique Designed to address constant change Large change on the outside of the model small change within Concepts: Objects Attributes Relationships Based on sixth normal form Highly decomposed 2 1

22 2 2

23 Populating the Warehouse 2 3

24 ETL Extract the process by which data is extracted from the data source Transform transformations and integrity checking Load loading the data into the warehouse Designing from the end backwards Business rules are coded into the transformation process Only the data required for the output is included in the extraction process Trying to design for current and future needs 2 4

25 ELT Extract, Load, Transform We don t need no stinking transforms interpretation Extract data sources into staging perform integrity checks Load source data into the warehouse (cleaned, offline copy) Transform create target output DW has a cleaned, offline copy of the source data Separates the transformation process from the Extract and Load. Separates loading from process design Removes the dependency on designing from the back Brought all the data and now available for future needs Tradeoff of using a lot more database server resources 2 5

26 Automation of ETL/ELT is KEY Brittle designs hold back changes and new development SSIS editing can be very frustrating Code Generation BIML / Bids Helper Write your own Code Generator will use metadata about sources and destinations Map the views/tables Map the columns Flag the Business keys Flag the Update type (dimensional modeling) Helper views on the source can provide simple transforms 2 6

27 ELT Automation Friendly Not getting bogged down by destination design decisions Metadata Mapping is straight-forward Source meet Target GO! 2 7

28 OLAP - On Line Analytical Processing 2 8

29 Analysis Services (SSAS) Options Multi-Dimensional mode Works well with star schemas Tabular Mode Forgiving, adaptable to non-star schema designs xvelocity in-memory analytics engine (VertiPaq) 2 9

30 Adding Data Scientists / Uber Analysts to the Mix Extreme users of data All about insights into revenue, profitability, customer satisfaction Working directly with senior management They need to spend their time understanding what data means and not creating clean, integrated data sets 3 0

31 Providing for the Analysts A Sandbox An Analyst Sandbox complements the data warehouse Environment for: Experiments New ideas Test hypotheses New data sources Evaluating and exploring new tools *Minimally Governed (Bob Becker Design Tip #174 Kimball Group) 3 1

32 Analytics Sandbox CAVEATS: Not for core business functions Not for ongoing reporting or analytics If an experiment pans out turn it over for implementation in the data warehouse 3 2

33 Traditional Data Warehouse Flow Audits STAGING ERP Reporting Web Services Data Warehouse - Source/Persistent - Star Schema OLAP Adhoc Reporting Power Pivot Power BI Analytics Sandbox Applications 3 3

34 Adapting to Changing Times 3 4

35 Old School DW very structured heavily designed subject to well-defined business rules tightly governed by the enterprise Synchronized with production via regularly scheduled loads. fairly rigid. Takes time to react to new data and analytic requests. 3 5

36 4 V s of Big Data Variety: new sources and types of data Volume: amounts of data increasing. Velocity: low-latency, real time delivery expected Veracity: uncertainty about the quality of data 3 6

37 Volume Volume expanding tenfold every five years Log Files Social Media Click Stream Device generated Remote monitoring sensor RFID Spatial and GPS coordinates Sometimes it s too big to copy Ralph Kimball - Whale in the swimming pool 3 7

38 Velocity Used to be good enough to load the data nightly We used to only analyze history, or near-history New way take action on current data 3 8

39 Hadoop Stores non-relational, unstructured data Handles quickly arriving data Distributed processing across clusters of commodity computers Cost effective for increased capacity and processing Popular with companies having to deal with PB size data Useful for < PB data as well 3 9

40 Map Reduce How Hadoop processes work among its nodes Map Split the data into pieces Processed in parallel on individual nodes Stores results locally Reduce Aggregate the data 4 0

41 HDFS Hadoop Distributed File System Where the data is stored Linked across all the nodes in the cluster 4 1

42 Azure HDInsight Hortonworks Hadoop hosted in Azure Integrates with Excel, SQL Server Analysis Services, and SQL Server Reporting Services Includes Implementations of: Hbase NoSQL database built on Hadoop Apache Storm Processes large streams of data fast Pig Simpler scripting for MapReduce Sqoop Data import and export (etl) Etc. 4 2

43 Spark for Azure HDInsight Significantly faster than MapReduce Can manipulate data in-memory Offers the interactivity of OLAP or DW column store Supports Batch and Interactive queries Real-time streaming Machine learning Graph processing Integrated with Power BI service 4 3

44 Why incorporate Streaming Data? Real Time Decisions for: Manufacturing Process Financial trading Web analytics Operational Analytics 4 4

45 Hybrid Data Warehouse Environment 4 5

46 What s in the Data Lake? Archives Operational data Journals and Audits Logs Analytics Sandbox 4 6

47 4 7 Relational Database aren t standing still

48 SQL Server 2016 BI Features Updateable nonclustered columnstore index support with columnar index in-memory or on-disk row store Polybase distributed queries to Hadoop, blobs, files and relational data in SQL Server reaches across on premise and cloud sources options to import Hadoop data or archive relational data into Hadoop Master data management improvements / additions Revolution Analytics R baked in made via T-SQL queries SSRS - improvements / additions Mobile-friendly reports 4 8

49 Columnstore Data that is: Logically organized as a table with row and columns Physically stored as columns Vertipaq Technology All in memory if possible If not, uses cached, recently used data 4 9

50 Columnstore Index Technology for storing, retrieving, and managing data using columnar data format Data is: Compressed Stored in column segments Query optimizer considers it for use just like a regular query Restrictions on data types that can be used e.g. no decimal with precision > 18, binary, text, varchar(max), etc. 5 0

51 NonClustered Columnstore Index Introduced in SQL Server 2012 Order of columns in the index does not matter cannot update table To update - drop and rebuild the index or swap out partitions 5 1

52 Clustered Columstore Index New to SQL Server 2014 Includes all columns in table No other indexes allowed on table Can update with Insert, Update, Delete operations Cannot use unique, primary key or foreign key constraints Cannot create on a table with computed or sparse columns 5 2

53 Columnstore Index Use Case Best for data warehouse workloads Read-only queries analyzing large sets of aggregated data Best for bulk load operations Not suited to seek operations for specific data 5 3

54 SQL Server 2016 SSIS (ETL) SSIS designer will support previous versions. Support for Power Query as a data source for self-service ETL Incremental deployment options Custom logging levels Package templates for ETL code reuse 5 4

55 SQL Server 2016 SSRS (Reporting) FINALLY - improvements and additional features Mobile-friendly reports, support for browsers on multiple platforms Native connectors for latest versions of Microsoft data sources - SQL Server and SSAS; third-party data sources Oracle Database; Oracle Essbase; SAP BW; Teradata ODBC and OLEDB connectors for many more New report themes and Styles New chart types Greater control over parameter prompts Dynamic parameterized report design options 5 5

56 SQL Server Analysis Services Tabular Mode Improvements in enterprise readiness Query engine optimizations Direct Query Parallel partition processing Advanced modeling New DAX functions DATEDIFF GEOMEAN PERCENTILE PRODUCT XIRR XNPV 5 6

57 Data Governance Come up with standards and process on handling data Implement master data management Involve the business in the process 5 7

58 Understand the New Unstructured / distributed database technologies ( NoSQL ) Pick the best technology for the purpose 5 8

59 What Does your DW look like? Find that compromise between your available resources and what YOUR users need Design with outcomes in mind Deliver initial results quickly, then adapt and iterate Build systems to change not last Pick a problem and get started! 5 9

60 References The Microsoft Modern Data Warehouse Josh Fennessy resource for BI and Hadoop information HDInsight Kimball Group Dimensional Modeling Biml Announcing Spark for Azure HDInsight 6 0

61 References cont d Book: Modeling the Agile Data Warehouse With Data Vault Hans Hultgren Data Vault Modeling Anchor Modeling 6 1

62 Las Cruces & El Paso SQL Server User Group Meets Second Thursday of the Month at Noon New Horizons Training Center 1625 Hawkins Blvd El Paso, TX * Talk to us if you re from Las Cruces with enough interest we may do alternating locations! 6 2