Enterprise Information Management with SQL Server 2016

Transcription

1 Enterprise Information Management with SQL Server 2016 Dejan February 27, 2016 #sqlsatpordenone #sqlsat495 Organizers February 27, 2016 #sqlsatpordenone #sqlsat495

2 Gold Sponsors February 27, 2016 #sqlsatpordenone #sqlsat495 Bronze Sponsors February 27, 2016 #sqlsatpordenone #sqlsat495

3 Speaker info 30 years of experience 13 books 10 courses Microsoft, SolidQ, Pluralsight Focus Data modeling Data mining Data quality February 27, 2016 #sqlsatpordenone #sqlsat495 Overview Data quality Data Quality Services Master data Master Data Services

4 Data Quality Issues Data is a key asset in a company Bad data problems in daily operations Bad data problems in strategic operations Line of business (LOB) applications and databases enforce data integrity Data complies with business rules Data quality is a stricter term Data must also be accurate, complete, etc. Data cleansing is a reactive approach Master data management (MDM) is proactive Data Quality Hard Dimensions Data quality aspects that you measure and improve define data quality dimensions Data profiling is about finding bad data Measure hard dimensions programmatically Hard dimensions include (not limited to): Completeness Accuracy Information Consistency

5 Data Quality Soft Dimensions Soft dimensions are not measurable programmatically Must communicate them with users Soft dimensions include (not limited to): Timeliness Ease of use Intention Trust Presentation quality Data Quality Schema Dimensions Schema dimensions are in part measurable programmatically Must understand the business Schema dimensions include (not limited to): Schema completeness Schema correctness Documentation Compliance with theoretical models Minimalization

6 Data Quality Level Data Quality Level Regular Data Cleansing Without DQS Amount of Work Needed for Cleansing With DQS Time Amount of Work Needed for Cleansing Time Data Quality Services Data Quality Services (DQS) is a knowledge-driven data quality solution, enabling IT pros and data stewards to easily improve the quality of their data

7 Knowledge Bases and Domains A knowledge base (KB) stores all the knowledge related to a specific type of data source Knowledge is contained in data domains A semantic representation of a type of data A list of trusted values, invalid values, and erroneous data Synonym associations Term relationships Validation and business rules Matching policies DQS Architecture

8 Batch Cleansing with SSIS SSIS Data Flow DQS Server Knowledge Base SSIS Package Values/Rules Source DQS Cleansing Component Destination Reference Data Definition Matching Policy Master Data Master data in a relational model Nouns of a business Entities (products, customers, categories) Master data in a dimensional model Dimensions that give context to measures Applications that include master data: Enterprise resource planning (ERP) Customer relationship management (CRM) Human resources management (HRM)

9 Master Data Factors that influence the definition of what is master data for a company Cardinality (number of rows) Complexity (number of attributes) Volatility (frequency of changes) Versioning, auditing, and history Data re-usage The more you reuse master data, the more value a MDM solution gives you Master Data Management Coordinated process of managing master data, involving policies, processes, procedures, and people

10 Data Quality Level Data Quality Level MDM Roles MDM applications should also support versioning, auditing, and workflows Data governance refers to data quality and MDM activities Data steward is the role that performs data governance Measure the improvements Communicate the results Master Data Management Without DQS and MDS Amount of Work Needed for Cleansing With DQS and MDS Time Amount of Work Needed for Cleansing Time

11 MDS Architecture MDS at a Glance The application and the managed database that contains the master data Windows Communication Foundation (WCF) APIs for all operations (programmability) Web and Excel UI to manage the model (schema) and data Validation rules Auditing and versioning Table and View bulk integration notifications and SharePoint and 3 rd party workflows

12 MDS Objects and Relationships 23 Entities and Attributes An entity represents a thing or a concept from the real world that you can identify and that is of business interest One entity is central to the model, and other entities relate to this central entity Attributes (at least Name and Code) Free-form Numeric limited to.net SqlDouble data type Domain-based File attributes

13 Derived Hierarchies Natural (derived) hierarchies stem from domain-based attributes Derived hierarchies can be recursive You can have only one recursive relationship in a derived hierarchy In a recursive hierarchy, you cannot assign member permissions You cannot have circular relationships Business Rules Business rules are If Then statements that ensure data quality Can specify a condition on a specific attribute value or whenever an attribute value changes Publish them after creation Can apply rules to a complete set of data, to a subset of data, or to a specific version of data Business Rules order 1. Default value 2. Change value 3. Validation 4. External action, like starting a workflow

14 MDS 2016 Improved performance Row compression on entities IIS Dynamic Content Compression Index defragmentation and log purging maintenance jobs Improved security Super User function permission (replaces Server Admin) Read, Create, Update, Delete instead of Read-Only, Update, Delete permissions Admin permissions on models and entities levels MDS 2016 Entity change logs on attribute (only option before 2016) or member level Tracing in the Logs folder under the WebApplication folder CSV format Improved manageability Attribute names longer than 50 characters Can rename and hide the Name and Code Member revision history Custom indexes nonclustered, filtered Entity sync relationship share lookup entity data between models

15 MDS 2016 Sync, index and log maintenance jobs Improved business rules Business rules extension with T-SQL UDFs as condition and SPs as action SQL objects are NOT part of package deployment Many-to-many relationships in derived hierarchies Automatically resolve merge conflicts Use change sets to save pending changes to an entity, and wait for an approval notifications when status changes Q&A Questions? February 27, 2016 #sqlsatpordenone #sqlsat495

16 #sqlsatpordenone #sqlsat495 THANKS! February 27, 2016 #sqlsatpordenone #sqlsat495