Please silence cell phones

Transcription

1 Building Better SSIS Packages Please silence cell phones Tim Mitchell Independent Business Intelligence Consultant 2 Explore Everything PASS Has to Offer Agenda FREE ONLINE WEBINAR EVENTS LOCAL USER GROUPS AROUND THE WORLD FREE 1-DAY LOCAL TRAINING EVENTS ONLINE SPECIAL INTEREST USER GROUPS VOLUNTEERING OPPORTUNITIES PASS COMMUNITY NEWSLETTER Reliability Resiliency Reusability Maintainability Performance Security 3 FREE ONLINE RESOURCES BUSINESS ANALYTICS TRAINING BA INSIGHTS NEWSLETTER 4 About Tim Mitchell Business intelligence consultant Microsoft Data Platform MVP TimMitchell.net Twitter.com/Tim_Mitchell Housekeeping Questions Breaks Lunch Resources: TimMitchell.net/BetterSSIS 5 6 1

2 Housekeeping Not everything applies to everyone, but everyone will get something Who are you? Job? First PASS Summit? User group members? Presenters or bloggers? 7 8 Building Better SSIS Packages: Reliability A dirty little ETL secret: Reliable SSIS packages are dull. 10 Reliability What would make you trust your ETL processes? Knowing that it completed Confirmation that it was successful Notification upon failure or anomaly Reliability What would make you trust your ETL processes? Audit trail to prove nothing is out of place Knowing that recent changes did not upset core functionality

3 Notification If a package fails and nobody knows, did it really fail? Notification of failure Capture and notify upon anomaly What is an anomaly? Detection methods Audit Trail The package succeeded, but did it really do what it was supposed to do? Data validation Row count match Logging Even less exciting than documentation. But Establishes patterns of normalcy Easier root cause identification Clearer path to resolution upon error Logging Logging options Catalog logging (SSIS 2012+) Package logging Custom logging Logging Package logging Package by package Very flexible Cumbersome Package Logging 17 3

4 Logging SSIS Catalog Version Very easy for catalog-deployed packages The SSIS Catalog History MSDB or Package Store File System Package logging Configurations The SSIS Catalog Overview Catalog released in 2012 Default Optional The SSIS Catalog Just a database (sort of) No more MSDB storage Dedicated database, tables, views, and sprocs Not installed by default SSIS Catalog Features Integrated logging SSIS environments Execution via T-SQL SSIS Catalog Structure User database: SSISDB Schemas Internal Catalog

5 SSIS Catalog Structure Database requires regular backup, etc. Modifications allowed but strongly discouraged Custom objects/schema SSIS Catalog Structure Integration Services Catalog Browsing node in SSMS One catalog per instance Folder structure Environments SSIS Catalog Deployment Granularity = project, not package Application vs. component Requires careful attention to source control Changes back to package deployment in 2016 SSIS Catalog Deployment Deployment tools Deployment from Visual Studio Deployment from SSMS dtutil(command line) SSIS Catalog Deployment Versioning Maintains a brief history of deployed projects Not a substitute for source control! Working with the SSIS Catalog 29 5

6 Logging Catalog logging Very easy Predefined logging levels Logging Catalog logging Least flexible * Changes coming in Catalog Logging Notifications Validation failure Package failure Non-failure anomaly Rowcount mismatch, etc. 34 Notifications Package notifications Send mail task Easy to use Clunky Notifications Scheduling tool notifications Most scheduling tools Removes notification from ETL process More reliable

7 Data Validation Check resulting data for Valid lookup values Rowcount match Total dollar value match Data signatures? Percent of nulls Average dollar value within known range Data Validation 37 Resiliency Building Better SSIS Packages: Resiliency What would happen if your most critical SSIS package were to encounter an error condition right now? Continue execution Fail gracefully Undo Notification 40 Resiliency Every process will eventually fail. -Me, 2015 Resiliency SSIS makes resilient packages easier to build Control flow Data flow Logging Restartability

8 Resiliency Design Patterns Fail Fail gracefully Log errors Notification Undo partial load Prevent failure and continue Fail at end? Control Flow Error prevention and recovery Precedence constraints With or without expression Event handlers Does not prevent error Control Flow Error prevention and recovery Propagation Error prevention Not suitable for all cases Logging / notification Data Flow Row-level errors Triage (split) Lookup Redirect Source Destination Transformation Data Flow Row-level errors Prevent error Row- or buffer-level? Relational operations: buffer-level Resiliency Demos: Error Handling and Prevention 47 8

9 Building Better SSIS Packages: Reusability Reusability Mitchell s Law: Never write the same code twice (TimMitchell.net/MitchellsLaw) 50 Reusability SSIS design patterns support reusability Parent/child structures Expressions Parameterization Reusability Avoid hard-coding anything Connection strings File paths Lookup ID values Reusability SSIS catalog Environments Applied to package parameters Reusability Atomic packages Do exactly one thing More packages, but easier to edit, test, deploy

10 Reusability Atomic packages One package per table load/operation Less time spent validating Multi-developer environment Building Better SSIS Packages: Maintainability 55 Maintainability Be kind to your fellow developers and to yourself. Who wrote this code? (Oh -it was me) Documentation The dirty D word: documentation Yes, it s dull Yes, it s necessary Documentation Internal documentation Annotations Description fields Quick ref: package explorer Documentation Internal documentation Naming conventions

11 Documentation Application name on DB connections Track package name, other info Documentation External documentation Manual Documentation tools (third party) Useful but not comprehensive Source control Package Maintainability Documentation Code integrity and evolution Versioning Labels Branching 64 Source control Important in most every dev scenario Yes, even in single-developer environments! Requires discipline but it s worth it Source control Tools required Source control system (TFS most common) Source control provider Separate install in SSDT

12 Biml Business Intelligence Markup Language Why edit when you can regenerate? Dozens or even hundreds of packages at once Biml Pattern-based automation Infer structure from metadata Source-to-target mapping Business rules Biml Biml BimlCode (XML) Biml interpreter (BIDS Helper) Independent Independent SSIS Independent SSIS Independent Packages SSIS Packages SSIS Packages Packages The good: Rapid development of lots of packages Reusable patterns Built into Visual Studio (with BIDS Helper) It s free! Biml The challenges: It s XML with C# (or VB.net) Intellisense Debugging Maintainability: Biml Not a solution to every ETL problem! Repetitive or similar patterns Frequent ETL changes, or a large number of ETL processes to create Staff capabilities

13 Package Maintainability Biml Building Better SSIS Packages: Well-Performing Performance Considerations Before focusing on performance, ask: Is performance really a concern? Is it really an SSIS concern? What task or component is most at risk (or causing the most trouble)? Performance Considerations Logging is critical! Yes, it s still dull Identify patterns or normalcy Runtimes Row counts Lookup Management Know your cache modes Full (default) Partial None Lookup Management Full cache Entire contents of the lookup table loads to memory Lookups done in memory

14 Lookup Management Partial cache Row-by-row lookup for each distinct value Then it loads to cache Subsequent same-value lookups done in memory Lookup Management No cache Row-by-row lookup for each value Narrow set of applications Package Performance Lookup Cache Modes Avoid Unnecessary Expenses Don t use the table drop-down list! It s convenient It s expensive SELECT * OPENROWSET to get metadata 82 Avoid Unnecessary Expenses Don t use the table drop-down list! Use the query window to get only the needed columns Use WHERE filtering if possible Package Performance Avoiding the Table List 83 14

15 Blocking Transforms Know the blocking properties of transforms Non-blocking Partially blocking Fully blocking Blocking Transforms Non-blocking: no buffers are held Row count Derived column transform Conditional split * Blocking Transforms Partially blocking: buffers are held as needed Merge join Union all Lookup * Blocking Transforms Fully blocking everything stops Sort Aggregate Fuzzy tools Blocking Transforms Blocking transforms are not evil! There s a tool for every job. Package Performance Blocking Transformations 89 15

16 Slow Transformations Some transformations are just slow by nature SCD wizard OLEDB command Sort Slow Transformations Know the workarounds Other transforms Third party SSIS add-ins Non-SSIS solutions T-SQL MERGE Package Performance Slow Transformations Incremental Loads Process only new and changed data Single load package for full, incremental No need to duplicate load logic Better performance 94 Incremental Loads Incremental Loads What is an incremental load? Incremental Load vs. Full Load Entire Data History Data Warehouse New and changed data Data Warehouse 16

17 Incremental Loads Incremental load methods Change tracking Change data capture Row hashing Insert/update date Brute force (column-by-column) Incremental Loads Why incremental loads? Small amount of changed data per execution Performance Source data no longer available Incremental Loads Processing time increases with data volume Scales linearly (assuming no bottlenecks) Processing Time vs. Data Volume Incremental Loads At the source Limit the retrieved data to only new or changed records Dependent on a reliable method at the source for new/changed records 100 Incremental Loads At the destination Filtering of insert/update is done at the destination prior to load Performance can be a concern Volume of data Brute force method T-SQL MERGE statement Source Incremental Load Patterns Update timestamp Relies on a solid method at the source of tracking the last modification of each row ETL will store the last load start date, and only process rows modified since

18 Source Incremental Load Patterns Update timestamp Can be risky! Often relies on app logic Locking source data during load Change Tracking Tracks adds, changes, deletes Change tracking version number Enabled at the table level, tracked at the column level 104 Change Tracking Pros: Lightweight and synchronous Minimal storage requirements Easy to set up and use Change Tracking Cons: SQL Server source only Requires setup and maintenance outside the realm of typical ETL Reusability Change Tracking Change Data Capture Change Data Capture (CDC) Relational source mechanism for persisting complete change history of tracked data Asynchronous handled by SQL Agent jobs SSIS tasks/components for managing CDC data (new to 2012) 18

19 Change Data Capture Change Data Capture Pros: Complete change history, not just the delta Some integration with non-sql Server data (such as Oracle) Change Data Capture Cons: Storage Performance Maintained outside the ETL environment Change Detection Using Hashing Change tracking or CDC not always available Calculate row hash before insertion Compare hash value AA John Smith 0x54 0xa4 0x9b 112 Change Detection Using Hashing Collision concerns Consistent algorithm Column order Combining fields AA vs. AA Change Detection Using Hashing Dealing with null, blank Tips Separators Include column length

20 Improving MERGE Performance MERGE INTO [DimProduct] AS FACT USING [Staging] AS SRC ON ( FACT.ProductKey = SRC.ProductKey ) WHEN MATCHED AND FACT.Hash_SHA1!= SRC.Hash_SHA1 THEN UPDATE SET FACT.[Description] = SRC.Description,FACT.[ReorderPoint] = SRC.ReorderPoint,FACT.[StockLevel] = SRC.StockLevel ; Destination Incremental Loads Check record status for insert, update, delete Managed in SSIS Lookup Merge join Conditional split 116 Destination Incremental Loads Destination Incremental Loads Very versatile most any source, destination Properly configured, can perform very well Destination Incremental Loads Slowly Changing Dimensions DimCustomer CustomerId GivenName FamilyName State HairColor Bob Smith FL MN Brown FactClaims ClaimId CustomerId Date ClaimCode ReasonCode AUTO W-SNOW AUTO W-SNOW

21 SCD Type 1 Changing Attributes SCD Type 2 Historical Attributes DimCustomer CustomerId GivenName FamilyName State HairColor Bob Smith MN Brown Grey DimCustomer CustomerId BusinessKey GivenName FamilyName State HairColor StartDate EndDate BOBSMITH01 Bob Smith MN Brown NULL BOBSMITH01 Bob Smith FL Brown NULL Change is made in place original row is updated Change is done in two steps Old row is marked as expired New row is added SCD Type 3 Historical Attribute Columns DimCustomer CustomerId BusinessKey GivenName FamilyName CurrentState PreviousState BOBSMITH01 Bob Smith MN FL NULL MN Change is done in two steps Old value is moved to history column New value is added to current column Destination Incremental Load Patterns Slowly changing dimension wizard Built into SSIS Relatively easy to implement for small loads Does not scale well 123 Incremental Loads Final thoughts Several ways to accomplish incremental loads Source methods Destination methods Native Third Party There is no silver bullet Trial and error Performance Incremental Loads

22 Security Building Better SSIS Packages: Security Package security Securing parameter values Runtime security credentials Securing the SSIS catalog 128 Package Digital Signature Runtime check Protect against package tampering Rarely used Package Protection Level Secure sensitive portions or all of the package Opt not to save sensitive info at all (recommended) Package Protection Level Levels: Encrypt Sensitive with User Key (default) Encrypt Sensitive with Password Encrypt All with User Key Encrypt All with Password Don t Save Sensitive Package Protection Level Only applies in Visual Studio or file system Server storage

23 Securing Parameter Values Project deployment only Sensitive flag on parameters Obfuscated in logging Securing Parameter Values Applies to variables in SSIS catalog environments Sensitivity level must match! Runtime Security Credentials Who is executing this package? The current logged-in user Database engine service account SQL Server Agent service account Proxy account Runtime Security Credentials Know your execution account! Database permissions File system permissions Notifications Runtime Security Credentials Execution source SSDT / BIDS SQL Server job dtexec.exe Runtime Security Credentials Proxy accounts Explicit substitution credentials Relies on SQL Server credential

24 Securing the SSIS Catalog SSIS security history Managed in MSDB Database roles Very broad Securing the SSIS Catalog Catalog security More sophisticated Windows ACL-like Scriptable Securing the SSIS Catalog Catalog security SSIS database roles not required User or group Edit, execute, manage permissions Security SSIS Security 141 Session Evaluations Your feedback is important and valuable. ways to access Submit by 5pm Friday November 6 th to WIN prizes Thank You Go to passsummit.com Downloadthe GuideBookApp and search: PASS Summit 2015 Follow the QR code link displayed on session signage throughout the conference venue and in the program guide