Seven Interesting Data Warehouse Ideas

Transcription

1 Seven Interesting Data Warehouse Ideas

2 Learning Objectives Take a detailed dive into some interesting ideas and concepts that can enhance your data warehouse or reporting database. Review some examples of code and focus on some key topics. Broad Topic, High Level View, Only 75 Minutes

3 Agenda (Estimated) Agenda/Introductions (5 Min.) Standards are a Great First Step (15 Min.) Integration of a Data Dictionary (10 Min.) The Order of All Things (15 Min.) Got functions? (10 Min.) Thousands of Parameters (5 Min.) Data Cleanup and Maintenance that Works (15 Min.) Odds and Sods (ask anything you want) Resources (Provided as Reference We Won t Cover) Questions Anytime (More Fun For All) brian.dowling@supportingadvancement.com

4 Experience, Expertise, BIAS Sr. Vice President for Finance & Information Systems VGH & UBC Hospital Foundation Principal and Founder SupportingAdvancement.Com SupportingFundraising.Com Finance, Information Systems, Operations (Most of Career) Large, Medium and Small Organizations Not for Profit and Private Sector Financial Statements, Audit, Investment Management Transaction Processing and Management Technical Infrastructure Database Management, Reporting and Data Warehousing Web, Online, Social Media, Marketing Part-Time Consulting

5 Lessons Learned Ongoing Support from Sponsors is Critical Planning is Not Always Fun But Essential Be Creative Change Management & Communication is Key Change in Jobs, Roles and Responsibilities Everything Takes a Lot of Time and Effort Every Organization has a Variety of Issues Try and Learn from Past Mistakes Have Fun

6 What is a data warehouse? Repository of an organization's electronically stored data. Data warehouses are designed to facilitate reporting and analysis. A data warehouse houses a standardized, consistent, clean and integrated form of data sourced from various operational systems in use in the organization, structured in a way to specifically address the reporting and analytic requirements. (Wikipedia) Where you can combine your data in ways that are not possible in transactional and other systems. Where there is one version of the truth. The Idea of Real Time. Any time you get information in less than real time, you need to significantly re-architect your information environment. Ralph Kimball

7 Standards are a Great First Step

8 Entropy The Amount of Order, Disorder or Chaos Project Documentation File and Folder Structures Databases, Tables, Columns and Other Objects Scripts and Program Code Reports (Including Sub-Reports), Visualizations and Other Output Report Requests Training, Communication and Collateral Standards for Deployment Encryption and Security

9 Standards for Complex Logic Break Into Smaller Pieces to Understand Keep Out of Reports

10 Standards for Naming Objects Ensure no collisions with vendor objects. dw_ Naming conveys information tmp_, BAK_ Standardize column names and rename vendor column names where appropriate.

11 Standards for Code Comment Blocks Throughout PRINT Outputs as Part of Job Process Test Cases Imbedded Within Code Simple Elements such as Bracketing Calculations Code is Lined up for Readability Punctuation Standards Use of Temporary Tables Agreement on Naming ix_ Index, f_ Function, pr_ Procedure

12 Standards for Missing Data Why use other values than nulls? Where will missing data be consumed? When to change nulls to zeros?

13 Standards for Testing

14 Class Exercise on Reporting What s wrong with this picture?

15 Standards are a Great First Step Are becoming more complex. Web deployment of reporting, data cubes, pivot tables Should be set at the beginning of a data warehouse project. Are critical to keeping your data warehouse organized. Are critical to the productivity of technical staff. Need to address new technologies such as data visualizations, mobile Comments or questions on standards?

16 Integration of a Data Dictionary

17 Design Goals Keep Track of Scripts and Objects Created by Scripts Error Reports if Objects are Not in Dictionary Run Times to Monitor Performance Row Counts and Space Used Report Catalog and Report Usage 100% of Objects Documented as they are Created Easily Done and Extensible Create Compiled Help Files for End Users Documentation in Word for Main Tables

18 Dictionary Documentation Scripts

19 Examples of Reports from Dictionary The following report examples come from the dictionary tables and database system tables. All reports are deployed in Reporting Services and have multiple prompts to increase flexibility.

20 Scripts and Objects they Create

21 Run Times for Each Script

22 Row Counts and Space Used

23 Jobs Run by User Across Fiscal Year

24 Counts of Objects

25 Report Catalog

26 Report Usage Counts

27 Report Performance

28 Extended Properties Document all objects in database. Generate documentation with third party tool.

29 Extended Properties

30 Extended Properties

31 The Order of All Things

32 Design Goals What do you create first? How do you decide the sequence? What do you create last? How do you make the sequence easy to change? How do you experiment with order? How do you integrate the order of scripts into a job scheduler?

33 Class Exercise on Order Choose a Logical Order and Naming Convention for Various Categories of Scripts

34 Got functions?

35 Design Goals Repeatable Code (Code Once - Use in Many Places) Saves Time (Logic in One Place Only Have to Think Once) Improves Flexibility (Change in One Place Everything Changes) Save Joins in Queries (Use for Table Lookups) You Need to Monitor Performance of Functions

36 Got functions? Fiscal Dates YYYY/YYYY, Fiscal Quarter, Fiscal Month. Sorted in Order, Substring(4, for Formatting

37 Got functions? f_dw_pyramid_amount_desc Usage, gifts, cumulative giving, proposal amounts, donor counts

38 Got functions? f_dw_code_tables_longdescription Returns descriptions for code table entries. Reduces coding.

39 Got functions? f_dw_gifts_days_since_date_recd_lapsed_desc Returns descriptions for lapsed donors.

40 Got functions? f_dw_pledge_paid_amount Returns pledge paid amount based on installments. Always check transactional system code before creating.

41 Got functions? f_dw_gifts_constant_dollars (More exotic.) Converts gifts to constant dollars based on consumer price index.

42 Got functions? More Examples Maximum and Minimum Amounts Aging of Amounts into Columns CFY, PFY, CFYTD, PFYTD Fund Descriptions Text Descriptions (Into 1 field Concatenation) Pledge Aging Overdue Description (Current, Past Due, Etc.) Your examples?

43 Thousands of Parameters

44 Design Goals Extreme Report Flexibility (Thousands of Views) Extensibility (Add Parameters Easily) Policy (One for Every Description in DW Tables) Naming Conventions (Same for All) Performance Considerations (Materialized Views) Report Tables vs. Cubes

45 Flexible Reporting

46 Stored Procedure Creates Parameters

47 Show What s Been Picked on Reports

48 Data Cleanup and Maintenance That Works

49 Old Style Data Cleanup

50 Old Style Data Cleanup Labor Intensive to Run One Report at a Time Not Easy to Repeat Logic Imbedded in Multiple Reports Hard to Maintain Not Organized by Department Not Grouped by User Not Easily Grouped by Subject, Table or Screen Hard to Organize into Specific Projects

51 Design Goals Automate Data Cleanup Reporting Encapsulate Complex Business Rules Provide Estimation of Data Cleanup Efforts Create Training Tools for End Users Ability to Utilize Reporting as Projects Monitor Over the Long Term Easily Extensible with a Minimum Effort Future Data Mining Tools to Predict?

52 New Approach to Data Cleanup Data Cleanup Script for Every Table Scripts Contain Business Rules Includes Priority and Time Estimate for Cleanup

53 End User Training Tools

54 Errors by Area of Focus

55 Track Error Trends Over Time

56 Odds and Sods

57 Odds and Sods Fiscal Dates Table Set of Test IDs Slowly Changing Dimensions and Snapshots The Kill File Monitoring Query Usage in Application Duplicate Scoring Data Quality Scoring Your examples?

58 A Few Resources

59 acmqueue

60 CASE Books on Advancement Services

61 CoolData Blog

62 Data Warehousing Institute

63 FlowingData

64 Good Is

65 Listservs

66 Many Eyes

67 PASS

68 Periodic Table of Visualization Methods

69 SupportingAdvancement.Com

70 Tableau Software

71 Other Vendors Substantial resources, white papers, newsletters, free webinars and much more. Check for local chapters in your area.

72 Questions?