Clarion Live Presentation. Employing SQL To Improve Data Quality October 23, 2015

Size: px

Start display at page:

Download "Clarion Live Presentation. Employing SQL To Improve Data Quality October 23, 2015"

Alvin Ferguson
6 years ago
Views:

1 WHITEMARSH DATA MANAGEMENT SERIES Clarion Live Presentation Employing SQL To Improve Data Quality October 23,

TOPICS Problem to be Solved Initial Scans of the 88 pages Construction of a Single File Building a Collection of Word Perfect Tables Building the Single word transactions

2 TOPICS Problem to be Solved Initial Scans of the 88 pages Construction of a Single File Building a Collection of Word Perfect Tables Building the Single word transactions Constructing the Source WP Tables Building the CSV and Loading into SQL Tables Employing the Transaction Building Screens within Midas Using SQL to Harmonize the data values 2

3 PROBLEM TO BE SOLVED Take 88 pages of 3-up Name and Address labels and convert them to data transactions Match transactions with existing voters in the Political Management System, Midas Allocated match voters with the list category, Senior Citizen Center members. Create the Senior Citizen Center voter list for the upcoming Bowie City election. 3

4 SCANS OF 88 PAGES OF 3-UP NAMES AND ADDRESSES 88 pages, 30 Per page total 4

5 CONSTRUCTION OF A SINGLE FILE Scans were saved as in Word to produce tables Spaces were stripped off the tables Created a single column of three-line entries Result was 88 pages of three-line entries Joined all 88 pages via Bulk File Merger Pro Bulk File Merger software Essex Software, LLC Result file had 10,400 or so lines. 5

6 BUILDING THE SINGLE WORD TRANSACTIONS Issues: Commas Periods Variable word quantity transactions 6

7 CONSTRUCTING THE SOURCE WP TABLES Converted to tables Deleted commas and periods Transactions ranged from 9 14 columns of data 7

8 BUILDING THE CSV 8

9 LOADED INTO A TPS BASED TABLE CSV Source TPS Table Target 9

10 INITIAL TPS-BASED DATA Benefits of Starting with TPS: Discrete files that are easy to backup (i.e., make copies) Easy to restore to earlier versions incase of Oh Feces! 10

11 ON THE WAY REFORMATTING TPS DATA 11

12 MORE ON DATA BY COLUMN MANIPULATION Specially created Procedure Routines: Resolved Three cases for House # and Street Name Subsequent joining of Columns 3&4, and 4&5 Final set of Generalized Joining and moving 12

13 DATA CLEANSING CLARION PROCEDURE RECORD PROCESSING 13

14 MORE PROCEDURE ROUTINES 14

15 FIXING/ADJUSTING INDIVIDUAL RECORDS 15

16 FIXING INDIVIDUAL RECORDS 16

17 FINAL REFORMATTING TPS DATA 17

18 DATA FORMAT FIXED. DATA QUALITY Col 05 Data Quality Issues: Rd, Road PL, Place St, Street Pky, Parkway Col 6 Issues # Apt <actual number> Col 8 issues MD Md Maryland 18

19 LOADED INTO A TPS BASED TABLE TPS Source SQL Table Target 19

20 USING SQL TO HARMONIZE THE DATA VALUES Select Count(distinct Col05) as count, Col05 as StreetType from dbo.voternameaddressloadtransactions group by Col05 order by Col05; 20

21 USING SQL TO HARMONIZE THE DATA VALUES Select Count(Col05) as count, Col05 as StreetType from dbo.voternameaddressloadtransactions group by Col05 order by Col05; 21

22 USING SQL TO HARMONIZE THE DATA VALUES Select Count(distinct Col06) as count, Col06 as Apt# from dbo.voternameaddressloadtransactions group by Col06 order by Col06; 22

23 USING SQL TO HARMONIZE THE DATA VALUES Select Count(Col06), Col06 from dbo.voternameaddressloadtransactions group by Col06 order by Col06; 23

24 USING SQL TO HARMONIZE THE DATA VALUES Select Col08 as State, Count(Distinct Col08) as Count from dbo.voternameaddressloadtransactions group by Col08 order by Col08; 24

25 USING SQL TO HARMONIZE THE DATA VALUES Select Col08 as State, Count(Col08) as Count from dbo.voternameaddressloadtransactions group by Col08 order by Col08; 25

26 USING SQL TO HARMONIZE THE DATA VALUES Select Count(Distinct Concat(col07, Col08, col09)) as Count, Concat(col07, Col08, col09) CityStateZip from dbo.voternameaddressloadtransactions group by Concat(col07, Col08, col09) order by Concat(col07, Col08, col09); 26

27 USING SQL TO HARMONIZE THE DATA VALUES Select Count(Concat(col07, Col08, col09)) as Count, Concat(col07, Col08, col09) as CityStateZip from dbo.voternameaddressloadtransactions group by Concat(col07, Col08, col09) order by Concat(col07, Col08, col09); 27

28 FIXING INDIVIDUAL RECORDS 28

29 BENEFITS FROM SQL Set processing of collections of records Special operators that allow for counting across sets Special operators that discover distinct (a.k.a., unique) values Special operators that allow for concatenation of multiple columns of value for production of sets. Discovery of same needed value but different formats (e.g., #, A-, Apt, Unit) Discovery of same needed value but different value precisions (e.g., vs ) 29

30 COMPARING CLARION-BASED ACCESS VS STRAIGHT SQL ACCESS Clarion-based TPS Access Individual record discovery Created procedure routine Individual record fixing (procedure routines). Groups of records fixing (record by records adjusting values based on existing values) through procedure routines Straight SQL-based Access Set of records discovery Not practically possible Only surgical changing of values column by column Not practically possible 30

31 SUMMARY AND WAY FORWARD Nearly neither Clarion & TPS or Straight SQL is THE answer Initial acquisition through CSV is the way in the door Conversion from CSV to TPS enables more sophisticated data value adjustments (e.g, table lookup changes, moving data, etc.) Conversion from TPS to SQL enables application of specialized operators such as Counts, Distinct, and Concatenation that enable even more sophisticated value adjustments (e.g., specific row identification and update. Decision of only ONE choice is not the right approach. Moving data from one access method to another is essential. Include all three (cvs, tps, and sql) are key for the road to data quality 31

32 QUESTIONS FOR THE GREAT GRAND PA? 32

Achieving Data Standardization

Achieving Data Standardization Whitemarsh Information Systems Corporation 2008 Althea Lane Bowie, Maryland 20716 Tele: 301-249-1142 Email: mmgorman@wiscorp.com Web: www.wiscorp.com 1.0 The Issue If database