Effective Data Engineering for Data Science (Analytics)

Transcription

1 Effective Data Engineering for Data Science (Analytics) Prashanth H Southekal, PhD Managing Principal DBP-Institute Data for Business Performance (DBP) 1

2 Before We Start Data for Business Performance (DBP) 2

3 Purpose of the Talk The talk is mainly on 3 key topics from my book Data for Business Performance. 1. The first part of the talk is on defining and understanding why Data Engineering is essential for Data Science. 2. The second part of the presentation will list out the key factors for effective Data Engineering. 3. The third part of the presentation will talk about a Data Engineering case study focus in on Data Standards. Data for Business Performance (DBP) 3

4 Agenda 1 Data Engineering and Data Science 2 Key factors for Effective Data Engineering 3 Data Engineering Case Study Data for Business Performance (DBP) 4

5 Business Data Definition Reference Data Master Data Metadata Transactional Data Business Data Technical Data Data for Business Performance (DBP) 5

6 Example of Integrated Business Data Transactional Data Reference Data Master Data Reference Data Master Data Reference Data Transactional Data Data for Business Performance (DBP) 6

7 Data Engineering and Data Science Data Engineering is preparing the data infrastructure i.e. designing, building, and integrating data from various resources to be analyzed by Data Scientists. Data Science is the processes where Data Scientists use data to answer questions and derive insights (Descriptive and Inferential) from the data supplied by Data Engineers Create Update Delete Read Data for Business Performance (DBP) 7

8 Note 1: Dominance in the Data Lifecycle (DLC) Data Security Origination Capture Validation Processing Distribution Aggregation Interpretation Consumption Data Storage Data Engineering Data Science 8 of the 10 stages in the DLC pertain to Data Engineering Data for Business Performance (DBP) 8

9 Note 2: What are Practitioners Saying? Deriving Insights Takes Effort Data for Business Performance (DBP) 9

10 Note 3: What do the Experts/Researchers say? The most important analytics skills are cleaning and organizing large data sets Andrew McAfee and Erik Brynjolfsson, MIT. Data is super messy, and data cleanup will always be literally 80 percent of the work. DJ Patil, U.S. Chief Data Scientist at White House Office of Science and Technology Policy Multiple data sources which are the ingredients for wellrounded insight need to be blended correctly for a comprehensive solution Anjali Lai, Forrester Data for Business Performance (DBP) 10

11 Agenda 1 Data Engineering and Data Science 2 Key factors for Effective Data Engineering 3 Data Engineering Case Study Data for Business Performance (DBP) 11

12 Key factors for Effective Data Engineering 1 Manage Core business processes in the SoR 2 Treat Business data as an Enterprise asset Enable Data Integration using Standards 3 4 Manage Reference and Master data with Standards Position Data Governance as a Business Function

13 12 Key Data Standards in Oil and Gas # Data Standard Oil and Gas Domain Key Data Elements 1 Energistics Upstream Exploration and Production 2 IOGP All EHS 3 MIMOSA All Manufacturing, Fleet and Facility 4 OGC Upstream Geospatial 5 OPC All Production 6 POSC Upstream Exploration and Production 7 PIDX All Business Transactions 8 UNSPSC All Commodities 9 PODS Midstream Pipelines 10 PPDM Upstream Well 11 SEG Upstream Exploration and Production 12 ISO 14224:2006 All Equipment All = Upstream + Midstream + Downstream Data for Business Performance (DBP) 13

14 Why Data Standards? Data standards give businesses confidence on the goods and services produced. Data Standards facilitate Interoperatablity and Portability across IT systems Data Standards are core building blocks for decision making Functionally Safety Reliability and Cost Effectiveness Data for Business Performance (DBP) 14

15 Agenda 1 Data Engineering and Data Science 2 Key factors for Effective Data Engineering 3 Data Engineering Case Study Data for Business Performance (DBP) 15

16 Mapping Problem to the Solution Problem Business Impact Data Standard Solution Design Duplicate Material Dysfunctional UNSPSC 4 Levels with Categories Category Segment, Family, Management Class, and Commodity Duplicate & Increased cost and PIDX Noun-Modifiers(s) Inconsistent Poor Spend Analytics format defined by Naming of Materials PIDX Data for Business Performance (DBP) 16

17 17 Data Standards Design UNSPSC PIDX Attributes (Attributes 1 to 12) Internal Attributes (Attributes 12 to 15) Segment: Fuel & Fuel Additives & Lubricants & Anti-corrosive Materials Family: Fuels Class: Petroleum & Distillates Commodity: Gasoline Noun: Gasoline Modifier: Regular Attribute 1: TYPE Attribute 2: FORM Attribute 3: OCTANE Attribute 4: GRADE Attribute 5: ADDITIVE Attribute 6: SULFUR PERCENTAGE Attribute 7: SULFUR CONTENT Attribute 8: MNEMONICS Attribute 9: CONTAINER TYPE Attribute 10: CONTAINER SIZE Attribute 11: TRADE NAME Attribute 12: SAFETY DETAILS Attribute 13: UNSPSC CODE Attribute 14: PLANTS Attribute 15: QA RELEVANT

18 Summary Business Data = Reference Data + Master Data + Transactional Data; Using data to answer questions is analytics; Good Analytics (Data Science) <- Quality Data (Data Engineering) <- Data Governance <- Data Standards Data governance and Data quality should be seen as continuous improvement initiative; Data for Business Performance is a journey and NOT a destination. Finally, there is no endeavour without a customer. Data for Business Performance (DBP) 18

19 Foreword: Dr Tom C Redman the data doctor I strongly believe this book will help to bridge the gaps among the data world and the business needs. Mario Faria Managing Vice President Gartner This book brings great value to Industry professionals. Brandon Rohrer Sr. Data Scientist Facebook 50% of the book sales today goes to PPDM Data for Business Performance (DBP) 19

20 We have no time to do it right; but we have all the time to do it again! John Wooden American basketball coach Connect with me in LinkedIn Prashanth Southekal Data for Business Performance (DBP) 20