GOVERNING HADOOP (AND THE DATA LAKE)

Transcription

1 GOVERNING HADOOP (AND THE DATA LAKE) DAMA-RMC Discussion Lowell W. Fryman, CBIP-CDMP Practice Principle April 20, Collibra Inc

2 DAMA-RMC Discussion Agenda Do we need to Govern Hadoop Data Lakes Why, where is the Value & Risk What will be our Governance processes Where do I start What Communications & Education When are we done 2 Collibra 2017

3 Why do we build an Hadoop architecture Seems we come down to 3 reasons 1. IT needs to implement less costly infrastructure Hadoop is cheap 2. Variety of data exceeds capacity of our resource capabilities (Social media, website, other external data, internal application variety) 3. We can integrate that additional data Capturing the data is not really the issue, Using the data is the issue, So who is our audience, Where is the Risk and Value proposition It is with our Consumers of the data Should we pay the Data Scientist to do Governance and data wrangling 3 Collibra 2017

4 The Approach Bring all data in is the basic mantra of a Data Lake The challenge is that your Data Lake will slowly turn into a Data swamp if data governance is not in place In broad terms, data lakes are marketed as enterprise-wide data management platforms for analyzing disparate sources of data in its native format. The idea is simple: instead of placing data in a purpose-built data store, you move it into a data lake in its original format. This eliminates the upfront costs of data ingestion, like transformation. Once data is placed into the lake, it's available for analysis by everyone in the organization. Nick Heudecker, research director at Gartner 4 Collibra 2017

5 What we want to prevent Unsecured Data Lake - no clear roles - no access controls - will become a dump yard to fulfill individual needs of the users - multiple copies of the same data - Users won t easily be able to find trustable data 5 Collibra 2017

6 Data Scientists Can t Do Their Jobs Accessing data is difficult and time-consuming Time to impact is slow all the wrangling to do Identifying ownership across on-premises, cloud, and external solutions is challenging Fostering collaboration and reuse is challenging Multiple data sources are in use; some are outdated Many variants and versions of the same data exist The same Consumer questions exist Where is the data integration! 6 Collibra 2017

7 They Spend Most of Their Time on the Least Enjoyable Tasks Data scientists spend 60% of their time cleaning and organizing data But 57% of data scientists view this as the least enjoyable part of their job Source: CrowdFlower via Forbes 7 Collibra 2016

8 From TDWI 8 Collibra 2017

9 Organizations need to reduce the time to find data they can trust BI analytics & Big Data is the top priority for 78% of CFOs. Their needs are: more accurate reports save time and money need for company big picture reporting Create an effective environment to share relevant information Despite spending $2.6bn per year, CFOs report that most BI analytics projects are: too tactical focus too much on technology do not address the fundamental issue of data quality and consistency Governance reduces the time spent searching for data by as much as 40% Data Lineage and Quality measures are required to enhance the TRUST 9 Collibra 2017

10 What we want Secured Data Lake - Marked data (data categories) - Classification, Taxonomy - Controls and measures based as well as lineage - Scorecards indicating quality, controls & reliability - Accountability and responsibilities 10 Collibra 2017

11 So how do we Govern the Data Lake as needed Identify critical data elements (CDE) Ensure we have Business Terms for the CDE Maintain the Business Glossary & Catalog for the CDE What does this mean, values & rules, security, constraints for usage Communications between Owner and Consumer are necessary to support governance as you need it Data Integration is still necessary Maintain a Data Dictionary of the data sets in the Lake What is available and from which data set What is the integration & Lineage What is the data quality and controls 11 Collibra 2017

12 Architecture with Governance Raw Data Lake Integrated Clean Lake Liberated Self Service Governance As you need it Foundation Business Terms (Stewardship, Rules, Security, Authoritative Source, Data Quality) Lineage & Traceability (data flow, transparency) Integration (Technical & Business metadata) Data Catalog (Assets for Governance, Data, Reporting) Data Dictionary (technical metadata alignment) Access Controls and knowledge 12 Collibra 2017

13 Collibra Community Discussion Summary Govern the critical data needed by Data Scientists May require a different DG approach(as you need, know of it) Define critical Business Terms by the Variety of the data sourced into the Lake Technology dependent data integration techniques exist and integration is important for the Consumer Ask the Data Consumers (Scientists) what are you looking for Create a clean Lake for Consumers/Scientists Data Governance is a program, never done Stay calm & allow your data to prosper 13 Collibra 2017

14 QUESTIONS?

15 THANK YOU 2017 Collibra Inc