1 25/07/2017 Big-Data- and Data-Science-Day 2017

Size: px

Start display at page:

Download "1 25/07/2017 Big-Data- and Data-Science-Day 2017"

Vivien Hart
6 years ago
Views:

1 1 25/07/2017

How to enable smooth Business on Big Data

2 How to enable smooth Business on Big Data considering Governance - Hochschule der Medien Stuttgart - Ralf Leipner - Principal Consultant - BASLE BERN BRUGG DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. GENEVA HAMBURG COPENHAGEN LAUSANNE MUNICH STUTTGART VIENNA ZURICH

3 Content 1. Trivadis 2. Preface 3. Challenges 4. Status quo 5. Solution proposal 6. Proof of concept 3 25/07/2017

4 Trivadis 4 25/07/2017

Die Trivadis kurz erklärt Trivadis ist führend bei der IT-Beratung, der Systemintegration, dem Solution Engineering und der Erbringung von IT-Services mit Fokussierung auf, und -Technologien in der

5 Die Trivadis kurz erklärt Trivadis ist führend bei der IT-Beratung, der Systemintegration, dem Solution Engineering und der Erbringung von IT-Services mit Fokussierung auf, und -Technologien in der Schweiz, Deutschland, Österreich und Dänemark. Trivadis erbringt ihre Leistungen in folgenden strategischen Geschäftsfeldern: Trivadis Betrieb, Support, Wartung Trivadis Services Trivadis Services stellt zentrale Organisation und Infrastruktur Service-Level Management zur Verfügung. 5 25/07/2017

Mit über 600 IT- und Fachexperten bei Ihnen vor Ort. KOPENHAGEN 14 Trivadis Niederlassungen mit über 600 Mitarbeitenden. HAMBURG Über 200 Service Level Agreements. Mehr als 4'000 Trainingsteilnehmer.

6 Mit über 600 IT- und Fachexperten bei Ihnen vor Ort. KOPENHAGEN 14 Trivadis Niederlassungen mit über 600 Mitarbeitenden. HAMBURG Über 200 Service Level Agreements. Mehr als 4'000 Trainingsteilnehmer. DÜSSELDORF FRANKFURT STUTTGART Forschungs- und Entwicklungsbudget: CHF 5.0 Mio. Finanziell unabhängig und nachhaltig profitabel. Erfahrung aus mehr als 1'900 Projekten pro Jahr bei über 800 Kunden. BASEL FREIBURG BRUGG ZÜRICH MÜNCHEN WIEN BERN GENF LAUSANNE 6 25/07/2017

Trivadis Unsere Mission. Trivadis makes IT easier: Wir unterstützen unsere Kunden massgeblich bei der intelligenten Nutzung von Daten im digitalen Zeitalter.

7 Trivadis Unsere Mission. Trivadis makes IT easier: Wir unterstützen unsere Kunden massgeblich bei der intelligenten Nutzung von Daten im digitalen Zeitalter. Wir reduzieren Komplexität für unsere Kunden durch herausragende Technologiekompetenz. Wir übernehmen Kernaufgaben der bestehenden und zukünftigen IT unserer Kunden. 7 25/07/2017

8 Trivadis makes IT easier. 8 25/07/2017

9 Traditional Business Digital Business Traditional Business Solutions Modernization Big Data & Data Science IoT EVERYTHING AS A SERVICE CLOUD COMPUTING 9 25/07/2017

10 Customers A choice within the previous months IoT / SmartMeter Smart Data Architecture Data Science Orientation Governance Training Technology Evaluation Infrastructure MeineStadt 10 25/07/2017

11 Preface 11 25/07/2017

12 Abstract Requirement & stakeholder analysis Logical partitioning of the cluster Standardization of the information repository within the partitions Metadata based resource- & role management Approval procedures and rules Setup infrastructure & implementation pilot RBAC 12 25/07/2017

13 Mandate example 30 Big Data Developers mostly internal 20 DWH / BI Developers external / internal HDP, Kafka, Kerberos, Apache Ranger, Teradata, MicroStrategy, SAS, PowerCenter, SailPoint Services & Applications no Security on Big Data simple Governance on Big Data Stakeholders Governing Board Legal responsible Security officer Department Manager Innovation Development teams 13 25/07/2017

14 Implementing Role-Based Access Control Authentication Authorization Processes Audit Data Protection Access roles Access rolesets Users User groups Resources Services Policies Access control Approvals Stakeholders Logging Exceptions Alarming Classification Identifier Masking Anonymization Encryption Kerberos Apache Knox Apache Ranger Apache Ranger SailPoint Apache KMS HDFS TDE 14 25/07/2017

15 Challenges 15 25/07/2017

16 Enabling smooth business protection building blocks DevOps privacy Security Architecture Business Development Legal principles DSG Flexibility Regulation T&C 16 25/07/2017

17 Requirements & concerns Priorities today tomorrow Business Creation of new innovative business models Information monetization Legal Conforming to the law Transparency of the approval processes Security Authentication, authorization & data protection Security principles fulfilled Development Flexible environment supporting agile development Simple administration of data & access Architecture Efficient management of enterprise information Readiness for hybrid solution approaches insufficient satisfying good coverage 17 25/07/2017

18 Status quo 18 25/07/2017

Applications & services on Big Data (#33) 6 4 5 18 Applications

19 Applications & services on Big Data (#33) Applications active Applications inactive Services active Services inactive 19 25/07/2017

20 Big Data Platform Man power 30 developer thereof 6 integration developers 2/3 within IT 1/3 within Business organized in Squads No standardized naming convention Frameworks not standardized on purpose Data is mostly organized by integration technology 20 25/07/2017

# of applications or service per team 7 6 5 4 3

21 # of applications or service per team /07/2017

# of users per application or service 10 9 8 7 6 5 4 3 2 1 0 9 8 7 7 6 5 4 4 3 3 3 3 2 2 2 2 2 2 2 2 2 2 1 1 Application 1 Application 2 Application 3 Application 4 Application 5 Application 6

22 # of users per application or service Application 1 Application 2 Application 3 Application 4 Application 5 Application 6 Application 7 Application 8 Application 9 Application 10 Application 11 Application 12 Application 13 Application 14 Application 15 Application 16 Application 17 Application 18 Application 19 Application 20 Application 21 Application 22 Application 23 Application /07/2017

23 Relation of needed application access # of users # of application access 23 25/07/2017

# of subject per application or service 18 16 14 12 10 15 16 8 6 8 7

24 # of subject per application or service C... Co E.. G H I M N O P R S Z /07/2017

# of data sources per application or service 25 25/07/2017 8 3 5 1 1

25 # of data sources per application or service 25 25/07/ C... Co E.. G H I M N O P R S Z..

26 Solution Proposal 26 25/07/2017

27 Kerberos Version 5 Authentication method originated 1980 at the Massachusetts Institute of Technology Athena project different operating systems Single-sign-on Parties involved Service User Service Provider Authentication Service Ticket Granting Service 27 25/07/2017

Service Ticket Granting Service I m user

(TGT) 1 Here is my TGT, give me a service

decrypt this response with your password

Service session initiated Service User 6 5

28 Kerberos authentication steps Key Distribution Centre (KDC) Authentication Service Ticket Granting Service I m user Sam and I need a ticket to get tickets (TGT) 1 Here is my TGT, give me a service ticket 3 Here is your TGT, if you can decrypt this response with your password hash 2 Here is your service ticket 4 Service session initiated Service User 6 5 Service Provider Here is my service ticket, authenticate me 28 25/07/2017

29 Context of security object models Hdfs Kafka Repository* Resource Type* Directory Topic Resource Resource Policy Bundle RANGER Resource Policy Resource Category Audit Event Trail Permission Type* Function Information Policy Role type User Permission Apache Ranger Company specific User Group Key Store Approvals Area bundling Group Role 29 25/07/2017

30 Data areas overview Service & application area is dedicated for application data & development. Each applications has his own sector. Discovery area common presentation and easy access layer that allows to gain new insights for business Raw data contains the complete datasets of the source systems. Information is grouped by concept & source system Innovation & experimental area allows to identify new business models and assess integration value of new data sources /07/2017

31 Data areas characteristics Service & application area purpose: dedicated business structure: dedicated subject integration effort: medium data maturity: medium, high Discovery area purpose: general business structure: domain, subject integration effort: medium data maturity: high Raw data purpose: restricted data reservoir structure: domain, source integration effort: low data maturity: low, medium Innovation & experimental area purpose: internal research structure: depends integration effort: different data maturity: variable 31 25/07/2017

32 Data flows & cross area access Service & application area Discovery area Raw data Innovation & experimental area data flow data link 32 25/07/2017

33 Data areas structure Service & application area WO RW RO create publish consume application prod subject 1 subject 2 subject dev Discovery area WO RW RO create publish consume domain subject anonymised disguised aggregated confidential Raw data WO RW RO create publish consume origin domain domain domain Innovation area WO RW RO create publish consume sandboxes * * users sam 33 25/07/2017

34 Illustration Raw data area naming convention Ranger resource raw.netalyzer.prod.network.events.<*> Role #5 Area #4 Origin #22 Environment #4 Domain #16 Subject #74 Placeholder developer service netalazer prod network events <*> integrator raw data Dwh dev customer basic-configuration urban-planer discovery om test marketing contract network-engineer innovation crm stage location-data segment raw_integrator Ranger group 34 25/07/2017

35 Illustration Service area naming convention Ranger resource service.bupper.dev.network.events.<*> Role #5 Area #4 Service #22 Environment #4 Domain #16 Subject #74 Placeholder developer service bupper prod network events <*> data-scientist raw data urbanizer dev customer basic-configuration urban-planer discovery claimdetector test marketing contract network-engineer innovation city stage location-data segment bupper_developer Ranger group 35 25/07/2017

36 Proof of concept 36 25/07/2017

37 Ranger Architecture Source: Hortonworks Documentation 37 25/07/2017

38 Use Cases 1. Configuration of roles & users in Apache Ranger a. define role b. define user c. assign user to role 2. Creation of resources a. Apache Kafka topics b. HDFS directory nested c. HDFS files in nested directories 3. Define policies a. for Kafka b. for HDFS 4. Evaluation of Kafka access policies a. consume Kafka topic by authorized user / role (no kerberos ticket) b. consume Kafka topic by authorized user / role c. consume Kafka topic by unauthorized user / role d. produce Kafka topic by authorized user / role 38 25/07/2017

39 Use Cases 5. Evaluation of HDFS access policies a. access directory by authorized user / role b. access directory by unauthorized user / role c. read file by authorized user / role d. read file by authorized user / role (no kerberos ticket) e. read file by unauthorized user / role 6. Audit Log a. show access denied b. show access allowed 7. Ranger REST API a. get policy b. create policy 39 25/07/2017

Ralf Leipner Principal Consultant Architecture Management trivadis AG Weltpoststrasse 5 3015 Bern Switzerland +41 58 459 54 24 +41 79 669 38 47 ralf.

40 Ralf Leipner Principal Consultant Architecture Management trivadis AG Weltpoststrasse Bern Switzerland ralf.leipner@trivadis.com /07/2017

Partitionierungsstrategien für Data Vault. Dani Schnider, Trivadis AG DOAG Konferenz, 23. November 2017

Partitionierungsstrategien für Data Vault Dani Schnider, Trivadis AG DOAG Konferenz, 23. November 2017 @dani_schnider DOAG2017 Unser Unternehmen. Trivadis ist führend bei der IT-Beratung, der Systemintegration,