Big Data Security. Facing the challenge

Size: px

Start display at page:

Download "Big Data Security. Facing the challenge"

Aubrey Parrish
6 years ago
Views:

1 Big Data Security Facing the challenge

2 Experience the presentation xlic.es/v/e98605

3 About me Father of a 5 year old child Technical leader in Architecture and Security team at Stratio Sailing skipper 3

4 In your opinion, how difficult is it to manage security in your projects? Very difficult Difficult Easy Very Easy What is security? 4

5 PROJECTS FOR EVER ONGOING IN BIG COMPANIES HUNDRED OF MILLIONS OF EUROS SPENT DURING THE YEARS IN GLOBAL IT CROSS INITIATIVES CRM Towers Watson In a monolithic application centric it with data silos these initiatives never get accomplished Earnix (Pricing) SAS Oracle Mainframe WebFocus DATA GOVERNANCE LOGS CENTRALIZATION MONITORING Data Warehouse ERP Lab H0 (Plataforma Big Data compartida por el grupo) DATA SECURITY SECURITY AUDIT

6 PROJECTS FOR EVER ONGOING IN BIG COMPANIES 1 4 DATA GOVERNANCE LOGS CENTRALIZATION 5 MONITORING 2 3 DATA SECURITY AUDIT

7 PROJECTS FOR EVER ONGOING IN BIG COMPANIES 1 4 DATA GOVERNANCE LOGS CENTRALIZATION 5 MONITORING 2 3 DATA SECURITY AUDIT

8 PROJECTS FOR EVER ONGOING IN BIG COMPANIES 1 4 DATA GOVERNANCE LOGS CENTRALIZATION 5 MONITORING 2 3 ETL DATA SECURITY AUDIT

9 GALGO CHASING ELECTRONIC RABBIT COMPANIES ALWAYS TRY TO GET THE RABBIT DATA GOVERNANCE LOGS DATA CENTRALIZATION MONITORING SECURITY SECURITY AUDIT In an application centric company with data silos you never will be able to achieve successfully those projects

10 STRUCTURAL INITIATIVES ARE SOLVED COMPLETELY WITH DATA CENTRIC Functionalities Implemented in the product DaaS (data as a service) Data Intelligence DATA GOVERNANCE LOGS CENTRALIZATION MONITORING Data DATA SECURITY SECURITY AUDIT

11 RABBIT IN A JAIL MINIMUM EFFORT AND COST TO GET THE RABBIT

12 Facing the challenge 12

13 SECURITY IN A DATA CENTRIC Protect the data Protect the service Perimeter security to access the cluster. Support identity management and authentication to prove that a user/service is who claims to be. In a multi-data store platform ACLs should be centralized to simplified the correct authorization to different data stores. Audit events must be centralized to control misuse of the cluster in real time. Data integrity and confidentiality in network communications to protect data on the fly. Perimeter security to access the cluster. Support identity management and authentication to prove that a user/service is who claims to be. A user/service should be authorized so more resources than expected are not used. A user/service should not interfere with other users/services when it is not needed. To control the use of resources, it should be audited. 13

14 INFRAS Stratio DataCentric A P P S Standalone Applications Standalone Applications Apps with Microservices Docker Apps with Microservices Apps Apps Docker Docker Docker DaaS Microservices Docker Data Intelligence as a Service Microservices Docker SQL VAULT STRATIO EOS (Enterprise Operating System) Kafka Zookeeper DATA CENTER OPERATING SYSTEM MESOS SERVICE ORCHESTATION MARATHON CONSUL DOCKER SERVICE DISCOVERY CONTAINERS TERRAFORM NODE PROVISIONING CALICO NETWORK ISOLATION BAREMETAL PRIVATE CLOUD PUBLIC CLOUD

15 INFRAS Stratio DataCentric A P P S Standalone Applications Standalone Applications Apps with Microservices Docker Apps with Microservices Apps Apps Docker Docker Docker DaaS Microservices Docker Data Intelligence as a Service Microservices Docker SQL VAULT STRATIO EOS (Enterprise Operating System) Kafka Zookeeper DATA CENTER OPERATING SYSTEM MESOS SERVICE ORCHESTATION MARATHON CONSUL DOCKER SERVICE DISCOVERY CONTAINERS TERRAFORM NODE PROVISIONING CALICO NETWORK ISOLATION BAREMETAL PRIVATE CLOUD PUBLIC CLOUD

16 INFRAS Stratio DataCentric A P P S Standalone Applications Standalone Applications Apps with Microservices Docker Apps with Microservices Apps Apps Docker Docker Docker DaaS Microservices Docker Data Intelligence as a Service Microservices Docker SQL VAULT STRATIO EOS (Enterprise Operating System) Kafka Zookeeper DATA CENTER OPERATING SYSTEM MESOS SERVICE ORCHESTATION MARATHON CONSUL DOCKER SERVICE DISCOVERY CONTAINERS TERRAFORM NODE PROVISIONING CALICO NETWORK ISOLATION BAREMETAL PRIVATE CLOUD PUBLIC CLOUD

17 INFRAS Stratio DataCentric A P P S Standalone Applications Standalone Applications Apps with Microservices Docker Apps with Microservices Apps Apps Docker Docker Docker DaaS Microservices Docker Data Intelligence as a Service Microservices Docker SQL VAULT STRATIO EOS (Enterprise Operating System) Kafka Zookeeper DATA CENTER OPERATING SYSTEM MESOS SERVICE ORCHESTATION MARATHON CONSUL DOCKER SERVICE DISCOVERY CONTAINERS TERRAFORM NODE PROVISIONING CALICO NETWORK ISOLATION BAREMETAL PRIVATE CLOUD PUBLIC CLOUD

18 INFRAS Stratio DataCentric A P P S Standalone Applications Standalone Applications Apps with Microservices Docker Apps with Microservices Apps Apps Docker Docker Docker DaaS Microservices Docker Data Intelligence as a Service Microservices Docker SQL VAULT STRATIO EOS (Enterprise Operating System) Kafka Zookeeper DATA CENTER OPERATING SYSTEM MESOS SERVICE ORCHESTATION MARATHON CONSUL DOCKER SERVICE DISCOVERY CONTAINERS TERRAFORM NODE PROVISIONING CALICO NETWORK ISOLATION BAREMETAL PRIVATE CLOUD PUBLIC CLOUD

19 SECURITY OVERVIEW In order to guide the security priorities in the product roadmap, we are focused on helping to comply with LOPD within the platform. Every release of the Stratio platform, the security status is notified through: Results of the OWASP tests for the main components of the platform. Results of additional general purpose security tests defined to assure the quality expected. Security Risk Report that includes the known issues found. When Critical and High issues are found: We explain how can be mitigated. We plan to solve them during the next release. 19

20 PERIMETER SECURITY: NETWORKING Public Network Admin network Admin Router Admin network Admin Router Public Agents The default network configuration allows a zone-based network security design: Public. Admin. Private. Using Mesos roles to identify nodes ensures that only tasks specifically configured with this role will be executed outside the Private zone. Master Nodes Private network Private Agents Using Marathon labels, endpoints can be registered dynamically: Admin Router for the Admin zone. Marathon LB for the Public zone. 20

AUTHENTICATION, AUTHORIZATION AND AUDIT The solution is integrated with LDAP and Kerberos owned by the company where Stratio DCS is installed. Authentication: Web: OAuth2.

21 AUTHENTICATION, AUTHORIZATION AND AUDIT The solution is integrated with LDAP and Kerberos owned by the company where Stratio DCS is installed. Authentication: Web: OAuth2. Services & Data Stores: Kerberos or TLS-Mutual. Authorization: OAuth2 gosec Management: API Rest and website used to manage roles, profiles and ACLs. Also it shows users, groups and audit data. Audit: authentication and authorization events are structured and stored in a data bus (Kafka) to be computed and collected. 21

22 AUTHENTICATION, AUTHORIZATION AND AUDIT Plugins are lightweight programs running within processes of each cluster component. They are responsible for: Authorization (using gosec ACLs). Audit of every request sent to the component. Currently plugins have been developed for: Crossdata Sparta Kafka Zookeeper Elasticsearch HDFS 22

23 KEY MANAGEMENT SYSTEM It is a good practice to manage secretes by key management system instead of store them locally. For this purpose Stratio DCS uses HashiCorp Vault 23

24 KEY MANAGEMENT SYSTEM the secret of secrets Can applications obtain authentication tokens in a secure way? Where applications save vault s tokens? How are tokens protected? How will I know if someone steal tokens? First secret management Mesos Application Admin Marathon 24

25 KEY MANAGEMENT SYSTEM the secret of secrets Can applications obtain authentication tokens in a secure way? Where applications save vault s tokens? How are tokens protected? How will I know if someone steal tokens? First secret management one time secret Mesos Application Admin Marathon Run Application Env: one time secret 25

26 KEY MANAGEMENT SYSTEM the secret of secrets Can applications obtain authentication tokens in a secure way? Where applications save vault s tokens? How are tokens protected? How will I know if someone steal tokens? token < - > ACL First secret management one time secret login Mesos Application Admin Marathon Run Application Env: one time secret 26

27 KEY MANAGEMENT SYSTEM the secret of secrets Can applications obtain tokens in a secure way? Where applications save vault s tokens? How are tokens guarded? How will I know if someone steal tokens? First secret management one time secret Mesos Application Admin Marathon Run Application Env: one time secret 27

28 KEY MANAGEMENT SYSTEM the secret of secrets Can applications obtain tokens in a secure way? Where applications save vault s tokens? How are tokens guarded? How will I know if someone steal tokens? First secret management one time secret login Mesos Application Admin Marathon Run Application Env: one time secret 28

29 KEY MANAGEMENT SYSTEM the secret of secrets Can applications obtain tokens in a secure way? Where applications save vault s tokens? How are tokens guarded? How will I know if someone steal tokens? First secret management one time secret login Mesos Application Admin Marathon Run Application Env: one time secret 29

KEY MANAGEMENT SYSTEM the secret of secrets Can applications obtain tokens in a secure way? Where applications save vault s tokens? How are tokens guarded?

30 KEY MANAGEMENT SYSTEM the secret of secrets Can applications obtain tokens in a secure way? Where applications save vault s tokens? How are tokens guarded? How will I know if someone steal tokens? Logs Alert First secret management one time secret login Mesos Application Admin Marathon Run Application Env: one time secret 30

31 DATA PROCESSING ENGINE: SPARK Spark jobs need access to multiple data stores so that Spark needs to support the security of Stratio DCS. Spark 2.x compilation has been modified by Stratio in order to: Access secrets that are stored in the KMS. Allow access to Kerberized HDFS. Allow access to PostgreSQL with TLS authentication. Allow access to Elasticsearch TLS authentication. Allow access to Kafka with TLS authentication. 31

32 PROTECT THE DATA - use case - Perimeter security Authentication, Authorization, Audit Ciphered communications TABLEAU Admin PUBLIC NETWORK ADMIN NETWORK MARATHON-LB PRIVATE NETWORK ZOOKEEPER KMS ZOOKEEPER GOSEC MANAGEMENT GOSSEC SSO ADMIN ROUTER LDAP KERBEROS AUDIT KAFKA HDFS 32

33 PROTECT THE DATA - use case - Perimeter security Authentication, Authorization, Audit Ciphered communications TABLEAU Admin PUBLIC NETWORK ADMIN NETWORK MARATHON-LB PRIVATE NETWORK ZOOKEEPER KMS ZOOKEEPER GOSEC MANAGEMENT GOSSEC SSO ADMIN ROUTER LDAP KERBEROS AUDIT KAFKA HDFS 33

34 PROTECT THE DATA - use case - Perimeter security Authentication, Authorization, Audit Ciphered communications TABLEAU Admin PUBLIC NETWORK ADMIN NETWORK MARATHON-LB PRIVATE NETWORK ZOOKEEPER KMS ZOOKEEPER GOSEC MANAGEMENT GOSSEC SSO ADMIN ROUTER LDAP KERBEROS AUDIT KAFKA HDFS 34

35 PROTECT THE DATA - use case - Perimeter security Authentication, Authorization, Audit Ciphered communications TABLEAU Admin PUBLIC NETWORK ADMIN NETWORK MARATHON-LB PRIVATE NETWORK ZOOKEEPER KMS ZOOKEEPER GOSEC MANAGEMENT GOSSEC SSO ADMIN ROUTER LDAP KERBEROS AUDIT KAFKA HDFS 35

36 PROTECT THE DATA - use case - Perimeter security Authentication, Authorization, Audit Ciphered communications TABLEAU Admin PUBLIC NETWORK ADMIN NETWORK MARATHON-LB PRIVATE NETWORK ZOOKEEPER KMS ZOOKEEPER GOSEC MANAGEMENT GOSSEC SSO ADMIN ROUTER LDAP KERBEROS AUDIT KAFKA HDFS 36

37 PROTECT THE DATA - use case - Perimeter security Authentication, Authorization, Audit Ciphered communications TABLEAU Admin PUBLIC NETWORK ADMIN NETWORK MARATHON-LB PRIVATE NETWORK ZOOKEEPER KMS ZOOKEEPER GOSEC MANAGEMENT GOSSEC SSO ADMIN ROUTER LDAP KERBEROS AUDIT KAFKA HDFS 37

38 MULTI-TENANCY CAPABILITIES: RESOURCES ISOLATION Stratio DCS cluster resources (memory, disk, cpus and port ranges) are managed by Mesos. Mesos, Marathon and Metronome security can be activated post-installation in order to limit the use of the available resources for each framework. Once it is activated, admins will be able to: Reserve resources for a Mesos role. Grant permissions for each user/framework to do actions such as register frameworks, run tasks, reserve resources, create volumes, etc. Grant a minimum set of resources to a specific mesos role Mesos Cluster MASTER Marathon AGENT 1 role=slave_public AGENT 2 role=* AGENT 3 role=postgresql AGENT 4 role=* AGENT 5 role=* 38

39 MULTI-TENANCY CAPABILITIES: NETWORKS ISOLATION What about network isolation into containerized world? For this purpose Stratio DCS uses Project Calico 39

Virtual networks can manage all Mesos supported containerized technologies.

40 MULTI-TENANCY CAPABILITIES: NETWORKS ISOLATION Virtual networks topologies can be created dynamically. Virtual networks topologies can be managed by network policies. Virtual networks can manage all Mesos supported containerized technologies. Virtual networks barely impacts big data performance. Frameworks/apps are authorized into a network. Frameworks/apps can be isolated into a virtual network. Frameworks/apps IP addresses and ports are managed by instance. 40

Network Isolation Virtual Networks Stratio

44 PROTECT THE SERVICE - use case - Framework authentication Check resources for the role Authorization to launch tasks Authorization to use the network Audit (logs and Mesos API) MESOS Admin CALICO & DOCKER ENGINE 44

45 PROTECT THE SERVICE - use case - Framework authentication Check resources for the role Authorization to launch tasks Authorization to use the network Audit (logs and Mesos API) At least 1 core, 1GB to framework 1 MESOS Admin CALICO & DOCKER ENGINE 45

46 PROTECT THE SERVICE - use case - Framework authentication Check resources for the role Authorization to launch tasks Authorization to use the network Audit (logs and Mesos API) At least 1 core, 1GB to framework 1 MESOS Admin net_2: Deny from framework 1 CALICO & DOCKER ENGINE 46

47 PROTECT THE SERVICE - use case - Framework authentication Check resources for the role Authorization to launch tasks Authorization to use the network Audit (logs and Mesos API) User 2. Launches FRAMEWORK 1 User 2. Launches FRAMEWORK 2 At least 1 core, 1GB to framework 1 MESOS Admin net_2: Deny from framework 1 CALICO & DOCKER ENGINE NETWORK A 0.5 CORES 1Gb RAM CONTAINER 1 CONTAINER 2 NETWORK B 2 CORES 5Gb RAM 47

48 PROTECT THE SERVICE - use case - Framework authentication Check resources for the role Authorization to launch tasks Authorization to use the network Audit (logs and Mesos API) User 2. Launches FRAMEWORK 1 User 2. Launches FRAMEWORK 2 At least 1 core, 1GB to framework 1 MESOS Admin net_2: Deny from framework 1 CALICO & DOCKER ENGINE NETWORK A 0.5 CORES 1Gb RAM CONTAINER 1 CONTAINER 2 NETWORK B 2 CORES 5Gb RAM 48

49 MULTI-DATA CENTER - a use case - 49

Service Mesh and Microservices Networking

Service Mesh and Microservices Networking WHITEPAPER Service mesh and microservice networking As organizations adopt cloud infrastructure, there is a concurrent change in application architectures towards