Big Data security, tools and tips to protect information assets

Transcription

1 Big Data security, tools and tips to protect information assets Eddie Garcia Chief Security Architect 1

2 A Big Data Revolution is Happening as We Speak Industrial Revolution Data Revolution 2

3 The Benefits of Apache Hadoop... One place for unlimited data All types More sources Faster, larger ingestion Unified, multi-framework data access More users More tools Faster changes 3

4 Can Create Information Security Challenges Business Manager Run high value workloads in cluster Quickly adopt new innovations Information Security Follow established policies and procedures Maintain compliance IT/Operations Integrate with existing IT investments Minimize end-user support Automate configuration 4

5 Levels of Security Where do you start? 5

6 Comprehensive, Compliance-Ready Security Authentication, Authorization, Audit, and Compliance Perimeter Access Visibility Data Guarding access to the cluster itself Defining what users and applications can do with data Reporting on where data came from and how it s being used Protecting data in the cluster from unauthorized visibility Technical Concepts: Authentication Network isolation Technical Concepts: Permissions Authorization Technical Concepts: Auditing Lineage Technical Concepts: Encryption, Tokenization, Data masking 6

7 Active Directory and Kerberos Active Directory Manages Users, Groups, and Services Provides username / password authentication Group membership determines Service access Kerberos Trusted and standard third-party Authenticated users receive Tickets Tickets gain access to Services User [ssmith] Password[***** ] User authenticates to AD Authenticated user gets Kerberos Ticket Ticket grants access to Services e.g. Impala 7

8 Authentication Kerberos authentication LDAP and SAML authentication for web Uis LDAP authentication for SQL access (Hive and Impala) 8

9 Network Isolation Firewall creates 2 tiers of access: Edge Nodes gateway services only / no HDFS data Most user access The rest of the cluster all other services / HDFS data Only a few admins allowed 9

10 Access Control Background The majority of hadoop users are dealing with structured data Applications that run on Hadoop (Datameer, Platfora, SAS, etc) now use Spark, in addition to Hive, Impala and MR. Customers say they want to... Keep one logical copy of data, to minimize storage and complexity of securing that data Set permissions once and analyze that data with any application or compute framework Apply permissions at the granularity of columns and rows 10

11 Use Cases for Fine-Grained Access Control Across All Hadoop Access Paths Columns: Sensitive column visibility varies by role (Ex. credit card numbers) Entitlement varies by subscription Rows: Different user groups need access to different records European privacy laws Government security clearance Financial information restrictions Subscription entitlement 11

12 Access Control with Storage Permissions Only Early Days of Hadoop APPS PLATFORA DATAMEER TABLEAU SAS ETC... Simple All or Nothing permissions for each file/table COMPUTE HIVE, IMPALA SPARK, MR But... In Big Data, tables often contain tens or hundreds of columns STORAGE Filesystem HDFS Not all users are allowed to see all columns and rows 12

13 Access Control with SQL Auth and Storage Permissions APPS STORAGE COMPUTE PLATFORA DATAMEER TABLEAU SAS ETC... Filesystem HDFS HIVE, IMPALA SQL Auth X SPARK, MR Adds column and row-level permissions But... The hive objects are not accessible outside of Hive and Impala. Many, many applications use MR and Spark So data needs to be extracted from the table and placed in separate files where varying HDFS permissions are applied => Duplicate data, duplicate permissions 13

14 Access Control with Apache Sentry and RecordService APPS COMPUTE PLATFORA DATAMEER TABLEAU SAS ETC... HIVE, IMPALA SPARK, MR Column and Row-level Permissions One copy of data One set of permissions APACHE SENTRY, RECORDSERVICE STORAGE Filesystem HDFS 14

15 Fine-Grained Access Control without Sentry & RecordService Split the original file Use HDFS permissions to limit access Date/time Accnt # SSN Asset Trade Country 09:33: :33: :12: :22: :55: :22: :45: :03: :55: AZP Sell US TBT Buy EU IDI Sell UK ICBD Buy US FWQ Buy US UAD Buy UK NZMA Sell EU TMV Buy US DRW Buy UK Date/time Accnt # SSN Asset Trade Country 09:33: :22: :55: :03: Date/time Accnt # SSN Asset Trade Country 11:33: :45: Date/time Accnt # SSN Asset Trade Country 14:12: :22: :55: AZP Sell US ICBD Buy US FWQ Buy US TMV Buy US TBT Buy EU NZMA Sell EU IDI Sell UK UAD Buy UK DRW Buy UK 15

16 Fine Grained Access Control With Sentry & RecordService Sentry: Define Row, column, and sub-column (masking) permissions Sentry + RecordService: Enforce these across all access paths Single HDFS file: Column-Level Controls Date/time Accnt # SSN Asset Trade Country What U.S. Brokers See Column-Level Controls Date/time Accnt # SSN Asset Trade Country 09:33: AZP Sell US 09:33: XXX-XX AZP Sell US 11:33: :12: :22: :55: :22: TBT Buy EU IDI Sell EU ICBD Buy US FWQ Buy US UAD Buy EU Row-Level Controls Hive, Impala, MR, Spark, Pig 11:33: :12: :22: :55: :22: XXX-XX XXX-XX TBT Buy group2 IBM Sell group3 ICBD Buy US FWQ Buy US UA Buy group3 Row-Level Controls 13:45: NZMA Sell EU 13:45: AMZN Sell group2 16

17 Additional Apache Sentry Features Protection for Hive Metastore Without this, users could modify views to point to unauthorized sensitive data Simple positive permissions Combining positive, negative and exceptions, permissions become very difficult to audit and track in a large environment True Role-Based Access Control RBAC Ties to Enterprise Directory (LDAP or Active Directory) for users and groups Permissions GUI Handles: Hive, Hive Metastore, Impala, Search, Kafka, MR, Spark, Pig 17

18 ABAC*: Enforce Top-level Policies while Delegating Administration Master File (M) Date/time Accnt # SSN Asset Trade Broker 09:33: :33: :12: :22: :55: :22: :45: :03: :55: PI I AAPL Sell group1 TBT Buy group2 HDP Sell group3 INTC Buy group1 F Buy group1 UA Buy group3 AMZN Sell group2 TMV Buy group1 MA Buy group3 Example Permissions based on Tags PI PII tag means: Only members of role A can see this I *Roadmap Item 18

19 Column Tag Follows the Data Master File Date/time Accnt # SSN Asset Trade Broker 09:33: :33: :12: :22: :55: :22: :45: :03: :55: PI I AAPL Sell group1 TBT Buy group2 HDP Sell group3 INTC Buy group1 F Buy group1 UA Buy group3 AMZN Sell group2 TMV Buy group1 MA Buy group3 Date/time Accnt # SSN Asset Trade Broker Date/time Accnt # SSN Asset Trade Broker Date/time Accnt # 09:33: SSN AAPL Asset Sell Trade Broker group1 09:33: AAPL Sell group1 09:33: AAPL Sell group1 09:22: INTC Buy group1 09:22: INTC Buy group1 09:22: INTC Buy group1 11:55: F Buy group1 Date/time Accnt # SSN Asset Trade Broker 11:55: F Buy group1 11:55: F Buy group1 09:03: TMV Buy group1 09:22: INTC Buy 09:03: TMV Buy group1 09:03: TMV Buy group1 group1 09:03: TMV Buy group1 Date/time Accnt # SSN Asset Trade Broker 09:33: PI I Copies, subsets, result-sets PI I AAPL Sell group1 09:22: INTC Buy group1 Date/time Accnt # SSN Asset Trade Broker 11:55: F Buy group1 11:55: F Buy group1 09:03: TMV Buy group1 09:03: TMV Buy 19 group1

20 Role-Based Access Control (RBAC) and Centralized Authorization Manage data access by role, instead of just by users & groups Customer Support Rep has read access to US Customers Broker Analyst has read access to US Transactions Relationships between users and roles are established via groups An RBAC policy is then uniformly enforced for all Hadoop services Provides unified authorization controls As opposed to tools for managing numerous, service specific policies Roles are a requirement of RBAC to administer policies at scale 20

21 Centralized Policy Management with Apache Sentry Sam Smith Group Tier 1 Customer Support Reps Sentry Role Sentry Perm. Read Access to Customers.Cust omerid Where Country = US Cust. ID SSN Phone Country US 09:22: EU US Martha Jones Group Tier 1 Broker Analysts U.S. Customer Transaction Analysis Sentry Perm. Read Access to Transactions.D ate Where Country = US Date/Time Cust. ID Trade Country 11:33: :22: Sell US EU 13:45: Buy US 21

22 Apache Hadoop for Cybersecurity 22

23 Data is constantly under threat Threat surface expanding Attacks are increasing Threats are adaptive 16 billion connected devices generating more data There has been a 250% increase in successful attacks Protection against attacks with known signatures no longer sufficient 23

24 Challenges with traditional threat detection Security Operations Security Analysts Security Responders Data left out of process Expensive to scale systems Proprietary tooling makes difficult to implement new Out of the box analytics Signature based is yesterdays threat Advanced analytics are add-ons Data can take weeks to retrieve Raw and historic data offline Reactive instead of predictive 24

25 Powering the next generation of cybersecurity Time (Months) SIEM (TBs) Machine Learning Advanced Statistics SQL Correlations Search Aggregated Events User Data Raw System Logs Network Flows/ DNS Data Types (MBs>PBs) Full Packet Capture Video, Text, Images 25

26 Powering the next generation of cybersecurity Time (Months) SIEM (TBs) Apache Hadoop Based Applications (PB) Machine Learning Advanced Statistics SQL Correlations Search Aggregated Events User Data Raw System Logs Data Types (MBs>PBs) Network Flows/ DNS Full Packet Capture Video, Text, Images 26

27 Benefits of modern threat detection Security Operations Security Analysts Security Responders Keep data online forever Process larger volumes of diverse data Native SQL, Statistical, Machine Learning capabilities Advanced Persistent Threat detection Make data accessible immediately Provide raw and enriched data access 27

28 Introducing Apache Spot (incubating) Apache Hadoop on Intel platform delivers unrivaled analytic performance and scale Network Endpoint Apache Hadoop Intel Platform User / Identity Apache Spot open data models place customer in control of the data unlocking tremendous value Apache Spot application framework accelerates development and delivery of adjacent use cases built on open data models Robust community 28

29 Apache Spot V1.0 Apache Spot for cybersecurity difference Analyze billions of network events per day leveraging machine learning in order to detect unknown events, insider attacks, and diagnose dark areas Reduce false positive alerts by triangulating the data with context to assure the alerts you receive are legitimate Provide meaningful insights by analyzing the data (e.g., flow, DNS packet) that is already being collected 29

30 Use Case: Hidden Networks and Suspicious Connects How can I find the bad mixed in with all the good? Port 80 HTTP Port 143 IMAP Suspicious! Advanced Analytics Algorithms Deliver suspicious connections in ranked order with multiple data points such as time, traffic flow, and more. Take action Monitor Ignore Human input helps the system evolve. Quickly eliminate false positives from the lineup. Identify the needle in the haystack with patterns that provide insight into potential threats. And make every item on the list worth your time to investigate. 30

31 Path to Enlightenment Apache Spot v 1.0 perimeter flows (Stealthy) Scanning Side-channel data escapes Reflection attacks Unusual data flows Beaconing perimeter flows + DNS DNS tunneling Covert DNS channels Internal DNS recon perimeter flows + DNS + internal flows Lateral movement Complete threat visibility 31

32 From raw packets to the most actionable events Network Flows (nfcapd) Each data source is a pipeline new pipelines can be added by following a short recipe Parallel Ingest Framework Machine Learning Operational Analytics DNS (pcap) Sensors feed ONI Open Source Decoders Creates CSV and Compressed data in HDFS Filters billions to thousands Baseline not required Unsupervised, no rules required Returns small number of credible threats from machine learning Visualization, Noise Filter, Attack Heuristics 32

33 Key Differentiators Open can be used turnkey but owners can build on it to expand capability Scale storage and machine learning go far beyond capabilities of most security products Best of Hadoop and Intel Architecture Spark is combined with software optimized for Intel chips (Math Kernel Libraries, Intel MPI library) Sophisticated machine learning and visualization are back of the book, ready for cloud and on-prem today 33

34 Spark and C for ML Use Spark for pre- and post-processing Use C/MPI for Latent Dirichlet Allocation (empirical bayes) Parameter estimation gamma and phi, then alpha E-step parallel, m-step local 3.2 * 10^7 docs, 2*10^5 words Sequential 96 hours, parallel 90 minutes Next step is to capture benchmarks from MKL improvements 34

35 Telemetry streaming ingestion and batch processing NetFlow Parse DNS Syslog Enrich Publish Alert Publisher Alert Monitoring Packet Captures SNMP Server Logs Malware High Throughput Telemetry Ingestion Framework Streaming Parse / Enrich Machine Learning Publish Threat Analysis Search ONI Dashboard Real Time Search Exploits Executables Batch Operational Analytics Visualization 35

36 Real world use cases 36

37 Managed security provider investigates ~1,000 potential security incidents and analyzes ~20,000 pieces of malicious software. 37

38 Consumer credit provider building detection models using a full year of comprehensive log and indicator data 38

39 US Bank leveraging user behavior analytics to detect fraud and inside user threats 39

40 A US national security organization identifies potentially suspicious activity across the worldwide web, and supply threat information to 700 commercial and federal organizations. 40

41 Thank you. 41