Boni Bruno, Chief Solutions Architect, EMC BLUETALON AUDITING AND AUTHORIZATION WITH HDFS ON ISILON ONEFS V8.0

Transcription

1 Boni Bruno, Chief Solutions Architect, EMC BLUETALON AUDITING AND AUTHORIZATION WITH HDFS ON ISILON ONEFS V8.0 1

2 Secure, Fast, Flexible Hadoop Data Security Solution for Enterprises Analyze data at any scale or speed with your favorite Hadoop framework Info Sec Data-Centric Security Hadoop Compute Data users Auditing HDFS Masking Authorizatio n Simplify data security with a central policy and audit Manage your enterprise data in a highperformance flexible grow-as-you-go storage system that scales-out 2

3 BlueTalon & Isilon provides Hadoop control & visibility at the Data Layer Transparent enforcement End users use existing apps without change Minimal performance overhead for security Contextual auditing Tagged with policy, role and actions Dynamic masking Selective for users without duplicating data Precise authorization Granular: file, sub-file, row, column, cell, sub-cell Decisions based on business data Security admins Auditors Audit Engine Policy Engine Data Stewards Analysts Developers Business users Any Application Hadoop Enforcement Points Data Scientists Machines 3

4 Example of policy, enforcement and audit in BlueTalon Security admins Data Stewards Data users (e.g. analysts, data scientists) Auditors 4

5 Performance benchmark with BlueTalon on Isilon Minute performance difference with large map reduce jobs without and with BlueTalon 1 TB data Job Elapsed Time (mins) Without BlueTalon With BlueTalon HDP 2.4, 7 compute nodes Terasort Audit Engine 40x7 cores, 252 GBx7 mem Hadoop Policy Engine Enforcement Points Teragen nodes, 100 TB, OneFS Teragen Measures Write I/O from Hadoop cluster to Isilon cluster. Terasort Measures entire MapReduce performance across HDFS I/O between Hadoop and Isilon, local disk I/O, CPUs usage, memory usage, etc. 5

6 BlueTalon Validation in SA Lab BlueTalon Enforcement Points in diagram Filesystem EP (installed on each compute node) Policy and Audit in diagram Policy Engine and UI Audit Engine and UI Clients shown in diagram FsShell (hdfs) Hive cli (mapreduce) Isilon node 1 This is how BlueTalon s customers using a Hadoop cluster for compute and Isilon cluster for HDFS storage deploy BlueTalon. Security admins create rules and view audit through UI (or API) that drive run-time Policy and Audit Engines on a management node of the compute cluster. All file system requests from the compute cluster go through the local FSEP, which proxies the Isilon NameNode over HDFS (not webhdfs) protocol. There is one instance of FSEP per compute node. The FS EP proxy connects to OneFS using SmartConnect to maintain scalability and performance. 6 6

7 Details of the BlueTalon validation in SA Lab Isilon storage cluster (Hopkington SA Lab) 4 node Isilon cluster with hdfs enabled and webhdfs not enabled. Webhdfs was disabled on Isilon to make sure BT only used HDFS on the backend with Isilon. It does. This means we can use BT to proxy both HDFS and WEBHDFS to Isilon HDFS on the backend! HDP compute cluster 8 node HDP cluster configured with HDFS, YARN, Ambari Metrics, etc. 7 compute nodes with 40 cores and 252 GB each Ambari UI: BlueTalon EPs : Filesystem EP installed on each compute node BlueTalon Policy and Audit Engines and UIs installed on Ambari node Policy UI : Audit UI : Tests validated (see screenshots) FsShell ls and cat commands Teragen and Terasort mapreduce jobs with 1GB and 1TB data Screen on the bottom right shows write throughput on a teragen mapreduce job running through BlueTalon EP 7 7

8 Details of Validation with OneFS Simulator Note: BlueTalon Engineering runs an HDFS command test suite as part of its release exit criteria on native HDFS clusters. We ran this checklist Jenkins job against the Isilon cluster. All 118 tests passed successfully. Test1. Functional validation of storage, compute and storage+compute jobs 3 Node Isilon OneFS 8.0 Simulator with HDFS enabled on a ESXi host o o HDFS license enabled FreeBSD OS Single Node HDP 2.3 cluster on EC2 instance o o o BlueTalon Policy, Audit and Filesystem EP HDFS clients (fs shell and yarn) CentOS 6.5 OS Ports opened between compute cluster on EC2 and Isilon storage cluster o Port 8020 for NameNode and port 585 for DataNode process Configuration on HDP cluster o o core-site.xml changed to point to FSEP or Isilon for different tests Filesystem EP configured in proxy authentication Sizing for functional testing o o Isilon VMWare Host: 8 vcpu, 32 GB mem, 500GB disk HDP EC2 instance: m3.xlarge = 4 vcpu, 15 GB mem, 80GB disk 8

9 Comparison of storage queries with and without BlueTalon Without BlueTalon With BlueTalon core-site.xml on the compute cluster configures the filesystem Without FSEP: Both alice & bob can list in alice s home folder With FSEP: bob can t list in alice s home folder Without FSEP: Both alice & bob can read data from a private file in alice s home folder With FSEP: bob can t access data from a private file in alice s home folder Compute cluster points to Isilon storage cluster directly Compute cluster points FSEP which points to Isilon storage cluster Without FSEP: alice can t move files in her home folder because filesystem is owned by hdfs & supergroup (required for Hadoop functionality) With FSEP: alice can move data from private location to public location to share with bob 9

10 Enforcement and Policies applied on Isilon storage cluster Enforcement of BlueTalon policies in HDFS backed by Isilon Policies created in BlueTalon Policy UI (or automated with rules API) Compute cluster points to FSEP which points to Isilon storage cluster alice can list her folder bob can t list her folder alice can view her private data bob can t view her private data alice can make her private data public by copying it to public folder bob can view alice s public data global_default policy applies to all users If no rule is applicable, then deny is enforced Allowing recursive execute on / enables traversing the filesystem meta-data without exposing data. Allowing recursive read on /user/<username>/public enables users to share data with others through their home folder <username>_default policy applies to only that user Allowing recursive read and write on /user/<username> enables users to maintain their private files in their home folders Each user gets the effect of permissions from both global_default and their <username>_default policies 10

11 Enforcement and Audit of requests on Isilon storage cluster Enforcement of BlueTalon policies in HDFS backed by Isilon Audit of the requests captured by BlueTalon Compute cluster points to FSEP which points to Isilon storage cluster alice can list her folder bob can t list her folder alice can view her private data bob can t view her private data alice can make her private data public by copying it to public folder bob can view alice s public data 11

12 MapReduce jobs through BlueTalon FSEP on Isilon storage cluster mapreduce compute cluster points to FSEP which points to Isilon storage cluster alice doesn t have any fs-test files Subset of the audit captured during the map reduce test in BlueTalon UI alice is running a mapreduce job that goes through BlueTalon FS EP the file system read test run by alice completes successfully the file system write test run by alice completes successfully 12

13 Policies in BlueTalon credit cards and social security are sensitive our contracts prohibits use of customer data outside west coast Data Stewards or Business Security Admins social security is selectively masked data is restricted to west coast locations Business Users, Data Scientists, Developers 13

14 GUI - BlueTalon HDFS Data Domain for Isilon OneFS 14

15 GUI - BlueTalon OpenLDAP User Domain for Users and Roles 15

16 Audit of HDFS with BlueTalon FS EP on Isilon OneFS Not only READ and WRITE, but also OPEN and GETFILESTATUS requests can be audited 16

17 Detailed audit of Hive user alice, beeline -e select * from accounts { } { } "Action": "LOGIN","AuditParams": "-", "Client": "-","ClientIp": " ", "ColumnList": "-","DataBase": "-", "Effect": "Authorized", "FinalQuery": "-", "GroupName": "bedrock", "LoggedUser": "alice", "OrignalQuery": "-", "PolicySet": "global_default,bedrock_default", "Schema": "-", "SessionID": "-", "Timestamp": " :09:34.354", "UniqueID": "-" "Action": "UNKNOWN","AuditParams": "", "Client": "","ClientIp": " ", "ColumnList": "-","DataBase": "default", "Effect": "Denied", "FinalQuery": "select * from ARCAccessDenied", "GroupName": "bedrock","loggeduser": "alice", "OrignalQuery": "select * from accounts", "PolicySet": "global_default,bedrock_default", "Schema": "default","sessionid": "", "Timestamp": " :09:34.711", "UniqueID": "711655_ _ " { }{ } user bluetalon, beeline -e select * from accounts "Action": "LOGIN","AuditParams": "-", "Client": "-","ClientIp": " ", "ColumnList": "-","DataBase": "-", "Effect": "Authorized","FinalQuery": "-", "GroupName": "bluetalon","loggeduser": "bluetalon", "OrignalQuery": "-","PolicySet": "bluetalon_default,global_default", "Schema": "-","SessionID": "-","Timestamp": " :09:39.819", "UniqueID": "-" "Action": "UNKNOWN","AuditParams": "", "Client": "","ClientIp": " ", "ColumnList": "ID,NAME,PHONE,BIRTHDATE,SOC_SEC_NO,ZIP,CREDIT_CARD,BALANCE", "DataBase": "default","effect": "Policy", "FinalQuery": "select accounts.id, accounts.name, accounts.phone, accounts.birthdate, hash(accounts.soc_sec_no) SOC_SEC_NO, accounts.zip, 0 CREDIT_CARD, accounts.balance from accounts WHERE (accounts.zip > /*<GCODE>WestCoastZips<GCODE>*/) ", "GroupName": "bluetalon","loggeduser": "bluetalon", "OrignalQuery": "select * from accounts", "PolicySet": "bluetalon_default,global_default", "Schema": "default", "SessionID": "", "Timestamp": " :09:40.214", "UniqueID": "214805_ _ " 17

18 Detailed audit of HDFS user alice, hdfs dfs -ls /bedrock user bluetalon, hdfs dfs -ls /bedrock { }{ } "audit_type": "audit", "database": "", "group_list": ["bedrock"], "ipaddress": " ", "modified_request": ["Allow ","GETFILESTATUS ","/bedrock"], "policy_list": [], "policy_type": "", "request": ["GETFILESTATUS ","/bedrock"], "schema": "", "time_stamp": " :16:11", "unique_key": "0f885bb e56-9d6c-90c000f24f78", "user": "alice" "audit_type": "audit", "database": "", "group_list": ["bedrock"], "ipaddress": " ", "modified_request": ["Allow ","READ ","/bedrock"], "policy_list": [], "policy_type": "", "request": ["READ ","/bedrock"], "schema": "", "time_stamp": " :16:11", "unique_key": "64d3e4b8-bbee-4e2a-a4f4-b6da6134e045", "user": "alice" { } "audit_type": "audit", "database": "", "group_list": ["users","bluetalon"], "ipaddress": " ", "modified_request": ["Allow ","GETFILESTATUS ","/bedrock"], "policy_list": [], "policy_type": "", "request": ["GETFILESTATUS ","/bedrock"], "schema": "", "time_stamp": " :16:15", "unique_key": "fa796aa9-1fb1-4ddc-8dfc-71dcc91981a5", "user": "bluetalon" Output from the bt-audit-kafka service 18

19 Verbosity in BlueTalon HDFS 19

20 Example of a Quick Report in BlueTalon Audit UI 20

21 Example of a Quick Report in BlueTalon Audit UI 21

22 Example of Short Filter Reports in BlueTalon Audit UI 22

23 List of Predefined Quick Reports in BlueTalon Audit UI 23

24 Quick Reports in BlueTalon Audit UI Exported to CSV 24

25 Create Customized Reports in BlueTalon Audit UI (I) 25

26 Use Customized Reports in BlueTalon Audit UI (II) 26

27 Run Customized Reports in BlueTalon Audit UI (III) 27

28