Securing Hadoop. Keys Botzum, MapR Technologies Jan MapR Technologies - Confiden6al

Transcription

1 Securing Hadoop Keys Botzum, MapR Technologies Jan 2014 MapR Technologies - Confiden6al 1

2 Why Secure Hadoop Historically security wasn t a high priority Reflec6on of the type of data and the type of organiza6ons using Hadoop Hadoop is now being used by more tradi6onal firms as well as organiza6ons with high security requirements Highly regulated Sensi6ve data sets People with experience with security in exis6ng enterprise technologies (e.g., databases) are asking for the same in Hadoop 2

3 Why Secure Hadoop Client opera6ng system is trusted to iden6fy user (weak authen6ca6on) If I can compromise client, I can run jobs or access HDFS as anyone Think about virtual machines with root access Hadoop servers trust anyone that can reach them on the network Could I falsify a data node, job tracker, etc.? Hive Server runs as system user All Hive Server submi\ed jobs run as that system user Intruders can see and modify all network traffic 3

4 Apache Hadoop Security Core goals Authen6cate network traffic Users authen6cate Servers authen6cate to each other Encrypt network traffic Note: Hadoop also has a lot of authoriza6on func6onality which I m not discussing here 4

5 Apache Hadoop Security Kerberos as core authen6ca6on technology Kerberos to access HDFS, JT, Oozie, etc. Kerberos for server to server traffic But Kerberos doesn t fit perfectly with Hadoop model Introduce delega6on tokens for carrying iden6ty in many scenarios Kerberos is complicated Need Kerberos iden6ty for every server in the cluster Lots to manage! Every user needs a Kerberos iden6ty to access cluster, Web UIs, etc. Lots of steps h\p:// content/cloudera- docs/ CDH4/4.3.0/CDH4- Security- Guide/cdh4sg_topic_3.html 5

6 Ecosystem Kerberos Ecosystem components also generally rely on Kerberos Need to create appropriate Kerberos SPNEGO iden66es for many services (Web UI access) Need to create service Kerberos iden6ty for cluster access for many services, oken for each node Lots to manage HBase, Oozie, Hive Server 2, Hive Meta Server, Flume, etc. 6

7 Apache Hadoop Security Addi:onal Items Kerberos only part of the puzzle More steps some examples Configure Web UI HTTPS Configure Encrypted Shuffle Configure Hive Server 2 Authen6ca6on using LDAP or Kerberos Impersona6on Authen6cate to HS2 (userid/password or Kerberos) HS2 executes job using secure impersona6on on cluster Now job runs as submiong user and can see/modify only what user can Encryp6on SSL can be used to protect userid & password authen6ca6on to HS2 7

8 MapR Distribu:on for Apache Hadoop Complete Hadoop distribu6on Comprehensive management suite Industry- standard interfaces Enterprise- grade dependability Higher performance Ease of Use 8

9 The Cloud Leaders Pick MapR Amazon EMR is the largest Hadoop provider in revenue and # of clusters Google chose MapR to provide Hadoop on Google Compute Engine 9

10 MapR Security Build on the work of the Apache community, but with improvements Goals Authen6cate network traffic Users authen6cate Servers authen6cate to each other Encrypt network traffic Low performance overhead Simple and easy to administer 10

11 MapR Na:ve Security Hadoop security without Kerberos But borrow heavily from Kerberos design Kerberos integra6on if desired 11

12 Architecture Shared secrets like Kerberos Managed at cluster level Iden6ty represented using a 6cket which is issued by MapR CLDB servers (Container Loca6on DataBase) 12

13 Tickets A 6cket represents a valid authen6cated iden6ty Contains An expira6on 6me, renewal life6me, and crea6on 6me A randomly generated secret key Informa6on about the iden6ty userid, group ids A client authen6cates to servers using the 6cket 13

14 User Experience User invokes maprlogin maprlogin connects to CLDB (over h\ps) Provide userid & password (or Kerberos 6cket) for valida6on by CLDB Ticket is returned, saved in file in /tmp file and accessible only by owning user file name is /tmp/mapr6cket_<uid> MapR PAM module Op6onal MapR provided PAM module creates MapR 6ckets automa6cally during Unix login All processes automa6cally pick up 6cket (nothing to do) Java and C/C++ clients implicitly look for valid 6cket and use it Clients op6onally use exis6ng Kerberos iden6ty to get MapR 6cket 14

15 Client First Contact Client sends the 6cket and data encrypted using secret key Receiving server Validates 6cket, including expira6on Extracts iden6ty informa6on from 6cket and uses that for authoriza6on Returns encrypted response to client No6ce that MapR user iden6ty is independent of host or opera6ng system iden6ty 15

16 Server First Contact When a trusted server starts it uses a local server 6cket to authen6cate to the CLDB CLDB verifies the 6cket s authen6city using secret key CLDB returns a server key that is used to create and validate user 6ckets The server is now a trusted member of the cluster 16

17 Maprlogin Primary user visible security tool Ac6ons are password - authen6cate to a MapR cluster using a valid password kerberos - authen6cate to a MapR cluster using Kerberos print - print informa6on on your exis6ng creden6als authtest - test authen6ca6on as a generic client end / logout - logout of cluster renew - renew exis6ng 6cket For example: % maprlogin password [Password for user 'fred' at cluster 'my.cluster.com': ] MapR creden6als of user 'fred' for cluster 'my.cluster.com' are wri\en to '/tmp/mapr6cket_1001' 17

18 Maprlogin Under the Covers maprlogin 1. username/passwd sent on h\ps MapR CLDB 2. uses PAM to authen6cate LDAP/ Kerberos/ NIS 4. 6cket + key saved in file in /tmp 3. 6cket + user key returned hadoop fs ls / 5. cmd picks up 6cket + key from file 6. client sends RPC encrypted with user- key + 6cket FileServer/ CLDB 7. server decrypts 6cket to authen6cate user and checks permissions on ACL 18

19 Cryptography Encrypted using current NIST standards AES- 256 in GCM mode for encryp6on and signing h\p://en.wikipedia.org/wiki/galois/counter_mode NIST standard - h\p://csrc.nist.gov/publica6ons/fips/fips140-2/fips1402annexa.pdf Leverage Intel hardware encryp6on where available, sokware otherwise Use the open source crypto++ library for our C++ cryptography h\p://cryptopp.com Random number genera6on Use secure random number genera6on as documented here h\p:// class_auto_seeded_random_pool.html#_details 19

20 MapR Security More by Default By default, out of the box HS2 supports password authen6ca6on Can configure Kerberos and SSL func6on, same as from Apache, including secure impersona6on Oozie supports MapR 6cket authen6ca6on Can configure Kerberos and SSL func6on, same as from Apache, including secure impersona6on MapR Tables (HBase APIs) use na6ve MapR security, no configura6on needed Most Web UIs enhanced to support userid & password authen6ca6on and HTTPS Can configure Kerberos SPNEGO, same as from Apache 20

21 Encrypted Shuffle (?) No need to special case encryp6ng shuffle MapR- FS is store for Map output Shuffle inherits the same encryp6on, authen6ca6on, and authoriza6on func6onality of the rest of MapR- FS 21

22 Let s Build a Secure Cluster! Node 1 apt- get install mapr. configure.sh C - Z - secure genkeys Generates all needed keys for MapR- RPC as well as for HTTPS Node N apt- get install mapr. scp rootormapr@node1:/opt/mapr/conf/ {cldb.key,maprserver:cket,ssl_keystore,ssl_truststore} /opt/mapr/conf configure.sh C - Z - secure Clients apt- get install mapr scp anyuser@noden:/opt/mapr/conf/ssl_truststore /opt/mapr/conf configure.sh - secure 22

23 MapR Advantage Vastly simpler Core secured by default in one step No requirement for Kerberos in core and associated complexity Easier integra6on Leverage exis6ng Linux authen6ca6on (PAM and NSSwitch) Faster Leverage Intel AES hardware cryptography 23

24 Further Reading MapR h\p://mapr.com MapR Na6ve Security h\p:// release/mapr- technologies- integrates- security- into- hadoop h\p:// with- mapr/mapr- integrates- security- into- hadoop Adding Security to Apache Hadoop h\p://hortonworks.com/wp- content/uploads/2011/10/security- design_withcover- 1.pdf The Evolu6on of Hadoop s Security Model h\p:// 24

25 Thank You MapR Technologies - Confiden:al 25