Ranger installation in Kerberized Environment

Transcription

1 Ranger installation in Kerberized Environment Summary Creating Keytab and principals For Ranger Admin For Ranger Lookup For Ranger Usersync For Ranger Tagsync Installation Steps for Ranger-Admin Installation Steps for Ranger-Usersync Installation Steps for Ranger-Tagsync Installation Steps for Ranger-KMS Installing Ranger Plugins Manually Installing/Enabling Ranger HDFS plugin Installing/Enabling Ranger HIVE plugin Installing/Enabling Ranger HBASE plugin Installing/Enabling Ranger YARN plugin Installing/Enabling Ranger KNOX plugin Installing/Enabling Ranger STORM plugin Installing/Enabling Ranger KAFKA plugin Summary In order to install Ranger in kerberized environment, user will have to enable kerberos on the cluster where Ranger is to be installed. Once, cluster is kerberized, user will have to create principals for each Ranger service and then follow below given steps to install Ranger. Creating Keytab and principals Note: Below steps required only for manual installation of ranger services and plugins Do some initial Checks : Login as ranger user: If ranger user not found then create it i.e. useradd ranger E.g : su - ranger Check for HTTP Principal -> kinit -kt <HTTP keytab path> HTTP/<FQDN_OF_Ranger_Admin_Cluster>@<REALM> E.g: kinit -kt /etc/security/keytabs/spnego.service.keytab HTTP/test-dummy-X.openstacklocal@EXAMPLE.COM (After above command there should not be any error. You can check using klist whether the above command was successful) -> kdestroy (Please don't miss kdestroy after above step) For Ranger Admin Create rangeradmin/<fqdn of Ranger Admin>@<REALM> -> kadmin.local -> addprinc -randkey rangeradmin/<fqdn of Ranger Admin> E.g: addprinc -randkey rangeradmin/test-dummy-x.openstacklocal@example.com -> xst -k /etc/security/keytabs/rangeradmin.keytab rangeradmin/<fqdn of Ranger Admin>@<REALM> -> exit

2 Check ranger-admin created principal -> kinit -kt /etc/security/keytabs/rangeradmin.keytab rangeradmin/<fqdn of Ranger E.g :kinit -kt /etc/security/keytabs/rangeradmin.keytab (After above command there should not be any error. You can check using klist whether the above command was successful) -> kdestroy (Please don t miss kdestroy after above step) For Ranger Lookup Create rangerlookup/<fqdn of Ranger Admin>@<REALM> -> kadmin.local -> addprinc -randkey rangerlookup/<fqdn of Ranger Admin> Eg: addprinc -randkey rangerlookup/test-dummy-x.openstacklocal@example.com -> xst -k /etc/security/keytabs/rangerlookup.keytab rangerlookup/<fqdn of Ranger Admin>@<REALM> -> exit Check ranger-lookup created principal -> kinit -kt /etc/security/keytabs/rangerlookup.keytab rangerlookup/<fqdn of Ranger Admin>@<REALM> E.g : kinit -kt /etc/security/keytabs/rangerlookup.keytab rangerlookup/test-dummy-x.openstacklocal@example.com (After above command there should not be any error u can check using klist whether the above command was successful) -> kdestroy (Please don t miss kdestroy after above step) For Ranger Usersync Create rangerusersync/<fqdn>@<realm> -> kadmin.local -> addprinc -randkey rangerusersync/<fqdn of Ranger usersync> E.g: addprinc -randkey rangerusersync/test-dummy-x.openstacklocal@example.com -> xst -k /etc/security/keytabs/rangerusersync.keytab rangerusersync/<fqdn>@<realm> -> exit Check rangerusersync created principal -> kinit -kt /etc/security/keytabs/rangerusersync.keytab rangerusersync/<fqdn of Ranger usersync>@<realm> E.g : kinit -kt /etc/security/keytabs/rangerusersync.keytab rangerusersync/test-dummy-x.openstacklocal@example.com (After above command there should not be any error u can check using klist whether the above command was successful) -> kdestroy (Please don t miss kdestroy after above step)

3 For Ranger Tagsync Create -> kadmin.local -> addprinc -randkey rangertagsync/<fqdn of Ranger tagsync> Eg: addprinc -randkey rangertagsync/test-dummy-x.openstacklocal -> xst -k /etc/security/keytabs/rangertagsync.keytab -> exit Check rangertagsync created principal -> kinit -kt /etc/security/keytabs/rangertagsync.keytab rangertagsync/<fqdn of Ranger E.g : kinit -kt /etc/security/keytabs/rangertagsync.keytab rangertagsync/test-dummy-x.openstacklocal@example.com (After above command there should not be any error u can check using klist whether the above command was successful) -> kdestroy (Please don t miss kdestroy after above step) Note: Change the keytab permission to read only and assign it to ranger user Installation Steps for Ranger-Admin Untar the ranger-<verison>-admin.tar.gz -> tar zxf ranger-<version>-admin.tar.gz Change directory to ranger-<version>-admin -> cd ranger-<version>-admin Edit install.properties (Enter appropriate values for the below given properties)

4 db_root_user=<username> db_root_password=<password of db> db_host=test-dummy-x.openstacklocal db_name= db_user= db_password= policymgr_external_url= authentication_method=unix or LDAP or AD spnego_keytab=<http keytab path> token_valid=30 cookie_domain=<fqdn_of_ranger_admin_cluster> cookie_path=/ admin_keytab=<rangeradmin keytab path> lookup_keytab=<rangerlookup keytab path> hadoop_conf=/etc/hadoop/conf Note: If kerberos server and admin are on different host then copy the keytab on admin host and assign permission to ranger user scp the rangeradmin keytab file to the respective path of another host chown ranger <rangeradmin keytab path> chmod 400 <rangeradmin keytab path> Run setup ->./setup.sh Start Ranger admin server ->./ranger-admin-services.sh start Installation Steps for Ranger-Usersync Untar the ranger-<verison>-usersync.tar.gz -> tar zxf ranger-<version>-usersync.tar.gz Change directory to ranger-<version>-usersync -> cd ranger-<version>-usersync Edit install.properties (Enter appropriate values for the below given properties) POLICY_MGR_URL = usersync_keytab=<rangerusersync keytab path> hadoop_conf=/etc/hadoop/conf

5 Note: If kerberos server and usersync are on different host then copy the keytab on usersync host and assign permission to ranger user scp the rangerusersync keytab file to the respective path of another host chown ranger <rangeusersync keytab path> chmod 400 <rangerusersync keytab path> Run setup ->./setup.sh Start Usersync server ->./ranger-usersync-services.sh start Installation Steps for Ranger-Tagsync Untar the ranger-<verison>-tagsync.tar.gz -> tar zxf ranger-<version>-tagsync.tar.gz Change directory to ranger-<version>-tagsync -> cd ranger-<version>-tagsync Edit install.properties (Enter appropriate values for the below given properties) TAGADMIN_ENDPOINT = tagsync_keytab=<rangertagsync keytab path> hadoop_conf=/etc/hadoop/conf TAG_SOURCE= (either 'atlas' or 'file' or 'atlasrest') Note: If kerberos server and tagsync are on different host then copy the keytab on tagsync host and assign permission to ranger user scp the rangertagsync keytab file to the respective path of another host chown ranger <rangetagsync keytab path> chmod 400 <rangertagsync keytab path> Run setup ->./setup.sh Start Ranger tagsync server ->./ranger-tagsync-services.sh start Installation Steps for Ranger-KMS Untar the ranger-<verison>-snapshot-kms.tar.gz -> tar zxf ranger-<version>-snapshot-kms.tar.gz Change directory to ranger-<version>-snapshot-kms -> Cd ranger-<version>-snapshot-kms Edit install.properties (Enter appropriate values for the below given properties)

6 KMS_MASTER_KEY_PASSWD=<Master Key Password> kms_principal=rangerkms/<fqdn of ranger kms kms_keytab=<ranger kms keytab path> hadoop_conf=<hadoop core-site.xml path> POLICY_MGR_URL= of ranger admin host>:6080 Note: if kerberos server and Ranger KMS are on different host then copy the keytab on Ranger KMS host and assign permission to kms user scp the rangerkms keytab file to the respective path chown ranger <rangekms keytab path> chmod 400 <rangerkms keytab path> Run setup ->./setup.sh Follow other setup required for kerberized cluster like creating keytab adding proxy user Start Ranger tagsync server ->./ranger-kms start Installing Ranger Plugins Manually Installing/Enabling Ranger HDFS plugin We ll start by extracting our build at the appropriate place -> copy ranger-<version>-snapshot-hdfs-plugin.tar.gz to namenode host in /usr/hdp/<hdp-version> directory -> cd ranger-<version>-snapshot-hdfs-plugin Untar the ranger-<verison>-snapshot-snapshot-hdfs-plugin.tar.gz ->cd ranger-<version>-snapshot-hdfs-plugin -> REPOSITORY_NAME=hadoopdev Enable the HDFS plugin by running the below commands -> export JAVA_HOME=/usr/lib/jvm/java-7.0-openjdk.x86_64 ->./enable-hdfs-plugin.sh After enabling plugin, follow the below steps to stop/start namenode. ->su hdfs -c "/usr/hdp/current/hadoop-client/sbin/hadoop-daemon.sh stop namenode" ->su hdfs -c "/usr/hdp/current/hadoop-client/sbin/hadoop-daemon.sh start namenode" Create the default repo for HDFS with proper configuration. -> In Custom repo config add component user (eg. hdfs) as value for below properties a. policy.download.auth.users OR policy.grantrevoke.auth.users

7 7. You can verify the plugin is communicating to ranger admin in Audit->plugins tab Installing/Enabling Ranger HIVE plugin We ll start by extracting our build at the appropriate place -> copy ranger-<version>-snapshot-hive-plugin.tar.gz to hiveserver2 host in /usr/hdp/<hdp-version> directory Untar the ranger-<verison>-snapshot-snapshot-hive-plugin.tar.gz -> cd ranger-<version>-snapshot-hive-plugin -> REPOSITORY_NAME=hivedev Enable the Hive plugin by running the below commands -> export JAVA_HOME=/usr/lib/jvm/java-7.0-openjdk.x86_64 ->./enable-hive-plugin.sh After enabling plugin, follow the below steps to stop/start hiveserver -> ps -aux grep hive grep -i hiveserver2 awk '{print $1,$2}' grep hive awk '{print $2}' xargs kill >/dev/null 2>&1 -> su hive -c "nohup /usr/hdp/current/hive-server2/bin/hiveserver2 -hiveconf hive.metastore.uris="" -hiveconf hive.log.dir=/var/log/hive -hiveconf hive.log.file=hiveserverlog >/var/log/hive/hiveserverout 2> /var/log/hive/hiveserver2err.log &" Create the default repo for Hive with proper configuration -> In Custom repo config add component user (eg. hive) as value for below properties a. b. policy.download.auth.users OR policy.grantrevoke.auth.users tag.download.auth.users 7. You can verify the plugin is communicating to ranger admin in Audit->plugins tab Installing/Enabling Ranger HBASE plugin We ll start by extracting our build at the appropriate place -> copy ranger-<version>-snapshot-hbase-plugin.tar.gz to Active Hbasemaster host in /usr/hdp/<hdp-version> directory Untar the ranger-<verison>-snapshot-snapshot-hbase-plugin.tar.gz -> cd ranger-<version>-snapshot-hbase-plugin -> REPOSITORY_NAME=hbasedev Enable the Hbase plugin by running the below commands -> export JAVA_HOME=/usr/lib/jvm/java-7.0-openjdk.x86_64

8 ->./enable-hbase-plugin.sh After enabling plugin, follow the below steps to stop/start hbase. -> su hbase -c "/usr/hdp/current/hbase-master/bin/hbase-daemon.sh stop regionserver; sleep 25" -> su hbase -c "/usr/hdp/current/hbase-regionserver/bin/hbase-daemon.sh stop master" -> su hbase -c "/usr/hdp/current/hbase-master/bin/hbase-daemon.sh start master; sleep 25" -> su hbase -c "/usr/hdp/current/hbase-regionserver/bin/hbase-daemon.sh start regionserver" Create the default repo for HBase with proper configurations -> In Custom repo config add component user (eg. hbase) as value for below properties a. policy.grantrevoke.auth.users 7. You can verify the plugin is communicating to ranger admin in Audit->plugins tab. Installing/Enabling Ranger YARN plugin We ll start by extracting our build at the appropriate place -> copy ranger-<version>-snapshot-yarn-plugin.tar.gz to Active ResourceManager host in /usr/hdp/<hdp-version> directory Untar the ranger-<verison>-snapshot-snapshot-yarn-plugin.tar.gz -> cd ranger-<version>-snapshot-yarn-plugin -> REPOSITORY_NAME=yarndev Enable the YARN plugin by running the below commands -> export JAVA_HOME=/usr/lib/jvm/java-7.0-openjdk-amd64 ->./enable-yarn-plugin.sh Make sure HADOOP_YARN_HOME and HADOOP_LIBEXEC_DIR is set. -> export HADOOP_YARN_HOME=/usr/hdp/current/hadoop-yarn-nodemanager/ -> export HADOOP_LIBEXEC_DIR=/usr/hdp/current/hadoop-client/libexec/ After enabling plugin, follow the below steps to stop/start. -> Stop/Start the ResourceManager on all your ResourceManager hosts. a. su yarn -c "/usr/hdp/current/hadoop-yarn-resourcemanager/sbin/yarn-daemon.sh stop resourcemanager" b. su yarn -c "/usr/hdp/current/hadoop-yarn-resourcemanager/sbin/yarn-daemon.sh start resourcemanager" c. ps -ef grep -i resourcemanager -> Stop/Start the NodeManager on all your NodeManager hosts. a. su yarn -c "/usr/hdp/current/hadoop-yarn-nodemanager/sbin/yarn-daemon.sh stop nodemanager" b. su yarn -c "/usr/hdp/current/hadoop-yarn-nodemanager/sbin/yarn-daemon.sh start nodemanager" c. ps -ef grep -i nodemanager Create the default repo for Yarn with proper configuration In Custom repo config add component user (eg. yarn) as value for below propertiesyou can verify the plugin is communicating to ranger admin in Audit->plugins tab. a. policy.download.auth.users OR policy.grantrevoke.auth.users You can verify the plugin is communicating to ranger admin in Audit->plugins tab.

9 Installing/Enabling Ranger KNOX plugin We ll start by extracting our build at the appropriate place -> copy ranger-<version>-snapshot-knox-plugin.tar.gz to Active ResourceManager host in /usr/hdp/<hdp-version> directory Untar the ranger-<verison>-snapshot-snapshot-knox-plugin.tar.gz -> cd ranger-<version>-snapshot-knox-plugin -> REPOSITORY_NAME=knoxdev Enable the Knox plugin by running the below commands -> export JAVA_HOME=/usr/lib/jvm/java-7.0-openjdk-amd64 ->./enable-knox-plugin.sh After enabling plugin, follow the below steps to stop/start knox gateway. -> su knox -c "/usr/hdp/current/knox-server/bin/gateway.sh stop" -> su knox -c "/usr/hdp/current/knox-server/bin/gateway.sh start" 7. Create the default repo for Knox with proper configuration -> In Custom repo config add component user (eg. hive) as value for below properties a. policy.grantrevoke.auth.users You can verify the plugin is communicating to ranger admin in Audit->plugins tab. Note: -> For Test Connection to be successful follow addition step Trusting Self Signed Knox Certificate -> Knox plugin must be enabled in all Knox instances (in HA env). Installing/Enabling Ranger STORM plugin We ll start by extracting our build at the appropriate place -> copy ranger-<version>-snapshot-storm-plugin.tar.gz to Active host in /usr/hdp/<hdp-version> directory Untar the ranger-<verison>-snapshot-snapshot-storm-plugin.tar.gz -> cd ranger-<version>-snapshot-storm-plugin -> REPOSITORY_NAME=stormdev Enable the Storm plugin by running the below commands -> export JAVA_HOME=/usr/lib/jvm/java-7.0-openjdk.x86_64 ->./enable-storm-plugin.sh After enabling plugin, follow the below steps to stop/start storm. -> su - storm -c 'source /usr/hdp/current/storm-nimbus/conf/storm-env.sh ; export PATH=/usr/jdk64/jdk8.0_60/bin:$PATH ; storm nimbus > /var/log/storm/nimbus.out'

10 -> su - storm -c 'source /usr/hdp/current/storm-client/conf/storm-env.sh ; export PATH=/usr/jdk64/jdk8.0_60/bin:$PATH ; storm drpc > /var/log/storm/drpc.out' -> su - storm -c 'source /usr/hdp/current/storm-client/conf/storm-env.sh ; export PATH=/usr/jdk64/jdk8.0_60/bin:$PATH ; storm ui > /var/log/storm/ui.out' -> su - storm -c 'source /usr/hdp/current/storm-supervisor/conf/storm-env.sh ; export PATH=/usr/jdk64/jdk8.0_60/bin:$PATH ; storm logviewer > /var/log/storm/logviewer.out' Create the default repo for Storm with proper configuration -> In Custom repo config add component user (eg. storm) as value for below properties a. policy.grantrevoke.auth.users OR policy.grantrevoke.auth.users 7. You can verify the plugin is communicating to ranger admin in Audit->plugins tab. Installing/Enabling Ranger KAFKA plugin We ll start by extracting our build at the appropriate place -> copy ranger-<version>-snapshot-kafka-plugin.tar.gz to Active host in /usr/hdp/<hdp-version> directory Untar the ranger-<verison>-snapshot-snapshot-kafka-plugin.tar.gz -> cd ranger-<version>-snapshot-kafka-plugin -> COMPONENT_INSTALL_DIR_NAME=/usr/hdp/<hdp-version>/kafka -> REPOSITORY_NAME=kafkadev Enable the KAFKA plugin by running the below commands -> export JAVA_HOME=/usr/lib/jvm/java-7.0-openjdk.x86_64 ->./enable-kafka-plugin.sh Create the default repo for Storm with proper configuration -> In Custom repo config add component user (eg. storm) as value for below properties a. policy.grantrevoke.auth.users OR policy.grantrevoke.auth.users You can verify the plugin is communicating to ranger admin in Audit->plugins tab. Note: If plugin is not able to communicate then check property authorizer.class.name in /usr/hdp/<hdp-version>/kafka/config/server.properties, value of authorizer.class.name should be org.apache.ranger.authorization.kafka.aut horizer.rangerkafkaauthorizer.