Enabling YARN ResourceManager High Availability

Transcription

1 Enabling YARN High Availability Note: The following steps are for development purposes only. Ambari and above exposes the ability to enable High Availability directly from the web client and should be used for real-world use cases. Prerequisites HDP 2.0+ YARN Installed At least 2 hosts in the cluster where YARN is not present on 1 host. These steps assume there are 3 hosts in the cluster. Assumptions This table lists the placeholders in the various steps below. Placeholder Description Example ambari-server The server running the Ambari web client c6401.ambari.apache.org:8080 cluster-name The name of the cluster cluster1 target-host rm-host-active rm-host-standby zk-host-1, zk-host-(n) The host that will have the additional The host that will run the active The host that will run the standby The ZooKeeper hosts configured on the cluster. c6403.ambari.apache.org c6402.ambari.apache.org c6403.ambari.apache.org c6401.ambari.apache.org:2181, c6402.ambari.apache.org:2181, c6403.ambari.apache.org:2181 yarn-cluster-name The YARN cluster name. yarn_cluster yarn-site-tag The tag property of Ambari's yarn-site version1 Installing 1. Stop all services except for HDFS. There are two methods that can be used to accomplish this: 1. Using the Ambari web client to manually stop each service. 2. Using the Ambari REST APIs directly. For this method, you can consult the documentation on starting and stopping services. 2. Add a component to YARN on a host that does not already have installed. This will not actually install, but will setup the host component associations. curl -u admin:$password -H "X-Requested-By: Ambari" -i -X POST -d '"host_components" : ["HostRoles":"component_name":" RESOURCEMANAGER"] ' /<cluster-name>/hosts?hosts/host_name=<target-host> 3. Install on the same target_host used in the previous step.

2 curl -u admin:$password -H "X-Requested-By: Ambari" -i -X PUT -d '"RequestInfo":"context":"Install "," operation_level":"level":"host_component","cluster_name":"<clustername>","host_name":"<target-host>","service_name":"yarn","body": "HostRoles":"state":"INSTALLED"' /clusters/<cluster-name>/hosts/<target-host>/host_components /RESOURCEMANAGER YARN Configuration 1. The yarn-site configuration group (yarn-site.xml) must now be updated to reflect the backup. property yarn.resourcemanager.ha.enabled yarn.resourcemanager.ha.rm-ids yarn.resourcemanager.hostname.rm1 yarn.resourcemanager.hostname.rm2 yarn.resourcemanager.recovery.enabled yarn.resourcemanager.store.class yarn.resourcemanager.zk-address value true rm1,rm2 <rm-host-active> <rm-host-standby> true org.apache.hadoop.yarn.server.resourcemanager.recovery. ZKRMStateStore <zk-host-1>, <zk-host-2>, <zk-host-3> yarn.resourcemanager.cluster-id < yarn-cluster-name> yarn.resourcemanager.ha.automatic-failover.zk-base-path /yarn-leader-election There are two methods to update these configuration properties: 1. /var/lib/ambari-server/resources/scripts/configs.sh 2. REST API configs.sh $./configs.sh set <ambari-server> <cluster-name> yarn-site <paramname> <param-value> example $./configs.sh set c6401.ambari.apache.org cl1 yarn-site yarn. resourcemanager.ha.enabled true This will need to be execute for each parameter listed. 4. REST API 1. Get the existing yarn-site configuration version and tag.

3 GET fields=clusters/desired_configs "href" : " "Clusters" : "cluster_name" : "c1", "version" : "HDP-2.1", "desired_configs" : "yarn-site" : "tag" : "version1", "user" : "_anonymous", "version" : 1, 2. Using the value from the tag property of yarn-site, request the current YARN configuration. In this example, <yarn-sitewould be " version1". tag>

4 GET /configurations?(type=yarn-site&tag=<yarn-site-tag>) "href" : " /configurations?(type=yarn-site&tag=version1)", "items" : [ "href" : " "tag" : "version1", "type" : "yarn-site", "version" : 1, "Config" : "cluster_name" : "c1", "properties" : "yarn.acl.enable" : "false", "yarn.admin.acl" : "", "yarn.timeline-service.ttl-enable" : "true", "yarn.timeline-service.ttl-ms" : " ", "yarn.timeline-service.webapp.address" : "c6402.ambari. apache.org:8188", "yarn.timeline-service.webapp.https.address" : "c6402. ambari.apache.org:8190" ] 3. The properties property contains the current yarn-site configuration. Append the new properties to this structure and then update yarn-site.

5 PUT "Clusters": "desired_config": "type":"yarn-site", "tag":"<new-yarn-site-tag>", "properties":: "yarn.acl.enable":"false", "yarn.admin.acl":"", "yarn.timeline-service.ttl-enable":"true", "yarn.timeline-service.ttl-ms":" ", "yarn.timeline-service.webapp.address":"c6402.ambari. apache.org:8188", "yarn.timeline-service.webapp.https.address":"c6402. ambari.apache.org:8190" <yarn-ha-site-properties> "yarn.resourcemanager.ha.enabled": "true", "yarn.resourcemanager.ha.rm-ids": "rm1,rm2", 5. Restart all stopped services.