Cluster Upgrade Procedure with Job Queue Migration.

Cluster Upgrade Procedure with Job Queue Migration. Zend Server 5.6 Overview Zend Server 5.6 introduces a new highly-reliable Job Queue architecture, based on a MySQL database storage backend. This document describes how to upgrade from existing clusters running previous versions of Zend Server to the latest version, including a procedure for the migration of Zend Job Queue to the new architecture General Guidelines and Considerations The upgrade procedure for Zend Server and Cluster Manager should be performed according to the instructions in the Installation Guide. This document explicitly focuses on issues and steps that are specific to Job Queue. All other functions and components are to be upgraded as usual. Two main setups are supported and covered by the step-by-step instructions below: 1. A Cluster of Zend Servers is configured to use one of the cluster nodes as a Job Queue server. This is the configuration used by default when Zend Server Cluster Manager is installed. 2. A Cluster of Zend Servers is configured to submit jobs to another Zend Server that is defined as dedicated Job Queue servers ( This Zend Sever is originally not part of the cluster). For simplicity purposes, the upgrade procedure is described using two-node clusters (ZS-CM and 2 ZS) on Linux, while it can be applied to cluster of any size. NOTE: It is highly recommended to perform an upgrade procedure on a cluster / server that does not actively serve user requests. This can be achieved by disconnecting the system from active clients using a load balancer, or by any other means, available in particular setups. 1

Scenario 1 - Job Queue server on one of the cluster nodes Setup The cluster in this scenario consists of: Zend Server Cluster Manager Two Zend Server nodes (server A and server B) Job Queue is configured to run on server A MySQL database (in use by Zend Monitor) is up, running and accessible from all computers above. Upgrade steps 1. Upgrade the Zend Server Cluster manager. This upgrade will reuse the MySQL server and schema that is configured for Zend Monitor, and will add tables for Job Queue to use. The upgrade process will launch a JQD that is configured to use this MySQL DB. This JQD will only serve job info / status requests, issued by ZS-CM GUI. After the upgrade, the ZS CM will detect that not all cluster nodes were upgraded to version 5.6, and will display a message about it. This message also informs the user that he will not be able to use Job Queue normally through the UI until all cluster nodes are upgraded. In the background, the PHP application will still be able to create and execute jobs as usual using the old SQLITE DB, but the ZSCM UI does not display them yet. 2. Upgrade nodes that did not have active JQD running (in our scenario, this is server B). The upgrade procedure will install new software and configure it to use MySQL DB as the JQ backend (using the same database details / credentials as Zend Monitor). From that point, jobs submitted from this server will be successfully created, stored in the MySQL database and executed. These jobs will be shown in the ZSCM UI. Note 1: Cluster Manager UI still shows that the upgrade is not completed. This is because the remaining server (server A) is still running older software and the Job Queue data was not migrated from its local DB. Note 2: Since old jobs were not migrated to the MySQL database yet, any reference to old jobs invoked from this server will fail until the next step is completed. 2

Zend Server 5.6 3. Upgrade the server that runs JQ for the cluster (in our case, server A). The upgrade procedure will install new software and configure it to use the MySQL DB as JQ backend (using the same database details / credentials as Zend Monitor). From this point, all the jobs in the cluster are submitted to the new MySQL database, their status and details are accessible from ZS-CM GUI and no warnings are shown. The next step (seamlessly) performed by the system is an upgrade of the current SQLITE DB. It will be converted to the Zend Server 5.6 format. The adjustments are due to bug fixes and due to changes that will be required by the MYSQL architecture. This procedure takes a couple of minutes, depending on the size of the DB. The final step performed by the upgrade is a migration of jobs from SQLITE DB to the new MySQL DB. First base data (queues, applications, schedules) is migrated, and then jobs. During the migration procedure, the MySQL will start to fill up with old jobs information, which will enable the ZSCM to show them and will also make it possible for incoming query requests (for old jobs info) to be successful. Once the upgrade and migration are finished, all the jobs info should be accessible to the UI and to incoming query requests. The UI notice that was shown before will not be necessary anymore. 3

Scenario 2 - Job Queue server outside the cluster Setup The cluster in this scenario consists of: Zend Server Cluster Manager Two Zend Server nodes (server A and server B) Job Queue is configured to run on server C, which is not part of the cluster. MySQL database (in use by Zend Monitor) is up, running and accessible from all computers above. Upgrade steps 1. Upgrade the Zend Server Cluster manager first. The upgrade procedure will create the MySQL schema and launch a JQD that is configured to use the MySQL DB. After the upgrade, the ZSCM will detect that not all cluster nodes were upgraded to version 5.6, and display a message about it. This message also informs the user that he will not be able to use Job Queue normally through the UI until all cluster nodes are upgraded. At this stage, the PHP application running on cluster nodes can create and execute jobs as usual using the old SQLITE DB, but the ZSCM UI does not display them yet. 2. Upgrade all cluster nodes. The upgrade procedure will install new software and configure it to use the MySQL DB as JQ backend (using the same database details / credentials as Zend Monitor). From that point, jobs submitted from these servers will be successfully created, stored in the MySQL database and executed. These jobs will be shown in the ZSCM UI. Note: Since old jobs were not migrated to MySQL database yet, any reference to old jobs invoked from freshly upgraded servers will fail, until data migration (next step) is completed. 3. Upgrade the server that runs JQ for the cluster (in this case, server C). This will install Zend Server 5.6 software on this dedicated JQ server. 4

Zend Server 5.6 4. Manually configure dedicated JQ server (server C, in our example) to use MySQL database. Edit jqd.ini file and add the following lines: zend_jobqueue.database.type = MYSQL zend_jobqueue.database.name = zend_jobqueue.database.host_name = zend_jobqueue.database.port = zend_jobqueue.database.user = zend_jobqueue.database.password = zend_jobqueue.node_id = Note: zend_jobqueue.node_id should be set with a value that will never collide with node id's in the cluster that 'owns' the MYSQL and with other JQ servers that are set in the same way. A good candidate value could be a high integer, such as 10000 or so. JQD should be restarted for the changes to take effect. After JQD is restarted, it will use these directives to connect to DB. 5. Manually initiate JQ data migration procedure In this setup, the database upgrade / migration script should be executed manually after the server was upgraded. The MYSQL DB settings should be provided to this script. Run the script as follows: /usr/local/zend/gui/lighttpd/sbin/php -c /usr/local/zend/gui/lighttpd/etc/phpfcgi.ini /usr/local/zend/share/scripts/zs_create_databases.php migratejq zsdir=/usr/local/zend dbhost=10.1.1.1 dbport=3306 dbusername=root dbpassword=1234 dbname=zend_monitor isactive=1 5