IBM Aspera Orchestrator Administrator Guide Version 2.3

Transcription

1 IBM Aspera Orchestrator Administrator Guide Version 2.3 [Aspera Software Services Team] ABSTRACT [This document describes the install, upgrade and configuration procedures for Aspera Orchestrator version 2.3.]

2 Contents INTRODUCTION... 3 PRE-REQUISITES... 3 INSTALLATION OF ASPERA COMMON PACKAGE... 3 ORCHESTRATOR INSTALLATION... 4 ORCHESTRATOR STARTUP... 5 ORCHESTRATOR CONFIGURATION... 5 ORCHESTRATOR INSTALLATION AS NON-ROOT USER... 6 ORCHESTRATOR CLUSTER INSTALLATION... 7 Overview... 7 Storage pre-requisites... 7 Other pre-requisites... 8 In the cloud... 8 Active/active cluster... 9 Alternative setups... 9 Checking the services Stop the services Sharing mysql data Sharing Orchestrator var data Configuring orchestrator.yml Setting up the database connectivity Start the services ACM setup MANUAL FAILOVER WITH ACM AUTOMATIC FAILOVER WITH ACM DISABLING ACM ORCHESTRATOR UPGRADE SINGLE NODE UPGRADE Orchestrator upgrade procedure to 2.3 from previous releases Orchestrator upgrade procedure from 2.3 to newer releases Upgrade on a cluster without service interruption Rollback MIGRATE A NODE VIA SNAPSHOTS MIGRATE A NODE VIA DB BACKUP MIGRATE + UPGRADE NODE(S) VIA DB BACKUP TROUBLESHOOTING GUIDE ORCHESTRATOR SINGLE NODE INSTALLATION

3 Introduction This document provides necessary information to install, configure, and run IBM Aspera Orchestrator on a supported host. The document assumes that the version of Orchestrator being installed is v2.3. Please contact the Aspera Support or Aspera PS team for legacy install instructions. Pre-requisites For a production grade system: - Physical server or Virtual Machine - 8 core CPUs - 8GB of RAM - 100GB of free disk space - 64 bits Linux RedHat EL or CentOS version 6.x - Firefox or Chrome browser For a development system, 4 core CPUs and 4GB RAM is sufficient. It is recommended that Orchestrator be housed in its own server. However, Orchestrator can be colocated with Aspera Console. It is advisable to not mix management systems (e.g. Console or Orchestrator) with managed systems like transfer servers and WEB application servers (e.g. Aspera Connect Server or Aspera Faspex). All the commands below must be performed with a root login. Installation of Aspera common package Ruby 1.8.7, MySQL and Apache are installed with this package. The package can be downloaded from the Aspera download site together with the Console product at The package version must be greater or equal to v * Please verify if aspera-common rpm is already installed on the host. The below command should not return any entries. # rpm qa grep aspera-common If aspera-common is already installed, please verify compatibility of this product with Aspera Orchestrator. If not present, copy the package onto the Orchestrator server (under /tmp for example) and install it with:

4 # cd /tmp # rpm -Uvh aspera-common-aspera-common x86_64.rpm Orchestrator installation Obtain and copy the Orchestrator rpm (assuming a package named aspera-orchestrator x86_64.rpm) to /tmp for example and do: # cd /tmp # rpm -ivh aspera-orchestrator x86_64.rpm Set Mongrel base port as 3000 for example Run Orchestrator setup: # asctl orchestrator:setup For questions that have a default option, For questions that have a default option, select the default option. When prompted for a root MySQL password, choose a secure password. Confirm your settings. Adding the Orchestrator license: The Orchestrator license must be copied into this directory: /opt/aspera/var/config/orchestrator/licenses/ The license file name must be obtained by the Sales contact or the PS contact. The license file must follow this pattern: orchestrator-license*.txt or -*-orchestrator.aspera-license where * is usually denotes contract number of client identification. Verify the consistency of the MySQL password among different products: If another Aspera product was pre-installed (e.g. Console), make sure that the MySQL password is the same in opt/aspera/var/config/orchestrator/database.yml and /opt/aspera/common/mysql/database.rb.yml If the passwords are different, update database.yml to reflect the one in database.rb.yml. In /opt/aspera/common/mysql/my.cnf, set this line with your time zone: default-time-zone= +01:00 or use the server time:

5 default-time-zone="system" Restart MySQL. Orchestrator startup Start Orchestrator: # asctl orchestrator:start In a WEB browser, access Orchestrator User Interface with the following URL: or HTTP is redirected internally to HTTPS. Replace the text <orchestrator-ipaddress> with the IP address or fully qualified name of the Orchestrator machine. You should be presented with the login screen. Use admin as the login. The password is not set by default. Check in Engine + Processes that all processes are running. Orchestrator configuration All Aspera plugins are enabled by default along with some generally used plugins. Enable other plugins you plan to use. Reload all plugins (Engine + Plugins + Reload Plugins) and click on Force Maintenance (Engine + Processes + Force maintenance). Enable all plugins (Engine + Plugins + Enable all plugins) if you want to browse through the list of plugins. Enable all portlets (Engine + Portlets + Enable all) to use them in the Dashboard. Perform the SMTP server(s) configuration in Engine + Mailer Configuration + New mail server if you plan to send notifications within the workflows. Test it with Send Test . Activate the journaling configuration in Engine + Journaling Configurations (click Activate in the default Saved Configurations) if you plan to use the journaling capability. Define the users (e.g. with roles developer, operator, etc.) and their permissions in Accounts + New User. Set an admin password in Accounts + select admin user + Change password. Add the Scheduler and Contributor groups to the admin user in Account + Modify Role / Group Memberships to monitor queues and tasks as admin. Define an Active Directory server in Accounts + Active Directory Setup if needed. The first time you log in with an AD user credentials, they are checked against the directory server and a user entry is created in Accounts. Set the permissions for this AD user as for regular users. Each time the AD user tries to log in Orchestrator, his credentials are checked against the directory server.

6 If you plan to use remote access capability with remote file operation and remote file watcher plugins, install Aspera Point-to-Point on the remote node, as the Aspera ascmd executable is required. No Pointto-Point license is actually needed. Note that this is not necessary for a remote execution plugin. Add the remote nodes entries in Workflows + Remote Nodes + New Remote Node. It may be an Aspera FASP node (e.g. remote file watcher, remote file operation, fasp transfer controlled node), a SSH node (e.g. ffmpeg transcoding or remote execution) or an IP node (e.g. Vantage transcoding via a SOAP API). If you plan to use the Aspera fasp transfer capability, install Aspera Point-to-Point and a regular Point-to- Point license on the controlled node (the one which initiates the file transfers). The controlled node may be Orchestrator itself. On the controlled node, make sure that /opt/aspera/etc/aspera.conf contains persistentstore lines such as: <central_server> <port>40001</port> <address> </address> <persistent_store>enable</persistent_store> <persistent_store_max_age>259200</persistent_store_max_age> <persistent_store_on_error>ignore</persistent_store_on_error> </central_server> Add a remote node entry for the controlled node with these features: Node type = Aspera FASP; ssh port = 33001; and your root login information. If you plan to change a parameter in /opt/aspera/orchestrator/config/orchestrator.yml, rather update it in /opt/aspera/var/config/orchestrator/orchestrator.yml The reason is that during an Orchestrator upgrade the former file will be overwritten with default values but the latter file will be kept unchanged. Only put the parameters you want to update in /opt/aspera/var/config/orchestrator/orchestrator.yml, not the ones that are unchanged. The parameters in this file take precedence over the ones in/opt/aspera/orchestrator/config/orchestrator.yml Orchestrator installation as non-root user Orchestrator can be installed with a user other than root. Use the procedures below to create a specific group + user and change the permissions on the Orchestrator folders (assuming Orchestrator_ is the installed release).

7 # asctl orchestrator:stop # groupadd aspera_orch_group # useradd -G aspera_orch_group aspera_orch # cd /opt/aspera # chown aspera_orch:aspera_orch_group orchestrator # chown -R aspera_orch:aspera_orch_group Orchestrator_ # chown -R aspera_orch:aspera_orch_group var # asctl orchestrator:restart Note that Orchestrator is still run as root or with sudo access. Orchestrator cluster installation Overview The Orchestrator platform is running in active/active mode for Apache and Orchestrator services, and in active/passive mode for MySQL. The Orchestrator cluster can run in premises or in the cloud. For Orchestrator in premises, a shared external storage must be set up for: /opt/aspera/var/run/orchestrator/ The file trigger history is stored there. Each node has the same vision of already triggered files. Note that the history may alternatively be kept in the shared MySQL data. The parameter to set is then persistence_mode ( db ) in orchestrator.yml /opt/aspera/var/config/orchestrator/ This is optional but recommended as the plugins are stored here (in the actions sub-directory). Importing a plugin can then be done on only one node. /opt/aspera/var/archive/orchestrator/ This is optional but recommended as the plugins are archived here, then nodes can share the same view of archived plugins. MySQL data ACM data NOTE: don t just share /opt/aspera/var/ instead of the 3 run, config and archive Orchestrator folders. The reason is that if Aspera Point to Point or other transfer products were installed on the same nodes as Orchestrator, the services would not work as expected. Storage pre-requisites The shared storage must be redundant (e.g. with RAID arrays) to avoid a single point of failure. NAS or SAN external storage is supported.

8 Note that Virtual Machines typically do not support SAN Fiber Channel storage. Thus make sure that your Orchestrator is not running on Virtual Machines if your storage is based on SAN Fiber Channel technology. A SAN storage requires a cluster file system (e.g. GFS) for our HA solution to work. A block level storage access would not work; only a file level storage access is suitable. Files must be visible from both nodes concurrently. If a file is changed from one node, the other node must be able to immediately see the changes. For example, a storage mount based on Raw Device Mapping is at the block level. RDM is a VMware server virtualization environment that enables storage to be directly connected to a virtual machine from the SAN. This would not work, as file updates on one node are not visible to the other node until the shared storage is remounted. A NAS storage based on NFS requires NFS version4. A solution based on NFS version 3 may cause troubles, as NFS3 does not automatically release the locks held by a client. This is critical for MySQL where only one node at a time must write to the shared disk (our MySQL version does not support locking on concurrent writes). With NFS3, if the active node (running MySQL) goes down, then the passive node is unlikely to be able to start MySQL when it takes over, as some MySQL files may still be locked on the shared storage. With the NFS4 technology, all the locks associated with a TCP session that times out will be automatically released, making MySQL automatic failover possible. A NAS storage based on CIFS (e.g. Samba) will work, as there would be no locking issues. MySQL data, ACM and Orchestrator var data may be configured on separate mount points, but this is not a required. Configure at least 20GB for the MySQL data, 1 GB for ACM and 20GB for the Orchestrator var data. As ACM is not migrating the mount points over failover, MySQL, Orchestrator var data and ACM data must be mounted and visible on both nodes simultaneously. Other pre-requisites Typically, the MySQL database runs on each Orchestrator node across shared MySQL data. This MySQL database is delivered with the Aspera common package and is supported by the Aspera Support team. An external MySQL database with an HA solution of its own is also a possible solution. But Aspera won t deliver support for it. Note that the Oracle MySQL Cluster solution comes with the NDB engine, which will not inter-operate with Orchestrator as only MySQL with the InnoDB engine is supported. Make sure that the date/time is identical on all nodes in the cluster (including the shared storage). Consider using NTP (Network Time Protocol) to synchronize the software clock of your Linux systems with Internet timeservers. In the cloud Regarding Orchestrator in the cloud, the setup may be the following: No shared storage as no file based storage is typically available (only object storage) External database with an HA solution of its own accessed via a unique virtual IP address File trigger history stored in the external database Plugins are not shared so importing and archiving a plugin must be done on both nodes of the cluster No ACM, so the services must be started and controlled by the Operating System

9 Active/active cluster Orchestrator and Apache software components run as active on all nodes. More than 2 nodes could be running these components. Orchestrator works natively in active/active mode. The workload is truly shared between the Orchestrator nodes in the cluster. A workflow instance may be executed on all nodes (e.g. one step in a node and the next step in another node). MySQL is run in active/passive mode. ACM (Aspera Cluster Manager) manages MySQL instances failover. ACM also starts Orchestrator services automatically at boot time. Upon failure, those services are also restarted automatically. Orchestrator accesses the single MySQL active instance via a primary/secondary IP address mechanism. An external load balancer may be used for WEB based Orchestrator operations (e.g. defining a workflow or doing user interaction) WEB requests may be done via a unique virtual IP address, managed by the load balancer. If watch folders are used in Orchestrator workflows, the files triggered by Orchestrator must be visible by all Orchestrator instances if the workflow logic relies on reading/writing the files after triggering them. Put the watch folders on a shared storage for this purpose. Alternative setups A multi-tenant setup is supported: 2 Orchestrators instances running on the same node, with different access URLs, different config directories and a shared database. Having multiple Orchestrator nodes operating independently is another alternative. Tracking issues may be simpler as each node may run its own dedicated workflows. Note that with the Federated Workflow plugin, an Orchestrator node may coordinate workflows running on multiple separate Orchestrator nodes.

10 Checking the services Make sure that these commands return off so that the services below are not started automatically by the Operating System: # chkconfig --list grep AsperaOrchestrator AsperaOrchestrator 0:off 1:off 2:off 3:off 4:off 5:off 6:off # chkconfig --list grep aspera_httpd aspera_httpd 0:off 1:off 2:off 3:off 4:off 5:off 6:off # chkconfig --list grep aspera_mysqld aspera_mysqld 0:off 1:off 2:off 3:off 4:off 5:off 6:off If not true, turn off a service with: # chkconfig <service name> off Stop the services Stop Orchestrator and MySQL on both nodes before doing the configuration steps described below: # asctl orchestrator:stop # asctl mysql:stop Sharing mysql data We assume that /mnt/shared/orchestrator/ is the Orchestrator mount point on the shared storage. Make sure that /mnt/shared/orchestrator/ has the rwxr-xr-x permissions. Otherwise change it with # chmod 755 /mnt/shared/orchestrator/ Create a mysql_data directory under this mount point for the MySQL shared data files. Make sure that mysql group and user id are the same on both nodes. Check in /etc/group and /etc/passwd files.

11 On both nodes, do: # cd /opt/aspera/common/mysql # mv./data./data.bak # ln -s /mnt/shared/orchestrator/mysql_data./data # chown -h mysql.mysql./data # ls -lah /opt/aspera/common/mysql drwxr-x--- 2 mysql mysql 4.0K Jan 18 16:13 bin lrwxrwxrwx 1 mysql mysql 4 Jun 12 15:25 data -> /mnt/shared/orchestrator/mysql_data drwxr-x--- 5 mysql mysql 4.0K Jan 18 16:26 data.bak -rw-r mysql mysql 14K Nov database_controller.rb Create an empty file under /opt/aspera/common/mysql/data/ and check that it comes with mysql ownership (rather than nobody ownership): # cd /opt/aspera/common/mysql/data/ # sudo -u mysql touch toto.txt # ll -rw-r--r-- 1 mysql mysql 0 May 12 08:19 toto.txt Then remove this file: # rm toto.txt On one node where the shared storage is accessible, copy the MySQL data into the shared volume: # cp -R /opt/aspera/common/mysql/data.bak/* /opt/aspera/common/mysql/data Then go on the other node and verify that you can see the data files in this directory (may be necessary to perform a failover to make the shared storage visible): # ls /opt/aspera/common/mysql/data/ -rw-rw mysql mysql Apr 23 11:04 ibdata1

12 -rw-rw mysql mysql Apr 23 11:04 ib_logfile0 -rw-rw mysql mysql Apr 22 16:01 ib_logfile1 drwx mysql mysql -rw-rw mysql mysql drwx mysql mysql drwx mysql mysql Apr 22 16:00 mysql 5536 Apr 23 11:04 mysqld.log Apr 22 17:52 orchestrator Apr 22 16:00 test Sharing Orchestrator var data We assume that /mnt/shared/orchestrator/ is the Orchestrator mount point on the shared storage. On one node, create the var/archive/orchestrator, var/config/orchestrator and var/run/orchestrator directories under this mount point. # cd /mnt/shared/orchestrator/ # mkdir -p var/archive/orchestrator # mkdir -p var/config/orchestrator # mkdir -p var/run/orchestrator On both nodes, create the symbolic links: # cd /opt/aspera/var/archive # mv orchestrator orchestrator.bak # ln -s /mnt/shared/orchestrator/var/archive/orchestrator orchestrator # cd /opt/aspera/var/config # mv orchestrator orchestrator.bak # ln -s /mnt/shared/orchestrator/var/config/orchestrator orchestrator # cd /opt/aspera/var/run # mv orchestrator orchestrator.bak # ln -s /mnt/shared/orchestrator/var/run/orchestrator orchestrator

13 On one node where the shared storage is accessible, copy the shared data: # cp -R /opt/aspera/var/archive/orchestrator.bak/* /opt/aspera/var/archive/orchestrator # cp -R /opt/aspera/var/config/orchestrator.bak/* /opt/aspera/var/config/orchestrator # cp -R /opt/aspera/var/run/orchestrator.bak/* /opt/aspera/var/run/orchestrator Then go on the other node and verify that you can see the shared files in these directories (may be necessary to perform a failover to make the shared storage visible): /opt/aspera/var/archive/orchestrator/ /opt/aspera/var/config/orchestrator/ /opt/aspera/var/run/orchestrator/

14 Configuring orchestrator.yml If existing, backup somewhere then remove /opt/aspera/var/config/orchestrator/orchestrator.yml as only /opt/aspera/orchestrator/config/orchestrator.yml will be read by Orchestrator. If there are differences with /opt/aspera/orchestrator/config/orchestrator.yml (e.g. in number of threads), update this one on each node with the values from /opt/aspera/var/config/orchestrator/orchestrator.yml On one node, edit /opt/aspera/orchestrator/config/orchestrator.yml to reflect this configuration: engine_instance: 1 use_db_time: 1 (add this element if absent) active_active: true install_config_file: /opt/aspera/orchestrator/config/orchestrator.yml aspera_dir: /opt/aspera orchestrator_dir: /opt/aspera/orchestrator run_dir: /opt/aspera/var/run/orchestrator config_dir: /opt/aspera/var/config/orchestrator archive_dir: /opt/aspera/var/archive/orchestrator install_images_dir: /opt/aspera/var/config/orchestrator/images install_pages_dir: /opt/aspera/var/config/orchestrator/pages install_css_dir: /opt/aspera/var/config/orchestrator/stylesheets log_file: /opt/aspera/orchestrator/log/orchestrator.log Repeat the same edit on the peer node with engine_instance set to 2 instead of 1. In some cases the HTML branding may be different between the nodes. This can customized in /opt/aspera/orchestrator/config/branding.yml on each node. In this case, remove /opt/aspera/var/config/orchestrator/branding.yml if existing. We specify the logs in the local folder /opt/aspera/orchestrator/log/ rather than on the shared folder /opt/aspera/var/run/orchestrator/log/ as there may be an issue when the shared storage is accessed via NFS (the logs won t be written in the shared log file).

15 Setting up the database connectivity For the active/passive MySQL setup within an active/active cluster, add in the database.yml file at each node under /opt/aspera/orchestrator/config/ the primary_host and alternate_host entries as shown below. development: host: localhost port: 4406 primary_host: alternate_host: adapter: mysql username: root password: aspera database: orchestrator production: host: localhost port: 4406 primary_host: alternate_host: adapter: mysql username: root password: aspera database: orchestrator Copy database.yml under /opt/aspera/var/config/orchestrator/ so that these changes are taken into account upon upgrade. # cp database.yml /opt/aspera/var/config/orchestrator

16 Start the services Start MySQL on one node: # asctl mysql:start Execute these commands on the same node to allow access via IP addresses: # asctl mysql:grant_remote_access " " # asctl mysql:grant_remote_access " " Restart Orchestrator on both nodes. # asctl orchestrator:start ACM setup ACM is an Aspera clustering software to handle: MySQL failover: detect when the node with the active MySQL instance fails and trigger a failover (start the MySQL instance on the passive node). Make sure that MySQL is active on one node only. Service restart: restart Apache, MySQL (on the active node only), AsperaOrchestrator instances if they fail. Also start them at boot time. If not already done, set hostnames on both nodes in /etc/sysconfig/network file. Example on first node: NETWORKING=yes HOSTNAME=orchestrator1 GATEWAY= Reboot on both nodes to take the hostname change into account: # shutdown r now To configure the ACM log file to be /var/log/aspera.log managed by the rsyslog service, on each node, modify /etc/rsyslog.conf with: Add this line at the end: # Aspera Logging local2.* -/var/log/aspera.log Replace the cron.none occurrence by cron.none;local2.none Replace the /var/log/messages occurrence by -/var/log/messages

17 Restart rsyslog with: # /etc/init.d/rsyslog restart We assume that /mnt/shared/orchestrator/acm_files is the ACM files mount point on the shared storage. Create it if it does not exist. Install ACM files on one node only (assuming a package name acm_orchestrator-0.3.tar.gz) under /tmp for example. Extract the files with: # cp /tmp/acm_orchestrator-0.3.tar.gz /mnt/shared/orchestrator/acm_files # cd /mnt/shared/orchestrator/acm_files # tar xvfz acm_orchestrator-0.3.tar.gz Create a symbolic link on both nodes # ln -s /mnt/shared/orchestrator/acm_files /opt/aspera/acm If database.xml is not existing under /opt/aspera/var/config/orchestrator/, copy it from /opt/aspera/orchestrator/config/ into /opt/aspera/var/config/orchestrator/ Copy database.yml into /opt/aspera/acm/config: # cp /opt/aspera/var/config/orchestrator/database.yml /opt/aspera/acm/config/ Make sure that the MySQL user and password defined in database.yml match MYSQL_USER and MYSQL_PWD values as defined in the following files: /opt/aspera/acm/bin/acm4orchestrator /opt/aspera/acm/bin/acmctl Install ACM in the crontab on both nodes so that it gets launched every minute: # crontab e * * * * * /opt/aspera/acm/bin/acm4orchestrator <node IP address> > /dev/null 2>&1 Example: * * * * * /opt/aspera/acm/bin/acm4orchestrator > /dev/null 2>&1 In this example, the node IP address is

18 The IP address can be obtained with this command: # ip addr grep "inet " inet /8 scope host lo inet /24 brd scope global eth0 Optionally, a partition id can be provided to the acm4orchestrator command line to make sure that the shared partition is mounted on Orchestrator. # crontab e * * * * * /opt/aspera/acm/bin/acm4orchestrator <node IP address> <shared partition device number> > /dev/null 2>&1 Example: * * * * * /opt/aspera/acm/bin/acm4orchestrator > /dev/null 2>&1 You can get the partition number with this command: # stat -c "%d" /mnt/shared/orchestrator/acm_files/acm_orchestrator/ 21 Note that this may cause troubles if the mount point is renamed. Perform an ACM safety check: # /opt/aspera/acm/bin/acmctl -s ACM sanity check Checking if an entry for ACM seems to exist in the crontab OK Checking that the Orchestrator master service is disabled in chkconfig OK Checking that SE Linux mode is not set to enforcing OK On one node, enable ACM: # /opt/aspera/acm/bin/acmctl -E ACM is enabled globally

19 Check the cluster status on the active node: # /opt/aspera/acm/bin/acmctl -i Checking current ACM status... Aspera Cluster Manager for Orchestrator - status Local hostname: Active node: Status of this node: Status file: Disabled globally: orchestrator1 orchestrator1 (me) active current no Disabled on this node: no Database configuration file Database host: localhost Orchestrator active/active services status Apache: running Orchestrator Status: -> Orchestrator Engine running with pid: > Mongrel serving orchestrator on port 3000 is running with pid: > Mongrel serving orchestrator on port 3001 is running with pid: > Mongrel serving orchestrator on port 3002 is running with pid: > Orchestrator Monitor running with pid: > Asynchronous Worker Process 0 is running with pid: > Asynchronous Worker Process 1 is running with pid: > Synchronous Worker Process 2 is running with pid: > Synchronous Worker Process 3 is running with pid: > Synchronous Worker Process 4 is running with pid: > Synchronous Worker Process 5 is running with pid: 25274

20 Orchestrator active/passive services status MySQL: running Check the status on the passive node:

21 bin]# /opt/aspera/acm/bin/acmctl -i Checking current ACM status... Aspera Cluster Manager for Orchestrator - status Local hostname: Active node: Status of this node: Status file: Disabled globally: orchestrator2 orchestrator1 passive current no Disabled on this node: no Database configuration file Database host: localhost Orchestrator active/active services status Apache: running Orchestrator Status: -> Orchestrator Engine running with pid: > Mongrel serving orchestrator on port 3000 is running with pid: > Mongrel serving orchestrator on port 3001 is running with pid: > Mongrel serving orchestrator on port 3002 is running with pid: > Orchestrator Monitor running with pid: > Asynchronous Worker Process 0 is running with pid: > Asynchronous Worker Process 1 is running with pid: > Synchronous Worker Process 2 is running with pid: > Synchronous Worker Process 3 is running with pid: > Synchronous Worker Process 4 is running with pid: 2664

22 -> Synchronous Worker Process 5 is running with pid: 2666 Orchestrator active/passive services status MySQL: stopped Check ACM log messages in this file: /var/log/aspera.log

23 Manual failover with ACM On the active node, disable ACM locally: # /opt/aspera/acm/bin/acmctl -d ACM is disabled locally Then stop all the services: # asctl orchestrator:stop Orchestrator is being stopped forcefully: -> Killing Orchestrator Engine process with pid: > Killing Mongrel serving orchestrator on port 3000 with pid: > Killing Mongrel serving orchestrator on port 3001 with pid: > Killing Mongrel serving orchestrator on port 3002 with pid: > Orchestrator Monitor running with pid: > Killing Orchestrator Monitor with pid: > Killing Asynchronous Worker Process 0 with pid > Killing Asynchronous Worker Process 1 with pid > Killing Synchronous Worker Process 2 with pid > Killing Synchronous Worker Process 3 with pid > Killing Synchronous Worker Process 4 with pid > Killing Synchronous Worker Process 5 with pid # asctl apache:stop Apache: Stop... done # asctl mysql:stop MySQL: Stop... done Check the status: # /opt/aspera/acm/bin/acmctl -i Checking current ACM status... Aspera Cluster Manager for Orchestrator - status Local hostname: Active node: orchestrator1 orchestrator1 (me)

24 Status of this node: Status file: Disabled globally: active current no Disabled on this node: yes Database configuration file Database host: localhost Orchestrator active/active services status Apache: stopped Orchestrator Status: -> Orchestrator Engine NOT running -> Mongrel serving orchestrator on port 3000 is NOT running -> Mongrel serving orchestrator on port 3001 is NOT running -> Mongrel serving orchestrator on port 3002 is NOT running -> Orchestrator Monitor NOT running -> Asynchronous Worker Process 0 is NOT running -> Asynchronous Worker Process 1 is NOT running -> Synchronous Worker Process 2 is NOT running -> Synchronous Worker Process 3 is NOT running -> Synchronous Worker Process 4 is NOT running -> Synchronous Worker Process 5 is NOT running Orchestrator active/passive services status MySQL: stopped Check that the peer node becomes active automatically after a while:

25 # /opt/aspera/acm/bin/acmctl -i Checking current ACM status... Aspera Cluster Manager for Orchestrator - status Local hostname: Active node: Status of this node: Status file: Disabled globally: orchestrator2 orchestrator2 (me) active current no Disabled on this node: no Database configuration file Database host: localhost Orchestrator active/active services status Apache: running Orchestrator Status: -> Orchestrator Engine running with pid: > Mongrel serving orchestrator on port 3000 is running with pid: > Mongrel serving orchestrator on port 3001 is running with pid: > Mongrel serving orchestrator on port 3002 is running with pid: > Orchestrator Monitor running with pid: > Asynchronous Worker Process 0 is running with pid: > Asynchronous Worker Process 1 is running with pid: > Synchronous Worker Process 2 is running with pid: > Synchronous Worker Process 3 is running with pid: > Synchronous Worker Process 4 is running with pid: 31723

26 -> Synchronous Worker Process 5 is running with pid: Orchestrator active/passive services status MySQL: running On the newly passive node, enable ACM to let it start automatically the active services: # /opt/aspera/acm/bin/acmctl -e ACM is enabled locally Check the status on the newly passive host:

27 # /opt/aspera/acm/bin/acmctl -i Checking current ACM status... Aspera Cluster Manager for Orchestrator - status Local hostname: Active node: Status of this node: Status file: Disabled globally: orchestrator1 orchestrator2 passive current no Disabled on this node: no Database configuration file Database host: localhost Orchestrator active/active services status Apache: running Orchestrator Status: -> Orchestrator Engine running with pid: > Mongrel serving orchestrator on port 3000 is running with pid: > Mongrel serving orchestrator on port 3001 is running with pid: > Mongrel serving orchestrator on port 3002 is running with pid: > Orchestrator Monitor running with pid: > Asynchronous Worker Process 0 is running with pid: > Asynchronous Worker Process 1 is running with pid: > Synchronous Worker Process 2 is running with pid: > Synchronous Worker Process 3 is running with pid: > Synchronous Worker Process 4 is running with pid: 26989

28 -> Synchronous Worker Process 5 is running with pid: Orchestrator active/passive services status MySQL: running Automatic failover with ACM The failover mechanism relies on periodical checks of a shared status file: /opt/aspera/acm/run/acm4orchestrator.status By default, the status file is kept up-to-date by the active node changing its last modification date every minute. The status file expires if there is no update by the active node within the last 2 minutes. If the active node detects that the status file is expired, it just updates it. If the passive node detects that the status file is expired, it performs an automatic failover as it assumes the peer node is out of service. Note that this will work only if MySQL database is really stopped on the peer node otherwise the failover will not be triggered. It usually takes 2 to 3 minute to detect a failover situation, plus 2 to 3 minutes to restart Orchestrator, depending on the number of workers to run.

29 Disabling ACM Disable ACM on all nodes: # /opt/aspera/acm/bin/acmctl -D ACM is disabled globally No status file update will then be performed by ACM. It will pass to the expired state. The services are let run. # /opt/aspera/acm/bin/acmctl -i Checking current ACM status... Aspera Cluster Manager for Orchestrator - status Local hostname: Active node: Status of this node: Status file: Disabled globally: orchestrator1 orchestrator1 (me) active expired yes Disabled on this node: no Database configuration file Database host: localhost Orchestrator active/active services status Apache: running Orchestrator Status: -> Orchestrator Engine running with pid: > Mongrel serving orchestrator on port 3000 is running with pid: > Mongrel serving orchestrator on port 3001 is running with pid: 29593

30 -> Mongrel serving orchestrator on port 3002 is running with pid: > Orchestrator Monitor running with pid: > Asynchronous Worker Process 0 is running with pid: > Asynchronous Worker Process 1 is running with pid: > Synchronous Worker Process 2 is running with pid: > Synchronous Worker Process 3 is running with pid: > Synchronous Worker Process 4 is running with pid: > Synchronous Worker Process 5 is running with pid: Orchestrator active/passive services status MySQL: running Re-enable ACM: # /opt/aspera/acm/bin/acmctl -E ACM is enabled globally If the active node calls acm4orchestrator before the passive node, it just updates the status file. If the passive node calls acm4orchestrator before the active node, it does not perform an automatic failover as the status file is passed in current state upon re-enabling ACM.

31 Orchestrator upgrade This section consists of guidance in upgrading Orchestrator to a newer release for a single node and an Orchestrator cluster. The upgrade is done with Orchestrator service interruption on a single node and without interruption in an active/active cluster model. Note that plugins can be upgraded live without restarting Orchestrator, on a single node or a cluster. When Orchestrator is stopped and restarted, the previously running workflow instances will pick up at the point where they were left just before the stop action. Orchestrator is backward compatible so that the existing workflows can still run with the newer Orchestrator version. If the workflows must be upgraded as well (e.g. to take advantage of a new plugin), it can be done live by exporting/importing manually each workflow. Check below the Migrate + upgrade node(s) via DB backup procedures to upgrade both the workflows and the Orchestrator release. Prerequisites - User access: o The document assumes that the commands will be performed as root on Unix machines or an equivalent super user - Aspera common installed and Aspera Orchestrator are already installed on the Unix server - Backup of MySQL data performed by MySQL dump o Refer to Orchestrator Admin guide on how to take a manual MySQL backup. # /opt/aspera/common/mysql/bin/mysqldump -h localhost u root p orchestrator > /tmp/orchestrator_db_backup_<timestamp>.sql The generated sql file may take several gigabytes of disk space depending on the database size, so make sure that enough space is available on the targeted partition (/tmp in the example above) before running the backup - Backup of Orchestrator data performed by Snapshot feature o Refer to Orchestrator Admin guide on how to take a Snapshot - Orchestrator services must be stopped Archive /opt/aspera/var: # cd /opt/aspera/ ; tar chvfz /tmp/aspera_orig_<timestamp>.tar.gz var/

32 Move all backup files onto a safe server after running the backup so that they don t take space on Orchestrator. /opt/aspera/orchestrator/ is overwritten with the new release files while files under /opt/aspera/var/config/orchestrator/ are left unchanged during the upgrade. During the install, custom images, branding files and style sheets are copied from /opt/aspera/var/config/orchestrator/ to /opt/aspera/orchestrator/public/. Thus make sure that any custom files as well as orchestrator.yml if the default Orchestrator configuration has been changed are placed under /opt/aspera/var/config/orchestrator/ before doing the upgrade. Backup /opt/aspera/orchestrator/config/database.yml if the DB configuration is not standard (e.g. non standard password) as it is overwritten by the default released file. Single node upgrade Orchestrator upgrade procedure to 2.3 from previous releases Aspera Orchestrator is now available via an RPM and hence the upgrade procedure is a bit difference when upgrading from prior versions to 2.3. The below procedure is only for users upgrading from a previous version of Orchestrator (1.9.x, 2.0, 2.1.x, 2.2.x) to Orchestrator v2.3. It is likely that asperacommon package installed with Orchestrator is incompatible with Orchestrator v2.3 so it is recommended to upgrade that first. Procedure Step 1 Obtain the new install files and copy them to the server. Copy the new release package (e.g. asperaorchestrator x86_64.rpm and aspera-common-aspera-common x86_64.rpm) into a local directory (e.g. /tmp/) Step 2 Shell/Login into the Unix machine that requires an upgrade and check if services were stopped # asctl all:status If not stopped: On Orchestrator hosts with Orchestrator pre v2.3 servers # cd /opt/aspera/orchestrator # bin/orchestrator all kill # asctl all:stop On On Orchestrator hosts with Orchestrator v2.3 or newer # asctl all:stop

33 Step 3 Upgrade Aspera Common RPM: # rpm UvH aspera-common x86_64.rpm Step 4 Upgrade MySQL and Apache services and follow default answers to the prompts. # asctl mysql:upgrade # asctl apache:upgrade Step 5 Check if MySQL and Apache are running post upgrade # asctl mysql:status # asctl apache:status Step 6 Remove the symlink of orchestrator under /opt/aspera/ which should be pointing to existing Orchestrator installation directory. # rm /opt/aspera/orchestrator Note: If you don t see a symlink, then move the orchestrator directory into something like orchestrator_backup # mv /opt/aspera/orchestrator /opt/aspera/orchestrator_bkp Step 7 Install Orchestrator RPM (It is an install because 2.3 is the first release of Orchestrator with an RPM) # rpm -ivh aspera-orchestrator x86_64.rpm Follow the steps under the Orchestrator installation (Single node installation) section for more detailed list of the steps to perform. Step 8

34 Check status of Orchestrator service, it should be stopped. # asctl orchestrator:status Step 9 Start Orchestrator service and Restart Apache # asctl orchestrator:start # asctl apache:restart Orchestrator upgrade procedure from 2.3 to newer releases Step 1 Copy the new release package (e.g. aspera-orchestrator x86_64.rpm) into a local directory (e.g. /tmp/) Step 2 Stop Orchestrator: # asctl orchestrator:stop Step 3 Install the new release (assumed to be in this example): # cd /tmp/ # rpm -ivh aspera-orchestrator x86_64.rpm Restore database.yml into /opt/aspera/orchestrator/config/ if the DB configuration is not standard and not present in /opt/aspera/var/config/orchestrator/. In this case only, execute the installation script: # cd /opt/aspera/orchestrator/ #./script/install.sh Step 4 Check if there are capsules present in the folder: /opt/aspera/var/config/orchestrator/capsules/ If present, remove the ones that are already integrated in the new Orchestrator code. Keeping old capsules may cause issues hard to detect.

35 Step 5 Start Orchestrator service and Restart Apache # asctl orchestrator:start # asctl apache:restart The Orchestrator previous releases are still present under /opt/aspera (example: /opt/aspera/orchestrator_ /). They can be used for rollback operations. During an upgrade, plugins are not upgraded automatically. You can upgrade all them via Engine + Plugins + Upgrade All Plugins. In a production system, upgrade the new ones individually only when necessary (compare Version versus Released Version). It s advisable to test in pre-production first to make sure that the new plugins operate well with the existing workflows. Check if some new plugins are not enabled (click on Not enabled only box) and enable them if necessary. Do Engine + Plugins + Reload Plugins and Engine + Processes + Force Maintenance to take the new plugins into account. During an upgrade, the new portlets are not enabled by default. Enable them if necessary with Engine + Portlets + Enable all. Test that your workflows work well after the upgrade.

36 Upgrade on a cluster without service interruption We do the upgrade on the node where MySQL is active first (without stopping the MySQL processes, so the Orchestrator on the peer node can still process the workorders), then we do it on the other node. Check the cluster manager status to learn where is the active node: # /opt/aspera/acm/bin/acmctl -i Checking current ACM status... Aspera Cluster Manager for Orchestrator - status Local hostname: Active node: Status of this node: Status file: Disabled globally: orchestrator1 orchestrator1 (me) active current no Disabled on this node: no Above, the orchestrator1 node holds the MySQL active instance. On each node, copy /opt/aspera/orchestrator/config/orchestrator.yml in a safe location (e.g. /tmp) before the upgrade. Custom files if any (e.g. /opt/aspera/orchestrator/public/images/aspera_logo_simple.jpg or /opt/aspera/orchestrator/public/stylesheets/aspera.css) may need to be preserved somewhere and restored after the upgrade. On the node where MySQL is active, disable acm: /opt/aspera/acm/bin/acmctl -d Do the Single node upgrade steps described above with this in mind: Before running install.sh, think of copying /opt/aspera/var/config/orchestrator/database.yml into /opt/aspera/orchestrator/config/

37 After running install.sh, copy the backup orchestrator.yml into /opt/aspera/orchestrator/config/ Enable acm: /opt/aspera/acm/bin/acmctl -e On the node where MySQL is not started, disable acm: /opt/aspera/acm/bin/acmctl -d Do the Single node upgrade steps described. Before running install.sh, think of copying /opt/aspera/var/config/orchestrator/database.yml into /opt/aspera/orchestrator/config After running install.sh, copy the backup orchestrator.yml into /opt/aspera/orchestrator/config/ Re-enable acm: /opt/aspera/acm/bin/acmctl -e On one node only, upgrade the plugins as necessary. On both nodes, reload the plugins and execute Force Maintenance.

38 Rollback This procedure is applicable to an active/active ACM based cluster. For a single node rollback. Just remove the ACM related steps. Check where is the active node: # /opt/aspera/acm/bin/acmctl -i Checking current ACM status... Aspera Cluster Manager for Orchestrator - status Local hostname: Active node: Status of this node: Status file: Disabled globally: orchestrator1 orchestrator1 (me) active current no Disabled on this node: no Above, the orchestrator1 node holds the MySQL active instance. On the node where MySQL is active, disable ACM: /opt/aspera/acm/bin/acmctl -d On this same node, do the following operations: Copy the backup packages (e.g. orchestrator_db_backup_<timestamp>.sql and aspera_orig_<timestamp>.tar) into a local directory (e.g. /tmp/) Stop Orchestrator: # asctl orchestrator:stop Use mysqldump to backup the current DB # /opt/aspera/common/mysql/bin/mysqldump h <IP address> u root p orchestrator > /tmp/orchestrator_db_backup_current_<timestamp>.sql Archive the existing /opt/aspera/var:

39 # cd /opt/aspera/ ; tar cvfz /tmp/aspera_current_<timestamp>.tar.gz var/ Import the backup DB: # /opt/aspera/common/mysql/bin/mysql u root p aspera orchestrator < /tmp/orchestrator_db_backup_orig_<timestamp>.sql Import the backup /opt/aspera/var archive: # cd /opt/aspera/ # rm rf var/ # tar xvfz /tmp/aspera_orig_<timestamp>.tar.gz Check if there are capsules present in the folder: /opt/aspera/var/config/orchestrator/capsules/ If present, remove them. Re-install the previous release (assumed to be in this example): # rpm -ivh aspera-orchestrator x86_64.rpm Restart Orchestrator: # asctl orchestrator:start In the user interface, do Engine + Plugins + Reload Plugins and Engine + Processes + Force Maintenance to reload the plugins. Re-enable acm: /opt/aspera/acm/bin/acmctl -e On the node where MySQL is passive, disable ACM: /opt/aspera/acm/bin/acmctl -d Stop Orchestrator: # asctl orchestrator:stop

40 Re-install the previous release (assumed to be in this example): # rpm -ivh aspera-orchestrator x86_64.rpm Restart Orchestrator: # asctl orchestrator:start In the user interface, do Engine + Plugins + Reload Plugins and Engine + Processes + Force Maintenance to reload the plugins. Migrate a node via snapshots The snapshot method takes the design time data from the DB (workflows, users, plugins, templates, monitors, portlets), but it does not take the existing run-time data (workorders). Snapshot doesn t take /opt/aspera/var/config/orchestrator configuration files. It has to be archived and copied on the new node manually. The snapshot based procedure works only between nodes with the same Orchestrator version. It cannot be used for a version upgrade as DB schema changes won t be taken. All commands below performed as root On the originating node: Log in Orchestrator GUI as admin user. Make a snapshot via the Orchestrator GUI as admin user: Engine + Deployment Configurations + New snapshot The snapshot file is created under: /opt/aspera/var/archive/orchestrator/snapshots/ On the new node: Make sure that IP addresses and firewall rules are OK. Install the software required by the plugins used in the workflows (e.g. MediaInfo) Install Orchestrator (with the same version as on the originating system) Make a backup of the DB on the new node: # /opt/aspera/common/mysql/bin/mysqldump uroot p orchestrator > /tmp/orchestrator_db_backup_new_<timestamp>.sql Untar the archive:

41 cd / ; tar xvfz /tmp/aspera_<timestamp>.tar.gz Start Orchestrator if not yet started: # asctl orchestrator:start Edit /opt/aspera/var/config/orchestrator/orchestrator.yml and add this parameter db_bin: /opt/aspera/common/mysql/bin Import the snapshot file with: # cd /opt/aspera/orchestrator # export PATH=/opt/aspera/common/ruby/bin:${PATH} # export GEM_HOME=/opt/aspera/orchestrator/vendor/dependencies/linux-gnu # ruby script/runner script/snapshot <snapshot file path> Careful: this will delete any existing workflows on the node where it is performed. Perform a maintenance via the Orchestrator GUI: Engine + Force Maintenance Perform a sanity check via the Orchestrator GUI: Check the workflow list and confirm that it matches the list of workflows on the origin server. Check the plugin list and confirm that it matches the list of plugins on the origin server. Check the accounts and confirm that they match the list of accounts on the origin server Careful: some files (e.g. csv files for table lookup or html user defined files) may not be stored under /opt/aspera and may need to be copied as well on new node. Note: instead of a snapshot, a manual export/import of all workflows (via the Orchestrator GUI) may be done.

42 Migrate a node via DB backup This consists in replicating (cloning) an Orchestrator node to a new node, using the database backup method. The new node must have been installed with the same Operating System and the same Orchestrator version as the originating node. Both design-time and run-time data are taken with this method. It may also be used as a base for backup/restore operations when Orchestrator versions stay identical and the run-time data need to be restored. For example, files that were already detected on the origin node will not be detected again on the new node and running workorders on the origin node will be restarted automatically on the new node. Caution: when restoring a DB backup, all existing workflows are deleted. Make sure that all software components (e.g. MediaInfo) used on the originating node are also installed on the destination nodes. All commands below must be performed as root. On the originating node If possible (node not in production), stop Orchestrator before running mysqldump: # asctl orchestrator:stop Use mysqldump to backup the DB # /opt/aspera/common/mysql/bin/mysqldump h <IP address> u root p orchestrator > /tmp/orchestrator_db_backup_orig_<timestamp>..sql The generated sql file may take several gigabytes depending on the database size, so make sure that enough space is available on the disk. Archive /opt/aspera/var: # cd /opt/aspera/ ; tar chvfz /tmp/cvf aspera_orig_<timestamp>.tar.gz var/ Copy dump and archive files on the new node (e.g. under /tmp). Restart Orchestrator: # asctl orchestrator:start