The most powerful engine for your analytics! EXAoperation 5.0.rc2 Reference

Transcription

1 The most powerful engine for your analytics! EXAoperation 5.0.rc2 Reference

2

3 EXAoperation 5.0.rc2 Reference Table of Contents 1. EXAoperation Components Logging Permissions Processes Installation Booting Export of backups via Samba shares Restore Restore (Storage) Renaming of databases Update servers Interfaces Storage archive volumes Maintenance user Failover License server failure Power outage and checksum mismatches Automatic node reordering for Storage databases Volume restore delay Using the EXAoperation browser interface Form "EXASolution Instances" Form "EXASolution Instance" Form "EXAStorage" Form "EXAStorage Volume Node Information" Form "EXAStorage Node Information" Form "EXAStorage Node Device Information" Form "Cluster Nodes" Form "EXACluster Scheduler Information" Form "EXASolution Backup Services Information" Form "Backups Information" Form "EXACluster Scheduler Job Information" Form "Users" Form "Versions" Form "JDBC Drivers" Form "EXACluster Debug Information" Form "Monitoring Services" Form "Threshold Values" Form "Network" Form "System Passwords" Form "Licence" EXAoperation Add/Edit Forms Form "Create EXACluster Node" Form "EXACluster Node Properties" Form "EXACluster Node Disk Properties" Form "Create EXACluster Virtual Node" Form "EXACluster Virtual Node Properties" Form "Edit EXASolution Instance" Form "Create EXASolution Instance" Form "Create Periodic Backup Job" Form "Periodic Backup Job" Form "Create EXACluster Scheduler" Form "EXACluster Logging Service" Form "Create Remote Volume Instance" Form "Create Script Extension" iii

4 iv Form "Create EXACluster Backup Service" Form "EXACluster Backup Service" Form "Create Jdbc Driver" Form "EXACluster Jdbc Drivers" Form "Create EXACluster Route" Form "EXACluster Route Properties" Form "Create EXACluster Vlan" Form "EXACluster Vlan Properties" Form "Create EXACluster Public Vlan" Form "EXACluster Public Vlan Properties" Form "Create EXACluster Ipmi Group" Form "EXACluster Ipmi Group Properties" Form "Create Key Store" Form "Key Store Properties" Form "EXACluster Default Disk Configuration" Form "EXACluster System Properties" Form "EXACluster Update Url" Form "EXACluster Monitor Thresholds" Form "EXACluster Password Properties" XMLRPC interface Fetch log messages Fetch only new log messages Get state of a database Get connection state of a database Get current connection string of a database Get current database nodes Get current operation of a database Start a database Stop a database Backup a Storage database Backup a database Upload and activate an iptables firewall configuration Get current firewall configuration Startup node Shutdown node Get hardware information Get list of cluster nodes Get current EXAoperation main node Get current EXASuite version Get list of archive volumes Get list of databases Get information about database Get list of database backups Get information about backup Get database statistics Start EXAStorage service Stop EXAStorage service Get state of EXAClusterOS services Get IPMI sensor status of a node Get disk state(s) of a node Show list of installed plugins Call plugin function Show plugin functions Hardware Security Modules Disk wipe Hugepages Compatibility and known issues Web browsers... 81

5 EXAoperation 5.0.rc2 Reference Pulling backups Synchronizing backups between EXASuite clusters SWRAID NTP symmetric key exchange Python and XMLRPC Block sizes Enlarging databases Uploading multiple backups at once Remote syslog servers v

6 vi

7 EXAoperation 5.0.rc2 Reference List of Figures 1.1. Show foreign database backups Example view: Form "EXASolution Instances" Example view: Form "EXASolution Instance" Example view: Form "EXAStorage" Example view: Form "EXAStorage Volume Node Information" Example view: Form "EXAStorage Node Information" Example view: Form "EXAStorage Node Device Information" Example view: Form "Cluster Nodes" Example view: Form "EXACluster Scheduler Information" Example view: Form "EXASolution Backup Services Information" Example view: Form "Backups Information" Example view: Form "EXACluster Scheduler Job Information" Example view: Form "Users" Example view: Form "Versions" Example view: Form "JDBC Drivers" Example view: Form "EXACluster Debug Information" Example view: Form "Monitoring Services" Example view: Form "Threshold Values" Example view: Form "Network" Example view: Form "System Passwords" Example view: Form "Licence" DB RAM and hugepages vii

8 viii

9 EXAoperation 5.0.rc2 Reference List of Tables 1.1. EXAoperation permissions... 3 ix

10 x

11 Chapter 1. EXAoperation Chapter 1. EXAoperation 1.1. Components The EXAoperation service is composed of following components: Application server This is the core component which implements the frontend and all backend processes. Configuration database The database is used to store data of the application server. Command execution service This service executes processes which are initiated from the frontend. Unix services For following standard unix services are used for booting and scheduling: DHCP Server XINET Service Cron daemon TFTPd SSH Syslog 1.2. Logging To show the information of the loggingd service, a periodical job is triggered with crond every minute. This job uses the logd_collect command to collect the data and write it to a file. This file is afterwards accessible with the EXAoperation frontend Permissions To manage permissions in EXAoperation you have following predefined roles: 1. Master This role has all possible rights. 2. Administrator As administrator you can manage the cluster, but you can not change the license, password for disk encryption or set a master role to a user. 3. Supervisor 1

12 1.3. Permissions A supervisor has the same rights as administrator without the posibility to change something. It is used to monitor the cluster. 4. User A user has the same rights as supervisor, but can only view the basic state of nodes and databases. In EXAoperation every user can have a different role for any object, so it is possible to have administrator rights on one database and user or supervisor rights on all other objects. 2

13 Chapter 1. EXAoperation Table 1.1. EXAoperation permissions Description View EXACluster password definitions Manager Anonymous Administrator Master Supervisor User Use EXACluster cluster manager operations Change EXACluster public network View EXACluster users. passwords and grants definitions Change EXACluster scheduler Add EXACluster logging service View EXASolution instance Use users view on EXACluster Change EXACluster backup password View EXACluster IPMI group View EXACluster node content Change remote volume instance Use EXACluster node disks view Add EXACluster node disk Edit EXACluster monitor thresholds Add EXASolution instance Change User Management folder content View EXACluster content Use EXACluster scheduler manager operations Add EXACluster JDBC driver View remote volume View EXACluster route Use EXACluster backup services views Use EXACluster operation field View EXACluster backup service Change EXACluster route Change script extension instance Use network view on EXACluster Change EXACluster node disk Use EXASolution manager operations Use EXACluster Software view Add EXACluster VLAN Use EXASolution backup View EXAoperation node priority list Explicitely delete logfiles/coredumps Change EXACluster license Change EXACluster IPMI group Use EXACluster virtual node manager views Use EXACluster JDBC driver view 3

14 1.3. Permissions Description Use EXACluster node manager operations Manager Anonymous Administrator Master Supervisor User Use EXASolutions view on EXACluster View EXACluster logging manager Change EXACluster backup job View EXACluster node Use EXACluster JDBC manager views Add EXACluster scheduler Use EXACluster scheduler manager view Use jobs view on EXACluster View EXACluster versions definitions Use storage view on EXACluster Use EXASolution manager view View EXACluster license Call plugin functions Change EXACluster logging service View EXACluster key store Change EXACluster content Change EXACluster default disk definitions Change EXACluster network definitions Add EXACluster node View User Management folder content Add EXACluster route View script extension View EXACluster scheduler Use EXACluster virtual node manager operations View EXACluster node disk Change EXACluster versions definitions Use software view on EXACluster Add EXACluster user Change EXACluster backup service Change EXACluster storage Use EXACluster cluster manager views Add EXACluster IPMI group Use EXASolution DB backup operations Use EXACluster node manager views Use EXACluster backup manager views Change EXACluster node content Add remote volume instance Download support information Add EXACluster instance 4

15 Chapter 1. EXAoperation Description Add EXACluster backup service Manager Anonymous Administrator Master Supervisor User Change EXACluster node Change EXACluster JDBC driver Change EXACluster key store Change EXASolution instance Use EXACluster node backup Add EXACluster public network Add script extension instance View EXACluster public network View EXACluster logging service View EXACluster scheduler View EXACluster backup job Use EXACluster backup manager operations View EXACluster storage Use EXACluster JDBC manager operations Use EXACluster scheduler backup View EXACluster network definitions Use nodes view on EXACluster Add EXACluster key store Change EXACluster VLAN Change EXACluster user Change EXACluster disk password Manage EXACluster storage Add EXACluster backup job Use logservices view on EXACluster View EXACluster default disk definitions Change EXACluster scheduler View EXACluster VLAN 1.4. Processes Installation If a node is in installation mode its boot process is shown as follows: 1. Boot node over ethernet. 2. Check node parameters. 3. Initialize and format disks. 4. Transfer installation packages. 5. Install software. 5

16 1.4. Processes 6. Configure node parameters. 7. Start all required services Booting On activation of a node its boot process is shown as follows: 1. Boot node over ethernet. 2. Check node parameters. 3. Reinitialize and check disks. 4. Configure node parameters. 5. Start all required services Export of backups via Samba shares Backups of databases are exported via a Samba share. This mechanism can also be used vice versa to import backups into a database. Each client node exports one or more Samba shares for each database. The appropriate share name is compound of the database name and the used data disk. In case the database name equals "testdb" and the data disk is named "/d02_data", the share name would be "testdb_02". The following hints should be kept in mind: After each database backup, the backup files are exported automatically into the appropriate share. The required user name for login is "exasolution". When changing backup passwords, it may take up to one minute to commit this change as this update is done by a Cron job every minute Restore For restoring a database, the database must be created in EXAoperation, but not be started. The steps of the restore process are the following: 1. Check whether enough files are available. This means having a node file for every node number and a metadata file. 2. Copy files from archive nodes to database nodes. 3. Trigger database to read backup files, i.e. start the restore process over EXAoperation. When using offline backups, the backup files must be moved to the archive nodes including an empty 'dontexpire' file. They have to be located into a directory whose name matches the backup name (which is usually a timestamp) The files also have to be located on the nodes where they were created first. If not, the metadata file ("backup.ini") has to be adjusted first Restore (Storage) Storage databases provide three different restore mechanisms: 1. Blocking restore: This restore mechanism loads all data into the database before setting the database into a mode in which it accepts connections. This is the fastest restore mechanism. 6

17 Chapter 1. EXAoperation 2. Nonblocking restore: This mechanism only loads the most necessary part of the data into the database and immediately sets the database into a mode in which it accepts connections. This mechanism is useful for decreasing the downtime of the database, but will load data slightly slower than the blocking restore mechanism. 3. Virtualaccess restore: The mechanism starts a database in a readonly mode. Thus, no write operations are possible. It is useful for restoring only a single object of a database backup into another database via IM PORT/EXPORT. Hint: Remote archive volumes can only be used for blocking restore processes. All other restore types require further functionality that is only available in internal cluster volumes. Thus, a remote backup must be moved to a cluster archive volume first in such a situation. Figure 1.1. Show foreign database backups To restore backups from other databases, use the "Show foreign database backups" button in the "EXASolution Database Backup List" form (see screenshot above). One can restore backups from arbitrary EXASolution databases as long as the number of nodes is similar Renaming of databases The following steps have to be done to rename a database: 1. Create a backup from the database with EXAoperation. You may stop it right afterwards. 2. Create a new database with a different name (and a different communication and connection port if the old database is still running) but with the same number of nodes. 3. Edit the backup properties and insert the new database into "Systems". Afterwards, restore the backup into the new database. 4. Now, the old database may be deleted. Remember the following aspects: 1. After deleting the old database, all backups of this system will be deleted. 2. Change the references from the old database to the new one, especially in the scheduler, monitor and backup view. 3. You may have to change the database name in external tools, e.g. in monitoring tools. 4. The name of the Samba (CIFS) share changes with the new database. 7

18 1.6. Update servers 1.6. Update servers EXAoperation is able to connect to remote FTP/SFTP servers to retrieve update packages. Thus, you may deliver such packages to several EXASuite clusters without being forced to upload it to every single EXAoperation instance. Consider the following file/link hierarchy on a specified remote server: 4.x.2/EXAClusterOS4.x.2_LSUpdateCentOS6.2_x86_64.pkg 4.x.3/EXAClusterOS4.x.2_LSUpdateCentOS6.2_x86_64.pkg >../4.x.2/EXAClusterOS4.x.2_LSUpdateCentOS6.2_x86_64.pkg 4.x.3/EXASolution4.x.3_x86_64.pkg This structure enables an EXASuite cluster in version 4.x.2 to find database version 4.x.3 (see the file link) and to show this version as applicable database version in the appropriate EXAoperation form. It would further enable an EXASuite cluster with a version smaller than 4.x.2 to update to EXASuite version 4.x.2. 1 A patchlevel for version 4.x.2 would have to be located in the 4.x.2 directory Interfaces The following interfaces are available with EXAoperation. EXAoperation frontend HTTP on port 80 and HTTPS on port 443 on all cluster nodes Database backups CIFS on all nodes with backups and virtual machine images (only available if required) Storage archive volumes Ports 2021 (FTP), 2022 (SFTP), 2080 (HTTP), and 2443 (HTTPS) on all cluster nodes Storage archive volumes As noted above, Storage archive volumes can be reached via SFTP, FTP, HTTP, and HTTPS. The username and password are specified in EXAoperation (see 'Users' form). A backup contains the following directory/file structure, given by the following example of a system "testdb", its first level 0 backup with id 0 and two online database nodes: testdb/id_0/level_0/node_0/metadata_{timestamp} testdb/id_0/level_0/node_0/backup_{timestamp} testdb/id_0/level_0/node_1/backup_{timestamp} To store this backup to an offline storage system, all these three files must be downloaded. Restoring this backup from an offline storage system requires all these files to be uploaded into exactly this file structure. Alternatively, you may choose to download this backup in a compressed form. Therefore, you would have to download the virtual file testdb/id_0.tar.gz (which must be uploaded to exactly the same location when restoring from an offline archive) Maintenance user An administrator may specify a dedicated maintenance on installation of a new EXASuite cluster. This user has the ability to see and modify the current EXAoperation status as well as the public network interface configuration 1 Even with a wrong file hierarchy, EXAoperation will be able to detect whether an update is applicable. Every package is signed and will be checked once the update process is in progress. 8

19 Chapter 1. EXAoperation via SSH or a login shell. When not specified (or updating from a version without support for the maintenance user), the root user can add and enable the maintenance user with help of the following commands: 1. adduser u 1100 s $COS_DIRECTORY/sbin/maintain_setup maintenance 2. passwd maintenance The password of the maintenance user can only be changed by the root user in a shell Failover License server failure EXAoperation is able to do a failover in case a license server fails. This has the following consequences: Every cluster node is able to host EXAoperation (not only license servers). The interfaces of EXAoperation (HTTP/HTTPS/XMLRPC) are reachable on all cluster nodes, but only one node has a running instance of EXAoperation (e.g. is able to boot nodes) and thus is the EXAoperation master node. All other nodes act as proxy servers and will redirect requests. In case of a failover, EXAoperation will not be connectable for a short period of time (up to one minute) and a user will experience "Connection refused" messages during this process. The default license server is named as 'n10'. This name should be used for XMLRPC functions such as call Plugin() as the former 'license' name only refers to the current EXAoperation master node. Additional license servers may be named as 'n1' up to 'n9' Power outage and checksum mismatches In case of a power outage, the database may not start because of checksum mismatch errors in EXAStorage. In that case, there are two possible solutions (available in EXAoperation after EXAStorage has been shut down): 1. Fix checksums: All checksums on the selected node(s) are verified against the data and if there is a mismatch, the affected checksum will be regenareted. This operation may take hours to complete (depending on the amount of data that has already been written). 2. Discard checksums: All checksums on the selected node(s) are reset to 0 (as if no data has ever been written). This operation takes only a few minutes, but the checksums are lost and only regenerated as new data is written Automatic node reordering for Storage databases In case of using a Storage database, the database reserve and active nodes will be shuffled to match the volume master nodes (if possible and necessary). Thus, proactively moving a database node of a running database would consist of the following steps: 1. Move appropriate volume node to target node and wait for completion. 2. Stop database. 3. Start database. 9

20 1.11. Volume restore delay Volume restore delay For Storage databases there is a so called restore delay. This delay can be used to specify a timeout after which a Storage data volume moves to the nodes of its database after a failover in case the failed node stays offline during that period of time. The recommended value is 10 minutes Using the EXAoperation browser interface This section provides a brief description of use cases that can be solved with the EXAoperation browser interface. For this reason, each use case has a test that can be executed by an administrator. To start all tests, the following command has to be used: $COS_DIRECTORY/var/exaoperation/inst/bin/test m exaoperation all. Note that this test requires more than one valid running client node to be configured, a logservice (logservice1) that fetches all EXAClusterOS service sources and valid EXASolution binaries. There must be no database and no scheduler or backup service. One more note: The backup password must be set to the default backup password Form "EXASolution Instances" Figure 1.2. Example view: Form "EXASolution Instances" Add a database A database with the configured name does not exist. The newly configured database is shown in the "EXASolution Instances Information" form. 1. Click "Add" button 2. A new form opens. Configure the database and click "Add" once again. Expected message in logservice(s) as regular expression: 1. User \d added system. Delete a database 10

21 Chapter 1. EXAoperation The selected database(s) are not running and there is no logservice or backup that is configured with this database. The selected database(s) will not show up anymore in the "EXASolution Instances Information" form. 1. Select database(s). 2. Click "Delete" button. 3. Answer question with "OK" Expected message in logservice(s) as regular expression: 1. User \d deleted system. Start a database The selected database(s) must exist and not be running. The selected database(s) will be started. 1. Select database(s). 2. Click "Start" button. Expected messages in logservice(s) as regular expressions: 1. User \d requests startup of system. 2. System started successfully in partition \d. Restart a database The selected database(s) must be running. The selected database(s) will be restarted. 1. Select database(s). 2. Click "Restart" button. Expected message in logservice(s) as regular expression: 1. Successfully restarted database \'\w+\' 11

22 1.12. Using the EXAoperation browser interface Stop a database The selected database(s) must be running. The selected database(s) will be stopped. 1. Select database(s). 2. Click "Shutdown" button. Expected message in logservice(s) as regular expression: 1. User \d requests shutdown of system Form "EXASolution Instance" Figure 1.3. Example view: Form "EXASolution Instance" 12

23 Chapter 1. EXAoperation Start database The database must exist and not be running. More than 50% of all cluster nodes and all necessary database nodes must be online. The database will be started. 1. Select operation "Startup". 2. Click "Submit" button. Expected messages in logservice(s) as regular expressions: 1. User \d requests startup of system. 2. System started successfully in partition \d. Restart database The database must be running. The database will be restarted. 1. Select operation "Restart". 2. Click "Submit" button. Expected message in logservice(s) as regular expression: 1. Successfully restarted database \'\w+\' Stop database The database must be running. More than 50% of all cluster nodes must be online. The database will be stopped. 1. Select operation "Shutdown". 2. Click "Submit" button. Expected message in logservice(s) as regular expression: 13

24 1.12. Using the EXAoperation browser interface 1. User \d requests shutdown of system. Create database The database must not be running or in any other operation (e.g. backup). More than 50% of all cluster nodes must be online. Each node that is member of the database system must be online or must have been online before. All database directories will have been created. The DWAd service knows about the database and will start it in create mode next time. Within the next minute, a Samba share for this system will have been exported. 1. Select operation "Create". 2. Click "Submit" button. Expected message in logservice(s) as regular expression: 1. User \d requests new system setup. Start database in maintenance mode The database must exist and not be running. More than 50% of all cluster nodes must be online. The database will be started in maintenance mode. 1. Select operation "Start maintenance". 2. Click "Submit" button. Expected message in logservice(s) as regular expression: 1. User \d requests startup of system with maintenance flag. Cleanup database The database must exist (filesystem based). All temporary used files for this database system will be removed and the operation of the database is "None". 1. Select operation "Cleanup". 2. Click "Submit" button. 14

25 Chapter 1. EXAoperation This action may be necessary after a failed backup. It should never be used during a running backup. Remove obsolete database files All database files (including temporary files) on all nodes not being active, failed or deactivated for this database will be removed. 1. Select operation "Remove obsolete files". 2. Click "Submit" button. Remove database The database must exist on not be running. More than 50% of all cluster nodes must be online. All files of the database (log files, data files and backups) will be removed. 1. Select operation "Delete". 2. Click "Submit" button. Expected messages in logservice(s) as regular expressions: 1. User \d deleted system. 2. User \d added system. Change EXASolution database parameters The database may be in any state if only changing reserve nodes. For all other parameters, the database has to be created. The database has the same state (created, running) as before. 1. Click "Properties" button. 2. Change database parameters appropriately and click "Apply". Enlarge EXASolution database 15

26 1.12. Using the EXAoperation browser interface The database has been created, is not running and has an appropriate number of reserve nodes. The database will have been enlarged and an appropriate number of reserve nodes are active nodes now. The database will have been started. 1. Select operation "Enlarge". 2. Click "Submit" button. 3. A new form opens. Enter the number of new active nodes and click "Apply". Expected messages in logservice(s) as regular expressions: 1. User \d requests to increase system size by one node (*[azaz09_\.]). 2. User \d requests startup of system. After startup, you should explicitely issue the command "REORGANIZE DATABASE" in the database. This assures that data are reorganized in a balanced approach over all database nodes. Get EXASolution statistics The database must be online (connectable). The browser will download a ZIP file containing the database statistics of the last month, which can be sent to EXASOL to provide useful usage graphs over its web portal. 1. Select "Get Statistics". 2. A new form opens. Insert database user and password. 3. Click "Get Statistics". 16

27 Chapter 1. EXAoperation Form "EXAStorage" Figure 1.4. Example view: Form "EXAStorage" Start EXAStorage EXAStorage has not been started yet. 1. Click "Startup storage service" button. Create a new data/archive volume An appropriate volume with a new ID will have been created. 1. Click "Create volume" button. 2. Set volume parameters. 3. Click "Apply" button. The number of volume nodes must be a multiple of the "Master nodes" property. In case of an archive volume, the size will be enhanced properly to match 4 GB boundaries per node and disk. 17

28 1.12. Using the EXAoperation browser interface Enlarge an archive volume The volume has been created and no database backup to this volume is in progress. The volume will be enlarged by at least the number of GiB entered in the form. 1. Click on archive volume. 2. Click "Enlarge SDFS". 3. Enter number of GiB this volume should be enlarged by. Only enlarge an archive volume in case no write operation (database backup) is currently made to it. Thus, check the backup state for all databases on the "EXASolution form" and, in case of no backup, afterwards enlarge the volume. Create a new remote archive volume An appropriate remote volume with a new ID will have been created. 1. Click "Create remote volume" button. 2. Set volume parameters. 3. Click "Apply" button. In contrast to data/archive volumes, remote archive volumes will be named as r0000/r0001/... (instead of v0000/v0001/...). Restore processes from remote archive volumes must be made in blocking mode. Thus, the nonblocking and virtual restore mode is not available. Furthermore, EXAoperation will not delete expired backups from remote volumes (it is under another control). To simplify automatic backup deletion processes on the server side, an expire file will be created. This is placed as {database name}/id_{x}/level_{y}/node_0/expire_{expiration timestamp}, where the expiration timestamp has the format "%Y%m%d%H%M". Remove a volume No database refers to the selected volume as a data volume. 1. Select volume(s). 2. Click "Remove volume" button. Add unused disks for one or more nodes 18

29 Chapter 1. EXAoperation All formerly unused Storage disks of the selected node(s) will have been configured to be usable by the Storage service. The selected node(s) must be online or suspended. 1. Select node(s). 2. Click "Add unused disks" button. Restart Storage service on one or more nodes The Storage service will have been restarted on the selected node(s). 1. Select node(s). 2. Click "Restart service on node" button. Shutdown Storage service on all nodes The Storage service will have been stopped. 1. Click "Shutdown service on all nodes" button Form "EXAStorage Volume Node Information" Figure 1.5. Example view: Form "EXAStorage Volume Node Information" Restore volume on node Volume segments will be restored on this node. 19

30 1.12. Using the EXAoperation browser interface 1. Click "Restore node" button. Stop restore of volume on node Any active restore of volume segments will be stopped on this node. 1. Click "Stop node restore" button Form "EXAStorage Node Information" Figure 1.6. Example view: Form "EXAStorage Node Information" Suspend/Resume node The node will be suspended/online after this action. 1. Click the "Suspend node" or "Resume node" button. Restart Storage service on one node 1. Click "Restart service on node" button. Enable Storage node background recovery 20

31 Chapter 1. EXAoperation 1. Click "Enable background recovery" button. Disable Storage node background recovery 1. Click "Disable background recovery" button. Enable Storage devices 1. Select device(s) which should be activated. 2. Click "Enable devices" button. Disable Storage devices 1. Select device(s) which should be deactivated. 2. Click "Disable devices" button. Remove Storage devices 1. Select device(s) which should be removed. 2. Click "Remove devices" button. Add Storage devices 1. Select device(s) which should be added. 2. Click "Add selected devices" button. 21

32 1.12. Using the EXAoperation browser interface Form "EXAStorage Node Device Information" Figure 1.7. Example view: Form "EXAStorage Node Device Information" Disable Storage device 1. Click "Disable device" button. Enable Storage device 1. Clock "Enable device" button. 22

33 Chapter 1. EXAoperation Form "Cluster Nodes" Figure 1.8. Example view: Form "Cluster Nodes" Upload node list No node of the node list file exists. All nodes of the node list file will be shown in the "EXACluster Nodes Information" form. 23

34 1.12. Using the EXAoperation browser interface 1. Click "Browse" button and select node list file. 2. Click "Submit" button. Add a node A node with the provided number and/or same private/public MAC address(es) does not exist. The newly configured node is shown in the "EXACluster Nodes Information" form and may be installed/booted after switching power on. 1. Click "Add" button. 2. A new form opens. Configure the node and click "Add" once again. If specifying a private and/or public failsafety network interface, network interfaces will be bonded in a activebackup mode fashion. Thus, these interfaces may be connected to two different switches and will always communicate over one active link. If specifying no RAID type on a Storage device, each disk device will contain the size of the disk. Thus, a size of 50 GB on disk devices /dev/sda and /dev/sdb will result in a total size of 100 GB for the usable Storage space. Delete a node The deleted node will not show up again in the "EXACluster Nodes Information" form and is not installed/booted after power on/reboot. 1. Select node(s). 2. Click "Delete" button. Copy node A node with the provided number and/or same private/public MAC address(es) does not exist. The newly configured nodes is shown in the "EXACluster Nodes Information" form and may be installed/booted after switching power on. 1. Click "Add" button. 2. A new form opens. Configure the node and click "Add" once again. 24

35 Chapter 1. EXAoperation 3. Open a newly created node and press "Copy" Button If specifying a private and/or public failsafety network interface, network interfaces will be bonded in a activebackup mode fashion. Thus, these interfaces may be connected to two different switches and will always communicate over one active link. If specifying no RAID type on a Storage device, each disk device will contain the size of the disk. Thus, a size of 50 GB on disk devices /dev/sda and /dev/sdb will result in a total size of 100 GB for the usable Storage space. Change node properties The node must exist. The node properties are changed appropriately. 1. Select node. 2. Click "Properties" button. 3. Change node properties appropriately and click "Apply". Toggle ID LED of a node The selected node(s) has/have a LOM card. The front panel identify light of the selected node(s) will light up for some time. 1. Select node(s). 2. Select operation "Toggle ID LED". 3. Click "Submit" button. Start a node The selected node(s) has/have a LOM card. A "power on" command will be sent to the LOM card of the selected node(s). 1. Select node(s). 2. Select operation "Startup". 25

36 1.12. Using the EXAoperation browser interface 3. Click "Submit" button. Reboot a node The node has been started before and can be reached via SSH. The node will be rebooted. 1. Select node(s). 2. Select operation "Reboot". 3. Click "Submit" button. Stop a node The node has been started before and can be reached via SSH. The node will be stopped. 1. Select node(s). 2. Select operation "Shutdown". 3. Click "Submit" button. Reset a node The selected node(s) has/have a LOM card. A "power reset" command will be sent to the LOM card of the selected node(s). 1. Select node(s). 2. Select operation "Shutdown". 3. Click "Submit" button. 26

37 Chapter 1. EXAoperation Power off a node The selected node(s) has/have a LOM card. A "power off" command will be sent to the LOM card of the selected node(s). 1. Select node(s). 2. Select operation "Power off". 3. Click "Submit" button. Install a node The selected node(s) will be installed during the next boot process. This includes deleting all former data. 1. Select node(s). 2. Select operation "Install". 3. Click "Submit" button. Activate a node The selected node(s) has/have been installed before. The selected node(s) will not be installed during the next boot process. Thus, all database data will remain. 1. Select node(s). 2. Select operation "Install". 3. Click "Submit" button. Force filesystem check on a node The selected node(s) has/have been installed before. The selected node(s) will do a filesystem check during the next boot process. 27

38 1.12. Using the EXAoperation browser interface 1. Select node(s). 2. Select operation "Force filesystem check". 3. Click "Submit" button. Apply default disk layout The selected node(s) must be marked to be installed. The selected node(s) will be configured the the default disk layout after the next installation. 1. Select node(s). 2. Select operation "Apply default disk layout". 3. Click "Submit" button. Start cluster services on a node The cluster node is online and can be used for starting databases. Logs of this node will be shown in monitoring services. 1. Select node(s). 2. Select "Start cluster services" as operation. 3. Click "Submit" button. Cluster services are started automatically on each node as part of the startup procedure. This action is only necessary after having stopped the cluster services of this node. Stop cluster services on a node The cluster node is offline and cannot be used for databases. Logs of this node will not be shown in monitoring services anymore. 1. Select node(s). 2. Select "Stop cluster services" as operation. 3. Click "Submit" button. 28

39 Chapter 1. EXAoperation This operation can be useful in case of a defect node. Thus, a node can be analyzed without the danger of using logs as if doing reboots Form "EXACluster Scheduler Information" Figure 1.9. Example view: Form "EXACluster Scheduler Information" Add a new scheduler The newly added scheduler is shown in the "EXACluster Scheduler Information" form with its name scheduler<num>. 1. Click "Add Scheduler" button. 2. A new form opens. Set "Enabled" field (or not). 3. Click "Add" button. Delete a scheduler 1. Select scheduler. 2. Click "Remove scheduler" button Form "EXASolution Backup Services Information" Figure Example view: Form "EXASolution Backup Services Information" 29

40 1.12. Using the EXAoperation browser interface Create EXACluster backup service At least one database has to exist to apply backups to. The backup service will be shown in the "EXASolution Backup Services Information" form. 1. Click "Add" button. 2. Fill in all appropriate form fields and click "Add" button. Remove EXACluster backup service The selected backup service will not show up anymore in the \"EXASolution Backup Services Information\" form. All backups/backup files that have been made within this backup service will remain. These will be shown again in case of creating a backup service with the same name later on. 1. Select backup service. 2. Click "Delete" button Form "Backups Information" Figure Example view: Form "Backups Information" Start a backup The selected database system has to be started beforehand. A backup process for the selected system will have been started. 30

41 Chapter 1. EXAoperation 1. Click "Start backup" button. Expected messages in logservice(s) as regular expressions: 1. Start backup 20[09][09][09][09][09][09]_[09][09][09][09]_[09][09] on system *[azaz09_\. ]. 2. User 1000 requests system backup. Remove backup The backup and appropriate backup files will have been deleted. 1. Click "Remove backup" button. Edit backup metadata The backup will be shown with the new metadata. 1. Select appropriate backup. 2. Click "Edit backup" button. 3. Change appropriate fields and click "Save" button. Restore a backup The selected database system must be created and not be started. A restore process for the selected system will have been started. 1. Select appropriate backup and system. 2. Click "Start restore" button. Expected messages in logservice(s) as regular expressions: 1. Start restore of backup 20[09][09][09][09][09][09] [09][09][09][09] [09][09] on system *[aza Z09_\.]. 2. User \d requests startup of system in restore mode. 3. System is ready to receive a restore event. 31

42 1.12. Using the EXAoperation browser interface 4. User ([09]*) requests restore of system. Restore offline backup The selected database system must be created and not be started. A restore process for the selected system will have been started. An backup.ini.out file will have been generated in each timestamp directory and contains error messages in case of a failure. 1. Add an empty dontexpire file to offline backup files under each timestamp directory. 2. Copy backup files to target database system backup samba share. 3. Refresh window. The backup should appear now. 4. Click "Start restore" button. Expected messages in logservice(s) as regular expressions: 1. Start restore of backup 20[09][09][09][09][09][09] [09][09][09][09] [09][09] on system *[aza Z09_\.]. 2. User \d requests startup of system in restore mode. 3. System is ready to receive a restore event. 4. User ([09]*) requests restore of system. Show backups of specific system The list of shown backups will show the backups of the selected system. 1. Select system from dropdown menu and click "Select system". 32

43 Chapter 1. EXAoperation Form "EXACluster Scheduler Job Information" Figure Example view: Form "EXACluster Scheduler Job Information" Add a new scheduler job A scheduler and an appropriate database exists. 1. Click "Create" button. 2. Fill form fields and click "Add" button. Remove scheduler job 1. Select scheduler job(s). 2. Click "Remove" button. Enable scheduler job The selected job will not be scheduled anymore. 1. Select scheduler job(s). 2. Click "Enable" button. Disable scheduler job 33

44 1.12. Using the EXAoperation browser interface The selected job will be scheduled. 1. Select scheduler job(s). 2. Click "Disable" button. Enable scheduler The jobs of the selected scheduler will be scheduled. 1. Click "Enable" button. Disable scheduler The jobs of the selected scheduler will not be scheduled anymore. 1. Click "Disable" button. 34

45 Chapter 1. EXAoperation Form "Users" Figure Example view: Form "Users" Add a new user A user with the configured name does not exist. The newly configured user is shown in the "EXACluster Users Information" form and may login with {cluster prefix}.{username} and his appropriate password. 1. Click "Add" button. 2. A new form opens. Configure the user and click "Add" once again. Delete a user The user is not shown anymore in the "EXACluster Users Information" and not able to login. 1. Select user. 35

46 1.12. Using the EXAoperation browser interface 2. Click "Delete" button. Change user properties The user can login with its new credentials. 1. Select user. 2. Click "Edit" button Form "Versions" Figure Example view: Form "Versions" List EXAClusterOS version and available EXASolution version(s) 1. All versions are shown in the "EXACluster Versions Information" form. Install new EXASolution version 36

47 Chapter 1. EXAoperation 1. Click "Browse" button and select EXASolution update file. 2. Click "Submit update file" button. 3. Change versions appropriately for EXASolution systems. Install new EXAClusterOS version All client nodes have to be stopped. 1. Click "Browse" button and select EXAClusterOS update file. 2. Click "Submit update file" button. 3. Wait for EXAoperation to issue a "Please shutdown databases and nodes and restart license server" message. 4. Shutdown databases, Storage and cluster nodes (including additional license servers). 5. Restart license server. 6. Wait until EXAoperation can be reached again via HTTP/HTTPS. 7. Change EXAoperation configuration appropriately, e.g. version of EXASolution instances. 8. Do a final restart of the license server. Remove EXASolution version No database is configured with the specified version. 1. Select EXASolution version. 2. Click "Remove EXASolution version". Upload EXAoperation patchlevel 1. Click "Browse" button and select patchlevel file. 2. Click "Submit update file" button. 3. Restart EXAoperation. Download software update from remote site. An update URL has been defined. 37

48 1.12. Using the EXAoperation browser interface 1. Click "List available updates" button. 2. Choose update. 3. Click "Update" button Form "JDBC Drivers" Figure Example view: Form "JDBC Drivers" Add new JDBC driver. A new JDBC driver has been uploaded and can be used by databases. 1. Click "Add" button. 2. Fill in appropriate driver meta information. 3. Click "Add" button. 4. Select radio button for new driver. 5. Select file to upload. 6. Click "Upload" button. You may upload more than one file for a JDBC driver. Just repeat steps 46 as often as required. 38

49 Chapter 1. EXAoperation Remove JDBC driver files. All driver files for the selected JDBC driver will be removed. 1. Select driver. 2. Click "Cleanup" button. Change JDBC driver properties. All properties will have been changed properly. 1. Select driver. 2. Click "Properties" button. 3. Change properties and apply with "Apply" button. Delete JDBC driver. The selected driver will have been removed and is not usable anymore by any database. 1. Select driver. 2. Click "Delete" button. 39

50 1.12. Using the EXAoperation browser interface Form "EXACluster Debug Information" Figure Example view: Form "EXACluster Debug Information" Download debug information 1. Select debug information to download from. 2. Select nodes to download debug information from. 3. Optionally, estimate the size of uncompressed debug information selected. This will be the maximum file size that you have to download. 4. Click "Download debug information". In case of trying to get Ramdisk logs from offline nodes (e.g. after a wipe of a node), one could choose "Ramdisk logs" and no nodes. This special case results in retrieving all those logs from nodes that are defined in the node view. 40

51 Chapter 1. EXAoperation Form "Monitoring Services" Figure Example view: Form "Monitoring Services" Add new logservice The new logservice will show up in the "EXACluster Logging Information" 1. Click "Add" button. 2. Select lowest priority that logservice should show, EXAClusterOS services and EXASolution systems that should be shown. 3. Click "Add" button. Currently, a user may choose between five different EXAClusterOS services. These are: (1) EXAoperation This service logs general information about the cluster, e.g. boot processes of client nodes. (2) DWAd The DWAd service logs general information about EXASolution systems, like database startup/shutdown. (3) Lockd This base service is necessary for the DWAd. (4) Load Each node in the cluster checks its load every minute. If it is above a defined limit, a warning/error message will be logged. Furthermore, every client node will log its load as an information each minute. (5) Storage Yet unused. The message priorities have the following meaning: (A) Information State confirmation. (B) Notice Some state of the system changed. (C) Warning Something unexpected happened, but this should not affect usability of the system. (D) Error An error occured that needs interventioni from an administrator. Delete logservice The logservice will not show up again in the "EXACluster Logging Information" 1. Select logservice. 2. Click "Delete" button. Show logentries of logservice 1. Select logservice. 2. Click "Entries" button. 41

52 1.12. Using the EXAoperation browser interface 3. Alternatively, just click on the link of the appropriate logservice. Search logentries of logservice 1. Select logservice. 2. Click "Entries" button. 3. Alternatively, just click on the link of the appropriate logservice. Change properties of logservice The logservice will use its new properties. 1. Select logservice. 2. Click "Properties" button. 3. A new form opens. Change logservice properties appropriately and click "Apply" Form "Threshold Values" Figure Example view: Form "Threshold Values" Change monitoring thresholds All thresholds will be changed to appropriate values. 1. Click "Edit". 2. A new form opens. Change values appropriately and click "Apply". 42

53 Chapter 1. EXAoperation Synchronize to NTP server(s) explicitely An ntpdate command will have been issued that synchronizes to the NTP server(s) specified. This is useful in cases where a license server has been left unsynchronized in a way that the NTPd will not change the system time but an explicit time synchronization is necessary. 1. Click "Synchronize time now!" Form "Network" Figure Example view: Form "Network" Add a network/host route Client nodes will take over the new configuration after a reboot/startup and the specified network/host will have been made reachable via the provided gateway. 1. Click "Add route". 2. A new form opens. Insert route properties and click "Add". 43

54 1.12. Using the EXAoperation browser interface Delete a network/host route Client nodes will take over the new configuration after a reboot/startup. 1. Select route(s). 2. Click "Delete route". Change default gateway, network address, NTP server and/or time zone Client nodes will take over the new configuration after a reboot/startup. 1. Click "Properties". 2. A new form opens. Change properties appropriately and click "Apply". Add a VLAN The new VLAN may be selected for database use. Network interfaces for this VLAN may be defined for any node and can be used after the next boot. 1. Click "Add VLAN". 2. Choose descriptive name for VLAN and confirm. Remove a VLAN The selected VLAN must not be in use by any database or node. 1. Select VLAN. 2. Click "Remove VLAN". When deleting VLANs and appropriate node interfaces, these node interfaces are accessible until the next reboot, if those nodes are online. Thus, any started database using all network interfaces will continue to work as before. Add an IPMI card group The new IPMI card group may be used for any existing or new node IPMI card. 44

55 Chapter 1. EXAoperation 1. Click "Add IPMI card group". 2. Insert description, user, and password information for group. Remove an IPMI card group The selected IPMI card group must not be in use by any node. 1. Select IPMI card group. 2. Click "Remove IPMI card group". Restart license server The license server will be rebooted. 1. Click "Restart license server". 2. Answer question with "OK". Stop license server The license server will start a shutdown. 1. Click "Shutdown license server". 2. Answer question with "OK". Move EXAoperation to another node EXAoperation will move to the specified node. 1. Select online node from list. 2. Click "Move EXAoperation to specified node". 3. Answer question with "OK". 45

56 1.12. Using the EXAoperation browser interface Change EXAoperation node priorities EXAoperation will try to move to nodes with higher priority in case it fails on the current main node. 1. Click "EXAoperation node priorities". 2. Move nodes up (higher priority) or down (lower priority). 3. Click "Apply" Form "System Passwords" Figure Example view: Form "System Passwords" Change backup password The backup password will be changed to the appropriate value within the next minute. 1. Insert backup password. 2. Click "Apply" button. 46

57 Chapter 1. EXAoperation Form "Licence" Figure Example view: Form "Licence" Upload license The new license is shown and the sum of memory sizes over all running databases may be less or equal to the allowed database memory. 1. Click "Browse" button and select license file. 2. Click "Upload license" button. 47

58 1.13. EXAoperation Add/Edit Forms EXAoperation Add/Edit Forms This section explains parameters and their meaning of add/edit forms in the EXAoperation browser interface Form "Create EXACluster Node" Disk devices: Disk devices to use on this node. If empty, the cluster Disk RAID: Software RAID type to use for node. RAID 10 Redundancy: Redundancy of RAID 10 (if used). Disk Encryption: Type of encryption to use on data disks. Number: Number of node in cluster. Each node number must be equal and above 10. This parameter can only be set when adding a new node. External Number: This number defines the external IP address. Node Unique Identification: Unique Node Identification in this cluster. Console Redirection: Enable/disable redirection of kernel messages to a TTY instead of the monitor. Spool Disk: Disk to use for spool data (data used for loader processes). MAC Private LAN: MAC address of first (private) LAN interface. MAC Public LAN: MAC address of second (public) LAN interface. MAC Private LAN Failsafety Interface: MAC address of private failsafety interface (if any). MAC Public LAN Failsafety Interface: MAC address of public failsafety interface (if any). MAC SrvMgmt: MAC address of Server Management interface. SrvMgmt Group: Group, that the Server Management Card of this node belongs to (if any). Extra Private Network Interfaces: Further private network interfaces of node. Extra Public Network Interfaces: Further public network interfaces of node. PXE Boot Interface: Interface to use for PXE boot. Install Node: Enable/disable to install node on next boot. Wipe Disks: Enable/disable wipe of disks of node on boot. Wiping may be a very timeconsuming process. Force Filesystem Check: Force filesystem check on next boot of this node. Use 4 KiB Sectors for Disks: Use 4 KiB alignment for hard disks, is required for hard disks without 512 byte sector size emulation and improves I/O performance on regular disks. Enable Virtualization: Enable usage of virtualization features on this node for startup of virtual machines. Label: Label of node, e.g. an ID string that identifies a node in a data center. CPU Scaling Governor: Governor to use for power saving of node. 48

59 Chapter 1. EXAoperation Hugepages (GiB): Amount of hugepages in GiB to use for databases. This is recommended for systems with large amounts of RAM (> 512 GiB) to save process memory and must be smaller than the amount of DB RAM on this node Form "EXACluster Node Properties" Disk devices: Disk devices to use on this node. If empty, the cluster Disk RAID: Software RAID type to use for node. RAID 10 Redundancy: Redundancy of RAID 10 (if used). Disk Encryption: Type of encryption to use on data disks. Number: Number of node in cluster. Each node number must be equal and above 10. This parameter can only be set when adding a new node. External Number: This number defines the external IP address. Node Unique Identification: Unique Node Identification in this cluster. Console Redirection: Enable/disable redirection of kernel messages to a TTY instead of the monitor. Spool Disk: Disk to use for spool data (data used for loader processes). MAC Private LAN: MAC address of first (private) LAN interface. MAC Public LAN: MAC address of second (public) LAN interface. MAC Private LAN Failsafety Interface: MAC address of private failsafety interface (if any). MAC Public LAN Failsafety Interface: MAC address of public failsafety interface (if any). MAC SrvMgmt: MAC address of Server Management interface. SrvMgmt Group: Group, that the Server Management Card of this node belongs to (if any). Extra Private Network Interfaces: Further private network interfaces of node. Extra Public Network Interfaces: Further public network interfaces of node. PXE Boot Interface: Interface to use for PXE boot. Install Node: Enable/disable to install node on next boot. Wipe Disks: Enable/disable wipe of disks of node on boot. Wiping may be a very timeconsuming process. Force Filesystem Check: Force filesystem check on next boot of this node. Use 4 KiB Sectors for Disks: Use 4 KiB alignment for hard disks, is required for hard disks without 512 byte sector size emulation and improves I/O performance on regular disks. Enable Virtualization: Enable usage of virtualization features on this node for startup of virtual machines. Label: Label of node, e.g. an ID string that identifies a node in a data center. CPU Scaling Governor: Governor to use for power saving of node. Hugepages (GiB): Amount of hugepages in GiB to use for databases. This is recommended for systems with large amounts of RAM (> 512 GiB) to save process memory and must be smaller than the amount of DB RAM on this node. 49

60 1.13. EXAoperation Add/Edit Forms Form "EXACluster Node Disk Properties" Disk label: Userspecified label for disk. Size of disk (GiB): Size of disk in GiB. Size of disk (GiB): Size of disk in GiB. Disk number: Number of disk used in the disk name Form "Create EXACluster Virtual Node" Number: Number of node in cluster. Each node number must be equal and above 10. This parameter can only be set when adding a new node. CPUs: Number of CPUs on this node. Memory (GiB): Size of RAM. External IP: This number defines the external IP address. Boot Device: Device to boot virtual machine from. Data Volume: Storage volume to use as disk (otherwise a file will be used). CDROM ISO: CDROM ISO for virtual machine (located under VM share). MAC Public LAN: MAC address of second (public) LAN interface. Label: Label of node, e.g. an ID string identifying a node in a data center. VNC port: Number of VNC port to use. May be left empty otherwise use Public_IP_of_physical node:vnc_port to access the monitor Form "EXACluster Virtual Node Properties" Number: Number of node in cluster. Each node number must be equal and above 10. This parameter can only be set when adding a new node. CPUs: Number of CPUs on this node. Memory (GiB): Size of RAM. External IP: This number defines the external IP address. Boot Device: Device to boot virtual machine from. Data Volume: Storage volume to use as disk (otherwise a file will be used). CDROM ISO: CDROM ISO for virtual machine (located under VM share). MAC Public LAN: MAC address of second (public) LAN interface. Label: Label of node, e.g. an ID string identifying a node in a data center. VNC port: Number of VNC port to use. May be left empty otherwise use Public_IP_of_physical node:vnc_port to access the monitor. 50

61 Chapter 1. EXAoperation Form "Edit EXASolution Instance" Add Reserve Nodes: List of nodes that should be added to the system. Normally, these nodes will be added as reserve nodes, but may be added as active database nodes in case of too many nodes failed and there are no enough active database nodes. Remove Reserve/Failed Nodes: List of nodes that should be removed from the system. Only inactive database nodes can be selected here. Deactivate Database Nodes: List of nodes that should be deactivated for this system. When the database is running, only reserve or failed nodes can be selected here. Reactivate Database Nodes: List of deactivated nodes that should be reactivated for this system. Add Redundancy: Add redundancy to the database system. Comment: User comment for database Storage data volume: Volume for EXASolution database data. Storage volume restore delay: Move failed volume nodes to use reserve nodes automatically after given amount of time, or disable it with no value. Max volume size (GiB): Maximal size of database data volume (in GiB). Network interfaces: List of network interfaces to use for database. Leave empty to use all possible network interfaces. Version: EXASolution Version to use for this system. LDAP Server URLs: LDAP server URL(s) to use for remote database authentication, e.g. ldap:// Multiple servers must be separated by commas. Extra DB parameters: Further EXASolution parameters. Use with care! This can only be set by a user with role "Master". Connection port: Port to use for client connections. Database RAM (GiB): Database RAM consumption of system over all nodes (memory will be shared evenly between nodes). Auditing: Enables/disables auditing for this database system Form "Create EXASolution Instance" Storage data volume: Volume for EXASolution database data. Storage volume restore delay: Move failed volume nodes to use reserve nodes automatically after given amount of time, or disable it with no value. Max volume size (GiB): Maximal size of database data volume (in GiB). Network interfaces: List of network interfaces to use for database. Leave empty to use all possible network interfaces. Version: EXASolution Version to use for this system. LDAP Server URLs: LDAP server URL(s) to use for remote database authentication, e.g. ldap:// Multiple servers must be separated by commas. Extra DB parameters: Further EXASolution parameters. Use with care! This can only be set by a user with role "Master". 51

62 1.13. EXAoperation Add/Edit Forms Connection port: Port to use for client connections. Database RAM (GiB): Database RAM consumption of system over all nodes (memory will be shared evenly between nodes). Auditing: Enables/disables auditing for this database system. DB Name: Name to use for database system. Comment: User comment for database Redundancy: Redundancy for database system. Warning: This name is misleading. In fact, this parameter defines the number of data file copies in the cluster. Thus, if specifying "2", the "real" redundancy is "1" (and one node (and its disk(s)) may fail and may be restarted automatically without data loss). Obsolete option for nonstorage databases. Number of online Nodes: Number of online database nodes. If specifying more nodes than this number, all further specified nodes will be used as reserve nodes. Node List: List of nodes to use for this database system. Disk Name: Logical disk to store data and log files to Form "Create Periodic Backup Job" Enabled: Enables/disables the job. Minutes: Start job at given minute(s). If empty, the job will be started every minute. Hours: Start job at given hour(s). If empty, the job will be started every hour. Days of Week: Start job at given week day(s). Days of Month: Start job at given day(s) of month. Months: Start job at given month(s) (or every month if left empty). Expire Backup After: Expire time for backups created with this job. DB Name: Databases system(s) to apply job for. Disk: Disk that should be used for backup(s). Nodes List: List of nodes for backups created with this job. Redundancy: Redundancy of backup. Warning: This is the number of copies per backup file, not the "real" redundancy Form "Periodic Backup Job" Enabled: Enables/disables the job. Minutes: Start job at given minute(s). If empty, the job will be started every minute. Hours: Start job at given hour(s). If empty, the job will be started every hour. Days of Week: Start job at given week day(s). Days of Month: Start job at given day(s) of month. 52

63 Chapter 1. EXAoperation Months: Start job at given month(s) (or every month if left empty). Expire Backup After: Expire time for backups created with this job. DB Name: Databases system(s) to apply job for. Disk: Disk that should be used for backup(s). Nodes List: List of nodes for backups created with this job. Redundancy: Redundancy of backup. Warning: This is the number of copies per backup file, not the "real" redundancy Form "Create EXACluster Scheduler" Enabled: Enables/disables the scheduler Form "EXACluster Logging Service" Minimum Log Priority: Lowest priority of messages that this logservice will show. E.g. if set to "Warning" it will also show "Error" but neither "Notice" nor "Information". EXAClusterOS Services: Specifies all EXAClusterOS services that will log into this monitor. EXASolution Systems: Specifies all EXASolution systems that will log into this monitor. Remote Syslog Server: Specifies IP address of remote syslog service to which messages of this logservice should be sent to via TCP. Remote Syslog Protocol: Specifies protocol to use to remote syslog (TCP/UDP). Default Time Interval: Default time interval that is used for this logservice. Description: Description of logservice. Can be chosen arbitrarily Form "Create Remote Volume Instance" Archive URL: Remote URL for archive volume. May be of type FTP/HTTP/HTTPS/SMB. URLs are specified like "ftp:// :12345" or "smb://// /backupshare". User: Name of user for accessing remote volume (if any). Password: Password for accessing remote volume (if any). Allowed Users: Users that can access this volume. Readonly Users: Users that have readonly access to this volume. Labels: Labels for remote volume. Options: Options for remote volume: (1) noverifypeer (do not check server certificate), (2) nocompression (write plain data), (3) forcessl (use STARTTLS in FTP connection), (4) webdav (use WebDAV for httpurl) Form "Create Script Extension" Name: Name for package, e.g. 3to2 53

64 1.13. EXAoperation Add/Edit Forms Script URL: URL for script installation, e.g. Description: Description of script HTTP proxy: URL of HTTP proxy (if any) HTTPS proxy: URL of HTTPS proxy (if any) Form "Create EXACluster Backup Service" Disk: Disk to use for backups created with this backup service. Systems: Systems that may be used for this backup service. Default Expire: Default expire time for backups created with this backup service. Nodes List: Nodes to use for backups created with this backup service. Redundancy: Redundancy of backup. Warning: This parameter is misleading. It does specify the number of backup file copies in the system. Thus, if specifying "2", there will be a "real" redundancy of "1" Form "EXACluster Backup Service" Disk: Disk to use for backups created with this backup service. Systems: Systems that may be used for this backup service. Default Expire: Default expire time for backups created with this backup service. Nodes List: Nodes to use for backups created with this backup service. Redundancy: Redundancy of backup. Warning: This parameter is misleading. It does specify the number of backup file copies in the system. Thus, if specifying "2", there will be a "real" redundancy of "1" Form "Create Jdbc Driver" Driver Name: Name of driver. JAR Files: List of JAR files, which should be used as JDBC Driver. Should be given as list of URLs. Main Class: Main class name of the JDBC Driver. Prefix: Prefix of the JDBC name, must begin with "jdbc:" and ends with ":", like in "jdbc:mysql:". Comment: Description of the driver (only for user) Form "EXACluster Jdbc Drivers" Driver Name: Name of driver. JAR Files: List of JAR files, which should be used as JDBC Driver. Should be given as list of URLs. Main Class: Main class name of the JDBC Driver. Prefix: Prefix of the JDBC name, must begin with "jdbc:" and ends with ":", like in "jdbc:mysql:". Comment: Description of the driver (only for user). 54

65 Chapter 1. EXAoperation Form "Create EXACluster Route" Type: Type of route (network or host). Destination: Network/host into which define a route. Gateway: Gateway to use for sending packets into given destination Form "EXACluster Route Properties" Type: Type of route (network or host). Destination: Network/host into which define a route. Gateway: Gateway to use for sending packets into given destination Form "Create EXACluster Vlan" Description: Description for VLAN. MTU: MTU to use in this VLAN (default, 1500, 9000) Form "EXACluster Vlan Properties" Description: Description for VLAN. MTU: MTU to use in this VLAN (default, 1500, 9000) Form "Create EXACluster Public Vlan" Description: Description for public network MTU: MTU to use in this public network (default, 1500, 9000). Network address: Network address, e.g / Form "EXACluster Public Vlan Properties" Description: Description for public network MTU: MTU to use in this public network (default, 1500, 9000). Network address: Network address, e.g / Form "Create EXACluster Ipmi Group" Description: Description for IPMI group. IPMI Type: Type of IPMI card(s) in this group. IPMI Username: Name of user to use for accessing IPMI card(s). IPMI Password: Password of IPMI card(s). 55

66 1.13. EXAoperation Add/Edit Forms IPMI Multiline Password: Multiline password for IPMI card(s). May be useful if using SSH keys. Public IP Addresses: If set, IP addresses are only reachable in the public net and no DHCP is used Form "EXACluster Ipmi Group Properties" Description: Description for IPMI group. IPMI Type: Type of IPMI card(s) in this group. IPMI Username: Name of user to use for accessing IPMI card(s). IPMI Password: Password of IPMI card(s). IPMI Multiline Password: Multiline password for IPMI card(s). May be useful if using SSH keys. Public IP Addresses: If set, IP addresses are only reachable in the public net and no DHCP is used Form "Create Key Store" Description: Description for key store. Type: Type of key store. Attributes: Attributes for key store Form "Key Store Properties" Description: Description for key store. Type: Type of key store. Attributes: Attributes for key store Form "EXACluster Default Disk Configuration" Device: List of disk devices to use on each node per default. Default RAID: Type of software RAID to use on disks per default. Default RAID 10 Redundancy: Software RAID 10 redundancy on each node per default (if used). Default Data Encryption: Default data encryption to use for data disks. Default OS Size (GiB): Default size of OS disk on each node. Default Swap Size (GiB): Default size of swap disk on each node. Default Data Disk Size (GiB): Size of default disk reserved for EXAStorage service Form "EXACluster System Properties" Cluster Name: Name of cluster. Can be chosen arbitrarily. Public Network: Network and appropriate network mask for external network interfaces of client nodes. Example: /24 56

67 Chapter 1. EXAoperation Gateway: Gateway to use for external network interfaces of client nodes. NTP Server 1: IP address of first NTP server to use for time synchronization. NTP Server 2: IP address of second NTP server to use for time synchronization. NTP Key: Key for NTP server (consisting of Key ID and Key [space separated]) DNS Server 1: IP address of first DNS server. DNS Server 2: IP address of second DNS server. Search Domain: Search domain to use with DNS servers. Backup Network Bandwidth per Node (MiB/s): Maximum bandwidth (MiB/s) a backup job is able to transfer backups from one node to another at once. OS Memory per Node (GiB): Memory that must not be used by EXASolution on each node. This value should be 2 for databases consuming up to 36 GiB/node, 4 for databases up to 72 GiB/node, 8 for databases up to 144 GiB, else 16. Time Zone: Time zone to use for cluster. Private Network MTU: MTU to use for private network Public Network MTU: MTU to use for public network Form "EXACluster Update Url" URL: URL at which to check updates. User: User name for accessing software update URL. Password: Password for accessing software update URL Form "EXACluster Monitor Thresholds" Warning Level for Disk Usage (in %): Level upon which warnings will be issued about disk usage. Error Level for Disk Usage (in %): Level upon which errors will be issued about disk usage. Warning Level for Swap Usage (in %): Level upon which warnings will be issued about swap usage. Error Level for Swap Usage (in %): Level upon which errors will be issued about swap usage. Warning Level for Load: Level upon which warnings will be issued about load. Error Level for Load: Level upon which errors will be issued about load. Coredump Deletion Time (in days): Number of days after which coredumps will be deleted Form "EXACluster Password Properties" Samba Shares Password: Password for backup samba shares. VM Shares Password: Password for VM shares. Disk Password: Password for disk(s). When changed, all nodes must be reinstalled. 57

68 1.14. XMLRPC interface XMLRPC interface EXAoperation provides an XMLRPC interface that may be used e.g. for automatically executed scripts Fetch log messages Function name: logentries() Parameter(s): 1. start (optional) of type (year, month, day, hour, minute, second, 0): Start time for log entries. 2. halt (optional) of type (year, month, day, hour, minute, second, 0): Stop time for log entries. Result type: [start, stop, [log_entry1, log_entry2,...]] An appropriate logservice exists in EXAoperation. Perl example: use Frontier::Client; my $server = Frontier::Client>new('url' => ' my $result = $server>call('logentries'); my $result = $server>call('logentries', ['2009', '10', '2', '0', '0', '0', '0'], ['2009', '10', '2', '17', '0', '0', '0']); Python example: import xmlrpclib s = xmlrpclib.serverproxy(" s.logentries() s.logentries([2009, 01, 01, 0, 0, 0, 0], [2009, 01, 01, 12, 0, 0, 0]) Python example output: >>> pprint.pprint(s.logentries([2010, 9, 29, 10, 59, 47, 0], [2010, 9, 29, 11, 00, 47, 0])) [[2010, 9, 29, 10, 59, 47, 0], [2010, 9, 29, 11, 0, 47, 0], [{'message': 'n0011.c0001.exacluster.local ', 'node': '1', 'priority': 'Information', 'strtime': ' :00: :00', 'system': 'load', 'timestamp': ' :00: '}, {'message': 'n0013.c0001.exacluster.local ', 'node': '4', 'priority': 'Information', 'strtime': ' :00: :00', 'system': 'load', 'timestamp': ' :00: '}, {'message': 'n0014.c0001.exacluster.local ', 'node': '3', 58

69 Chapter 1. EXAoperation 'priority': 'Information', 'strtime': ' :00: :00', 'system': 'load', 'timestamp': ' :00: '}, {'message': 'n0012.c0001.exacluster.local ', 'node': '2', 'priority': 'Information', 'strtime': ' :00: :00', 'system': 'load', 'timestamp': ' :00: '}]] Fetch only new log messages Function name: logentriestagged() Parameter(s): 1. tag of type string or number: Tag for identifying source. Result type: [start, stop, [log_entry1, log_entry2,...]] An appropriate logservice exists in EXAoperation. Perl example: use Frontier::Client; my $server = Frontier::Client>new('url' => ' my $result = $server>call('logentriestagged', 'my source id'); Python example: import xmlrpclib s = xmlrpclib.serverproxy(" s.logentriestagged(3) s.logentriestagged('my source id') Python example output: >>> pprint.pprint(s.logentriestagged('my source id')) [[2010, 9, 29, 10, 59, 47, 0], [2010, 9, 29, 11, 0, 47, 0], [{'message': 'n0011.c0001.exacluster.local ', 'node': '1', 'priority': 'Information', 'strtime': ' :00: :00', 'system': 'load', 'timestamp': ' :00: '}, {'message': 'n0013.c0001.exacluster.local ', 'node': '4', 'priority': 'Information', 'strtime': ' :00: :00', 59

70 1.14. XMLRPC interface 'system': 'load', 'timestamp': ' :00: '}, {'message': 'n0014.c0001.exacluster.local ', 'node': '3', 'priority': 'Information', 'strtime': ' :00: :00', 'system': 'load', 'timestamp': ' :00: '}, {'message': 'n0012.c0001.exacluster.local ', 'node': '2', 'priority': 'Information', 'strtime': ' :00: :00', 'system': 'load', 'timestamp': ' :00: '}]] Get state of a database Function name: getdatabasestate() Result type: string in ('none', 'setup', 'starting', 'running', 'shutdown') Perl example: use Frontier::Client; my $server = Frontier::Client>new('url' => ' my $result = $server>call('getdatabasestate'); Python example: import xmlrpclib s = xmlrpclib.serverproxy(" s.getdatabasestate() Python example output: >>> s.getdatabasestate() 'running' >>> s.getdatabasestate() 'shutdown' Get connection state of a database Function name: getdatabaseconnectionstate() Result type: string in ('', 'No') Perl example: 60

71 Chapter 1. EXAoperation use Frontier::Client; my $server = Frontier::Client>new('url' => ' my $result = $server>call('getdatabaseconnectionstate'); Python example: import xmlrpclib s = xmlrpclib.serverproxy(" s.getdatabaseconnectionstate() Python example output: >>> s.getdatabaseconnectionstate() 'No' >>> s.getdatabaseconnectionstate() '' Get current connection string of a database Function name: getdatabaseconnectionstring() Result type: connection string Perl example: use Frontier::Client; my $server = Frontier::Client>new('url' => ' my $result = $server>call('getdatabaseconnectionstring'); Python example: import xmlrpclib s = xmlrpclib.serverproxy(" s.getdatabaseconnectionstring() Python example output: >>> s.getdatabaseconnectionstring() ' :8563' Get current database nodes Function name: getdatabasenodes() Result type: Lists of active, reserve and failed nodes of database Perl example: 61

72 1.14. XMLRPC interface use Frontier::Client; my $server = Frontier::Client>new('url' => ' my $result = $server>call('getdatabasenodes'); Python example: import xmlrpclib s = xmlrpclib.serverproxy(" s.getdatabasenodes() Python example output: >>> s.getdatabasenodes() {'active': ['n0011', 'n0012', 'n0013'], 'failed': [], 'reserve': ['n0014']} Get current operation of a database Function name: getdatabaseoperation() Result type: string in ('None', 'Create', 'Remove', 'Startup', 'Shutdown', 'Cleanup', 'Backup', 'Restore', 'Failed') Perl example: use Frontier::Client; my $server = Frontier::Client>new('url' => ' my $result = $server>call('getdatabaseoperation'); Python example: import xmlrpclib s = xmlrpclib.serverproxy(" s.getdatabaseoperation() Python example output: >>> s.getdatabaseoperation() 'None' >>> s.getdatabaseoperation() 'Backup' Start a database Function name: startdatabase() Result type: This function does not return a result. In case something goes wrong, an exception will be raised. 62

73 Chapter 1. EXAoperation The database is already created and not running. Expected messages in logservice(s) as regular expression(s): 1. User \d requests startup of system. 2. System started successfully in partition \d. Perl example: use Frontier::Client; my $server = Frontier::Client>new('url' => ' $server>call('startdatabase'); Python example: import xmlrpclib s = xmlrpclib.serverproxy(" s.startdatabase() Stop a database Function name: stopdatabase() Result type: This function does not return a result. In case something goes wrong, an exception will be raised. The database must be running. The database will be stopped. Expected messages in logservice(s) as regular expression(s): 1. User \d requests shutdown of system. Perl example: use Frontier::Client; my $server = Frontier::Client>new('url' => ' $server>call('stopdatabase'); Python example: import xmlrpclib s = xmlrpclib.serverproxy(" s.stopdatabase() Backup a Storage database Function name: 63

74 1.14. XMLRPC interface backupstoragedatabase() Parameter(s): 1. volume of type string: Volume to backup data into (e.g. 'v0000' or 'r0010') 2. level of type number: Level of backup 3. expire of type string: Expire time of backup (e.g. '1w 3d') The database must be running. The archive volume must be configured and (if level is larger than 0) a base backup must exist in the archive. The backup process will be started. Perl example: use Frontier::Client; my $server = Frontier::Client>new('url' => ' $server>call('startstoragebackup', 'v0000', 2, '3d'); Python example: import xmlrpxlic s = xmlrpclib.serverproxy(" s.startstoragebackup('v0000', 2, '3d') Backup a database Function name: startbackup() Parameter(s): 1. database_name of type string: Name of database to backup from. Result type: This function does not return a result. In case something goes wrong, an exception will be raised. The database must be running and a backup service must be configured. The backup process will be started. Expected messages in logservice(s) as regular expression(s): 1. Start backup 20[09][09][09][09][09][09]_[09][09][09][09]_[09][09] on system *[azaz09_\. ]. 2. User 1000 requests system backup. 64

75 Chapter 1. EXAoperation Perl example: use Frontier::Client; my $server = Frontier::Client>new('url' => ' $server>call('startbackup', 'exa_db1'); Python example: import xmlrpclib s = xmlrpclib.serverproxy(" s.startbackup('exa_db1') Upload and activate an iptables firewall configuration. Function name: submitfirewallconfiguration() Result type: This function returns 'OK' or an error code. Python example: import xmlrpclib s = xmlrpclib.serverproxy(" s.submitfirewallconfiguration('n11', file('fw.conf').read()) Python example output: >>> s.submitfirewallconfiguration('n11', file('fw.conf').read()) 'OK' Get current firewall configuration. Function name: getfirewallconfiguration() Result type: IPTables configuration file of current firewall configuration. Python example: import xmlrpclib s = xmlrpclib.serverproxy(" s.getfirewallconfiguration('n11') Python example output: >>> print s.getfirewallconfiguration('n11') *filter :INPUT DROP [0:0] :FORWARD DROP [0:0] :OUTPUT DROP [477:28620] 65

76 1.14. XMLRPC interface A INPUT m state state RELATED,ESTABLISHED j ACCEPT A INPUT i eth0 m state state NEW j ACCEPT A INPUT i eth1 p tcp m tcp dport 8563 m state state NEW j ACCEPT A INPUT i eth1 j DROP A FORWARD m state state RELATED,ESTABLISHED j ACCEPT A FORWARD i eth0 m state state NEW j ACCEPT A FORWARD o eth0 m state state NEW j ACCEPT A FORWARD i eth1 p tcp m tcp dport 8563 m state state NEW j ACCEPT A FORWARD i eth1 j DROP A OUTPUT m state state RELATED,ESTABLISHED j ACCEPT A OUTPUT o eth0 m state state NEW j ACCEPT COMMIT Startup node Function name: startupnode() Result type: This function returns nothing on success and an exception on error. Python example: import xmlrpclib s = xmlrpclib.serverproxy(" s.startupnode("n0011") Shutdown node Function name: shutdownnode() Result type: This function returns nothing on success and an exception on error. Python example: import xmlrpclib s = xmlrpclib.serverproxy(" s.shutdownnode("n0011") Get hardware information Function name: gethardwareinformation() Result type: This function returns the output of dmidecode on the specified node. Python example: 66

77 Chapter 1. EXAoperation import xmlrpclib s = xmlrpclib.serverproxy(" s.gethardwareinformation("n11", 1) Python example output: >>> import xmlrpclib >>> s = xmlrpclib.serverproxy(" >>> # Get dmidecode information of type 0 ("BIOS"). See man dmidecode for details. >>> print s.gethardwareinformation("n11", 0) # dmidecode 2.11 SMBIOS 2.4 present. Handle 0x0000, DMI type 0, 24 bytes BIOS Information Vendor: Bochs Version: Bochs Release Date: 01/01/2011 Address: 0xE8000 Runtime Size: 96 kb ROM Size: 64 kb Characteristics: BIOS characteristics not supported Targeted content distribution is supported BIOS Revision: Get list of cluster nodes Function name: getnodelist() Result type: This function returns a list of all defined cluster nodes. Perl example: use Frontier::Client; my $server = Frontier::Client>new('url' => ' my $result = $server>call('getnodelist'); Python example: import xmlrpclib s = xmlrpclib.serverproxy(" s.getnodelist() Python example output: >>> s.getnodelist() ['n0011', 'n0012', 'n0013', 'n0014'] Get current EXAoperation main node Function name: 67

78 1.14. XMLRPC interface getexaoperationmaster() Result type: This function returns a string with the name of the node currently serving EXAoperation. Perl example: use Frontier::Client; my $server = Frontier::Client>new('url' => ' my $result = $server>call('getexaoperationmaster'); Python example: import xmlrpclib s = xmlrpclib.serverproxy(" s.getexaoperationmaster() Python example output: >>> s.getexaoperationmaster() 'n0010' Get current EXASuite version Function name: getexasuiteversion() Result type: This function returns the version of the EXASuite version currently installed. Perl example: use Frontier::Client; my $server = Frontier::Client>new('url' => ' my $result = $server>call('getexasuiteversion'); Python example: import xmlrpclib s = xmlrpclib.serverproxy(" s.getexasuiteversion() Python example output: >>> s.getexasuiteversion() '5.0.0' Get list of archive volumes Function name: getarchivefilesystems() Result type: 68

79 Chapter 1. EXAoperation This function returns a list of archive volumes that can be used by the calling user. Python example: import xmlrpclib s = xmlrpclib.serverproxy(" s.getarchivefilesystems() Python example output: >>> s.getarchivefilesystems() {'v0002': ['volume', 2, ['read', 'write']]} Get list of databases Function name: getdatabaselist() Result type: This function returns a list of all defined databases. Perl example: use Frontier::Client; my $server = Frontier::Client>new('url' => ' my $result = $server>call('getdatabaselist'); Python example: import xmlrpclib s = xmlrpclib.serverproxy(" s.getdatabaselist() Python example output: >>> s.getdatabaselist() ['exa_db1', 'testdb'] Get information about database Function name: getdatabaseinfo() Result type: This function returns a dictionary of keyvalue pairs with information about the specified database. Perl example: use Frontier::Client; my $server = Frontier::Client>new('url' => ' my $result = $server>call('getdatabaseinfo'); Python example: 69

80 1.14. XMLRPC interface import xmlrpclib s = xmlrpclib.serverproxy(" s.getdatabaseinfo() Python example output: >>> pprint.pprint(s.getdatabaseinfo()) {'connectible': 'No', 'connection string': ' :8563', 'name': 'exa_db1', 'nodes': {'active': ['n0011', 'n0012'], 'failed': [], 'reserve': []}, 'operation': 'None', 'state': 'setup'} Get list of database backups Function name: getbackuplist() Result type: This function returns a list of backup IDs for the specified database. Python example: import xmlrpclib s = xmlrpclib.serverproxy(" s.getbackuplist() Python example output: >>> # NonStorage example >>> s.getbackuplist() [' ', ' '] >>> # Storage example >>> s.getbackuplist() [[' :29', 1], [' :29', 2], [' :30', 3], [' :30', 4]] >>> s.getbackupinfo(' :29')... >>> s.getbackupinfo(1)... >>> s.getbackupinfo((1, 'v0001')) Get information about backup Function name: getbackupinfo() Result type: This functions returns a dictionary with keyvalue pairs describing the specified backup. Perl example: 70

81 Chapter 1. EXAoperation use Frontier::Client; my $server = Frontier::Client>new('url' => ' my $result = $server>call('getbackupinfo', ' '); Python example: import xmlrpclib s = xmlrpclib.serverproxy(" s.getbackupinfo(' ') Python example output: >>> # NonStorage example >>> pprint.pprint(s.getbackupinfo(' ')) {'details': ['OK'], 'expire': 'n/a', 'expire date': 'n/a', 'files': {'n0011': ['/d03_data/exa_db2/backups/ _1034_00/backup.ini', '/d03_data/exa_db2/backups/ _1034_00/node0', '/d03_data/exa_db2/backups/ _1034_00/node2'], 'n0012': ['/d03_data/exa_db2/backups/ _1034_00/backup.ini', '/d03_data/exa_db2/backups/ _1034_00/node0', '/d03_data/exa_db2/backups/ _1034_00/node2'], 'n0013': ['/d03_data/exa_db2/backups/ _1034_00/backup.ini', '/d03_data/exa_db2/backups/ _1034_00/node1'], 'n0014': ['/d03_data/exa_db2/backups/ _1034_00/backup.ini', '/d03_data/exa_db2/backups/ _1034_00/node1']}, 'note': 'Backup from system exa_db2', 'offline media': '', 'public_ips': {'n0011': ' ', 'n0012': ' ', 'n0013': ' ', 'n0014': ' '}, 'size': {'n0011.c0001.exacluster.local': , 'n0012.c0001.exacluster.local': , 'n0013.c0001.exacluster.local': , 'n0014.c0001.exacluster.local': }, 'state': 100} >>> # Storage example >>> pprint.pprint(s.getbackupinfo(' :29')) {'comment': '', 'dependencies': [], 'expire date': ' :29', 'files': ['exa_db1/id_1/level_0/node_2/backup_ ', 'exa_db1/id_1/level_0/node_1/backup_ ', 'exa_db1/id_1/level_0/node_0/backup_ ', 'exa_db1/id_1/level_0/node_0/metadata_ '], 'id': 1, 'level': 0, 'nodes': 3, 'size': , 'system': 'exa_db1', 'timestamp': ' :29', 'usable': True, 'usage': , 'volume': ['v0001', 1]} 71

82 1.14. XMLRPC interface >>> s.getbackupinfo(1)... >>> s.getbackupinfo((1, 'v0001')) Get database statistics Function name: getdatabasestatistics() Result type: This functions returns a base64 encoded zip file containing database statistics for the last month. This can be used by customer service to provide useful usage graphs etc. Perl example: use Frontier::Client; my $server = Frontier::Client>new('url' => ' my $result = $server>call('getdatabasestatistics', 'user', 'pass'); Python example: import xmlrpclib s = xmlrpclib.serverproxy(" s.getdatabasestatistics('dbuser', 'pass') s.getdatabasestatistics('dbuser', 'pass', '201301', '201306') Python example output: >>> import xmlrpclib, base64 >>> s = xmlrpclib.serverproxy(" >>> zipfile = base64.b64decode(s.getdatabasestatistics('dbuser', 'dbpassword')) >>> file("statistics.zip", "w").write(zipfile) Start EXAStorage service Function name: startexastorage() Result type: This function returns 'OK' or an exception in case of a failure. Perl example: use Frontier::Client; my $server = Frontier::Client>new('url' => ' my $result = $server>call('startexastorage'); Python example: 72

83 Chapter 1. EXAoperation import xmlrpclib s = xmlrpclib.serverproxy(" s.startexastorage() Python example output: >>> s.startexastorage() 'OK' Stop EXAStorage service Function name: stopexastorage() Result type: This function returns 'OK' or an exception in case of a failure. Perl example: use Frontier::Client; my $server = Frontier::Client>new('url' => ' my $result = $server>call('stopexastorage'); Python example: import xmlrpclib s = xmlrpclib.serverproxy(" s.stopexastorage() Python example output: >>> s.stopexastorage() 'OK' Get state of EXAClusterOS services Function name: getservicestate() Result type: This function returns a list of tuples, each tuple consisting of a service name and the appropriate service state. The service state is described with 'OK', 'not running' or (for DWAd) 'DWAd has no quorum'. Perl example: use Frontier::Client; my $server = Frontier::Client>new('url' => ' my $result = $server>call('getservicestate'); Python example: 73

84 1.14. XMLRPC interface import xmlrpclib s = xmlrpclib.serverproxy(" s.getservicestate() Python example output: >>> s.getservicestate() [['Loggingd', 'OK'], ['Lockd', 'OK'], ['Storaged', 'OK'], ['DWAd', 'OK']] Get IPMI sensor status of a node Function name: getipmisensorstatus() Result type: This function returns a list of IPMI keyvalue pairs with a value classification depending on the underlying IPMI card. Perl example: use Frontier::Client; my $server = Frontier::Client>new('url' => ' my $result = $server>call('getipmisensorstatus'); Python example: import xmlrpclib s = xmlrpclib.serverproxy(" s.getipmisensorstatus() Python example output: >>> pprint.pprint(s.getipmisensorstatus()) [['CPU Temp 1', '47 degrees C', 'ok'], ['CPU Temp 2', '47 degrees C', 'ok'], ['CPU Temp 3', 'no reading', 'ns'], ['CPU Temp 4', 'no reading', 'ns'], ['Sys Temp', '29 degrees C', 'ok'], ['CPU1 Vcore', '1.14 Volts', 'ok'], ['CPU2 Vcore', '1.14 Volts', 'ok'], ['3.3V', '3.33 Volts', 'ok'], ['5V', '4.99 Volts', 'ok'], ['12V', '12 Volts', 'ok'], ['12V', '12.10 Volts', 'ok'], ['1.5V', '1.49 Volts', 'ok'], ['5VSB', '4.94 Volts', 'ok'], ['VBAT', '3.23 Volts', 'ok'], ['Fan1', '0 RPM', 'nr'], ['Fan2', '6300 RPM', 'ok'], ['Fan3', '6400 RPM', 'ok'], ['Fan4', '6100 RPM', 'ok'], ['Fan5', '0 RPM', 'nr'], ['Fan6', '0 RPM', 'nr'], ['Fan7/CPU1', '0 RPM', 'nr'], ['Fan8/CPU2', '0 RPM', 'nr'], 74

85 Chapter 1. EXAoperation ['Intrusion', '0 unspecified', 'nc'], ['Power Supply', '0 unspecified', 'ok'], ['CPU0 Internal E', '0 unspecified', 'ok'], ['CPU1 Internal E', '0 unspecified', 'ok'], ['CPU Overheat', '0 unspecified', 'ok'], ['Thermal Trip0', '0 unspecified', 'ok'], ['Thermal Trip1', '0 unspecified', 'ok']] Get disk state(s) of a node Function name: getdiskstates() Result type: This function returns a list of dictionaries (Python)/hashes (Perl), whereas each entry describes one disk of the specified node. Perl example: use Frontier::Client; my $server = Frontier::Client>new('url' => ' my $result = $server>call('getdiskstates'); Python example: import xmlrpclib s = xmlrpclib.serverproxy(" s.getdiskstates() Python example output: >>> pprint.pprint(s.getdiskstates()) [{'devices': 'Default', 'encr': 'diskencraes256', 'mount_count': '2/28', 'name': 'd00_os', 'next_fsck': 'Mon Mar 28 08:51: ', 'raid': 'diskraidnone', 'size': '50', 'state': 'None', 'type': 'disktypeos', 'free': '41.8'}, {'devices': 'Default', 'encr': 'diskencrnone', 'mount_count': '', 'name': 'd01_swap', 'next_fsck': '', 'raid': 'diskraidnone', 'size': '4', 'state': 'None', 'type': 'disktypeswap', 'free': '3.8'}, {'devices': 'Default', 'encr': 'diskencrnone', 'mount_count': '2/27', 75

86 1.14. XMLRPC interface 'name': 'd02_data', 'next_fsck': 'Mon Mar 28 08:52: ', 'raid': 'diskraidnone', 'size': '47', 'state': 'None', 'type': 'disktypedata', 'free': '45.2'}] Output details: The state of a disk can vary between 'Online', 'Offline', and 'Degraded' for software RAIDs. Hardware RAID systems may use proprietary interfaces to retrieve the current operation. Thus, such disks will show up with state 'None'. The size of a disk is always shown in GiB (as usage is, too) or declared as 'Rest' (partition grows with disk size). The used devices for a disk may be set to 'Default' (all devices of a node are used) or it is a list of devices, e.g. "['/dev/sda', '/dev/sdb']" Show list of installed plugins Function name: showpluginlist() Result type: List of installed plugins Perl example: use Frontier::Client; my $server = Frontier::Client>new('url' => ' my $funcs = $server>call('showpluginlist'); Python example: import xmlrpclib s = xmlrpclib.serverproxy(" funcs = s.showpluginlist() Python example output: >>> s.showpluginlist() ['RAID.tw_cli10.1', 'RAID.arcconf6.30'] Call plugin function Function name: callplugin() Parameter(s): 1. plugin of type string: Name of plugin to call. 2. node of type string: Name of node on which to call plugin. 3. command of type string: Command to call. 76

87 Chapter 1. EXAoperation 4. extra parameters (optional) of type string: Extra parameters for command. Result type: [return code, output] The appropriate plugin is installed in the cluster. Perl example: use Frontier::Client; my $server = Frontier::Client>new('url' => ' my $result = $server>call('callplugin', 'RAID.tool', 'n11', 'SHOW_LOGS'); Python example: import xmlrpclib s = xmlrpclib.serverproxy(" ret, output = s.callplugin('raid.tool', 'n11', 'SHOW_LOGS') Python example output: >>> status, result = s.callplugin('raid.arcconf6.30', 'license', 'SHOW', '1') >>> print result Controllers found: 1 Controller information Controller Status : Optimal Channel description : SAS/SATA Controller Model : Adaptec 5805Z Controller Serial Number : 0A Physical Slot : 4 Temperature : 49 C/ 120 F (Normal) Installed memory : 512 MB Copyback : Disabled Background consistency check : Disabled Automatic Failover : Enabled Global task priority : High Performance Mode : Default/Dynamic Stayawake period : Disabled Spinup limit internal drives : 0 Spinup limit external drives : 0 Defunct disk drive count : 0 Logical devices/failed/degraded : 1/0/0 SSDs assigned to MaxIQ Cache pool : 0 Maximum SSDs allowed in MaxIQ Cache pool : 8 NCQ status : Enabled Controller Version Information BIOS : 5.20 (17544) Firmware : 5.20 (17544) Driver : 1.15 (2456) Boot Flash : 5.20 (17544) 77

88 1.14. XMLRPC interface Controller ZMM Information Status : ZMM Optimal Logical device information Logical device number 0 Logical device name : ADAP1 RAID level : 0 Status of logical device : Optimal Size : MB Stripeunit size : 256 KB Readcache mode : Enabled MaxIQ preferred cache setting : Enabled MaxIQ cache setting : Disabled Writecache mode : Enabled (writeback) Writecache setting : Enabled (writeback) Partitioned : No Protected by HotSpare : No Bootable : Failed stripes : No Power settings : Disabled Logical device segment information Segment 0 : Present (0,4) WDWMAKE Segment 1 : Present (0,5) WDWMAKE Physical Device information Device #0 Device is a Hard drive State : Online Supported : Transfer Speed : SATA 1.5 Gb/s Reported Channel,Device(T:L) : 0,4(4:0) Reported Location : Connector 1, Device 0 Vendor : WDC Model : WD740GD00FL Firmware : 33.08F33 Serial number : WDWMAKE Size : MB Write Cache : Enabled (writeback) FRU : None S.M.A.R.T. : No S.M.A.R.T. warnings : 0 Power State : Full rpm Supported Power States : Full rpm,powered off SSD : No MaxIQ Cache Capable : No MaxIQ Cache Assigned : No NCQ status : Disabled Device #1 Device is a Hard drive 78

89 Chapter 1. EXAoperation State : Online Supported : Transfer Speed : SATA 1.5 Gb/s Reported Channel,Device(T:L) : 0,5(5:0) Reported Location : Connector 1, Device 1 Vendor : WDC Model : WD740GD00FL Firmware : 33.08F33 Serial number : WDWMAKE Size : MB Write Cache : Enabled (writeback) FRU : None S.M.A.R.T. : No S.M.A.R.T. warnings : 0 Power State : Full rpm Supported Power States : Full rpm,powered off SSD : No MaxIQ Cache Capable : No MaxIQ Cache Assigned : No NCQ status : Disabled Command completed successfully Show plugin functions Function name: showpluginfunctions() Parameter(s): 1. plugin of type string: Name of plugin to call. Result type: Dictionary of functions and function descriptions The appropriate plugin is installed in the cluster. Perl example: use Frontier::Client; my $server = Frontier::Client>new('url' => ' my $funcs = $server>call('showpluginfunctions', 'RAID.tool'); Python example: import xmlrpclib s = xmlrpclib.serverproxy(" funcs = s.showpluginfunctions('raid.tool') Python example output: 79

90 1.15. Hardware Security Modules >>> pprint.pprint(s.showpluginfunctions('raid.tw_cli10.1')) {'SHOW': 'Show information about controllers and units. May be called with a controller/unit ID as argument, e.g. "/c0", "/c0/u0"', 'SHOW_AENS': 'Show automatic event notifications of controllers. May be called with a controller ID as argument, e.g. "/c0".', 'SHOW_ALARMS': 'Show alarms of controllers. May be called with a controller ID as argument, e.g. "/c0".', 'SHOW_DIAG': 'Show diagnostic information of controllers. May be called with a controller ID as argument, e.g. "/c0".', 'SHOW_EVENTS': 'Show events of controllers. May be called with a controller ID as argument, e.g. "/c0".', 'SHOW_REBUILD': 'Show rebuild schedules for controllers. May be called with a controller ID as argument, e.g. "/c0".', 'SHOW_SELFTEST': 'Show information about controller selftests. May be called with a controller ID as argument, e.g. "/c0".', 'SHOW_VER': 'Show API/CLI version of tw_cli.', 'SHOW_VERIFY': 'Show verify schedules. May be called with a controller ID as argument, e.g. "/c0".'} Hardware Security Modules To enhance security for stored disk passwords, one may use HSM (hardware security module) devices. These devices can be used via a PKCS#11API enabled library. To couple these devices, do the following: 1. Configure your hardware device and install the necessary PKCS#11 library in the cluster. This may include using a command line shell in the EXASuite cluster. See your HSM manual for further details. 2. Enable the cluster node(s) to use the specified key and create an encryption key in that slot. To do this on the commandline easily, we provide a tool that does that for you. This one can be called with pkcs11handler l {PKCS#11 library} k {keylabel} S {slot number} c 3. Create a key store in EXAoperation via "Access Management" > "Key Stores" > "Add". Provide an identifier for that key store, the label of the key and some attributes. An example for how to use attributes is LIB={PKCS#11 library};slot={slot number} If only one slot is provided, the SLOT parameter may be skipped. Click "Apply" to save this new key store. 4. Select the newly created key store (via the radio button in the table) and choose "Unlock". Now, provide the slot PIN and choose the time duration this key should be accessible. 5. Browse to "Access Management" > "System Passwords" and choose this newly created key store ("Disk Key Store"). As long as the used key store is unlocked, nodes may be booted. Otherwise, a boot process will fail Disk wipe When wiping disks via EXAoperation, the following happens at boot time of a node: 1. Determine disks to wipe (as specified in EXAoperation). 2. For each disk, start the shred command. This is a standard tool on UNIX systems and will be used with the command line parameters n 7 z. Thus, it will overwrite each disk with random data (seven times) and start a further run, overwriting the disk with zeroes Hugepages Since EXAoperation version it is possible to define the amount of socalled hugepages for cluster nodes. The hugepage feature has been introduced into machine hardware and the Linux kernel several years ago and allows efficient management of memory especially for situations with large amounts of physical RAM. It allows to shrink the necessary kernel data structures for handling process memory dramatically 2.We recommend to define a reas 2 Especially page table data structures can benefit. These normally reference pages with a size of 4 KB, but when used with hugepages (whose size is 2MB), their size can be shrinked upto 500 times. In a realcase scenario, this can save dozens of Gigabytes. 80

91 Chapter 1. EXAoperation onable amount of hugepages for cluster nodes with at least 512 GiB RAM. These hugepages can be used for "hot database data" as shown in Figure 1.22, DB RAM and hugepages. As only those data can be used here, it is necessary that the amount of hugepages per node is smaller than the amount of DB RAM used on it. Typically, 216 GiB of database RAM on any node will be used for other data necessary to be held in memory. This amount may be larger in case of databases with many open client connections at one time. When using this feature, we recommend to define at least 70 percent of DB RAM for hugepages, but to leave at least 64 GiB of physical node RAM untouched. Figure DB RAM and hugepages When starting multiple databases on one node, hugepages will be shared ondemand between database instances. When changing the hugepage setting for a cluster node, it has to be restarted Compatibility and known issues Web browsers The EXAoperation web interface has been tested successfully against the following browsers: Mozilla Firefox 12 on a GNU/Linux system Mozilla Firefox 22 on a Windows system Internet Explorer 10 on a Windows system Other common browsers as well as Mozilla Firefox or Internet Explorer installations in higher versions should also work as expected. Konqueror is known not to deselect checkboxes in "Unselect All" dialogs. 81