Rundeck. A smart software for lazy people. Claudio IMGT - IGH Montpellier BioInformatics Meeting

Transcription

1 Rundeck A smart software for lazy people Claudio IMGT - IGH Montpellier BioInformatics Meeting

2 This is my family 1

3 And this is the my Sicilian family network 2

4 Thank you aunt Teresa! 3

5 Teresa : Sicilian family = Rundeck : computer network Rundeck is an open source software that helps to automate ad-hoc and routine procedures in data center or cloud environments. The Rundeck Server connects to the configured nodes through SSH protocol, while a web interface allows clients to connect and operate through HTTP protocol. 4

6 Protocols allow the communication between nodes By Google Trends 5

7 SSH protocol Different protocols are designed to satisfy different needs. SSH (Secure SHell) is a cryptographic network protocol that provides a secure channel over an unsecured network. It allows, for example, remote login to computer systems. Depending on your OS, there are several possible clients: UNIX based systems: OpenSSH, essh client, PuTTY, Dropbear,... Windows: PuTTY, MobaXTerm, essh Client,... 6

8 HTTP protocol HypeText Transfer Protocol (HTTP) is an application protocol to exchange or transfer hypertext. In this context, the client can be a browsers (Firefox, Chrome, Opera, Explorer...), while the server is the source of a website, such as an application running on a computer hosting. 7

9 And rundeck? 7

10 Rundeck is an Orchestrator It is designed to create, run, schedule, share and monitor standard operating procedures on remote nodes. Those node sources can be physical machines or virtual nodes in clouds or clusters environments. For example, rundeck supports Amazon s Elastic Cloud Compute (EC2) nodes, that are managed by a dedicated plug-in. 8

11 How to install rundeck Rundeck server is based on Java technology and is available for any GNU/LINUX distribution through: 1..jar A self contained launcher install 2..deb package for Debian based OS 3. YUM repository And after that... there is a wonderful Administrator Guide! 9

12 Rundeck is a powerful tool, but... With great power comes great responsibility Uncle Ben 10

13 Security: from rundeck server to nodes From the rundeck manual: When Rundeck executes a command on a node, it does so via a Node Executor. The most common built-in Node Executor is the SSH implementation, which uses SSH to connect to the remote node, however other implementations can be used.... The Key Storage facility provides an API and GUI to manage public, private keys and passwords. These stored keys are used by both Node Executor and File Copier plugins. 11

14 Security: from clients to rundeck server Rundeck uses Container Authentication to determine the logged in user name and the user s authorized roles. (Default Servlet Container: Jetty - JAAS). Authentication can be defined in 3 ways: 1. PropertyfileLoginModule: the easiest method. It s a plain file containing usernames, passwords and role definitions. It s the least safe, even if it s possible to encrypt the passwords. 2. LDAP (Lightweight Directory Access Protocol): is an application protocol for accessing and maintaining distributed directory information services over an Internet Protocol (IP) network. 3. PAM (Pluggable Authentication Module): is a generalized API for authentication-related services which allows a system administrator to add new authentication methods simply by installing new PAM modules, and to modify authentication policies by editing configuration files. The 3 methods can be combined! 12

15 Security: from clients to rundeck server Once authenticated, Users privileges are restricted by Access Control Policy (ACP). Users can belongs to lists of group or roles, that grants them to perform actions against rundeck resources like projects, jobs, nodes, commands and API. 13

16 Ok, we are safe enough! 13

17 What are the standard operating procedures? The standard operating procedures are a set of command lines with default and customizable parameters, organized in 3 layers: 1. Step: is the unit procedure. It defines an action that the rundeck server will run on a node or on the server itself. 2. Job: is a stack of Steps. They are configured as pipelines (or workflows). 3. Project: is the wider layer. It s an isolated environment. Rundeck Server step1 step1 step1 step1 step1 step2 step2 step2 step2 step3 step3 step3 step4 step4 step5 step5 step6 Job 1 Job 2 Job Project 3 Job 1 A Job Project 2 B 14

18 The Projects environment The resources (nodes) are defined for each project independently (file resources.xml ) They can be exported and imported. They can have global project variables, callable in any step of the project with: Command line Arguments: ${globals.example} Script Bash: $RD_GLOBALS_EXAMPLE Jobs in the project can be organized in folders 15

19 My IMGT-Dev Project 16

20 The Jobs Have to contain at least 1 step. They can be scheduled. Notifications can be sent in case of starting, success or/and failure. Notifications are by default mails, but there is also a plugin for Telegram Resources can be fixed or tunable by the user at each run. Between the settings, there is also the thread count, that gives the maximum number of parallel threads to use. Running informations (stdout and stderr) are given as logs in the web interface and stored as activity. Jobs are made of two elements: Steps and Options 17

21 The Options 18

22 Steps 19

23 Steps features It s possible to use invocation string (eg. sudo or python or awk -f or Rscript ) A step can be another Job (of the same project), to whom you can give the options through a string like this: -option1 "argument" -option2 "a,list,of values" Very flexible and versatile: scripts can be stored everywhere they can be reached by rundeck server (accepts also URL). Scripts can also be written inside the rundeck job. In this case, strings will be interpreted as rundeck variables (global or project options). 20

24 Let s see if we can reach IMGT-rundeck from here

25 An example of Job - the running 21

26 An example of Job - the pipeline 22

27 6 reasons to use rundeck 1. It s Open Source. The rundeck community is active, under the updates, plugins and bugfixs point of view. 2. Lots of universities, enterprises, research centres use or are going to use cloud or cluster remote computational facility. 3. It s user friendly: let commandline-phobic users to run programs on remote servers. 4. Monitoring the executions of all your jobs on all your nodes in an unique interface. 5. Organize your scripts and write the descriptions for each component is not only good for you, but allows your colleagues to understand better your work and the final user to use it in few clicks. 6. It s completely customizable and there is an API that allows to integrate it with other tools. 23

28 Thank you! 23

29 References Rundeck: rundeck.org L A T E X theme : github.com/matze/mtheme TiκZ/PGF: cba Orchestrator image: damco.com 24

30 Supplementary 24

31 An example of admin.aclpolicy - part 1 The project context allows access to actions on resources within a specific project. description: Limited user access for adm restart action context: project: '.*' for: job: - equals: group: 'adm' name: 'Restart' allow: [run,read] - equals: group: 'adm' name: 'stop' allow: [run] - equals: group: 'adm' name: 'start' allow: [run] by: group: [restart_user] 25

32 An example of admin.aclpolicy - part 2 The application level context allows access to things like creating projects, access to projects, managing users, and access to system information. description: Limited user access for adm restart action. context: application: 'rundeck' for: resource: - equals: kind: system allow: [read] # allow read of system info project: - match: name: '.*' allow: [read] # allow view of all projects by: group: [restart_user] 26