Arigi Handbook

Transcription

1 Arigi Handbook

2 Table of Contents 1 Introduction Functionality Principle of Operation Getting Help Installation Requirements Licensing Procedure Database Deployment Options Configuration Variables Installing Running Authentication & HTTPS Docker Image Dashboard Summary Indicators Devices Table Device Management Configuration Management Template Management The Template Editor Configuration Fragments Examples List of Available Template Functions CLI Examples...19

3 Arigi Handbook 3/19 1 Introduction 1.1 Functionality arigi verb (Esperanto) to amass to gather to put together Syncthing is an inherently distributed solution lacking a built in form of centralized management. Arigi provides the following centralized functionality on top of Syncthing: A dashboard overview of all configured devices and their folders, with indications of online/offline state, in sync/out of date status, and configuration sync status. Device name, tag, and folder label filtering and searching for the status dashboard. Device configuration management based on tags and templated configuration fragments. Devices are automatically configured based on configuration fragments matching device tags. An extensive JSON REST API for integrating Arigi with third party systems or dashboards. Aggregation and forwarding of events to external sources like ElasticSearch for further processing and dashboarding. 1.2 Principle of Operation Arigi simplifies deployment and management of multiple Syncthing devices. Instead of configuring devices one by one, Arigi gives you a single place in which to monitor all devices and apply configuration changes. Device configurations are based on templating and tag matching - you can easily add a new folder to a group of devices, and Arigi will ensure the devices are appropriately reconfigured. Arigi is a server component that provides a web based interface. It does not require an external web server, although it can be placed behind one for aggregation and authentication purposes.

4 Arigi Handbook 4/19 All communication between Arigi and the managed Syncthing devices is initiated from Arigi and carried securely over HTTPS. Syncthing devices are located using standard Syncthing discovery mechanisms or manually configured addresses. Syncthing devices are enrolled with Arigi, using either of the following methods: By entering the GUI address (IP and port) plus a valid username and password. Arigi will read the API key from the device configuration and use that for further communication. By entering the GUI address (IP and port) plus a valid API key. By generating a Syncthing configuration file in Arigi and transferring this to the Syncthing instance to be enrolled. This overrides all settings on the Syncthing instance and is most suitable when doing initial setup on Syncthing devices. By having Arigi act as a private Syncthing Discovery Server and automatically enroll devices registering with the discovery server. Once enrolled, a Syncthing device is monitored and controlled by Arigi. 1.3 Getting Help The latest version of Arigi and the product documentation can be downloaded from arigi/. When logged into an Arigi instance, the About page also provides access to the matching version of the product documentation, the list of known issues, and contact information to support. Should you be unable to access the GUI, please use the following information to contact support at any time: Direct to support@kastelo.net. Web portal at 2 Installation 2.1 Requirements Please ensure the following requirements prior to commencing installation. A Unixy server. Arigi runs fine on Debian, Ubuntu, CentOS and RedHat Enterprise Linux. It probably runs fine on most other flavors of Linux too. FreeBSD, macos ("Darwin") and Solaris are also good choices. The server should have Internet access. The exact resource requirements are dependent on the number of devices and their configuration; however, a baseline of 1 GiB of RAM and one CPU core is sufficient in most cases. A virtual machine is fine.

5 Arigi Handbook 5/19 A valid license. A license is required to use Arigi. Arigi ships with a built in "free edition" license limited to five devices. Potentially, an external database. Arigi stores all information either in RAM (saved to a file for persistency) or an external database. Currently only PostgreSQL is supported, although support for other databases is planned based on demand. One or more Syncthing devices to manage. Syncthing version 0.14 and above are supported. 2.2 Licensing Each Arigi instance requires a license. A license entitles you to use Arigi to manage the licensed number of Syncthing devices in perpetuity. It also entitles you to one year of free software updates from the date it is issued. The license is distributed as a file called license.pem. This file must be transferred to the server running Arigi. The license file shows the entitlements granted in clear text, in addition to the required cryptographic signature. For example: This license allows managing up to one thousand devices and is valid for any version of Arigi released on or before , in perpetuity. Any given License ID may be used on only one server BEGIN LICENSE----- Devices: 1000 ID: ed0be65da8f2d7e2 Issuer: support@kastelo.net Licensee: customer@example.com Support: :00:00 UTC 7QvmXajy1+Jc6UU1LRBhuSZOFDq5ReiycJP34NqdHaao1vIQ+dUsLkI+NsoYvjxR... hx5un2had/+05nnk2r7j8eolihvmg3igtyjpinm+bprkv8jj3khy -----END LICENSE----- An example license 2.3 Procedure Arigi is distributed as a generic.tar.gz file containing the arigisrv binary and supporting scripts and documentation.

6 Arigi Handbook 6/ Database Deployment Options Arigi ships with two separate database deployment options. Both are stable and appropriate for production, with separate tradeoffs External PostgreSQL Database All information is kept persistently in PostgreSQL. Arigi will manage the database schema and create the required tables on first startup. You will however need to create a user and database beforehand. We give general suggestions for the setup of the PostgreSQL database. There are however many options here and we defer to your or your DBA's expertise on the particulars. Create a user (role) for Arigi and give it a password: CREATE USER arigi WITH PASSWORD '...'; Create a database owned by the new user: CREATE DATABASE arigi OWNER arigi; Remember the user name, database name and password for the setup below In-memory Database All information is kept in RAM while running and periodically serialized to disk. The database is read at startup to populate the in-memory store. Database updates are append-only with periodic compaction. To use the in-memory database you need only enable the option ARIGI_USE_RAMDB. The database will be stored in $ARIGI_HOME (default /opt/var/arigi) Configuration Variables All configuration options are set as environment variables prior to starting arigisrv, or as command line flags. All such variables share the prefix ARIGI_. The following variables are available: ARIGI_HOME The directory where Arigi keeps persistent data. (default: "/opt/var/ arigi") ARIGI_LICENSE Sets the path to the license.pem file. (default: "/opt/etc/arigi/ license.pem") ARIGI_LISTEN The address where Arigi serves the web UI. (default: ":2525")

7 Arigi Handbook 7/19 ARIGI_LISTEN_INSECURE Listen on HTTP, not the defaults of HTTPS. (default: "false") ARIGI_EXTERNAL_URL Set the URL used to access Arigi. Used to construct URLs when sending s.(default: " ARIGI_USE_RAMDB Whether to use the built-in in-memory database (default: "false") ARIGI_POSTGRES_HOST The database host. (default: "localhost") ARIGI_POSTGRES_DATABASE The database name. (default: "arigi") ARIGI_POSTGRES_USER The database user name. ARIGI_POSTGRES_PASSWORD The database password. (no default value) ARIGI_POSTGRES_SSL_MODE The database SSL mode. (default: "disable") ARIGI_SMTP_SERVER An SMTP server address (used for password recovery s). (default: "smtp.example.com:587") ARIGI_SMTP_USER The SMTP authentication user. (default: ARIGI_SMTP_PASSWORD The SMTP authentication password. (no default value) ARIGI_SMTP_FROM The SMTP sender. (default: "Arigi TODO: Document Elastic and auto enrollment variables Installing Untar the distribution into a suitable directory. Our recommended default is /opt, resulting in the default Arigi installation directory /opt/arigi. # mkdir -p /opt # cd /opt # tar zxvf ~/arigi-linux-amd64-v1.0.tar.gz Create the configuration directory and move the license file there. If you're using only the free edition this step can be skipped. # mkdir -p /opt/etc/arigi # mv ~/license.pem /opt/etc/arigi You also need a directory for persistent data. The recommended location is the default, /opt/var/arigi. Make sure this directory is owned by the non-root user that will be running Arigi.

8 Arigi Handbook 8/19 # mkdir -p /opt/var/arigi # chown arigi:arigi /opt/var/arigi Any other existing, local user can be used instead of arigi:arigi. Edit the default variables in /opt/arigi/bin/start.sh to set the directory paths in use and the database parameters. To run Arigi you use the start.sh script. It is highly recommended to run Arigi as a separate, non-root user. # su - arigi $ /opt/arigi/bin/start.sh In summary, you should now have the following files and directories in place: /opt/etc/arigi/ Directory for read-only configuration. /opt/etc/arigi/license.pem The license file. /opt/arigi/ Directory for the distribution. /opt/arigi/bin/ Directory for executable files. /opt/arigi/bin/arigi A CLI interface to Arigi. /opt/arigi/bin/arigisrv The main Arigi binary. /opt/arigi/bin/start.sh The startup helper script. /opt/var/arigi/ Directory for read-write data Running Use your favorite operating system method to keep Arigi running. Possibilities include systemd, runit, daemon-tools, and cron. The arigi binary, and hence the start.sh script, will not exit unless Arigi encounters a fatal error Authentication & HTTPS Arigi by default uses HTTPS and creates a certificate pair on startup if one is missing. It is recommended that this certificate either be replaced with a CA issued certificate, or added to the local trust store. It is also possible to run Arigi in HTTP only mode behind a secure reverse proxy, such as Nginx, Apache or Caddy. Please note that the arigi command line tool requires that a trusted certificate is used, whether CA issued or by local trust policy. As an alternative, plain HTTP can be

9 Arigi Handbook 9/19 used with the --listen-insecure flag to arigisrv and the --api-insecure flag to arigi. The default credentials on initial login are username "admin" and password "arigi". 2.4 Docker Image For quick deployment and testing there is a Docker image available. This image uses the in-memory database by default. # docker run -d --name arigi --restart=always \ -p 2525:2525 kastelo/arigi Give the container a few seconds to start, and then visit (or the address of the host running the container if not localhost ). Data in the container is stored in /opt/var/arigi. You can bind a volume in order to get persistent storage independent of the container. To use HTTPS with the Docker image setup, start Arigi while setting the ARIGI_LISTEN_INSECURE variable to false and, ideally, a volume containing HTTPS certificate and key: # docker run -d --name arigi --restart=always \ -p 2525:2525 -e ARIGI_LISTEN_INSECURE=false \ -v /srv/arigidata:/opt/var/arigi kastelo/arigi Arigi will generate and use a self signed key pair if one is not already present on the volume. 3 Dashboard The Dashboard is the initial view or front page of Arigi. It shows a summary of the following items: Number of devices per tag with online, offline, in sync, syncing, errored, with config up to date, or with config pending update status. Number of devices per folder with online, offline, in sync, syncing, or errored status. Devices that are experiencing an error or that are offline due to being unreachable by Arigi, with the corresponding error.

10 Arigi Handbook 10/19 Each view can be filtered by tags, folder ID or folder label. Folder labels can be edited directly in the folder list by clicking on the corresponding label / ID. 3.1 Summary Indicators At the top of the dashboard there are several graphical indicators that give a quick overview of the current status of all devices.

11 Arigi Handbook 11/19 On the left is the device connection status graph. It indicates the fraction of configured devices that are currently online and reachable by Arigi. In the example, one device out of three is unreachable, resulting in one third of the indicator being red. The colors in the connection status graph are green for online devices, and red for offline devices. In the middle is the device sync status graph. It indicates the sync status of the configured devices. The colors in this graph are green for online devices that are in sync, blue for online devices that are syncing, yellow for online devices that are experiencing a sync error, and greyfor offline devices (unknown sync status). To the right is the device configuration sync status graph. It indicates the configuration sync status of the configured devices. The colors in this graph are green for online devices where the configuration is in sync, yellow for online devices that are pending an update to the configuration, and grey for offline devices (unknown configuration sync status). 3.2 Devices Table The table below the indicators contain the same information in a more detailed manner, broken down per tag. Please note that a given device may be accounted under more than one tag. The coloring in the table is green for positive values (online, in sync), red for values indicating a problem (offline devices), yellow for values indicating a possible but less serious problem (sync error, configuration pending). Note that devices that are not online do not have a determined sync or configuration status and are thus not accounted in those sections.

12 Arigi Handbook 12/19 4 Device Management The Device tab used to enroll (add) and edit devices. For Arigi to be able to manage a device it must know: The device ID (Action > Show ID in Syncthing) The device address, which may be an IP address, a domain name, or blank to use the standard Syncthing global discovery infrastructure. The device API port. This is the GUI/API listen port in Syncthing and defaults to The API key (also shown in Action > Settings) in Syncthing. Each device can have a label. The label is strictly for display purposes and doesn't affect the Arigi functionality. Once a device is saved you can add tags to describe it and link it to templates. When opening an existing device, the device tab includes some extra information picked up from the device: its version, CPU utilization, etc.

13 Arigi Handbook 13/19 5 Configuration Management Arigi manages Syncthing configurations in JSON format. In principle the configuration format is that described in the Syncthing REST API documentation: docs.syncthing.net/rest/system-config-get.html. There is however one transformation applied to simplify management: the lists of folders and devices are turned into objects keyed on the folder and device ID, respectively. This has the effect that each individual config item can be identified by its "path" in the configuration, like folders.gxwxf-3zgnu for a folder with the ID GXWxf-3zgnU, instead of having to iterate over a list to find the correct object. The overall configuration structure then becomes the following: { } "version": 21, "folders": { "folder ID": { "label": "...", "path": "...", "type": "...", "devices": [{ "deviceid": "..."}], // more options } }, "devices": { "device ID": { "name": "...", "addresses": [ "dynamic" ], // more options } }, "gui": { // gui options }, "options": { // general options }, "ignoreddevices": [], "ignoredfolders": []

14 Arigi Handbook 14/ Template Management Templating is the mechanism by which we configure devices in Arigi. In short, a device has one or more tags attached to it. These tags in turn match one or more configuration fragments. Arigi evaluates these fragments, assembles them in the order dictated by their priority, and thus creates a complete device configuration. If this configuration differs from what the device in question currently has, Arigi updates the device configuration accordingly. This is a powerful mechanism that allows reuse of configuration fragments between many devices, and automatically ensures that all affected devices are updated when a fragment changes. Configuration fragments, in turn, can be dynamically built based on the information in Arigi. For example, a single fragment might define a folder shared with all devices having a certain tag. When a new device is added to this tag the fragment will change to accommodate it, and all devices attached to the fragment will be reconfigured to share the folder with the new device. 5.2 The Template Editor The template editor is the environment in which you create and modify the templates. The template editor provides validation and evaluation of the given configuration fragment, showing visually the effect that it would have have on matched devices. It also shows which devices would be affected by the tags currently assigned and their status. 5.3 Configuration Fragments The device configuration is in JSON format, and the configuration fragments are likewise in a form of JSON. The actual format is "HJSON", for "Human friendly JSON". This is a variant of JSON that is more forgiving of punctuation and quoting than regular JSON, thus being more pleasurable to use. Regular, strict, JSON is valid HJSON and can be used if you prefer. A configuration fragment consists of the following parts: Name This gives us a friendly way to refer to the fragment in the configuration. Priority This sets the order in which fragments are evaluated and applied, lower priority value coming first. The priority is relevant when one fragment overrides or amends values in another.

15 Arigi Handbook 15/19 Key This is the configuration key that is to be altered by the fragment, for example folders or options. It can also refer to specific options or the whole configuration in general. Op This sets the operation that will be used to apply the fragment. The choices are Set (to set or change an option, or a whole configuration segment) and Remove (to remove a device or folder). Template This is the actual config fragment contents. It is in JSON or HJSON format, optionally with Go templating Examples Here follows some examples to get you started on how to build configuration fragments. The first is a simple fragment to change an option. It targets a specific option with the key, uses the replace operation, and the template is simply the value to use, verbatim, which in this case is the integer negative one. Name: Disable progress updates Priority: 100 Key: options.progressupdateintervals Op: Set Template: -1 A more advanced option adds a folder and makes sure it's shared among all devices with a certain tag. Name: Add common folder Priority: 100 Key: folders.thefolderid Op: Set Template: { id: TheFolderID label: This is a label path: ~/somepath devices: [ {{range DevicesWithTag "common" }} { deviceid: "{{.DeviceID}}" }

16 Arigi Handbook 16/19 } ] {{end}} Here we can see the template being a bit more involved. There are a few things to note about it: The Key indicates the folder ID to set. The Template is now an object, enclosed by curly braces, and not just a simple value like in the previous example. We use the HJSON syntax to avoid typing extra commas, quotation marks, etc. The folder ID, which is in the Key, is repeated in the template itself as the id attribute. This is necessary because we are creating or replacing the entire folder object, and it needs an ID. There are Go-style template directives enclosed in {{double curlies}}. There is a range-loop and a call of the function DevicesWithTag to enumerate other devices. Inside the range loop we use an DeviceID attribute of the looped over devices. Arigi performs the following steps when this fragment is to be applied to a device: 1. Evaluates the double curly brace templating. a. Run the DevicesWithTag function. It looks up all devices matching the given tag and returns them a list of device objects. b. Iterate over the returned list of device objects using the range. c. For each item, extract the DeviceID attribute and print the device line. The result is a HJSON object looking something like this: { id: TheFolderID label: This is a label devices: [ { deviceid: "abcdef " } { deviceid: "bcdefg " } { deviceid: "cdefgh " } ] } 2. Pass the resulting text to the HJSON processor to turn it into a real JSON object:

17 Arigi Handbook 17/19 { } "id": "TheFolderID", "label": "This is a label", "devices": [ { "deviceid": "abcdef " }, { "deviceid": "bcdefg " }, { "deviceid": "cdefgh " } ] 3. Apply this object to the device configuration using the specified Op, in this case add. Since we can key on the folder ID it's also easy to ensure that a certain device does not share a given folder: Name: Remove forbidden folder Priority: 500 Key: folders.some-secret-folder Op: Remove Here we use a higher Priority number, ensuring that the template is evaluated later. In this manner we ensure that it removes the folder from the configuration, had it been added by any other template with a lower priority number. It's also possible to access the current configuration and perform operations on the values therein. This happens implicitly when we specify a key to a specific object as above, but sometimes it's necessary to change (but not replace) folders or devices without knowing their ID. For example, we might want to set the allowednetworks attribute on all devices, regardless of their ID. Name: Add common folder Priority: 100 Key: devices Op: Set Template: [ {{ range $device := Current.devices }} {{ $device set "allowednetworks" (strings " /24" " /8") json }} {{ end }} ]

18 Arigi Handbook 18/19 This template replaces the entire devices list with a new list. The list is built by iterating over the list of current devices. For each item (device) the allowednetworks attribute is set to /24, /8. The expression {{ range $device := Current.devices }} starts a loop over all items in the devices list of the Current configuration. On every iteration the device config is available in the variable named $device. Inside the loop there is a "pipeline", delimited by the characters. It begins with the device, performs a set to change the attribute, and in the end formats the object as JSON so that it becomes a valid part of the template. 5.4 List of Available Template Functions The following functions are available in the templating language: AllDevices, returning the list of all devices known to Arigi. DevicesWithTag tag, returning the list of devices matching the given tag. Current, returning the configuration of the device being processed, prior to applying the template. Access attributes using dot notation, such as Current.options, Current.gui. MyID, returning the device ID of the device beinf processed. json object, formats the given object as JSON text. set path value object, sets the value at path to value in object and returns the result. strings string..., creates a list of strings from the given arguments. Pipelines work by sending the preceding object as last argument to the next stage. This allows easily chaining operations like set. For example, this takes the current options, changes two NAT-related attributes, and formats it as JSON: $Current.options set "natenabled" true set "natleastminutes" 1440 json In addition, all actions, pipelines and functions documented as part of the Go text/ template package are available. Please refer to for full documentation of the template language.

19 Arigi Handbook 19/19 6 CLI The installation package (.tar.gz) includes a command line client called simply arigi. This utility can be used to manage Arigi installation in most ways the graphical dashboard can. For example, you can: Enroll new devices, List and inspect status for current devices, List and inspect status for synchronized folders, Add and modify device tags, Add and modify configuration fragments,... and so on. To see all available commands and options, please run arigi â long-help. 6.1 Examples $ arigi login admin my-admin-password $ arigi --long-help $ arigi device add abcdef apikeyapikey $ arigi device list $ arigi device get abcdef $ arigi device tag abcdef mytag