Dynamic Edge Processing Streaming and Persistence Service - Configuration Guide Version 3.0

Transcription

1 Dynamic Edge Processing Streaming and Persistence Service - Configuration Guide Version 3.0

2 TABLE OF CONTENTS 1 INTRODUCTION ARCHITECTURE Installation Options SUPPORTED OPERATING SYSTEMS SUPPORTED WINDOWS VERSIONS WINDOWS PREREQUISITES AND INSTALLATION Java SE Runtime (version 1.8.0_ bit) Python (version 2.7.x 64-bit) WINDOWS - STREAMING AND PERSISTENCE SERVICE INSTALLATION Download Required Components from SAP Service Marketplace EDGE DB 3.0 Component SAP ESP SERVER 5.1 Component DEP_IOT_EDGE 3.0 Component WINDOWS - POSTINSTALLATION SECURITY TASKS WINDOWS - MANAGING STREAMING AND PERSISTENCE SERVICE PROCESSES WINDOWS - UNINSTALLING STREAMING AND PERSISTENCE SERVICE SUPPORTED LINUX VERSIONS LINUX PREREQUISITES AND INSTALLATION Java SE Runtime (version 1.8.0_31+) Python (version 2.7.x) LINUX - STREAMING AND PERSISTENCE SERVICE INSTALLATION Download Required Components from SAP Service Marketplace EDGE DB 3.0 Component STREAMING LITE 2.0 Component DEP_IOT_EDGE 3.0 Component LINUX - POSTINSTALLATION SECURITY TASKS LINUX - MANAGING STREAMING AND PERSISTENCE SERVICE PROCESSES LINUX - SERVICE CREATION DATABASE BACKUPS AND MAINTENANCE CUSTOMIZING SSL CONFIGURATION CUSTOMIZING EDGE CONSOLE PORTS LOG FILE CONFIGURATION CONNECTING TO THE STREAMING AND PERSISTENCE SERVICE DATABASE RUNTIME SETTINGS CONFIGURATION BUSINESS ESSENTIAL FUNCTIONS SERVICE CONFIGURATION ENTERPRISE PLUGIN CONFIGURATION Configuring an Enterprise Plugin using dbisql

3 24 PROTOCOL PLUGIN CONFIGURATION Configuring a Protocol or Engine Plugin using dbisql Sending Data to a Protocol Plugin Field Message Action to a Protocol Plugin Subprotocols for HTTP Protocol Plugin WebSocket Server ENGINE PLUGIN CONFIGURATION - ADVANCED Configuring Custom Engine Plugins WHITELIST CONFIGURATION ODATA CONFIGURATION IMPORT/EXPORT OF CONFIGURATION Exporting Configuration Importing Configuration TROUBLESHOOTING KNOWN ISSUES

4 1 INTRODUCTION SAP Dynamic Edge Processing contains two distinct sets of edge application services. They are installed separately, and can work together or independently. We will refer to these two sets of edge application services as: Business Essential Functions Service Streaming and Persistence Service The following definitions explain the main functionality in this product: Term Abbreviation Digital core (or just core) The customers core enterprise systems running in their data center or in the cloud including S/4HANA, Business Suite or HANA. Edge The location of customers business at the edge of their networks, outside their data centers. Typically this is the location of the IoT sensor. Edge application services The functionality and data exposed through defined interfaces used to enable edge applications (Business Essential Functions Service and Streaming and Persistence Service). This document focuses on the configuration of Streaming and Persistence Service. 2 ARCHITECTURE The Streaming and Persistence Service architecture is shown below. The area within the dashed line is considered the boundary of Streaming and Persistence Service. Streaming and Persistence Service is comprised of several processes: IoT Adapter Edge Console Streaming a. SAP Streaming Lite (Linux) b. SAP Event Stream Processor (Windows) Persistence a. SAP SQL Anywhere (Database) SAP Plant Connectivity (PCo) optional - provides additional connectivity to devices and sensors 4

5 2.1 Installation Options You can install Streaming and Persistence Service either on the same host as Business Essential Functions Service, or on an independent host. Streaming and Persistence Service can integrate with the Business Essential Functions Service on the same host, or can also integrate with the Business Essential Functions Service on a separate host. Streaming and Persistence Service can also be installed without the Business Essential Functions Service in the landscape. In summary, the installation options for Streaming and Persistence Service are the following: Same host as Business Essential Functions Service. Different host than Business Essential Functions Service. Without Business Essential Functions Service in the landscape. 3 SUPPORTED OPERATING SYSTEMS Streaming and Persistence Service supports both the Windows (x86 64-bit) and Linux (x86 64-bit and ARM 32- bit) operating systems. Separate installation procedures are provided for Windows and Linux. Note: On Windows x86 64bit and Linux x86 64bit (but not Linux ARM), Streaming and Persistence Service provides an OData interface to query data. OData interface is not supported on Linux ARM release of Streaming and Persistence Service. Note: This document refers to the Streaming and Persistence Service installation folder as <dep_iot_edge>. This folder is different on Windows and Linux: Windows must be located in a folder with no spaces in the path: for example: c:\dep_iot_edge or c:\sap\dep_iot_edge Linux must be located in /opt : /opt/dep_iot_edge 4 SUPPORTED WINDOWS VERSIONS Streaming and Persistence Service supports the following Windows versions: Windows 8.1 (x86 64-bit) Windows 10 (x86 64-bit) Windows 2012 Server (x86 64-bit) Windows 7 (x86 64-bit) 5 WINDOWS PREREQUISITES AND INSTALLATION Streaming and Persistence Service require several software packages to already be installed on the host. Prerequisites include the following: Java SE Runtime version 1.8.0_ bit (this is installed by SQL Anywhere 17) Python version 2.7.x 64-bit The detailed installation requirements are discussed below. 5.1 Java SE Runtime (version 1.8.0_ bit) SAP SQL Anywhere 17 automatically installs the required Java SE Runtime (recommended); or a compatible JRE can be downloaded and installed separately. Note: If using SAP SQL Anywhere 17 version of JRE, then the JAVA_HOME system environment variable must be manually configured (it can point to a different valid JRE version 1.8.0_31+) 5.2 Python (version 2.7.x 64-bit) Python can be downloaded from Python can be installed to any recommended directory. 5

6 Note: When installing Python, select the option to add Python in the SYSTEM PATH environment variable. 6 WINDOWS - STREAMING AND PERSISTENCE SERVICE INSTALLATION 6.1 Download Required Components from SAP Service Marketplace Streaming and Persistence Service requires several software components to be downloaded and installed from SAP Service Marketplace under Dynamic Edge Processing 3.0 : EDGE DB 3.0 Component (SAP SQL Anywhere 17 version bit) SAP ESP SERVER 5.1 Component (SAP Event Stream Processor 5.1 SP12 PL6 version / /SP12) DEP_IOT_EDGE 3.0 Component ( Streaming and Persistence Service ) Go to SAP Service Marketplace - Software Downloads 1. Search for SAP Dynamic Edge Processing In search results, click on SAP Dynamic Edge Processing 3.0 Maintenance Product 3. Select Downloads and then Comprised Software Component Versions a. Select EDGE DB 3.0 i. Select Windows on x64 64bit and download the latest package (remaining instructions assume file is named sql_anywhere-x64.zip). b. Select SAP ESP SERVER 5.1 i. Select Windows on x64 64bit and download the latest package (remaining instructions assume file is named esp_server-x64.zip). c. Select DEP_IOT_EDGE 3.0 i. Select Windows on x64 64bit and download the latest package (remaining instructions assume file is named dep_iot.zip). The installation of these components is described below. 6.2 EDGE DB 3.0 Component To install EDGE DB 3.0 Component (SAP SQL Anywhere 17), unzip sql_anywhere-x64.zip and install the enclosed msi file. Note: SAP SQL Anywhere is OEM licensed only for use by Dynamic Edge Processing. 6.3 SAP ESP SERVER 5.1 Component The setup process for SAP ESP SERVER 5.1 is as follows: 1. Decide where to put esp_server-x64.zip (C:\SAP\dep_streaming is recommended) 2. Unzip esp_server-x64.zip to that location 3. After unzip, the following folder should exist: C:\SAP\dep_streaming\ESP-5_1 4. Create a system environment variable for STREAMING_HOME (this must be a system environment variable not a user environment variable): STREAMING_HOME = C:\SAP\dep_streaming\ESP-5_1 Note: SAP Event Stream Processor is installed in such a way that only Streaming and Persistence Service can use the installed version of SAP Event Stream Processor. SAP Event Stream Processor is OEM licensed only for use by Dynamic Edge Processing. 6.4 DEP_IOT_EDGE 3.0 Component Note: Open and run all command prompts as Administrator. Note: DEP_IOT_EDGE 3.0 component must be installed to path with no spaces in the path, such as, c:\dep_iot_edge or c:\sap\dep_iot_edge. Note: DEP_IOT_EDGE 3.0 component is referred to as Streaming and Persistence Service below. 6

7 1. Ensure that all prerequisite software is successfully installed. A reboot is required after installing the prerequisite software. 2. Streaming and Persistence Service must be installed by a member of the local Administrators group. 3. Copy dep_iot.zip to the Windows host (for example, copy to C:\SAP\) 4. Unzip the file to a location without a space in the path. The path c:\sap\ is recommended (this creates a c:\sap\dep_iot_edge folder). 5. Run Streaming and Persistence Service setup script (setup.bat). setup.bat does the following: Allows you to keep or delete an existing Streaming and Persistence Service database o For delete existing database, all configuration and IoT data is wiped. o For keep existing database, all configuration and IoT data is maintained. For a new database o Requests the initial password for the Edge Console admin user. o Creates a new Streaming and Persistence Service database and required database users, and populates the database with required data. Note: All database user credentials are randomly generated. Requests the initial password for the Streaming and Persistence Service odata data access technical user. o Note: the odata data access technical user is used to authenticate odata queries to Streaming and Persistence Service. Creates ODBC entries for the Streaming and Persistence Service database. The SQL Anywhere dbdsn utility obfuscates the database user s password inside the Data Source Name (DSN). o DEP_IOT_ADMIN: used by Streaming and Persistence Service Java runtime + admin utilities o DEP_IOT_STREAMING: used by Streaming Lite Copies Java libraries from Event Stream Processor and SQL Anywhere into c:\sap\dep_iot_edge\java_lib\ Creates SSL certificates to be used by the built-in plugins: o Edge Console web server for HTTPS o HttpProtocolPlugin to enable HTTPS and WSS o Creates a NodeId (if not supplied) and writes it to NODE_SETTINGS table To run Streaming and Persistence Service setup script, execute the following commands (in command prompt as Administrator): cd c:\sap\dep_iot_edge\bin setup.bat Note: The final output of setup.bat, if successful, is INFO: Setup completed successfully. Note: All user prompts for setup.bat are also available as command line options. For complete details run setup.bat -h 7 WINDOWS - POSTINSTALLATION SECURITY TASKS By default, the Streaming and Persistence Service setup program applies the following permissions: c:\sap\dep_iot_edge only allows access to Administrators group and LOCAL_SERVICE account All Streaming and Persistence Service Windows services are only controllable by a member of the local Administrators group 8 WINDOWS - MANAGING STREAMING AND PERSISTENCE SERVICE PROCESSES Streaming and Persistence Service creates 4 Windows Services all are installed to run as Windows services that start automatically when the machine boots. The required inter-service dependencies are configured by default. 1. SAP Dynamic Edge Processing Server 2. SAP Dynamic Edge Processing Web Server 7

8 3. SAP Dynamic Edge Processing Streaming Server 4. SAP Dynamic Edge Processing Database Use the Windows Services control panel applet to start and stop services. 9 WINDOWS - UNINSTALLING STREAMING AND PERSISTENCE SERVICE Streaming and Persistence Service provides an uninstall script for Windows. This script prompts (twice) for confirmation to remove each the following: 1. First Prompt: Streaming and Persistence Service Windows Services 2. Second Prompt: Streaming and Persistence Service Database + Files a. Database file b. Database ODBC entries c. Program files Warning: If the Streaming and Persistence Service Database + Files are deleted, then all configuration, data, history, etc. is deleted from that host machine. To run the uninstall script on Windows, open a command prompt (as Administrator) and run the following: cd <dep_iot_edge>\bin uninstalldep.bat Note: If the database and files are selected to delete, then the script finishes with The system cannot find the path specified. This is expected and can safely be ignored. 10 SUPPORTED LINUX VERSIONS Linux is available from several vendors. In addition, a wide variety of Linux installations are possible, which poses a challenge for companies such as SAP who want to maintain an open and flexible approach to supporting Linux, yet require a practical means of identifying which Linux installations they support. Support for Linux is dependent on the Streaming and Persistence Service prerequisites support for Linux. For example, if all of the prerequisites support a specific version of Linux, then theoretically Streaming and Persistence Service should also support that version of Linux; however untested versions of Linux may have slightly different configurations that the Streaming and Persistence Service installer does not recognize. Please contact your SAP representative for further details. SAP SQL Anywhere for Linux Supported Platforms: SAP Streaming Lite Platform and Installation: m Note: Streaming Lite supports Linux x86 64-bit and Linux ARM 32-bit. Streaming Lite does not support Linux x86 32-bit. Tested Linux distributions for deployment: Ubuntu Desktop LTS (x86 64-bit) Ubuntu Desktop LTS (x86 64-bit) RHEL 7.3 (x86 64-bit) (See Known Issues 1, 2, 3) SUSE 12 (x86 64-bit) (See Known Issues 1, 2, 3, 4) Intel Intelligent Device Platform (IDP) 3.1 Wind River 7 (x86 64-bit) (See Known Issues 1, 2) Tested Linux distributions for prototyping: Raspbian v7 (Jessie) (ARM 32-bit) 11 LINUX PREREQUISITES AND INSTALLATION Streaming and Persistence Service requires several software packages to already be installed on the host. Prerequisites include the following: Java SE Runtime version 1.8.0_31+ (this is installed by SQL Anywhere 17) Python version 2.7.x 8

9 The detailed installation requirements are discussed below Java SE Runtime (version 1.8.0_31+) SAP SQL Anywhere 17 automatically installs the required Java SE Runtime (recommended); or a compatible JRE can be downloaded and installed separately. Note: If the JAVA_HOME environment variable is not already configured, then the Streaming and Persistence Service control script (dep.sh) correctly sets JAVA_HOME to the SQL Anywhere JRE folder (dep.sh calls the SQL Anywhere sa_config file, which sets the JAVA_HOME environment variable if it is not already configured) Python (version 2.7.x) Python can be installed either by downloading it from or by using the package manager local to the Linux distribution (apt, yum, or rpm). Python can be installed to any recommended directory and must be included in the PATH environment variable. 12 LINUX - STREAMING AND PERSISTENCE SERVICE INSTALLATION 12.1 Download Required Components from SAP Service Marketplace Streaming and Persistence Service requires several software components to be download and installed from SAP Service Marketplace under Dynamic Edge Processing 3.0 : EDGE DB 3.0 Component (SAP SQL Anywhere 17 version ) STREAMING LITE 2.0 Component (SAP Streaming Lite 2.0 SP00 Rev version / /SP00 Rev001) DEP_IOT_EDGE 3.0 Component ( Streaming and Persistence Service ) Go to SAP Service Marketplace - Software Downloads 1. Search for SAP Dynamic Edge Processing In search results, click on SAP Dynamic Edge Processing 3.0 Maintenance Product 3. Select Downloads and then Comprised Software Component Versions a. Select EDGE DB 3.0 i. Select appropriate Linux version (x86 64bit or ARM 32-bit) and download the latest package (remaining instructions assume file is named sql_anywhere-x64.gz). b. Select Streaming Lite 2.0 i. Select appropriate Linux version (x86 64bit or ARM 32-bit) and download the latest package (remaining instructions assume the Streaming Lite installation file is named SDSLITE_Linux_x86_64.gz). c. Select DEP_IOT_EDGE 3.0 i. Select appropriate Linux version (x86 64bit or ARM32) and download the latest package (remaining instructions assume file is named dep_iot.zip). The installation of these components is described below EDGE DB 3.0 Component EDGE DB 3.0 Component (SQL Anywhere 17) must be installed to /opt/sqlanywhere17 (exact folder name required). Installation instructions are: 1. Copy the EDGE DB 3.0 installation file to the Linux host. 2. Copy and extract the EDGE DB 3.0 installation file in the /opt folder (exact folder name required). Rename any /opt/sqlanywhere17 folder if it already exists. sudo cp sql_anywhere-x64.gz /opt/ cd /opt sudo tar xvf sql_anywhere-x64.gz 3. Remove the SQL Anywhere installation file: rm sql_anywhere-x64.gz 9

10 Note: SAP SQL Anywhere is OEM licensed only for use by Dynamic Edge Processing STREAMING LITE 2.0 Component STREAMING LITE 2.0 Component must be installed to /opt/streaminglite (exact folder name required). Installation instructions are: 1. Copy SDSLITE_Linux_x86_64.gz to the /opt/streaminglite folder (exact folder name required) on the Linux host. Create the /opt/streaminglite folder if it does not exist. sudo mkdir /opt/streaminglite sudo cp SDSLITE_Linux_x86_64.gz /opt/streaminglite/ 2. Extract SDSLITE_Linux_x86_64.gz into /opt/streaminglite: cd /opt/streaminglite sudo tar xvf SDSLITE_Linux_x86_64.gz 3. Remove SDSLITE_Linux_x86_64.gz: rm SDSLITE_Linux_x86_64.gz 4. Once extracted, verify that the Streaming Lite binaries are in /opt/streaminglite/install/bin (required). Note: STREAMING_HOME environment variable is automatically set by the Streaming and Persistence Service control script (dep.sh). Note: SAP Streaming Lite is licensed only for use by Dynamic Edge Processing DEP_IOT_EDGE 3.0 Component Note: DEP_IOT_EDGE 3.0 component must be installed to /opt/dep_iot_edge Note: DEP_IOT_EDGE 3.0 component is referred to as Streaming and Persistence Service below. 1. Ensure all prerequisite software is successfully installed. 2. Setup installation user: a. If Streaming and Persistence Service will be running as Linux services, you can install it either as a root user, or a non-root user. Consult your Linux documentation for more details. b. If Streaming and Persistence Service is not running as Linux services, it is recommended that you create a non-root user to install and run it. If you use a non-root user, it must be in the /etc./sudoers file. 3. Copy dep_iot.tar.gz to the Linux host (assume copy to ~/install/dep_iot.tar.gz) 4. Un-tar the file as follows: sudo tar -xzf ~/install/dep_iot.tar.gz -C /opt 5. Run Streaming and Persistence Service setup script (setup.sh) as root user or via sudo (required for Linux user creation). setup.sh does the following: Creates a Linux user (streamingliteuser) for Streaming Lite authentication. This user is created with a random password. Streaming Lite uses Linux Pluggable Authentication Modules (PAM) for authentication. If the Linux platform does not support PAM, then you must run setup.sh using the flag --noauth, to skip creating the Linux user. cd /opt/dep_iot_edge sudo./bin/setup.sh noauth o The Linux user (streamingliteuser) user name and encrypted password is written to /opt/dep_iot_edge/store/slauth.properties. Streaming and Persistence Service also reads this file at runtime and uses the credentials when publishing to Streaming Lite. Allows you to keep or delete an existing Streaming and Persistence Service database o For delete existing database, all configuration and IoT data is wiped. o For keep existing database, all configuration and IoT data is maintained. For a new database o Requests the initial password for the Edge Console admin user. 10

11 o Creates a new Streaming and Persistence Service database and required database users, and populates the database with required data. Note: All database user credentials are randomly generated. Requests the initial password for the Streaming and Persistence Service odata data access technical user. o Note: The odata data access technical user is used to authenticate odata queries to Streaming and Persistence Service. Creates ODBC entries for the Streaming and Persistence Service database. The SQL Anywhere dbdsn utility obfuscates the database user s password inside the Data Source Name (DSN). o DEP_IOT_ADMIN: used by Streaming and Persistence Service Java runtime + admin utilities o DEP_IOT_STREAMING: used by Streaming Lite Copies Java libraries from Streaming Lite and SQL Anywhere into /opt/dep_iot_edge/java_lib/ Creates SSL certificates to be used by the built-in plugins: o Edge Console web server for HTTPS. o HttpProtocolPlugin to enable HTTPS and WSS o Creates a NodeId (if not supplied) and writes it to NODE_SETTINGS table To run Streaming and Persistence Service setup script, execute the following commands: cd /opt/dep_iot_edge sudo./bin/setup.sh Note: The final output of setup.sh, if successful, is INFO: Setup completed successfully. Note: All user prompts for setup.sh are also available as command line options. For complete details, run sudo./bin/setup.sh -h 13 LINUX - POSTINSTALLATION SECURITY TASKS The /opt/dep_iot_edge/store folder contains sensitive information, and therefore must be secured via Linux file permissions. Only the user running Streaming and Persistence Service services should have read/write access to this folder and its contents. For example, if running Streaming and Persistence Service as root: chown R root:root /opt/dep_iot_edge/store chmod 700 /opt/dep_iot_edge/store chmod 600 /opt/dep_iot_edge/store/* The Streaming and Persistence Service setup program (setup.sh) creates a Linux user (streamingliteuser) for Streaming Lite authentication. This user is created with a random password. Streaming Lite uses Linux PAM for authentication. You can modify either the username or password of this Linux user: 1. Stop Streaming and Persistence Service. 2. Change the Linux user password (or create a new Linux user). 3. Modify the file containing the user credentials: /opt/dep_iot_edge/store/slauth.properties STREAMING_USERNAME=<Linux_User> STREAMING_ AUTHENTICATION =<Password Base64 encoded> 4. Start Streaming and Persistence Service. To configure Streaming Lite to support PAM authentication: 1. Log in as root. 2. Create the file /etc/pam.d/sp. The contents of this file determine the type of authentication required. For Linux password authentication, the contents of the file /etc/pam.d/sp should be: auth required pam_permit.so auth required pam_warn.so auth required pam_unix.so 14 LINUX - MANAGING STREAMING AND PERSISTENCE SERVICE PROCESSES Streaming and Persistence Service includes processes based on the following technology: 1. Java (IoT Adapter) 11

12 2. Java (Edge Console) 3. Streaming Lite (Streaming Lite) 4. SQL Anywhere (Database) The command script that manages the Streaming and Persistence Service processes is called dep.sh. To run dep.sh, execute the following commands: cd /opt/dep_iot_edge sudo./bin/dep.sh <command> <component> The following commands are available: start stop restart status Starts the Streaming and Persistence Service processes Stops the Streaming and Persistence Service processes Restarts the Streaming and Persistence Service processes Shows the status of all Streaming and Persistence Service processes For example, to list the status of the Streaming and Persistence Service processes, execute the following commands: cd /opt/dep_iot_edge sudo./bin/dep.sh status The following components are available for the start and restart commands: all Starts or restarts all Streaming and Persistence Service allnoauth Starts or restarts all Streaming and Persistence Service without Streaming Lite authentication DB Starts or restarts the Streaming and Persistence Service database IoTAdapter Starts or restarts the Streaming and Persistence Service IoT Adapter Web Starts or restarts the Streaming and Persistence Service Web server StreamingLite Starts or restarts the Streaming and Persistence Service Streaming Lite server StreamingLiteNoAuth Starts or restarts the Streaming and Persistence Service Streaming Lite server without authentication The following components are available for the stop commands: All Stops all Streaming and Persistence Service DB Stops the Streaming and Persistence Service database IoTAdapter Stops the Streaming and Persistence Service IoT Adapter Web Stops the Streaming and Persistence Service Web server StreamingLite Stops the Streaming and Persistence Service Streaming Lite server For example, to start all of the Streaming and Persistence Service processes, execute the following commands: cd /opt/dep_iot_edge sudo./bin/dep.sh start all 15 LINUX - SERVICE CREATION In a production rollout, configure all Streaming and Persistence Service processes to run as background processes that start automatically when the Linux instance boots. There are multiple ways to do this; here, we ll focus on using systemd Linux services. systemd allows you to restart processes in case of failure, define dependencies between services, and more easily configure your system. This section discusses only how to create systemd services for Streaming and Persistence Service, and deploy and configure them to start 12

13 automatically. Refer to the systemd documentation for additional information, including details about specific parameters. systemd services are defined by creating "unit" files in the /lib/systemd/system directory. On some Linux distributions such as SUSE Linux Enterprise 12, this directory does not exist; therefore, use the /etc./systemd/system directory instead. If both /lib/systemd/system and /etc./systemd/system are present, use /lib/systemd/system. The unit files specify how to start or stop services, any dependencies, what action to take if the process fails, and, if the service is auto-started while Linux boots, at what point it should be started. Below is the unit file for the SQL Anywhere database used by Streaming and Persistence Service. Name the file dep_sqlany.service, and copy it to the /lib/systemd/system directory on the Linux system. [Unit] Description=dep_sqlany [Service] Type=forking ExecStart=/bin/bash -c 'cd /opt/dep_iot_edge/bin &&./dep.sh start DB' ExecStop=/bin/bash -c 'cd /opt/dep_iot_edge/bin &&./dep.sh stop DB' Restart=always RestartSec=15 [Install] WantedBy=multi-user.target Restart always restart the process if it is not running. RestartSec restart the process a specified number of seconds (15 in the example above) after it fails. ExecStart specifies the start command to execute. ExecStop specifies the stop command to execute. Systemd executes the command specified in ExecStart and monitors the entire process tree that remains. Type=forking specifies that the command specified in ExecStart executes, launches a background process, and then exits. This process determines whether the service has failed and needs to be restarted WantedBy specifies at which point in the Linux boot process to start this service. Once the unit file is in the required location, execute the following command to get systemd to pick-up the new service definition. You must re-run this command any time you edit any unit file. systemctl daemon-reload The following are standard system commands available to start and stop services: systemctl status <servicename> Display the current status of the service systemctl start <servicename> systemctl stop <servicename> systemctl enable <servicename> systemctl disable <servicename> Start the service manually Stop the service manually Configure the service to auto-start when Linux boots Configure the service to no longer auto-start when Linux boots Below is the unit file for SAP Streaming Lite used by Streaming and Persistence Service. Name the file dep_streaminglite.service and place it in /lib/systemd/system. [Unit] Description=dep_streaminglite 13

14 BindsTo=dep_sqlany.service After=dep_sqlany.service [Service] Type=forking ExecStart=/bin/bash -c 'cd /opt/dep_iot_edge/bin &&./dep.sh start StreamingLite' ExecStop=/bin/bash -c 'cd /opt/dep_iot_edge/bin &&./dep.sh stop StreamingLite' Restart=always RestartSec=15 [Install] WantedBy=multi-user.target The After parameter specifies that the specified service, dep_streaminglite in this case, should be started after the "dep_sqlany" service has started. The BindsTo service specifies that if the dep_sqlany service stops, then the dep_streaminglite service should also be stopped. The After and BindsTo parameters depend on the dep_sqlany service. All the commands specified for dep_sqlany also apply to the After service. Below is the unit file for Edge Console Web Server used by Streaming and Persistence Service. Name the file dep_web.service, and place it in the /lib/systemd/system directory. [Unit] Description=dep_web [Service] Type=forking ExecStart=/bin/bash -c 'cd /opt/dep_iot_edge/bin &&./dep.sh start Web' ExecStop=/bin/bash -c 'cd /opt/dep_iot_edge/bin &&./dep.sh stop Web' Restart=always RestartSec=15 [Install] WantedBy=multi-user.target Below is the unit file for IoT Adapter. Name the file dep_iotadapter.service and place it in /lib/systemd/system. If you are planning on using MQTT for ingesting data and are running a MQTT broker such as Mosquito on the same Linux instance, you can also add a dependency on the MQTT broker service in this unit file. [Unit] Description=dep_IoTadapter BindsTo=dep_sqlany.service BindsTo=dep_streaminglite.service BindsTo=dep_web.service After=dep_sqlany.service After=dep_streaminglite.service After=dep_web.service [Service] Type=forking ExecStart=/bin/bash -c 'cd /opt/dep_iot_edge/bin &&./dep.sh start IoTAdapter' ExecStop=/bin/bash -c 'cd /opt/dep_iot_edge/bin &&./dep.sh stop IoTAdapter' Restart=always RestartSec=15 [Install] WantedBy=multi-user.target 16 DATABASE BACKUPS AND MAINTENANCE Streaming and Persistence Service provides a database that is essentially self-tuning. This database stores configuration, edge events and edge fidelity events. 14

15 The database consists of a database file and a database transaction log file. Database best practices dictate that the database transaction log file must be maintained. Additionally, the database may be configured for backups as required. WARNING: By default, Streaming and Persistence Service is not configured to perform database backups. Streaming and Persistence Service truncates the database transaction log file when it reaches 1GB. This is accomplished via a database event DEP_Admin.TruncateLogFile. If there is a need for separate database backups, then this event should be disabled and an alternate backup and transaction log file maintenance plan should be implemented. Consult SQL Anywhere 17 product documentation for details on database backup options. 17 CUSTOMIZING SSL CONFIGURATION The Streaming and Persistence Service setup (setup.sh/setup.bat) generates self-signed SSL certificates for use by: Streaming and Persistence Service web server for HTTPS HttpProtocolPlugin to enable HTTPS and WSS This section describes how to replace both of the self-signed certificates in the following Java keystore: <dep_iot_edge>/store/dep.jks By default, the Streaming and Persistence Service setup (setup.sh/setup.bat) uses Java Keytool to generate a 2048-bit RSA key pair with a validity period of 10,000 days in dep.jks under the alias httpprotocolplugin. The key pair is then duplicated under the alias com.sap.dep.webserver. The HttpProtocolPlugin makes use of the former and the Streaming and Persistence Service webserver makes use of the latter. Both enable secure communications. Note: If the certificate being placed in the keystore is not a trusted certificate (for example, if it is self-signed), then it should be the same for both aliases so that the WebSocket communication used by the emulated device on the Live Sensors page works without requiring you to go to the web server hosted by HttpProtocolPlugin and manually accepting the untrusted certificate in the browser. If the certificates are trusted by the browser, they can be different without causing any issues. To connect to a secured enterprise endpoint using RESTEnterprisePlugin or WebSocketEnterprisePlugin and default configuration, the required certificates must be put into this cacerts keystore. To add a certificate to the cacerts keystore, use the following command with $keytool and $cacerts being the full path for the current platform: $keytool -import -alias $certificate_alias -keystore $cacerts -file $your_certificate_file To manually generate a key pair with Java keytool for use by HttpProtocolPlugin, use the following command: $keytool -genkeypair -keyalg RSA -alias "httpprotocolplugin" -keystore <dep_iot_edge>/store/dep.jks" -storepass $password_for_keystore -validity keysize 2048 After doing this, the value of the KEY_STORE_AUTHENTICATION configuration for HttpProtocolPlugin in the database should be updated to match $password. Refer to the Protocol Plugin Configuration section for instructions. To manually export the HttpProtocolPlugin key pair into the alias used by the webserver, use the following command: $keytool -importkeystore -srckeystore <dep_iot_edge>/store/dep.jks" -srcalias "httpprotocolplugin" -srcstorepass $password_for_keystore -src-keypass $password_for_keystore -destkeystore <dep_iot_edge>/store/dep.jks" -destalias 15

16 "com.sap.dep.webserver " -deststorepass $password_for_keystore -destkeypass $password_for_webserver_cert Note: To obfuscate password properties that can optionally be obfuscated using Jetty, run the command java -cp jetty-all v jar org.eclipse.jetty.util.security.password $password from the Streaming and Persistence Service java_lib directory, replacing $password with the desired password. The obfuscated version of the password produced by the command can be identified by the OBF: prefix. When using the obfuscated password, you must include the OBF: prefix. After doing this, the value of the com.sap.dep.webserver.ssl.cert.auth property in <dep_iot_edge>/store/webserver.properties should be updated to be the value of $password_for_webserver_cert either Jetty obfuscated value or Base64 encoded. In addition the value of com.sap.dep.webserver.ssl.keystore.auth property in the same file should be updated to the value of $password_for_keystore Jetty obfuscated value or Base64 encoded. To import a PKCS12 file into the Streaming and Persistence Service keystore, use the following command: $keytool -importkeystore -srcalias "1" -srckeystore $pkcs12_filename -srcstorepass $pkcs12_password -srcstoretype PKCS12 -destalias $dest_alias -destkeystore "<dep_iot_edge>/store/dep.jks" -deststorepass $password_for_keystore The value for $dest_alias should be either httpprotocolplugin or com.sap.dep.webserver depending on which certificate you want to import. 18 CUSTOMIZING EDGE CONSOLE PORTS The Jetty web server ports for http and for https are configurable by modifying the file <dep_iot_edge>\store\webserver.properties and restarting the Streaming and Persistence Service. By default, the ports are: com.sap.dep.webserver.httpport=3000 com.sap.dep.webserver.httpsport= LOG FILE CONFIGURATION All Streaming and Persistence Service log files are stored in <dep_iot_edge>/log/ The following is a summary of the log files generated by Streaming and Persistence Service. File Name Description misc_rootlogger.log This is the log4j root log file. IoTAdapter.log Logging for the main Streaming and Persistence Service process IoTAdapter. StreamingLitePlugin.log Logging for the Streaming Lite Engine Plugin. Webserver.log Logging for the Jetty webserver hosting the Edge Console. EdgeConsole.log Logging for the Edge Console program. StreamingLite.log (Linux only) The Streaming Lite log file stout and sterr are redirected to this file. sapdepserver-svc.yyyy-mm-dd.log (Windows The Streaming and Persistence Service Server only) windows service log file service control messages. sapdepweb-svc.yyyy-mm-dd.log (Windows only) The Streaming and Persistence Service Web Server windows service log file service control messages. sapdepesp-stdout.yyyy-mm-dd.log (Windows The Streaming and Persistence Service Streaming only) Server windows service log file sterr error messages. 16

17 sapdepesp-stderr.yyyy-mm-dd.log (Windows only) sapdepesp-svc.yyyy-mm-dd.log (Windows only) HttpProtocolPlugin.log MQTTPlugin.log RESTEnterprisePlugin.log WebSocketEnterprisePlugin.log The Streaming and Persistence Service Streaming Server windows service log file stout output messages. The Streaming and Persistence Service Streaming Server windows service log file service control messages. Logging for the HTTP Plugin. Logging for the MQTT Protocol Plugin. Logging for the REST Enterprise Plugin. Logging for the Web Socket Enterprise Plugin. All files except StreamingLite.log roll over when they reach a size of 5MB. When a file has rolled over five times, the oldest file begins to be overwritten. The 7 Windows service files ending in YYYY-MM-DD.log are by default kept for 5 days and then deleted (can be overridden with service.log.keep.days setting in NODE_SETTING). The configuration of the logging level is stored in the Streaming and Persistence Service database. See RUNTIME SETTINGS Configuration for instructions about changing the global logging level. Additionally, each plugin can have its own log level that overrides the global value for that plugin. See the appropriate plugin Configuration section for instructions about changing a Protocol or Enterprise Plugin log level. 20 CONNECTING TO THE STREAMING AND PERSISTENCE SERVICE DATABASE For advanced configuration and troubleshooting, access to the underlying database may be beneficial. Warning: Direct database access should be performed with caution. To run SQL statements against the Streaming and Persistence Service database, log in to the Streaming and Persistence Service host, open a command prompt and complete the following: Linux only: Source the SQL Anywhere sa_config.sh file source /opt/sqlanywhere17/bin64/sa_config.sh Windows only: Open a command prompt as administrator (since Streaming and Persistence Service DSN s are secured to administrators only) Execute the desired SQL as follows: dbisql -c "DSN=DEP_IOT_ADMIN" "<SQL Statement>" For example: dbisql -c "DSN=DEP_IOT_ADMIN" "SELECT * FROM DEP_Admin.NODE_SETTING" Note: To connect to the database from a different computer, a dedicated database user must be created where the password is known. For more information, see the SQL Anywhere documentation for the GRANT command. 21 RUNTIME SETTINGS CONFIGURATION The Edge Console contains screens for viewing and editing Runtime Settings. Runtime Settings configuration can be updated in the Edge Console, or by using the dbisql command line interface. For a list of settings, please refer to the Runtime Settings section of the Streaming and Persistence Service Administration Guide. Command-line options for configuration Runtime Settings are as follows: To view the current configuration, run the following command via dbisql: SELECT * FROM DEP_Admin.v_node_setting To update the current configuration, run the following command via dbisql: CALL DEP_Admin.sp_UpsertNodeSetting (<CONFIG_NAME>, <CONFIG_VALUE>, 1) To delete a configuration item, run the following command via dbisql: CALL DEP_Admin.sp_DeleteNodeSetting (<CONFIG_NAME>) 17

18 22 BUSINESS ESSENTIAL FUNCTIONS SERVICE CONFIGURATION To support the Create Work Order action, Streaming and Persistence Service requires the configuration of the Business Essential Functions Service host. Please refer to the Streaming and Persistence Service Administration Guide for instructions on configuring the host, port, and authentication parameters. 23 ENTERPRISE PLUGIN CONFIGURATION Enterprise Plugin configuration can be viewed and updated using the Edge Console, or the dbisql command line interface. Initially, there are no configured Enterprise Plugins for a deployed edge instance. The following Enterprise Plugin types are supported: RESTEnterprisePlugin: outbound WebsocketEnterprisePlugin (Client): outbound to WebSocket server Custom (created by following the instructions in the Developer Guide) Note: Please refer to the Streaming and Persistence Service Administration Guide for a detailed list of configuration options for RESTEnterprisePlugin and WebsocketEnterprisePlugin. Every configured Enterprise Plugin must be listed in the Enterprise Plugin Table. The configuration of an Enterprise Plugin is contained in both the Enterprise Plugin Table and the Enterprise Plugin Configuration Table. Note: To obfuscate password properties that can optionally be obfuscated, run the command java -cp jetty-all v jar org.eclipse.jetty.util.security.password $password from the Streaming and Persistence Service java_lib directory and replace $password with the desired password. The obfuscated version of the password produced by the command can be identified by the OBF: prefix. When using the obfuscated password, you must include the OBF: prefix Configuring an Enterprise Plugin using dbisql 1. View and update the existing configured Enterprise Plugin. To view the existing Enterprise Plugin configuration, run the following command via dbisql: SELECT * FROM DEP_Admin.v_enterprise_plugin To configure an Enterprise Plugin, run the following command via dbisql: CALL DEP_Admin.sp_UpsertEnterprisePlugin (<ENTERPRISE_PLUGIN_ID>, <PLUGIN_CLASS>, <DESCRIPTION>, <PROCESS_IN_ORDER>, <THREADS>, <RETRY_COUNT>, <RETRY_DELAY>, <LOG_MESSAGE_ON_FAIL>) To delete an Enterprise Plugin, run the following command via dbisql: CALL DEP_Admin.sp_DeleteEnterprisePlugin (<ENTERPRISE_PLUGIN_ID>) 2. View and update the current configuration for the Enterprise Plugin. To view the configuration of an Enterprise Plugin, run the following command via dbisql: SELECT * FROM DEP_Admin.v_enterprise_plugin_configuration To configure an Enterprise Plugin, run the following command via dbisql: CALL DEP_Admin.sp_UpsertEnterprisePluginConfig (<ENTERPRISE_PLUGIN_ID>, <CONFIG_NAME>, <CONFIG_VALUE>) Parameter Type Example 18

19 <ENTERPRISE- Varchar(128) com.sap.dep.protocolplugin.mqtt.mqttplugin PLUGIN-ID> <CONFIG-NAME> Varchar(128) LOGGERLEVEL <CONFIG-VALUE> Long Varchar DEBUG In order to delete the configuration of an Enterprise Plugin, run the following via dbisql: CALL DEP_Admin.sp_DeleteEnterprisePluginConfig (<ENTERPRISE_PLUGIN_ID>, <CONFIG_NAME>) 3. (Optional) To connect to a secured enterprise endpoint using RESTEnterprisePlugin or WebSocketEnterprisePlugin, you must place the required certificates into the cacerts keystore. The keytool used by Streaming and Persistence Service is in /opt/sqlanywhere17/bin64/jre180/bin/keytool, and the cacerts file is in /opt/sqlanywhere17/bin64/jre180/lib/security/cacerts. 4. Restart Streaming and Persistence Service services for the updates to the Enterprise Plugin configuration to take effect. 24 PROTOCOL PLUGIN CONFIGURATION Existing Protocol Plugin configuration can be viewed and updated via the Edge Console. Additional Protocol Plugins can only be added using the dbisql command line interface. Initially, only the HTTP Protocol Plugin is enabled on a deployed Streaming and Persistence Service instance. No other Protocol Plugins are enabled. The following Protocol Plugin types are supported by Streaming and Persistence Service: HTTP: supports REST inbound and WebSocket Server (bi-directional); pre-configured; required for the Edge Console MQTT: to receive data via MQTT Protocol Custom (created following the instructions in the Developer Guide) Security Notes The provided HTTP Protocol Plugin supports token-based authentication (please refer to the Streaming and Persistence Service Administration Guide for instructions on configuring authentication tokens). If additional authentication is required, then a custom Protocol Plugin must be developed. A custom Protocol Plugin is responsible for implementation of authentication and authorization of the inbound messages. This implementation will be dependent on the protocol support and also the level of desired security in the solution. Each Protocol Plugin supports a unique set of properties, and these properties are stored in Streaming and Persistence Service in the Protocol Plugin Configuration Table. Note: To obfuscate password properties that can optionally be obfuscated using Jetty, run the command java -cp jetty-all v jar org.eclipse.jetty.util.security.password $password from the Streaming and Persistence Service java_lib directory, replacing $password with the desired password. The obfuscated version of the password produced by the command can be identified by the OBF: prefix. When using the obfuscated password, you must include the OBF: prefix. Settings for Protocol Plugin Configuration Table (LOCAL_PLUGIN_CONFIGURATION) Plugin Setting Name <CONFIG_NAME> Description Sample Value <CONFIG_VALUE> HTTP Please refer to the Streaming and Persistence Service Administration Guide for HTTP Protocol Plugin configuration options. MQTT CLEAN_SESSION_ON_ RECONNECT Flag for whether the MQTT client should be run with or without true 19

20 (optional) PERSISTENCE_METHO D (required, when CLEAN_SESSION_ON_ RECONNECT is false) PERSISTENCE_DIRECT ORY_PATH (required, when PERSISTENCE_METHO D is file) HOST (required) LOGGER_LEVEL (required) PUBLISHING_TOPIC (required) QOS (optional) RECONNECT_ATTEMP T_INTERVAL (optional) RECONNECT_ATTEMP T_LIMIT (optional) maintaining state information when a new session is started. If not supplied, defaults to true. Determines how to store state information across sessions. Used when CLEAN_SESSION_ON_RECONNE CT is set to false. Options are: - memory - file If set to file, then the PERSISTENCE_DIRECTORY_PAT H is required. The full path to where state information is saved. MQTT server address Controls the verbosity of logging within plugin. Available log levels from most to least logging: - TRACE - DEBUG - INFO - WARN - ERROR - FATAL - OFF The topic to which the MQTT Plugin publishes. Field messages routed to MQTTPlugin go out over this topic. Quality of service level set for publishing a topic. If not supplied, defaults to At most once At least once Exactly once Refer to: Interval, in milliseconds, to wait until next reconnection attempt by MQTT client. If not supplied, defaults to Maximum number of times the MQTT client attempts to reconnect. Once the maximum number of tries is reached, no more attempts are made. If not supplied, defaults to 0, meaning it does not try to reconnect. memory <dep_iot_edge>/tmp tcp://localhost:1883 INFO depevent

21 SUBSCRIBING_TOPIC (required) USER (required) Topic to which the MQTT client subscribes. Sensor reading input should come in over this topic. Unique identifier that MQTT client uses to connect to broker. depmessage com.sap.dep.protocolplugin. mqtt.mqttplugin Configuring a Protocol or Engine Plugin using dbisql To view the current list of provisioned Protocol and Engine plugins, and to retrieve the LOCAL_PLUGIN_ID values, run the following command via dbisql: SELECT * FROM DEP_Admin.v_local_plugin To add a Protocol or Engine Plugin to Streaming and Persistence Service, run the following command via dbisql: CALL DEP_Admin.sp_UpsertLocalPlugin (<LOCAL_PLUGIN_ID>, <PLUGIN_NAME>, <IS- FIELD-PLUGIN>, <PLUGIN_TYPE_ID>) Parameter Type Description Example <LOCAL_PL Varchar(128) UGIN_ID> <PLUGIN_N AME> <IS- FIELD- PLUGIN> <PLUGIN_T YPE_ID> ID of plugin, canonical name of the class that implements IProtocolPlugin Varchar(128) Name of plugin MQTTPlugin Bit Varchar(10) Whether plugin supports field messages that are sent to it Type of plugin, either PROTOCOL or ENGINE com.sap.dep.protocolplugin.mqtt.mqttp lugin can handle Field Message output cannot handle field message output. PROTOCOL To delete a Protocol or Engine Plugin from Streaming and Persistence Service, run the following command via dbisql. Deleting a plugin automatically deletes all associated configuration and routing. CALL DEP_Admin.sp_DeleteLocalPlugin (<LOCAL_PLUGIN_ID>) 1. View and update current configuration for the Protocol or Engine Plugin. To view the current configuration of all Protocol or Engine Plugins, run the following command via dbisql: SELECT * FROM DEP_Admin.v_local_plugin_configuration To view the current configuration of a specific Protocol or Engine Plugin, run the following command via dbisql: SELECT * FROM DEP_Admin.v_local_plugin_configuration WHERE LOCAL_PLUGIN_ID = <LOCAL_PLUGIN_ID> To update the current configuration of a specific Protocol or Engine Plugin, run the following command via dbisql: CALL DEP_Admin.sp_UpsertLocalPluginConfig (<LOCAL_PLUGIN_ID>, <CONFIG_NAME>, <CONFIG_VALUE>) Parameter Type Description Example <LOCAL- PLUGIN-ID> Varchar (128) ID of plugin com.sap.dep.protocolplugin.mqtt.m QTTPlugin <CONFIG- NAME> Varchar (128) Name of property LOGGERLEVEL 21