SAP Edge Services, on-premise edition Streaming Service - Configuration Guide Version 3.0 FP01

Transcription

1 SAP Edge Services, on-premise edition Streaming Service - Configuration Guide Version 3.0 FP01

2 TABLE OF CONTENTS 1 INTRODUCTION ARCHITECTURE SUPPORTED OPERATING SYSTEMS SUPPORTED WINDOWS VERSIONS WINDOWS PREREQUISITES AND INSTALLATION Java SE Runtime (version 1.8.0_ bit) Python (version 2.7.x 64-bit) WINDOWS - STREAMING SERVICE INSTALLATION Download Required Components from SAP Service Marketplace EDGE DB 3.0 Component SAP ESP SERVER 5.1 Component DEP_IOT_EDGE 3.0 Component WINDOWS - POSTINSTALLATION SECURITY TASKS WINDOWS - MANAGING STREAMING SERVICE PROCESSES WINDOWS - UNINSTALLING STREAMING SERVICE SUPPORTED LINUX VERSIONS LINUX PREREQUISITES AND INSTALLATION Java SE Runtime (version 1.8.0_31+) Python (version 2.7.x) LINUX - STREAMING SERVICE INSTALLATION Download Required Components from SAP Service Marketplace EDGE DB 3.0 Component STREAMING LITE 2.0 Component DEP_IOT_EDGE 3.0 Component LINUX - POSTINSTALLATION SECURITY TASKS LINUX - MANAGING STREAMING SERVICE PROCESSES LINUX - SERVICE CREATION DATABASE BACKUPS AND MAINTENANCE CUSTOMIZING SSL CONFIGURATION CUSTOMIZING EDGE CONSOLE PORTS LOG FILE CONFIGURATION ODATA CONFIGURATION (DEPRECATED) CONNECTING TO THE STREAMING SERVICE DATABASE TROUBLESHOOTING KNOWN ISSUES

3 1 INTRODUCTION This document focuses on the installation and configuration of SAP Edge Services - Streaming Service. 2 ARCHITECTURE The Streaming Service architecture is shown below. The area within the dashed line is considered the boundary of Streaming Service. Streaming Service is comprised of several processes: IoT Adapter Edge Console Streaming a. SAP Streaming Lite (Linux) b. SAP Event Stream Processor (Windows) Persistence a. SAP SQL Anywhere (Database) SAP Plant Connectivity (PCo) optional - provides additional connectivity to devices and sensors (PCo Product Documentation: 3 SUPPORTED OPERATING SYSTEMS Streaming Service supports Windows x64 (x86 64-bit), Linux x64 (x86 64-bit) and Linux ARM (32-bit) operating systems. Separate installation procedures are provided for Windows and Linux. Note: On Windows x64 and Linux x64 (but not Linux ARM), Streaming Service provides an OData interface to query data. OData interface is not supported on Linux ARM release of Streaming Service. Note: This document refers to the Streaming Service installation folder as <dep_iot_edge>. This folder is different on Windows and Linux: Windows must be located in a folder with no spaces in the path: for example: c:\dep_iot_edge or c:\sap\dep_iot_edge Linux must be located in /opt : /opt/dep_iot_edge 4 SUPPORTED WINDOWS VERSIONS Streaming Service supports the following Windows x64 versions: 3

4 Windows 8.1 x64 (x86 64-bit) Windows 10 x64 (x86 64-bit) Windows 2012 Server x64 (x86 64-bit) Windows 7 x64 (x86 64-bit) 5 WINDOWS PREREQUISITES AND INSTALLATION Streaming Service require several software packages to already be installed on the host. Prerequisites include the following: Java SE Runtime version 1.8.0_ bit (this is installed by SQL Anywhere 17) Python version 2.7.x 64-bit The detailed installation requirements are discussed below. 5.1 Java SE Runtime (version 1.8.0_ bit) SAP SQL Anywhere 17 automatically installs the required Java SE Runtime (recommended); or a compatible JRE can be downloaded and installed separately. Note: If using SAP SQL Anywhere 17 version of JRE, then the JAVA_HOME system environment variable must be manually configured (it can point to a different valid JRE version 1.8.0_31+) 5.2 Python (version 2.7.x 64-bit) Python can be downloaded from Python can be installed to any recommended directory. Note: When installing Python, select the option to add Python in the SYSTEM PATH environment variable. 6 WINDOWS - STREAMING SERVICE INSTALLATION 6.1 Download Required Components from SAP Service Marketplace Streaming Service requires several software components to be downloaded and installed from SAP Service Marketplace under SAP Edge Services : EDGE DB 3.0 Component (SAP SQL Anywhere 17 version bit) SAP ESP SERVER 5.1 Component (SAP Event Stream Processor 5.1 SP12 PL13 version / /SP12) DEP_IOT_EDGE 3.0 Component ( Streaming Service ) Go to SAP Service Marketplace - Software Downloads 1. Search for SAP Edge Services 2. In search results, click on SAP Edge Services, OP 3.0 Maintenance Product 3. Select Downloads and then Comprised Software Component Versions a. Select EDGE DB 3.0 i. Select Windows on x64 64bit and download the latest package (remaining instructions assume file is named sql_anywhere-x64.zip). b. Select SAP ESP SERVER 5.1 i. Select Windows on x64 64bit and download the latest package (remaining instructions assume file is named esp_server-x64.zip). c. Select DEP_IOT_EDGE 3.0 i. Select Windows on x64 64bit and download the latest package (remaining instructions assume file is named dep_iot.zip). The installation of these components is described below. 4

5 6.2 EDGE DB 3.0 Component To install EDGE DB 3.0 Component (SAP SQL Anywhere 17), unzip sql_anywhere-x64.zip and install the enclosed msi file. Note: SAP SQL Anywhere is OEM licensed only for use by SAP Edge Services. 6.3 SAP ESP SERVER 5.1 Component The setup process for SAP ESP SERVER 5.1 is as follows: 1. Decide where to put esp_server-x64.zip (C:\SAP\dep_streaming is recommended) 2. Unzip esp_server-x64.zip to that location 3. After unzip, the following folder should exist: C:\SAP\dep_streaming\ESP-5_1 4. Create a system environment variable for STREAMING_HOME (this must be a system environment variable not a user environment variable): STREAMING_HOME = C:\SAP\dep_streaming\ESP-5_1 Note: SAP Event Stream Processor is installed in such a way that only Streaming Service can use the installed version of SAP Event Stream Processor. SAP Event Stream Processor is OEM licensed only for use by SAP Edge Services. 6.4 DEP_IOT_EDGE 3.0 Component Note: Open and run all command prompts as Administrator. Note: DEP_IOT_EDGE 3.0 component must be installed to path with no spaces in the path, such as, c:\dep_iot_edge or c:\sap\dep_iot_edge. Note: DEP_IOT_EDGE 3.0 component is referred to as Streaming Service below. 1. Ensure that all prerequisite software is successfully installed. A reboot is required after installing the prerequisite software. 2. Streaming Service must be installed by a member of the local Administrators group. 3. Copy dep_iot.zip to the Windows host (for example, copy to C:\SAP\) 4. Unzip the file to a location without a space in the path. The path c:\sap\ is recommended (this creates a c:\sap\dep_iot_edge folder). 5. Run Streaming Service setup script (setup.bat). setup.bat does the following: Allows you to keep or delete an existing Streaming Service database o For delete existing database, all configuration and IoT data is wiped. o For keep existing database, all configuration and IoT data is maintained. For a new database o Requests the initial password for the Edge Console admin user. o Creates a new Streaming Service database and required database users, and populates the database with required data. Note: All database user credentials are randomly generated. Requests the initial password for the Streaming Service odata data access technical user. o Note: the odata data access technical user is used to authenticate odata queries to Streaming Service. Creates ODBC entry for the Streaming Service database. The SQL Anywhere dbdsn utility obfuscates the database user s password inside the Data Source Name (DSN). o DEP_IOT_ADMIN: used by Streaming Service Java runtime + admin utilities Copies Java libraries from Event Stream Processor and SQL Anywhere into c:\sap\dep_iot_edge\java_lib\ Creates SSL certificates to be used by the built-in plugins: o Edge Console web server for HTTPS 5

6 o o HttpProtocolPlugin to enable HTTPS and WSS Creates a NodeId (if not supplied) and writes it to NODE_SETTINGS table To run Streaming Service setup script, execute the following commands (in command prompt as Administrator): cd c:\sap\dep_iot_edge\bin setup.bat Note: The final output of setup.bat, if successful, is INFO: Setup completed successfully. Note: All user prompts for setup.bat are also available as command line options. For complete details run setup.bat -h 7 WINDOWS - POSTINSTALLATION SECURITY TASKS By default, the Streaming Service setup program applies the following permissions: c:\sap\dep_iot_edge only allows access to Administrators group and LOCAL_SERVICE account All Streaming Service Windows services are only controllable by a member of the local Administrators group 8 WINDOWS - MANAGING STREAMING SERVICE PROCESSES Streaming Service creates 4 Windows Services all are installed to run as Windows services that start automatically when the machine boots. The required inter-service dependencies are configured by default. 1. SAP Edge Services Server 2. SAP Edge Services Web Server 3. SAP Edge Services Streaming Server 4. SAP Edge Services Database Use the Windows Services control panel applet to start and stop services. 9 WINDOWS - UNINSTALLING STREAMING SERVICE Streaming Service provides an uninstall script for Windows. This script prompts (twice) for confirmation to remove each the following: 1. First Prompt: Streaming Service Windows Services 2. Second Prompt: Streaming Service Database + Files a. Database file b. Database ODBC entries c. Program files Warning: If the Streaming Service Database + Files are deleted, then all configuration, data, history, etc. is deleted from that host machine. To run the uninstall script on Windows, open a command prompt (as Administrator) and run the following: cd <dep_iot_edge>\bin uninstalldep.bat Note: If the database and files are selected to delete, then the script finishes with The system cannot find the path specified. This is expected and can safely be ignored. 10 SUPPORTED LINUX VERSIONS Linux is available from several vendors. In addition, a wide variety of Linux installations are possible, which poses a challenge for companies such as SAP who want to maintain an open and flexible approach to supporting Linux, yet require a practical means of identifying which Linux installations they support. Support for Linux is dependent on the Streaming Service prerequisites support for Linux. For example, if all of the prerequisites support a specific version of Linux, then theoretically Streaming Service should also support 6

7 that version of Linux; however untested versions of Linux may have slightly different configurations that the Streaming Service installer does not recognize. Please contact your SAP representative for further details. SAP SQL Anywhere for Linux Supported Platforms: SAP Streaming Lite Platform and Installation: m Note: Streaming Lite supports Linux x64 (x86 64-bit) and Linux ARM 32-bit. Streaming Lite does not support Linux x86 32-bit. Tested Linux distributions for deployment: Ubuntu Desktop LTS x64 (x86 64-bit) Ubuntu Desktop LTS x64 (x86 64-bit) RHEL 7.3 x64 (x86 64-bit) (See Known Issues 1, 2, 3) SUSE 12 x64 (x86 64-bit) (See Known Issues 1, 2, 3, 4) Intel Intelligent Device Platform (IDP) 3.1 Wind River 7 x64 (x86 64-bit) (See Known Issues 1, 2) Tested Linux distributions for prototyping: Raspbian v7 (Jessie) (ARM 32-bit) 11 LINUX PREREQUISITES AND INSTALLATION Streaming Service requires several software packages to already be installed on the host. Prerequisites include the following: Java SE Runtime version 1.8.0_31+ (this is installed by SQL Anywhere 17) Python version 2.7.x The detailed installation requirements are discussed below Java SE Runtime (version 1.8.0_31+) SAP SQL Anywhere 17 automatically installs the required Java SE Runtime (recommended); or a compatible JRE can be downloaded and installed separately. Note: If the JAVA_HOME environment variable is not already configured, then the Streaming Service control script (dep.sh) correctly sets JAVA_HOME to the SQL Anywhere JRE folder (dep.sh calls the SQL Anywhere sa_config file, which sets the JAVA_HOME environment variable if it is not already configured) Python (version 2.7.x) Python can be installed either by downloading it from or by using the package manager local to the Linux distribution (apt, yum, or rpm). Python can be installed to any recommended directory and must be included in the PATH environment variable. 12 LINUX - STREAMING SERVICE INSTALLATION 12.1 Download Required Components from SAP Service Marketplace Streaming Service requires several software components to be download and installed from SAP Service Marketplace under SAP Edge Services : EDGE DB 3.0 Component (SAP SQL Anywhere 17 version ) STREAMING LITE 2.0 Component (SAP Streaming Lite 2.0 SP01 PL04 version / /SP01 Rev012 PL04) DEP_IOT_EDGE 3.0 Component ( Streaming Service ) Go to SAP Service Marketplace - Software Downloads 1. Search for SAP Edge Services 2. In search results, click on SAP Edge Services, OP 3.0 Maintenance Product 3. Select Downloads and then Comprised Software Component Versions 7

8 a. Select EDGE DB 3.0 i. Select appropriate Linux version (x86 64bit or ARM 32-bit) and download the latest package (remaining instructions assume file is named sql_anywhere-x64.gz). b. Select Streaming Lite 2.0 i. Select appropriate Linux version (x86 64bit or ARM 32-bit) and download the latest package (remaining instructions assume the Streaming Lite installation file is named SDSLITE_Linux_x86_64.gz). c. Select DEP_IOT_EDGE 3.0 i. Select appropriate Linux version (x86 64bit or ARM32) and download the latest package (remaining instructions assume file is named dep_iot.zip). The installation of these components is described below EDGE DB 3.0 Component EDGE DB 3.0 Component (SQL Anywhere 17) must be installed to /opt/sqlanywhere17 (exact folder name required). Installation instructions are: 1. Copy the EDGE DB 3.0 installation file to the Linux host. 2. Copy and extract the EDGE DB 3.0 installation file in the /opt folder (exact folder name required). Rename any /opt/sqlanywhere17 folder if it already exists. sudo cp sql_anywhere-x64.gz /opt/ cd /opt sudo tar xvf sql_anywhere-x64.gz 3. Remove the SQL Anywhere installation file: rm sql_anywhere-x64.gz Note: SAP SQL Anywhere is OEM licensed only for use by SAP Edge Services STREAMING LITE 2.0 Component STREAMING LITE 2.0 Component must be installed to /opt/streaminglite (exact folder name required). Installation instructions are: 1. Copy SDSLITE_Linux_x86_64.gz to the /opt/streaminglite folder (exact folder name required) on the Linux host. Create the /opt/streaminglite folder if it does not exist. sudo mkdir /opt/streaminglite sudo cp SDSLITE_Linux_x86_64.gz /opt/streaminglite/ 2. Extract SDSLITE_Linux_x86_64.gz into /opt/streaminglite: cd /opt/streaminglite sudo tar xvf SDSLITE_Linux_x86_64.gz 3. Remove SDSLITE_Linux_x86_64.gz: rm SDSLITE_Linux_x86_64.gz 4. Once extracted, verify that the Streaming Lite binaries are in /opt/streaminglite/install/bin (required). Note: STREAMING_HOME environment variable is automatically set by the Streaming Service control script (dep.sh). Note: SAP Streaming Lite is OEM licensed only for use by SAP Edge Services DEP_IOT_EDGE 3.0 Component Note: DEP_IOT_EDGE 3.0 component must be installed to /opt/dep_iot_edge Note: DEP_IOT_EDGE 3.0 component is referred to as Streaming Service below. 1. Ensure all prerequisite software is successfully installed. 2. Setup installation user: a. If Streaming Service will be running as Linux services, you can install it either as a root user, or a non-root user. Consult your Linux documentation for more details. 8

9 b. If Streaming Service is not running as Linux services, it is recommended that you create a nonroot user to install and run it. If you use a non-root user, it must be in the /etc./sudoers file. 3. Copy dep_iot.tar.gz to the Linux host (assume copy to ~/install/dep_iot.tar.gz) 4. Un-tar the file as follows: sudo tar -xzf ~/install/dep_iot.tar.gz -C /opt 5. Run Streaming Service setup script (setup.sh) as root user or via sudo (required for Linux user creation). setup.sh does the following: Creates a Linux user (streamingliteuser) for Streaming Lite authentication. This user is created with a random password. Streaming Lite uses Linux Pluggable Authentication Modules (PAM) for authentication. If the Linux platform does not support PAM, then you must run setup.sh using the flag --noauth, to skip creating the Linux user. cd /opt/dep_iot_edge sudo./bin/setup.sh noauth o The Linux user (streamingliteuser) user name and encrypted password is written to /opt/dep_iot_edge/store/slauth.properties. Streaming Service also reads this file at runtime and uses the credentials when publishing to Streaming Lite. Allows you to keep or delete an existing Streaming Service database o For delete existing database, all configuration and IoT data is wiped. o For keep existing database, all configuration and IoT data is maintained. For a new database o Requests the initial password for the Edge Console admin user. o Creates a new Streaming Service database and required database users, and populates the database with required data. Note: All database user credentials are randomly generated. Requests the initial password for the Streaming Service odata data access technical user. o Note: The odata data access technical user is used to authenticate odata queries to Streaming Service. Creates ODBC entry for the Streaming Service database. The SQL Anywhere dbdsn utility obfuscates the database user s password inside the Data Source Name (DSN). o DEP_IOT_ADMIN: used by Streaming Service Java runtime + admin utilities Copies Java libraries from Streaming Lite and SQL Anywhere into /opt/dep_iot_edge/java_lib/ Creates SSL certificates to be used by the built-in plugins: o Edge Console web server for HTTPS. o HttpProtocolPlugin to enable HTTPS and WSS o Creates a NodeId (if not supplied) and writes it to NODE_SETTINGS table To run Streaming Service setup script, execute the following commands: cd /opt/dep_iot_edge sudo./bin/setup.sh Note: The final output of setup.sh, if successful, is INFO: Setup completed successfully. Note: All user prompts for setup.sh are also available as command line options. For complete details, run sudo./bin/setup.sh -h 13 LINUX - POSTINSTALLATION SECURITY TASKS The /opt/dep_iot_edge/store folder contains sensitive information, and therefore must be secured via Linux file permissions. Only the user running Streaming Service services should have read/write access to this folder and its contents. For example, if running Streaming Service as root: chown R root:root /opt/dep_iot_edge/store chmod 700 /opt/dep_iot_edge/store chmod 600 /opt/dep_iot_edge/store/* 9

10 The Streaming Service setup program (setup.sh) creates a Linux user (streamingliteuser) for Streaming Lite authentication. This user is created with a random password. Streaming Lite uses Linux PAM for authentication. You can modify either the username or password of this Linux user: 1. Stop Streaming Service. 2. Change the Linux user password (or create a new Linux user). 3. Modify the file containing the user credentials: /opt/dep_iot_edge/store/slauth.properties STREAMING_USERNAME=<Linux_User> STREAMING_ AUTHENTICATION =<Password Base64 encoded> 4. Start Streaming Service. To configure Streaming Lite to support PAM authentication: 1. Log in as root. 2. Create the file /etc/pam.d/sp. The contents of this file determine the type of authentication required. For Linux password authentication, the contents of the file /etc/pam.d/sp should be: auth required pam_permit.so auth required pam_warn.so auth required pam_unix.so 14 LINUX - MANAGING STREAMING SERVICE PROCESSES Streaming Service includes processes based on the following technology: 1. Java (IoT Adapter) 2. Java (Edge Console) 3. Streaming Lite (Streaming Lite) 4. SQL Anywhere (Database) The command script that manages the Streaming Service processes is called dep.sh. To run dep.sh, execute the following commands: cd /opt/dep_iot_edge sudo./bin/dep.sh <command> <component> The following commands are available: start stop restart status Starts the Streaming Service processes Stops the Streaming Service processes Restarts the Streaming Service processes Shows the status of all Streaming Service processes For example, to list the status of the Streaming Service processes, execute the following commands: cd /opt/dep_iot_edge sudo./bin/dep.sh status The following components are available for the start and restart commands: all Starts or restarts all Streaming Service allnoauth Starts or restarts all Streaming Service without Streaming Lite authentication DB Starts or restarts the Streaming Service database IoTAdapter Starts or restarts the Streaming Service IoT Adapter Web Starts or restarts the Streaming Service Web server StreamingLite Starts or restarts the Streaming Service Streaming Lite server StreamingLiteNoAuth Starts or restarts the Streaming Service Streaming Lite server without authentication The following components are available for the stop commands: 10

11 All DB IoTAdapter Web StreamingLite Stops all Streaming Service Stops the Streaming Service database Stops the Streaming Service IoT Adapter Stops the Streaming Service Web server Stops the Streaming Service Streaming Lite server For example, to start all of the Streaming Service processes, execute the following commands: cd /opt/dep_iot_edge sudo./bin/dep.sh start all 15 LINUX - SERVICE CREATION In a production rollout, configure all Streaming Service processes to run as background processes that start automatically when the Linux instance boots. There are multiple ways to do this; here, we ll focus on using systemd Linux services. systemd allows you to restart processes in case of failure, define dependencies between services, and more easily configure your system. This section discusses only how to create systemd services for Streaming Service, and deploy and configure them to start automatically. Refer to the systemd documentation for additional information, including details about specific parameters. systemd services are defined by creating "unit" files in the /lib/systemd/system directory. On some Linux distributions such as SUSE Linux Enterprise 12, this directory does not exist; therefore, use the /etc./systemd/system directory instead. If both /lib/systemd/system and /etc./systemd/system are present, use /lib/systemd/system. The unit files specify how to start or stop services, any dependencies, what action to take if the process fails, and, if the service is auto-started while Linux boots, at what point it should be started. Below is the unit file for the SQL Anywhere database used by Streaming Service. Name the file dep_sqlany.service, and copy it to the /lib/systemd/system directory on the Linux system. [Unit] Description=dep_sqlany [Service] Type=forking ExecStart=/bin/bash -c 'cd /opt/dep_iot_edge/bin &&./dep.sh start DB' ExecStop=/bin/bash -c 'cd /opt/dep_iot_edge/bin &&./dep.sh stop DB' Restart=always RestartSec=15 [Install] WantedBy=multi-user.target Restart always restart the process if it is not running. RestartSec restart the process a specified number of seconds (15 in the example above) after it fails. ExecStart specifies the start command to execute. ExecStop specifies the stop command to execute. Systemd executes the command specified in ExecStart and monitors the entire process tree that remains. Type=forking specifies that the command specified in ExecStart executes, launches a background process, and then exits. This process determines whether the service has failed and needs to be restarted WantedBy specifies at which point in the Linux boot process to start this service. Once the unit file is in the required location, execute the following command to get systemd to pick-up the new service definition. You must re-run this command any time you edit any unit file. systemctl daemon-reload The following are standard system commands available to start and stop services: 11

12 systemctl status <servicename> systemctl start <servicename> systemctl stop <servicename> systemctl enable <servicename> systemctl disable <servicename> Display the current status of the service Start the service manually Stop the service manually Configure the service to auto-start when Linux boots Configure the service to no longer auto-start when Linux boots Below is the unit file for SAP Streaming Lite used by Streaming Service. Name the file dep_streaminglite.service and place it in /lib/systemd/system. [Unit] Description=dep_streaminglite BindsTo=dep_sqlany.service After=dep_sqlany.service [Service] Type=forking ExecStart=/bin/bash -c 'cd /opt/dep_iot_edge/bin &&./dep.sh start StreamingLite' ExecStop=/bin/bash -c 'cd /opt/dep_iot_edge/bin &&./dep.sh stop StreamingLite' Restart=always RestartSec=15 [Install] WantedBy=multi-user.target The After parameter specifies that the specified service, dep_streaminglite in this case, should be started after the "dep_sqlany" service has started. The BindsTo service specifies that if the dep_sqlany service stops, then the dep_streaminglite service should also be stopped. The After and BindsTo parameters depend on the dep_sqlany service. All the commands specified for dep_sqlany also apply to the After service. Below is the unit file for Edge Console Web Server used by Streaming Service. Name the file dep_web.service, and place it in the /lib/systemd/system directory. [Unit] Description=dep_web [Service] Type=forking ExecStart=/bin/bash -c 'cd /opt/dep_iot_edge/bin &&./dep.sh start Web' ExecStop=/bin/bash -c 'cd /opt/dep_iot_edge/bin &&./dep.sh stop Web' Restart=always RestartSec=15 [Install] WantedBy=multi-user.target Below is the unit file for IoT Adapter. Name the file dep_iotadapter.service and place it in /lib/systemd/system. If you are planning on using MQTT for ingesting data and are running a MQTT broker such as Mosquito on the same Linux instance, you can also add a dependency on the MQTT broker service in this unit file. [Unit] Description=dep_IoTadapter BindsTo=dep_sqlany.service BindsTo=dep_streaminglite.service BindsTo=dep_web.service 12

13 After=dep_sqlany.service After=dep_streaminglite.service After=dep_web.service [Service] Type=forking ExecStart=/bin/bash -c 'cd /opt/dep_iot_edge/bin &&./dep.sh start IoTAdapter' ExecStop=/bin/bash -c 'cd /opt/dep_iot_edge/bin &&./dep.sh stop IoTAdapter' Restart=always RestartSec=15 [Install] WantedBy=multi-user.target 16 DATABASE BACKUPS AND MAINTENANCE Streaming Service provides a database that is self-tuning. This database stores configuration, edge events and edge fidelity events. The database consists of a database file and a database transaction log file. Database best practices dictate that the database transaction log file must be maintained. Additionally, the database may be configured for backups as required. WARNING: By default, Streaming Service is not configured to perform database backups. Streaming Service truncates the database transaction log file when it reaches 1GB. This is accomplished via a database event DEP_Admin.TruncateLogFile. If there is a need for separate database backups, then this event should be disabled and an alternate backup and transaction log file maintenance plan should be implemented. Consult SQL Anywhere 17 product documentation for details on database backup options. 17 CUSTOMIZING SSL CONFIGURATION The Streaming Service setup (setup.sh/setup.bat) generates self-signed SSL certificates for use by: Streaming Service web server for HTTPS HttpProtocolPlugin to enable HTTPS and WSS This section describes how to replace both of the self-signed certificates in the following Java keystore: <dep_iot_edge>/store/dep.jks By default, the Streaming Service setup (setup.sh/setup.bat) uses Java Keytool to generate a 2048-bit RSA key pair with a validity period of 10,000 days in dep.jks under the alias httpprotocolplugin. The key pair is then duplicated under the alias com.sap.dep.webserver. The HttpProtocolPlugin makes use of the former and the Streaming Service webserver makes use of the latter. Both enable secure communications. Note: If the certificate being placed in the keystore is not a trusted certificate (for example, if it is self-signed), then it should be the same for both aliases so that the WebSocket communication used by the emulated device on the Live Sensors page works without requiring you to go to the web server hosted by HttpProtocolPlugin and manually accepting the untrusted certificate in the browser. If the certificates are trusted by the browser, they can be different without causing any issues. To connect to a secured enterprise endpoint using RESTEnterprisePlugin or WebSocketEnterprisePlugin and default configuration, the required certificates must be put into this cacerts keystore. To add a certificate to the cacerts keystore, use the following command with $keytool and $cacerts being the full path for the current platform: $keytool -import -alias $certificate_alias -keystore $cacerts -file $your_certificate_file To manually generate a key pair with Java keytool for use by HttpProtocolPlugin, use the following command: $keytool -genkeypair -keyalg RSA -alias "httpprotocolplugin" -keystore 13

14 <dep_iot_edge>/store/dep.jks" -storepass $password_for_keystore -validity keysize 2048 After doing this, the value of the KEY_STORE_AUTHENTICATION configuration for HttpProtocolPlugin in the database should be updated to match $password. Refer to the Protocol Plugin Configuration section for instructions. To manually export the HttpProtocolPlugin key pair into the alias used by the webserver, use the following command: $keytool -importkeystore -srckeystore <dep_iot_edge>/store/dep.jks" -srcalias "httpprotocolplugin" -srcstorepass $password_for_keystore -src-keypass $password_for_keystore -destkeystore <dep_iot_edge>/store/dep.jks" -destalias "com.sap.dep.webserver " -deststorepass $password_for_keystore -destkeypass $password_for_webserver_cert Note: To obfuscate password properties that can optionally be obfuscated using Jetty, run the command java -cp jetty-all v uber.jar org.eclipse.jetty.util.security.password $password from the Streaming Service java_lib directory, replacing $password with the desired password. The obfuscated version of the password produced by the command can be identified by the OBF: prefix. When using the obfuscated password, you must include the OBF: prefix. After doing this, the value of the com.sap.dep.webserver.ssl.cert.auth property in <dep_iot_edge>/store/webserver.properties should be updated to be the value of $password_for_webserver_cert either Jetty obfuscated value or Base64 encoded. In addition the value of com.sap.dep.webserver.ssl.keystore.auth property in the same file should be updated to the value of $password_for_keystore Jetty obfuscated value or Base64 encoded. To import a PKCS12 file into the Streaming Service keystore, use the following command: $keytool -importkeystore -srcalias "1" -srckeystore $pkcs12_filename -srcstorepass $pkcs12_password -srcstoretype PKCS12 -destalias $dest_alias -destkeystore "<dep_iot_edge>/store/dep.jks" -deststorepass $password_for_keystore The value for $dest_alias should be either httpprotocolplugin or com.sap.dep.webserver depending on which certificate you want to import. 18 CUSTOMIZING EDGE CONSOLE PORTS The Streaming Service web server ports for http and for https are configurable by modifying the file <dep_iot_edge>\store\webserver.properties and restarting the Streaming Service. By default, the ports are: com.sap.dep.webserver.httpport=3000 com.sap.dep.webserver.httpsport= LOG FILE CONFIGURATION All Streaming Service log files are stored in <dep_iot_edge>/log/ The following is a summary of the log files generated by Streaming Service. File Name Description misc_rootlogger.log This is the log4j root log file. IoTAdapter.log Logging for the main Streaming Service process IoTAdapter. StreamingLitePlugin.log Logging for the Streaming Lite Engine Plugin. Webserver.log Logging for the Jetty webserver hosting the Edge Console. 14

15 EdgeConsole.log WhitelistManager.log StreamingLite.log (Linux only) sapdepserver-svc.yyyy-mm-dd.log (Windows only) sapdepweb-svc.yyyy-mm-dd.log (Windows only) sapdepesp-stdout.yyyy-mm-dd.log (Windows only) sapdepesp-stderr.yyyy-mm-dd.log (Windows only) sapdepesp-svc.yyyy-mm-dd.log (Windows only) HttpProtocolPlugin.log MQTTPlugin.log RESTEnterprisePlugin.log WebSocketEnterprisePlugin.log Logging for the Edge Console program. Logging of messages from non-whitelisted devices. The Streaming Lite log file stout and sterr are redirected to this file. The Streaming Service Server windows service log file service control messages. The Streaming Service Web Server windows service log file service control messages. The Streaming Service Streaming Server windows service log file sterr error messages. The Streaming Service Streaming Server windows service log file stout output messages. The Streaming Service Streaming Server windows service log file service control messages. Logging for the HTTP Plugin. Logging for the MQTT Protocol Plugin. Logging for the REST Enterprise Plugin. Logging for the Web Socket Enterprise Plugin. All files except StreamingLite.log roll over when they reach a size of 5MB. When a file has rolled over five times, the oldest file is overwritten. The 7 Windows service files ending in YYYY-MM-DD.log are by default kept for 5 days and then deleted (can be overridden with service.log.keep.days setting in NODE_SETTING). The configuration of the logging level is maintained in the Streaming Service Edge Console. Refer to the Runtime Settings section of Streaming Service Administration Guide for instructions to change the global logging level. Additionally, each plugin can have its own log level that overrides the global value for that plugin. See the appropriate plugin configuration section for instructions about changing a protocol or enterprise plugin log level. 20 ODATA CONFIGURATION On Windows x64 and Linux x64 (but not Linux ARM), Streaming Service provides an OData interface to query data. Note: OData interface is not supported on Linux ARM release of Streaming Service. The OData feature has two layers of authentication: 1. The external layer is managed by Streaming Service web server a. A technical user is created named odata_user and the password is input during Streaming Service setup installation. 2. The database layer is managed by Streaming Service database server (SAP SQL Anywhere) which is the destination of the forwarded OData call from the web server. a. The username for database layer is DEP_OData and the password is generated during the setup process. Users do not need to work with the database layer authentication directly. The major steps relating to OData configuration are the following: During setup, an initial password is required for the technical user odata_user. The password for the technical user odata_user can be changed in the Edge Console => Security => passwords => Change OData Credentials (see Streaming Service Administration Guide for details) Note: The DEP_OData database user has read-only permission to only two tables in the database: EDGE_EVENT and EDGE_FIDELITY_SENSOR_READING. OData Queries 15

16 Notes on using the OData services: - The OData endpoint is - Authentication is via a basic authentication header for the OData technical user (odata_user) - EDGE_EVENT and EDGE_FIDELITY_SENSOR_READING Entity Sets expose the same-named database tables. - edgeeventswithnames and edgesensorfidelityreadingswithnames Entity Sets expose the same information but with the addition of sensor profile and rule names rather than only the Ids - One additional sample function, GetAggregateSensorFidelity, has been exposed. o Input Filters (AGGREGATE is required; all others are optional): SENSOR_PROFILE_ID DEVICE_ID DEVICE_TAG SENSOR_ID SENSOR_TAG AGGREGATE (required) One of ('MIN', 'MAX', 'AVG', 'SUM', 'COUNT', 'VARIANCE', 'STDDEV') TIME_WINDOW Maximum age of readings in seconds. Ignored if not greater than 0. NUM_RECORDS o Maximum number of records to retrieve. Ignored if not greater than 0. Output: Corresponding calculated value in JSON format. Sample OData Queries (Note: The following queries require a basic authentication header for OData user.) Sample call for SELECT * FROM EDGE_EVENT Sample call for SELECT * FROM EDGE_FIDELITY_SENSOR_READING Sample call for SELECT * FROM EDGE_FIDELITY_SENSOR_READING WHERE READING_VALUE > 2 LUE) gt '2')&$format=json Sample call for SELECT READING_VALUE FROM EDGE_FIDELITY_SENSOR_READING &$format=json Sample call for GetAggregateSensorFidelity function OFILE_ID='<Your sensor Profile ID>'&AGGREGATE='MAX'&TIME_WINDOW=0&NUM_RECORDS=0 21 (DEPRECATED) CONNECTING TO THE STREAMING SERVICE DATABASE Deprecated The ODBC DSN DEP_IOT_ADMIN will be removed in a future version. Please make use of the Edge Console and available OData API s to interact with the Streaming Service. For advanced configuration and troubleshooting, access to the underlying database may be beneficial. Warning: Direct database access should be performed with caution. 16

17 To run SQL statements against the Streaming Service database, log in to the Streaming Service host, open a command prompt and complete the following: Linux only: Source the SQL Anywhere sa_config.sh file source /opt/sqlanywhere17/bin64/sa_config.sh Windows only: Open a command prompt as administrator (since Streaming Service DSN s are secured to administrators only) Execute the desired SQL as follows: dbisql -c "DSN=DEP_IOT_ADMIN" "<SQL Statement>" Note: On Linux ARM use dbisqlc instead of dbisql. Note: To connect to the database from a different computer, a dedicated database user must be created where the password is known. For more information, see the SQL Anywhere documentation for the GRANT command. 22 TROUBLESHOOTING If you have problems using Streaming Service, start your troubleshooting process by checking the following items: Issue Solution Ensure JAVA_HOME is set Ensure the JAVA_HOME environment variable is set correctly. At a command line type: set JAVA_HOME Ensure STREAMING_HOME Ensure the STREAMING_HOME environment variable is set correctly. is set At a command line type: set STREAMING_HOME /opt/streaminglite/install Inspect java_libs folder Inspect the java_libs folder located at: <dep_iot_edge>/java_lib/ Check all components are running Check all logs Under heavy load, messages could get dropped with the error message [Streaming Lite Plugin] Failed to handle message as the processing queue was full. This message exists in the StreamingLitePlugin.log file. Ensure there are no duplicate libraries by looking for multiple versions of the following: WebSocketEnterprisePlugin StreamingLitePlugin RESTEnterprisePlugin MQTTPlugin IoTAdapter HttpProtocolPlugin DEPPluginClient Linux: Run sudo./bin/dep.sh status. Windows: Ensure 4 Windows services are running. View logs in <dep_iot_edge>/log/ Increase the following configuration values in the StreamingLitePlugin Engine Plugin configuration: PUBLISHING_MAX_POOL_SIZE PUBLISHING_PUBLISHER_POOL_SIZE PUBLISHING_WORK_QUEUE_CAPACITY 23 KNOWN ISSUES The following error messages are known issues. If required, use the suggested commands as workarounds. Issue # Description Workaround 17

18 2 Linux - Streaming Lite lack of PAM authentication support Use: dep.sh start allnoauth Symptom: You see the following message in StreamingLitePlugin.log in the log folder: StreamingLite plugin is not able to connect and fails with error javax.security.auth.login.loginexception: Server error (4) :Authorization failed: 3 Linux - Streaming Lite missing library files Symptom: /opt/streaminglite/install/bin/streamingproje ct: error while loading shared libraries: libicui18n.so.44: cannot open shared object file: No such file or directory /opt/streaminglite/install/bin/streamingproje ct: error while loading shared libraries: libicuuc.so.44: cannot open shared object file: No such file or directory /opt/streaminglite/install/bin/streamingproje ct: error while loading shared libraries: libicudata.so.44: cannot open shared object file: No such file or directory 4 Linux - SQL Anywhere missing library files when attempting to use dbisql rather than: dep.sh start all Or use: dep.sh start StreamingLiteNoAuth rather than: dep.sh start StreamingLite links ln -s /opt/streaminglite/install/lib/libicui 18n.so.44.1 /opt/streaminglite/install/lib/libicui 18n.so.44 ln -s /opt/streaminglite/install/lib/libicuu c.so.44.1 /opt/streaminglite/install/lib/libicuu c.so.44 ln -s /opt/streaminglite/install/lib/libicud ata.so.44.1 /opt/streaminglite/install/lib/libicud ata.so.44 ln -s /opt/streaminglite/install/lib/libsasl 2.so /opt/streaminglite/install/lib/libsasl 2.so.2 SUSE 12 Install the missing libraries using: zypper search libxtst libxi libxi6 5 Windows - SQL Anywhere unable to use SQL Central to connect to database Run SQL Central as administrator as DSN s are secured and require administrator permissions. 6 Windows Streaming errors STREAMING_HOME must be a "system" environment variable, not a "user" environment variable 18

19 SAP SE or an SAP affiliate company. All rights reserved. No part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of SAP SE or an SAP affiliate company. SAP and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP SE (or an SAP affiliate company) in Germany and other countries. Please see for additional trademark information and notices. Some software products marketed by SAP SE and its distributors contain proprietary software components of other software vendors. National product specifications may vary. These materials are provided by SAP SE or an SAP affiliate company for informational purposes only, without representation or warranty of any kind, and SAP SE or its affiliated companies shall not be liable for errors or omissions with respect to the materials. The only warranties for SAP SE or SAP affiliate company products and services are those that are set forth in the express warranty statements accompanying such products and services, if any. Nothing herein should be construed as constituting an additional warranty. In particular, SAP SE or its affiliated companies have no obligation to pursue any course of business outlined in this document or any related presentation, or to develop or release any functionality mentioned therein. This document, or any related presentation, and SAP SE s or its affiliated companies strategy and possible future developments, products, and/or platform directions and functionality are all subject to change and may be changed by SAP SE or its affiliated companies at any time for any reason without notice. The information in this document is not a commitment, promise, or legal obligation to deliver any material, code, or functionality. All forward-looking statements are subject to various risks and uncertainties that could cause actual results to differ materially from expectations. Readers are cautioned not to place undue reliance on these forward-looking statements, which speak only as of their dates, and they should not be relied upon in making purchasing decisions.