SAP Edge Services, cloud edition Streaming Service - Configuration Guide Version 1803

Transcription

1 SAP Edge Services, cloud edition Streaming Service - Configuration Guide Version 1803

2 Public TABLE OF CONTENTS 1 INTRODUCTION PREREQUISITES JAVA_HOME Python (version 2.7.x 64-bit) INSTALLATION CONNECTING TO THE STREAMING SERVICE EDGE CONSOLE URL Admin Password IOT SERVICES DATA MAPPING IoT Services Naming IoT Services to Streaming Service Data Mapping Streaming Service to IoT Services Data Mapping IOT SERVICES CONFIGURATION Configuration DATABASE BACKUPS AND MAINTENANCE CUSTOMIZING SSL CONFIGURATION CUSTOMIZING EDGE CONSOLE PORTS LOG FILE CONFIGURATION ODATA CONFIGURATION (DEPRECATED) CONNECTING TO THE STREAMING SERVICE DATABASE TROUBLESHOOTING

3 1 INTRODUCTION This document focuses on the installation and configuration of SAP Edge Services - Streaming Service. 2 PREREQUISITES In addition to Java 1.8.x 64-bit, Streaming Service requires the following software package to already be installed on the host: Python version 2.7.x 64-bit The detailed installation requirements are discussed below. 2.1 JAVA_HOME The JAVA_HOME environment variable must refer to the same version of Java that the SAP IoT Services Gateway Edge process is started under (and is found on the system path). Note: SAP Edge Services requires JAVA_HOME to be configured on Windows x64. JAVA_HOME can point to a JRE or a JDK folder. 2.2 Python (version 2.7.x 64-bit) You can install Python either by downloading it from or on Linux by using the package manager local to the Linux distribution (apt, yum, or rpm). Python can be installed to any recommended directory and must be included in the PATH environment variable. 3 INSTALLATION Streaming Service installs to <edgeservices>/dep_iot_edge. This location is referred to as <dep_iot_edge> in this document. <edgeservices> refers to the location where SAP Edge Services is installed. By default, SAP Edge Services installs in a folder named edgeservices in the parent folder of the IoT Services Gateway Edge folder (i.e. at the same folder level as Gateway Edge folder). For more details on changing the installation folder of SAP Edge Services, refer to the SAP Edge Services Overview Guide. Streaming Service is installed through the Edge Services Console (Policy Service). The Edge Services Console (Policy Service) pushes all the bundles required by Streaming Service to the gateway and starts the bundles. Starting the bundle the first time starts the setup for Streaming Service. 4 CONNECTING TO THE STREAMING SERVICE EDGE CONSOLE 4.1 URL The URL for the Streaming Service Edge Console should be entered in the following format: Admin Password The setup for Streaming Service generates a secure random admin password for the Streaming Service Edge Console and copies it to the IoT Services Cockpit gateway properties with the key STREAM_SECRET in the corresponding network. The password can and should be changed using the Streaming Service Edge Console and cannot be done through the IoT Services Cockpit. Changes to the admin password made through Streaming Service Edge Console are not reflected in the IoT Services Cockpit properties. The STREAM_SECRET property can safely be deleted once the password is changed through the Streaming Service Edge Console. Process Flow Initial Login 1. The Streaming Service install is requested from the Policy Server. 2. The Streaming Service install program generates a secure random admin password. 3. The Streaming Service install program copies the admin password to the STREAM_SECRET gateway property. 4. The Administrator logs into the IoT Services Cockpit and retrieves the admin password in the STREAM_SECRET gateway property. 3

4 5. The Administrator logs into the Streaming Service Edge Console using the admin password stored in STREAM_SECRET. 6. The Administrator is advised to change the Streaming Service Edge Console admin password: a. The Administrator uses the Streaming Service Edge Console to change the admin password. The Administrator must keep track of the admin password in a separate secure system. b. At this point STREAM_SECRET is out of date and should be deleted to avoid future confusion. 5 IOT SERVICES DATA MAPPING 5.1 IoT Services Naming Note: In this section, we refer to IoT Services nomenclature using the values from IoT Services json payload column. IoT Services Data Model profileid capabilities/configaddr IoT Services json payload sensortypealternateid capabilityalternateid 5.2 IoT Services to Streaming Service Data Mapping The IoT services protocol plugin comes preconfigured with the Streaming Service. All data sent to the IoTServices gateway is intercepted and published by the EdgeInterceptor. The IoT services protocol plugin receives this data and maps the data into Streaming Service format as follows: Streaming Service Value from IoT Services Notes Data Model SensorProfileName {Label}_{sensorTypeAlternateId}_{capabilityAlternateId} ** Case Sensitive! SensorProfileName {Label}_{sensorTypeAlternateId}_{capabilityAlternateId} _{PropertyName} ** Case Sensitive! (relevant for Capabilities with Property Names defined) DeviceId {NSAP/MAC Address from IoT services} DeviceTag {sensortypealternateid} SensorId {TSAP from IoT services} SensorTag Unmapped Timestamp {ArrivalTime} ReadingValue {measures} Note: The sensortypealternateid/capabilityalternateid combination is used to lookup the Label in the IoT Services data model. Note: The mapping of SensorProfileName is case-sensitive! If Property Names for Capabilities are provided in the IoT Services data model, then the mapping from IoT Services to Streaming service will happen as follows: 1. The data will be mapped at the Capability Level where SensorProfileName = {Label}_{sensorTypeAlternateId}_{capabilityAlternateId} 2. The data will be mapped a second time at the individual Propery Level for each Property, where the SensorProfileName = {Label}_{sensorTypeAlternateId}_{capabilityAlternateId}_{PropertyName} Therefore, if a Capability has 3 Properties (Latitude, Longitude, Quote), then the data will be mapped four times into Streaming Service. In this case, it is important to only use the SensorProfile at the Capability level to define Fidelity Rules, as this ensures the outgoing payload contains all the data. If Streaming Service does not contain a Sensor Profile Name that matches, the message is dropped from further processing by Streaming Service. 4

5 Example data flow (mqtt): 1. An mqtt device with MAC Address AA:BB:CC:DD sends data to IoT Services: {"sensortypealternateid":"2", "capabilityalternateid": [100], "measures": [[89.5]], "sensoralternateid": "TempSensor103"} 2. Edge Interceptor forwards data to the IoT services protocol plugin. 3. The sensortypealternateid/capabilityalternateid combination is used to lookup the Label in the IoT Services data model (i.e. FreezerTemperature ) 4. IoT services protocol plugin maps the message into the Streaming Service Data Model as follows: Streaming Service Data Value from IoT Services Notes Model SensorProfileName FreezerTemperature_2_100 DeviceId AA:BB:CC:DD DeviceTag 2 sensortypealternateid = 2 SensorId Converted from lognodeaddr SensorTag Timestamp {ArrivalTime} ReadingValue Streaming Service to IoT Services Data Mapping The IoT services enterprise plugin comes preconfigured with the Streaming Service (refer to the Streaming Service - Administration Guide for details on configurating the IoT services enterprise plugin). The IoT services enterprise plugin is responsible for publishing both enterprise events and fidelity readings back to IoT Services. The Streaming Service maps data back into IoT Services data format as follows: IoT Services Data Model Enterprise Event Mapping Fidelity Event Mapping capabilityalternateid (capabilities/configaddr) Configured IoT services enterprise plugin value for ENTERPRISE_MEASURE_ID is used. The value must be defined in the IoT Services data model in sensortypealternateid defined by ENTERPRISE_PROFILE_ID. capabilityalternateid value remains the same as the incoming value DeviceAddress (NSAP/MAC Address) sensortypealternateid (profileid) If ENTERPRISE_PROFILE_ID is not defined, then the value for ENTERPRISE_MEASURE_ID must be defined in every profile for which enterprise events will be generated. DeviceAddress remains the same as the incoming value Configured IoT services enterprise plugin value for ENTERPRISE_PROFILE_ID is used. If not provided, then the outgoing sensortypealternateid value is same as the incoming sensortypealternateid that generated the event. DeviceAddress remains the same as the incoming value sensortypealternateid remains the same as the incoming value 5

6 SensorAddress (TSAP from IoT services) ArrivalTime measures New sensor address for Enterprise event in format (sensoraddress + SAPES + measureid) ArrivalTime remains the same as the incoming value The default ENTERPRISE_TEMPLATE for Enterprise Events is used. (Refer to the ENTERPRISE_TEMPLATE section in Streaming Service - Administration Guide.) SensorAddress remains the same as the incoming value ArrivalTime remains the same as the incoming value measures remains the same as the incoming value 6 IOT SERVICES CONFIGURATION 6.1 Configuration The Streaming Service has the following configuration values that can be set in the IoT Services Cockpit. Property Name: refreshtimeperiodfordatamodel Unit of Measure: seconds Default Value: 3600 Streaming Service polls for data model updates at the interval set by refreshtimeperiodfordatamodel. Any updates to the data model are consumed at this interval. Updates to the refreshtimeperiodfordatamodel value are also consumed at the same time. If refreshtimeperiodfordatamodel value changes, then the next interval uses the new value. refreshtimeperiodfordatamodel value (nothing) Behavior Data model (and refreshtimeperiodfordatamodel value) is read at the default interval of 3600 seconds. 600 Data model (and refreshtimeperiodfordatamodel value) is read at an interval of 600 seconds. Note: It is strongly recommended that you start the IoT Services Gateway Edge with offline support ( Disconnected Startup ) enabled. Refer to the SAP Edge Services Overview Guide for more information about enabling this feature. 7 DATABASE BACKUPS AND MAINTENANCE Streaming Service provides a database that is self-tuning. This database stores configuration, edge events and edge fidelity events. The database consists of a database file and a database transaction log file. Database best practices dictate that the database transaction log file must be maintained. Additionally, the database may be configured for backups as required. WARNING: By default, Streaming Service is not configured to perform database backups. Streaming Service truncates the database transaction log file when it reaches 1GB. This is accomplished via a database event DEP_Admin.TruncateLogFile. If there is a need for separate database backups, then this event should be disabled and an alternate backup and transaction log file maintenance plan should be implemented. Consult SQL Anywhere 17 product documentation for details on database backup options. 6

7 8 CUSTOMIZING SSL CONFIGURATION The Streaming Service setup (setup.sh/setup.bat) generates self-signed SSL certificates for use by: Streaming Service web server for HTTPS HttpProtocolPlugin to enable HTTPS and WSS This section describes how to replace both of the self-signed certificates in the following Java keystore: <dep_iot_edge>/store/dep.jks By default, the Streaming Service setup (setup.sh/setup.bat) uses Java Keytool to generate a 2048-bit RSA key pair with a validity period of 10,000 days in dep.jks under the alias httpprotocolplugin. The key pair is then duplicated under the alias com.sap.dep.webserver. The HttpProtocolPlugin makes use of the former and the Streaming Service webserver makes use of the latter. Both enable secure communications. Note: If the certificate being placed in the keystore is not a trusted certificate (for example, if it is self-signed), then it should be the same for both aliases so that the WebSocket communication used by the emulated device on the Live Sensors page works without requiring you to go to the web server hosted by HttpProtocolPlugin and manually accepting the untrusted certificate in the browser. If the certificates are trusted by the browser, they can be different without causing any issues. To connect to a secured enterprise endpoint using RESTEnterprisePlugin or WebSocketEnterprisePlugin and default configuration, the required certificates must be put into this cacerts keystore. To add a certificate to the cacerts keystore, use the following command with $keytool and $cacerts being the full path for the current platform: $keytool -import -alias $certificate_alias -keystore $cacerts -file $your_certificate_file To manually generate a key pair with Java keytool for use by HttpProtocolPlugin, use the following command: $keytool -genkeypair -keyalg RSA -alias "httpprotocolplugin" -keystore <dep_iot_edge>/store/dep.jks" -storepass $password_for_keystore -validity keysize 2048 After doing this, the value of the KEY_STORE_AUTHENTICATION configuration for HttpProtocolPlugin in the database should be updated to match $password. Refer to the Protocol Plugin Configuration section for instructions. To manually export the HttpProtocolPlugin key pair into the alias used by the webserver, use the following command: $keytool -importkeystore -srckeystore <dep_iot_edge>/store/dep.jks" -srcalias "httpprotocolplugin" -srcstorepass $password_for_keystore -src-keypass $password_for_keystore -destkeystore <dep_iot_edge>/store/dep.jks" -destalias "com.sap.dep.webserver " -deststorepass $password_for_keystore -destkeypass $password_for_webserver_cert Note: To obfuscate password properties that can optionally be obfuscated using Jetty, run the command java -cp jetty-all v uber.jar org.eclipse.jetty.util.security.password $password from the Streaming Service java_lib directory, replacing $password with the desired password. The obfuscated version of the password produced by the command can be identified by the OBF: prefix. When using the obfuscated password, you must include the OBF: prefix. After doing this, the value of the com.sap.dep.webserver.ssl.cert.auth property in <dep_iot_edge>/store/webserver.properties should be updated to be the value of $password_for_webserver_cert either Jetty obfuscated value or Base64 encoded. In addition the value of com.sap.dep.webserver.ssl.keystore.auth property in the same file should be updated to the value of $password_for_keystore Jetty obfuscated value or Base64 encoded. 7

8 To import a PKCS12 file into the Streaming Service keystore, use the following command: $keytool -importkeystore -srcalias "1" -srckeystore $pkcs12_filename -srcstorepass $pkcs12_password -srcstoretype PKCS12 -destalias $dest_alias -destkeystore "<dep_iot_edge>/store/dep.jks" -deststorepass $password_for_keystore The value for $dest_alias should be either httpprotocolplugin or com.sap.dep.webserver depending on which certificate you want to import. 9 CUSTOMIZING EDGE CONSOLE PORTS The Streaming Service web server ports for http and for https are configurable by modifying the file <dep_iot_edge>\store\webserver.properties and restarting the Streaming Service. By default, the ports are: com.sap.dep.webserver.httpport=3000 com.sap.dep.webserver.httpsport= LOG FILE CONFIGURATION All Streaming Service log files are stored in <dep_iot_edge>/log/ The following is a summary of the log files generated by Streaming Service. File Name Description misc_rootlogger.log This is the log4j root log file. IoTAdapter.log Logging for the main Streaming Service process IoTAdapter. StreamingLitePlugin.log Logging for the Streaming Lite Engine plugin. Webserver.log Logging for the Jetty webserver hosting the Streaming Service Console. EdgeConsole.log Logging for the Streaming Server Console program. WhitelistManager.log Logging of messages from non-whitelisted devices. StreamingLite.log The Streaming Lite log file stout and sterr are redirected to this file. HttpProtocolPlugin.log Logging for the HTTP plugin. RESTEnterprisePlugin.log Logging for the REST Enterprise plugin. MQTTPlugin.log Logging for the MQTT Protocol Plugin. WebSocketEnterprisePlugin.log Logging for the Web Socket Enterprise plugin. All files except StreamingLite.log roll over when they reach a size of 5MB. When a file has rolled over five times, the oldest file is overwritten. The configuration of the logging level is maintained in the Streaming Service Edge Console. Refer to Runtime Settings section of Streaming Service Administration Guide for instructions to change the global logging level. Additionally, each plugin can have its own log level that overrides the global value for that plugin. See the appropriate plugin Configuration section for instructions about changing a protocol or enterprise plugin log level. 11 ODATA CONFIGURATION Streaming Service provides an OData interface to query data. The OData feature has two layers of authentication: 1. The external layer is managed by Streaming Service web server. a. A technical user is created named odata_user. 2. The database layer is managed by Streaming Service database server (SAP SQL Anywhere), which is the destination of the forwarded OData call from the web server. 8

9 a. The username for database layer is DEP_OData and the password is generated during the setup process. Users do not need to work with the database layer authentication directly. The major steps relating to OData configuration are as follows: The password for the technical user odata_user can be changed in the Streaming Service Console. Go to Security => passwords => Change OData Credentials (refer to the Streaming Service - Administration Guide for details). Note: The DEP_OData database user has read-only permission to only two tables in the database: EDGE_EVENT and EDGE_FIDELITY_SENSOR_READING. OData Queries Notes on using the OData services: - The OData endpoint is - Authentication is via a basic authentication header for the OData technical user (odata_user) - EDGE_EVENT and EDGE_FIDELITY_SENSOR_READING Entity Sets expose the same-named database tables. - edgeeventswithnames and edgesensorfidelityreadingswithnames Entity Sets expose the same information but with the addition of sensor profile and rule names rather than only the Ids - One additional sample function, GetAggregateSensorFidelity, has been exposed. o Input Filters (AGGREGATE is required; all others are optional): SENSOR_PROFILE_ID DEVICE_ID DEVICE_TAG SENSOR_ID SENSOR_TAG AGGREGATE (required) One of ('MIN', 'MAX', 'AVG', 'SUM', 'COUNT', 'VARIANCE', 'STDDEV') TIME_WINDOW Maximum age of readings in seconds. Ignored if not greater than 0. NUM_RECORDS o Maximum number of records to retrieve. Ignored if not greater than 0. Output: Corresponding calculated value in JSON format. Sample OData Queries (Note: The following queries require a basic authentication header for OData user.) Sample call for SELECT * FROM EDGE_EVENT Sample call for SELECT * FROM EDGE_FIDELITY_SENSOR_READING Sample call for SELECT * FROM EDGE_FIDELITY_SENSOR_READING WHERE READING_VALUE > 2 LUE) gt '2')&$format=json Sample call for SELECT READING_VALUE FROM EDGE_FIDELITY_SENSOR_READING &$format=json Sample call for GetAggregateSensorFidelity function 9

10 OFILE_ID='<Your sensor Profile ID>'&AGGREGATE='MAX'&TIME_WINDOW=0&NUM_RECORDS=0 12 (DEPRECATED) CONNECTING TO THE STREAMING SERVICE DATABASE Deprecated The ODBC DSN DEP_IOT_ADMIN will be removed in a future version. Please make use of the Edge Console and available APIs to interact with the Streaming Service. For advanced configuration and troubleshooting, access to the underlying database may be beneficial. Warning: Direct database access should be performed with caution. To run SQL statements against the Streaming Service database, log in to Streaming Service host, open a command prompt and complete the following: Linux only: Update the SQL Anywhere install path in the sa_config.sh file located at <edgeservices>/ /sqlanywhere/ Linux only: Source the SQL Anywhere sa_config.sh: source <edgeservices>/sqlanywhere/17.0.8/bin64/sa_config.sh Windows only: Open a command prompt as administrator (since Streaming Service DSN is secured to administrators only) Execute the desired SQL as follows: <edgeservices>/sqlanywhere/17.0.8/bin64/dbisql -c "DSN=DEP_IOT_ADMIN" "<SQL Statement>" Note: To connect to the database from a different computer, a dedicated database user must be created where the password is known. For more information, see the SQL Anywhere documentation for the GRANT command. 13 TROUBLESHOOTING For advanced troubleshooting, check the following items: Issue Solution Check all components are running Linux: Run sudo./<dep_iot_edge>/bin/dep.sh status. Check all logs View logs in <dep_iot_edge>/log/ Under heavy load, messages could get dropped Increase the following configuration values in the with the error message [Streaming Lite Plugin] Failed to handle message as the processing StreamingLitePlugin Engine Plugin configuration: queue was full. This message exists in the StreamingLitePlugin.log file. PUBLISHING_MAX_POOL_SIZE PUBLISHING_PUBLISHER_POOL_SIZE PUBLISHING_WORK_QUEUE_CAPACITY 10

11 SAP SE or an SAP affiliate company. All rights reserved. No part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of SAP SE or an SAP affiliate company. SAP and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP SE (or an SAP affiliate company) in Germany and other countries. Please see for additional trademark information and notices. Some software products marketed by SAP SE and its distributors contain proprietary software components of other software vendors. National product specifications may vary. These materials are provided by SAP SE or an SAP affiliate company for informational purposes only, without representation or warranty of any kind, and SAP SE or its affiliated companies shall not be liable for errors or omissions with respect to the materials. The only warranties for SAP SE or SAP affiliate company products and services are those that are set forth in the express warranty statements accompanying such products and services, if any. Nothing herein should be construed as constituting an additional warranty. In particular, SAP SE or its affiliated companies have no obligation to pursue any course of business outlined in this document or any related presentation, or to develop or release any functionality mentioned therein. This document, or any related presentation, and SAP SE s or its affiliated companies strategy and possible future developments, products, and/or platform directions and functionality are all subject to change and may be changed by SAP SE or its affiliated companies at any time for any reason without notice. The information in this document is not a commitment, promise, or legal obligation to deliver any material, code, or functionality. All forward-looking statements are subject to various risks and uncertainties that could cause actual results to differ materially from expectations. Readers are cautioned not to place undue reliance on these forward-looking statements, which speak only as of their dates, and they should not be relied upon in making purchasing decisions.