ArcGIS GeoEvent Server: A Developer's Guide. Mark Bramer Esri Professional Services Vienna, VA

Transcription

1 ArcGIS GeoEvent Server: A Developer's Guide Mark Bramer Esri Professional Services Vienna, VA

2 Agenda Connectors - Inbound - Outbound GeoEvent SDK Transports and Adapters Processors Maven Scripting upstream

3 Real-Time GIS Integration and exploitation of streaming data Web Device Desktop Integrates real-time streaming data into ArcGIS Performs continuous processing and real-time analytics Sends updates and alerts to those who need it where they need it GeoEvent Server ArcGIS Enterprise

4 ArcGIS as an IoT Platform ArcGIS Enterprise with real-time & big data capabilities Web Device Desktop Ingest high velocity real-time IoT data into ArcGIS. live features live & historic aggregates & features Perform continuous analytics on IoT events as they are received. Store IoT observations in a spatiotemporal big data store. stream service map & feature service Visualize high velocity & volume IoT data: - as an aggregation - or as discrete features. ingestion actuation visualization ArcGIS Enterprise Notify about IoT patterns of interest. Adjust behavior of things in our environment through actuation. GeoEvent Server analytics spatiotemporal big data store storage GeoAnalytics Server analytics

5 Connectors

6 Connectors What is a connector? Connectors get event data into, and out of, GeoEvent Server Every connector has two components - Transport - Adapter Transports and Adapters can be inbound or outbound: - Inbound transport connects to and gets raw bytestream - Inbound adapter converts raw bytestream into a geoevent - Outbound adapter converts geoevent into a byte array, formatted for an output type - Outbound transport accepts arrays of bytes from the adapter and transmits them

7 Connectors Configuring a Connector The connector helps the user by Providing default values Re-label properties to be appropriate to the context Move properties under an advanced area to discourage modification Completely hide properties that the user should not see

8 Out of the Box Esri Gallery Partner Gallery Inputs Outputs Receiving Real-Time Data Easily integrate real-time streaming data into ArcGIS using an Input Connector GeoEvent Server GeoEvent Services You can create your own connectors. Poll an ArcGIS Server for Features ActiveMQ CompassLDE HTTP Poll an external website for GeoJSON, JSON, or XML CAP envirocar REST Receive Features, GeoJSON, JSON, or XML on a REST endpoint Cursor-on-Target exactearth AIS WS Receive GeoJSON or JSON on a WebSocket Exploitation Support Data FAA (ASDI) * Receive RSS Instagram GNIP * Receive Text from a TCP or UDP Socket KML Networkfleet * WS Subscribe to an external WebSocket for GeoJSON or JSON Kafka OSIsoft *.csv Watch a Folder for new CSV, GeoJSON, or JSON Files MQTT Valarm NMEA 0183 RabbitMQ Sierra Wireless (RAP) Zonar * Trimble (TAIP) Twitter

9 Esri Gallery Esri Gallery Inputs Outputs Receiving Real-Time Data Input Connector = Transport + Adapter GeoEvent Server GeoEvent Services you can create your own connectors Connectors Transport Adapter Receive Feature-JSON from Kafka Kafka Feature-JSON Receive GeoJSON on a REST endpoint HTTP GeoJSON ActiveMQ IRC Kafka MQTT RabbitMQ Twitter Transports Feature Service File HTTP HTTP+BasicAuth HTTP+OAuth TCP UDP Waze WebSocket Adapters Feature-JSON GeoJSON JSON RSS Text XML CAP Cursor-on-Target GeoMessage Instagram NMEA Sierra Wireless (RAP) Trimble (TAIP) Twitter VMF Waze

10 Out of the Box Esri Gallery Inputs Outputs Sending Real-Time Data Easily disseminate notifications, alerts, and updates using an Output Connector GeoEvent Server GeoEvent Services You can create your own connectors. Add or Update a feature ActiveMQ Publish Text to a UDP Socket Cursor-on-Target HTTP Push GeoJSON or JSON to an external Website Hadoop WS Push GeoJSON or JSON to an external WebSocket Kafka Push Text to an external TCP Socket MongoDB Send a Text Message RabbitMQ Send an Twitter im Send an Instant Message Send Features to a Stream Service.csv Write to a CSV or JSON File

11 Esri Gallery Esri Gallery Inputs Outputs Sending Real-Time Data Output Connector = Adapter + Transport GeoEvent Server GeoEvent Services you can create your own connectors Connectors Adapter Transport Send an Message Formatter SMTP Publish Feature-JSON to Kafka Feature-JSON Kafka Cursor-on-Target Adapters Cache Feature-JSON GeoJSON JSON Message Formatter RSS Text WebSocket Transports Big Data Store SMTP Feature Service Stream Service File TCP HTTP UDP HTTP+BasicAuth WebSocket HTTP+OAuth XMPP SMS ActiveMQ Hadoop IRC Kafka MongoDB MQTT RabbitMQ TCP-Squirt Twitter

12 Outputs Create Input Create Input Creating Inputs Inputs are configured using Connectors RSS Connector National Weather Service Connectors JSON over REST US Geological Survey GeoEvent Server Live Traffic Services Inputs Storms Earthquakes GeoEvent Services Traffic

13 Connector A Connector s Purpose By choosing a Connector, the user implicitly selects components from GeoEvent Server that know: HOW to move data (Transport) WHAT the data looks like (Adapter) Example Input Input (Receive text from a TCP Socket) Start Stop Byte [ ] GeoEvent GeoEvent Services TCP Transport Text Adapter

14 GeoEvent Server SDK

15 Inputs Outputs Extending GeoEvent Software Development Kit (SDK) You can create your own custom transports, adapters and processors using the GeoEvent Software Development Kit (SDK). GeoEvent Server Your connectors GeoEvent Services Your connectors Your processors A note on semantics: the SDK is used to build transports, adapters and processors. Input and output connectors are built by pairing a transport and adapter in GeoEvent Manager.

16 Extending GeoEvent Server Software Development Kit (SDK) - api: JavaDoc content associated with GeoEvent Server SDK - repository: Local maven repository - samples: Sample processors, transports, adapters - GeoEvent Server Developer Guide

17 Transports

18 Transport What makes up a Transport? Transport Service Provides a Transport Definition Instantiates new Transports Transport Definition Defines transport metadata and properties Transport Implements transport application logic

19 Transport Behavior Transports Transports are given - Properties that define behavior - A ByteListener where bytes should be sent Transport is started by the server and it sends bytes to the receiver Transport is stopped by the server and it stops sending bytes

20 Transport Lifecycle Transports Transports have a lifecycle that determines if they are producing data Server calls start() Transport reads properties and starts STARTING Internal error occurs STOPPED STARTED STOPPING ERROR Transport closes resources Server calls stop()

21 Outbound Transports Transports Outbound Transports accept arrays of bytes from the Adapter and transmit them. Occasionally the destination for the bytes depends on content in the GeoEvent. - The Transport has the option of looking back at the GeoEvent that generated the bytes, and using it to route the data. - address - Phone number

22 Adapters

23 Adapter What makes up an Adapter Adapter Service Provides an Adapter Definition Instantiates new Adapters Adapter Definition Defines adapter metadata and properties Adapter Implements adapter application logic

24 Adapter Behavior Adapters Adapters are given - Properties that define behavior - A GeoEventListener where the GeoEvents should be sent Adapters are DATA DRIVEN - No start/stop calls - The adapter is handed a byte array and pushes any generated GeoEvents to the Listener

25 Custom Connectors

26 Custom Connectors With Development Transport + Adapter - Use SDK to build transports or adapters - Custom connector made by pairing transport with an adapter - Both transport and adapter can be custom, or custom transport only or custom adapter only

27 Custom Connectors Without Development Transport + Adapter - You actually may not need the SDK to make a custom connector - Familiarize yourself with all out-of-box transports and adapters - Out of box connectors do not cover all unique permutations of adapter and transport pairings - You may already have all building blocks to make your custom connector

28 Custom Connectors Without Development Clone an existing connector & tailor it s properties to make a more specific connector

29 Custom Connectors Without Development Clone an existing connector & tailor it s properties to make a more specific connector

30 More Options on the GeoEvent Gallery Extend GeoEvent Server by downloading additional transports, adapters and processors

31 DEMO Custom Adapter: regextext-adapter developed using SDK

32 DEMO Custom Connector: XML over TCP no development required

33 Processors

34 Inputs Outputs Processors What is a processor? Processors perform some action on each GeoEvent passed to it Processors can be used to: - Modify existing fields or the geometry - Add new fields - Filter - Create new GeoEvent(s) - Perform GeoFencing GeoEvent Server GeoEvent Services Processors

35 Out of the Box Esri Gallery Inputs Outputs Applying real-time analytics GeoEvent Processing You can perform continuous analytics on GeoEvents as they are received using a processor. GeoEvent Server GeoEvent Services You can create your own processors. Buffer Creator Geotagger Add XYZ Range Fan Convex Hull Creator Incident Detector Bearing Reverse Geocoder Difference Creator Intersector Ellipse Service Area Creator Envelope Creator Projector Event Volume Control Symbol Lookup Field Calculator Simplifier Extent Enricher Track Idle Detector Field Enricher Symmetric Difference Field Grouper Unit Converter Field Mapper Track Gap Detector GeoNames Lookup Visibility Field Reducer Union Creator Motion Calculator Query Report

36 Processor Lifecycle Review GeoEvent Server GeoEvent Service Processor Creation Shutdown Initialization Processing Validation

37 Anatomy of a Processor What makes up a Processor? Processor Service Instantiates processor definition, processor and applies configuration Processor Definition Processor Defines processor metadata, properties and GeoEvent Definition(s) Implements processor lifecycle Configuration Defines processor, it s dependencies and metadata

38 Maven

39 For those of us in closed, secure environments Maven - Maven is an Apache project for automated software building, dependency management, and testing - Using Maven in secure environments raises some unique and potentially extremely frustrating challenges - Internet connectivity is at the core of most Maven installations - All of the GeoEvent SDK samples are Maven projects - Recommended you start with an existing sample - Using the GeoEvent SDK means using Maven - It is possible set up a GeoEvent project without Maven, but it can be very very frustrating!

40 For those of us in closed, secure environments Maven - So with all this caution, why and how would I even do this? - It s not that bad, just know these things: Read the Developer Guide, especially: - Overview of the Sample Projects - Using Eclipse to Edit and Build Custom Components - Starting a New Project - Building and Deploying the Sample Projects - At least one sample in The Sample Projects - Continued

41 For those of us in closed, secure environments Maven Build outside your work environment, where you have internet connectivity - Use one of the samples in the SDK to get started one from the Developer Guide - This downloads dependencies to your local machine - Burn your entire.m2 folder to CD - Hidden by default - C:\Users\username\.m2 on Widows - /root/.m2 on Linux - Place.m2 in equivalent location in secure environment - Create.m2/settings.xml file and point to local repository

42 For those of us in closed, secure environments In-house Maven - Find out if your agency or department has a repository manager - Artifactory - Sonatype - Archiva - If so, deploy the artifacts to the repository manager - Update your local /.m2/settings.xml to reference the repository manager

43 Scripting Upstream

44 Scripting upstream Alternate pattern for ingesting real-time data What if I don t know Java? What if I don t have time to learn a new SDK? What if I don t want to (maybe) have to re-compile my connector at each new GeoEvent version?

45 Scripting upstream Alternate pattern for ingesting real-time data Possible and easy to write your own app Runs upstream from GeoEvent Consumes real-time feed Parses and transforms data Sends data to GeoEvent in easy out-of-box form, like csv

46 Scripting upstream CTfastrak GTFS = General Transit Feed Specification - Common format for public transportation schedules GTFS-realtime - Trip updates - Alerts - Vehicle positions CTfastrak - Bus Rapid Transit system in Connecticut - GTFS-rt is now native to GeoEvent (But this slide section and demo still convey the point)

47 DEMO Python for CT Transit GTFS-rt GTFS-rt-to-GeoEvent

48 Summary Connectors get data in and out of GeoEvent. Connectors pair a transport and an adapter. You can develop your own transports and adapters. You can configure your own connectors by pairing transports and adapters (custom developed or out-of-box). Processors perform analytics on geoevents. You can build your own processors, as well. Maven is a key component to development with the GeoEvent Server SDK. You can also employ alternate tactics for custom GeoEvent development, such as writing scripts that send data to GeoEvent.

49 Where to learn more? Resources To learn more, visit the Get Started area of the GeoEvent Server page: Introduction - Big Data Store - Resiliency - Multiple-Machine Sites - Notifications - Stream Services - RSS, HTTP, Files - REST Admin API Ask questions on the GeoEvent Forum: -

50 Please Take Our Survey on the Esri Events App! Download the Esri Events app and find your event Select the session you attended Scroll down to find the feedback section Complete answers and select Submit

51 GeoEvent Server: A Developer's Guide Mark Bramer Esri Professional Services Vienna, VA Questions?