ArcGIS GeoEvent Server Overview. Thomas Paschke

Size: px

Start display at page:

Download "ArcGIS GeoEvent Server Overview. Thomas Paschke"

Morris Butler
5 years ago
Views:

1 ArcGIS GeoEvent Server Overview Thomas Paschke

2 Agenda GeoEvent Development GeoEvent Server Real-Time & Big 10.5.x Visualization Road Ahead (10.6) R&D Project Trinity & Other

3 1 GeoEvent Development

4 GIS Data What has happened, what is happening, what will happen Credit: istockphoto/chris_lemmens The current snapshot is outdated almost as soon as it s created

5 Real-Time GIS Requirements Ingestion, Analytics, Notifications and Alerting Requirement #1 Continuous Analysis Inside Boundary Features Position Alert Applications Requirement #2 Requirement #3

6 ArcGIS Server with real-time capabilities ingest real-time & big data visualize analyze store GeoEvent Server BYO-RDBMS

7 Web GIS with real-time capabilities ingest 3K e/s analyze 3K e/s real-time & big data visualize thousands X store 200 e/s Web GIS GeoEvent Server X 200 e/s Stream Service BYO-RDBMS

8 Web GIS with real-time capabilities ingest 3K e/s analyze 3K e/s real-time & big data visualize thousands X store 200 e/s Web GIS GeoEvent Server X 200 e/s relational data store

9 Web GIS with real-time capabilities ingest 4K e/s real-time & big data visualize millions analyze 4K e/s store 10Ks e/s Web GIS IoT 4K e/s GeoEvent Server spatiotemporal big data store

10 2 GeoEvent Server

11 ArcGIS Enterprise with real-time capabilities ArcGIS Enterprise IoT GeoEvent Server

12 Out of the Box Esri Gallery Partner Gallery Inputs Outputs Ingestion use an existing input connector GeoEvent Server GeoEvent Services you can create your own connectors Poll an ArcGIS Server for Features ActiveMQ Amazon IoT HTTP Poll an external website for GeoJSON, JSON, or XML CAP Azure IoT REST Receive Features, GeoJSON, JSON, or XML on a REST endpoint Cursor-on-Target CompassLDE WS Receive GeoJSON or JSON on a WebSocket Exploitation Support Data envirocar Receive RSS Instagram exactearth AIS Receive Text from a TCP or UDP Socket KML FAA (ASDI) * WS Subscribe to an external WebSocket for GeoJSON or JSON Kafka GNIP *.csv Watch a Folder for new CSV, GeoJSON, or JSON Files MQTT Networkfleet * NMEA 0183 OSIsoft * RabbitMQ Valarm Sierra Wireless (RAP) Waze Trimble (TAIP) Zonar * Twitter

13 Esri Gallery Esri Gallery Inputs Outputs Ingestion configure a input connector by pairing a transport & adapter together GeoEvent Server GeoEvent Services you can create your own connectors Connectors Transport Adapter Receive Feature-JSON from Kafka Kafka Feature-JSON Receive GeoJSON on a REST endpoint HTTP GeoJSON Amazon IoT Azure IoT IRC Kafka MQTT RabbitMQ Twitter Transports Feature Service File HTTP HTTP+BasicAuth HTTP+OAuth TCP UDP Waze WebSocket Adapters Feature-JSON GeoJSON JSON RSS Text XML CAP Cursor-on-Target GeoMessage Instagram NMEA Sierra Wireless (RAP) Trimble (TAIP) Twitter VMF Waze

Out of the Box Esri Gallery Inputs Outputs Dissemination use an existing output connector GeoEvent Server GeoEvent Services you can create your own connectors Add or Update a Feature Publish Text to

14 Out of the Box Esri Gallery Inputs Outputs Dissemination use an existing output connector GeoEvent Server GeoEvent Services you can create your own connectors Add or Update a Feature Publish Text to a UDP Socket Amazon IoT Azure IoT HTTP Push GeoJSON or JSON to an external Website ActiveMQ WS Push GeoJSON or JSON to an external WebSocket Cursor-on-Target Push Text to an external TCP Socket Hadoop Send a Text Message Kafka Send an MongoDB im Send an Instant Message MQTT Send Features to a Stream Service RabbitMQ.csv Write to a CSV, GeoJSON, or JSON File Twitter Add a Feature to a spatiotemporal big data store Update a feature in a spatiotemporal big data store

15 GeoEvent Server Java SDK Create your custom GeoEvent components

16 Applying Real-Time Analytics GeoEvent Services A GeoEvent Service defines the flow of GeoEvents - The Filtering and Processing steps to perform - what input(s) to apply them to - and what output(s) to send the results to

17 Out of the Box Esri Gallery Inputs Outputs Performing continuous analytics in ArcGIS Processors You can perform continuous analytics on events as they are received using a processor. GeoEvent Extension GeoEvent Services You can create your own processors. Buffer Creator Geotagger Add XYZ Range Fan Convex Hull Creator Incident Detector Bearing Reverse Geocoder Difference Creator Intersector Ellipse Service Area Creator Envelope Creator Projector Event Volume Control Symbol Lookup Field Calculator Simplifier Extent Enricher Track Idle Detector Field Enricher Symmetric Difference Field Grouper Unit Converter Field Mapper Track Gap Detector GeoNames Lookup Visibility Field Reducer Union Creator Motion Calculator Query Report

18 Additional Real-Time Spatial Analytics Spatial condition evaluation Spatial operators: inside enter intersects touches crosses overlaps outside exit disjoint contains equals within

19 Additional Real-Time Spatial Analytics Example Impending weather notifications

20 Additional Real-Time Spatial Analytics Example Convoy separation alerting

21 Additional Real-Time Spatial Analytics GeoFence scope Specify a GeoFence scope: Any or All Any GeoFence All GeoFences

22 Additional Real-Time Spatial Analytics Example Territory adherence alerting

23 GeoEvent Server REST API documentation New at

24 3 Real-Time & Big 10.5.x

25 ArcGIS Enterprise with real-time & big data capabilities ingest 4K e/s real-time & big data visualize millions analyze 4K e/s store 10Ks e/s ArcGIS Enterprise IoT Big Data GeoEvent Server spatiotemporal big data store GeoAnalytics Server

Writing analytic results from GeoAnalytics Server GeoAnalytics Server has the ability to: - perform analytics against data sources in the SBDS - and

26 Writing analytic results from GeoAnalytics Server GeoAnalytics Server has the ability to: - perform analytics against data sources in the SBDS - and write the analytic results to the SBDS new Web GIS layers ArcGIS Enterprise IoT Big Data GeoEvent Server spatiotemporal big data store GeoAnalytics Server

Writing analytic results from GeoAnalytics Server GeoAnalytics Server has the ability to: - perform analytics against files in an external big data file share - and write the analytic results to the

27 Writing analytic results from GeoAnalytics Server GeoAnalytics Server has the ability to: - perform analytics against files in an external big data file share - and write the analytic results to the spatiotemporal big data store ArcGIS Enterprise new Web GIS layers Web GIS layers.csv text files Hive.shp shapefiles Hadoop file system big data file shares GeoEvent Server spatiotemporal big data store GeoAnalytics Server feature service

28 4 Visualization

for real-time data - you can perform exploratory analysis via any combination

29 On-the-fly aggregations of observations The spatiotemporal big data store enables you to aggregate data on-the-fly - aggregating on-the-fly accommodates for real-time data - you can perform exploratory analysis via any combination of space, time and attribute queries - toggle from aggregation to raw feature view

pointy & flat hexagon/triangle - this is in addition to a temporal index on the time field - and an

30 Preparing data for on-the-fly aggregations spatial indexing as data is written to a data source in the spatiotemporal big data store: - up to four types of spatial indices are supported: geohash, square, pointy & flat hexagon/triangle - this is in addition to a temporal index on the time field - and an inverted index on each of the attribute fields geohash pointy triangle pointy hexagon square flat triangle flat hexagon

31 Getting Real-Time Data into Web Apps Two patterns Feature layers pull from feature services - Web apps poll to get periodic updates - Must be backed by an enterprise geodatabase (EGDB) or spatiotemporal big data store (SBDS) Stream layers subscribe to stream services - Web apps subscribe to immediately receive data - Low latency and high throughput GeoEvent Server Update a Feature Add a Feature feature layers Map Services Feature Services EGDB/ SBDS Polling (Pull) Your Applications Stream Services Send Features to a Stream Service Feature Layer Stream Layer

ArcGIS Enterprise with real-time & big data capabilities Web Device Desktop Ingest high velocity real-time data into ArcGIS Perform continuous analytics on events as they are received live features

32 ArcGIS Enterprise with real-time & big data capabilities Web Device Desktop Ingest high velocity real-time data into ArcGIS Perform continuous analytics on events as they are received live features live & historic aggregates & features Store observations in a spatiotemporal big data store Run batch analytics on stored observations Visualize high velocity & volume data: - as an aggregation - as discrete features ingestion analytics stream service visualization ArcGIS Enterprise map & feature service analytics Notify those who need to know about patterns of interest GeoEvent Server spatiotemporal big data store storage GeoAnalytics Server

33 5 Road Ahead (10.6.x)

34 ArcGIS Enterprise with real-time & big data capabilities ArcGIS Enterprise IoT Big Data GeoEvent Server spatiotemporal big data store GeoAnalytics Server

35 ArcGIS Enterprise with real-time & big data capabilities GeoEvent Resiliency (high availability) & scalability is only possible if User s bring their own gateway. - Barrier to entry is HIGH & typically requires a professional services engagement for success. RESILIENT environment ArcGIS Enterprise IoT spatiotemporal big data store GeoAnalytics Server Big Data bring your own gateway GeoEvent Server

36 ArcGIS Enterprise with real-time & big data capabilities GeoEvent Provides users with a resilient & scalable Real-Time GIS deployment OUT-OF-THE-BOX. ArcGIS Enterprise IoT gateway gateway gateway Big Data GeoEvent Server spatiotemporal big data store GeoAnalytics Server

37 GeoEvent Server 10.6 features RASP (Reliability, Availability, Scalability, Performance) - Kafka as a new gateway service (dedicated process that runs Apache Kafka + Zookeeper in it s own life-cycle). - Create a multi-machine GeoEvent Server site - Uses ArcGIS Server framework for creating multi-machine site

38 GeoEvent Server 10.6 features

39 GeoEvent Server 10.6 features

40 GeoEvent Server 10.6 features

41 GeoEvent Server 10.6 features

42 GeoEvent Server 10.6 features RASP (Reliability, Availability, Scalability, Performance) - Kafka as a new gateway service (dedicated process that runs Apache Kafka + Zookeeper in it s own life-cycle). - Create a multi-machine GeoEvent Server site - Uses ArcGIS Server framework for creating multi-machine site 3 rd party library updates for Apache Karaf, Elasticsearch & Spark Official Esri positioning on IoT (first time presented at UC, see session this afternoon) New and updated Connectors to Azure IoT, Kafka, Waze, GTFS, OAuth2 HTTP inbound

43 Spatiotemporal big data store 10.6 enhancements Enhanced data retention purge rules and feature deletion - with ability to specify a WHERE clause On-the-fly aggregation supports polygon, polyline & multipoint

44 Spatiotemporal big data store On-the-fly aggregation supports polygon, polyline & multipoint

45 Spatiotemporal big data store 10.6 enhancements Enhanced data retention purge rules and feature deletion - with ability to specify a WHERE clause On-the-fly aggregation supports polygon, polyline & multipoint Added space-time volume on-the-fly aggregation support

46 project on-the-fly & interactive space-time aggregation leverage existing geohash, square & hexagon aggregation capabilities of the spatiotemporal big data store to construct on-the-fly space-time volumes. - enables exploratory & interactive analysis of real-time & historic observation data ArcGIS Enterprise lodtype=flathexagon&lod=2 &time= , &timeinterval=1&timeunit=hour ArcGIS API for JavaScript & WebGL client-side rendering of space-time volumes via WebGL IoT GeoEvent Server spatiotemporal big data store

project on-the-fly & interactive space-time aggregation feature service lodtype=geohash&lod=2 &returngeometry=false feature service lodtype=geohash&lod=2 &time=1474675200000,1475279999000

47 project on-the-fly & interactive space-time aggregation feature service lodtype=geohash&lod=2 &returngeometry=false feature service lodtype=geohash&lod=2 &time= , &timeinterval=1&timeunit=hour&f=json &returngeometry=false geohash aggregation response space-time volume (based on a geohash aggregation) response an array of time-slices of aggregated features

48 ArcGIS Enterprise Technologies ArcGIS Enterprise IoT gateway gateway gateway Big Data GeoEvent Server spatiotemporal big data store GeoAnalytics Server

49 6 R&D Project Trinity, Polly & Banzai

is continuously updated state polygon county polygon polygon aggregation

50 project Polly spatial indexing: polygon as data is written to the spatiotemporal big data store: - a spatial index for polygon aggregation is continuously updated state polygon county polygon polygon aggregation (based on a state polygon index) polygon aggregation (based on a county polygon index)

51 Spatial, into geohash, square or hexagon visualization on-the-fly aggregation capabilities Spatiotemporal, into geohash, square or hexagon Spatial, into polygons Spatiotemporal, into polygons

52 project Trinity an ArcGIS Enterprise bundled with massive real-time & big data capabilities project Trinity is a capability that enables existing customers and prospects with the ability to: Web Device Desktop - Reliably ingest, analyze & store up to millions of sensor events per second. - Quickly visualize, replay & explore observations that are billions of features. - Perform fast analytics on billions of observations. Sensors ArcGIS Enterprise with real-time & big data sources hubs project Trinity spatiotemporal archive real-time batch

53 project Trinity bringing geospatial insights to your Web Device Desktop on Amazon C2S on Amazon on Azure project Trinity ArcGIS Enterprise with real-time & big data 50 nodes 400 cores, 1.4 TB ram, 20 TB ssd 30 nodes 480 cores, 1.6 TB ram, 30 TB ssd Internet of your Things Sources + Hubs + Spatiotemporal Archive f( Real-Time Analytics ) f( + Batch Analytics ),, 16 cores, 56 GB ram, 1 TB ssd 2 cores, 4 GB ram, 50 GB ssd 2 cores, 10 GB ram, 200 GB ssd 4 cores, 14 GB ram, 150 GB ssd,, 800 GB ssd,, 200 GB ssd sources hubs real-time analytics spatiotemporal archive batch analytics

54 project Trinity bringing geospatial insights to your Web Device Desktop on Amazon C2S on Amazon on Azure project Trinity ArcGIS Enterprise with real-time & big data 880 cores, 3TB memory, 50TB storage ArcGIS Enterprise GB ssd GB ssd GB ssd GB ssd GB ssd GB ssd GB 3ssd 400 GB 3ssd 400 GB 3ssd 400 GB 4ssd 400 GB 4ssd 400 GB 5ssd 400 6GB ssd GB ssd GB ssd GB ssd7 400 GB 2ssd 400 GB 2ssd 400 GB 2ssd 400 GB 2ssd 400 GB 2ssd 400 GB 2ssd 400 GB 2ssd 400 GB 2ssd 400 GB 2ssd 400 GB 2ssd 4001GB ssd GB ssd GB ssd GB ssd GB ssd GB 2ssd 400 GB 2ssd 400 GB 2ssd 400 GB 2ssd 400 GB 2ssd Internet of your Things sources hubs real-time analytics spatiotemporal archive batch analytics

55 project Trinity big data analytic pipelining & scheduling monitor Trinity task manager admin Banzai 240 cores, 1.1TB ram, 18TB storage ArcGIS Enterprise 28 Trinity GB ram Map service 0 to many a lightweight map & feature service distributed micro-service Source ingestion Event hub Real-Time analytic Spatiotemporal archive Batch analytic 0 to many 0 to few 0 to many 0 to few 0 to many a lightweight GeoEvent Input distributed micro-service a lightweight GeoEvent event hub distributed micro-service a lightweight GeoEvent Service distributed micro-service a lightweight spatiotemporal big data store distributed micro-service a lightweight GeoAnalytics distributed micro-service

56 Thank You to Our Generous Sponsor

GeoEvent Server: An Introduction. Adam Ziegler, Solution Engineer

GeoEvent Server: An Introduction Adam Ziegler, Solution Engineer Agenda 1 2 3 4 5 What is Real-Time GIS? Working with Real-Time Data Applying Real-Time Analytics Visualizing Real-Time Data Wrap-up 1 What