Real-Time & Big Data GIS: Best Practices. Suzanne Foss Josh Joyner

Transcription

1 Real-Time & Big Data GIS: Best Practices Suzanne Foss Josh Joyner

2 ArcGIS Enterprise With Real-time Capabilities Desktop Apps APIs visualization ingestion dissemination & actuation analytics storage

3 Agenda: Architecture Recommendations Big Data Storage Performance, Resiliency, & Scalability Stream Services Service Design Considerations Troubleshooting

4 1 Architecture Recommendations

5 GeoEvent Server What are the primary factors I should consider? Operating environment: m5.2xlarge - virtual machines beware! resources need to be shared in an effective way, like EC2 or Azure. - dedicated bare metal machines or public cloud instances are much more deterministic. Network - speed the faster the better. 1 GBit/s Memory - size 8GB has been required since GiB, default JVM max heap size is 4 GB - type minimum of DDR3 is recommended. - clock speed (MHz) and transfer rate (Mbps) the faster the better. Processors - # of cores the more the better. 8 vcpu - speed (GHz) the faster the better. Disk - 700MB required for installation 10GB recommended minimum (10.6+) - amount of disk space needed will vary based on quantity of deployed input connectors - each input can utilize up to a maximum of 600 MB of disk space before clean up

6 spatiotemporal big data store What are the primary factors I should consider? Operating environment: m5.2xlarge - virtual machines beware! resources need to be shared in an effective way, like EC2 or Azure. - dedicated bare metal machines or public cloud instances are much more deterministic. Disk - speed the faster the better 1,000 Mbps EBS, note: local SSD is much better Network - speed the faster the better. 1 GBit/s Memory - size 16GB minimum. 32GiB, big data store allocates 8GiB by default - type DDR3 is recommended. - clock speed (MHz) and transfer rate (Mbps) the faster the better. Processors - # of cores the more the better. 8 vcpu - speed (GHz) the faster the better.

7 ArcGIS Enterprise with real-time capabilities ArcGIS Enterprise 1 MINIMUM environment 3 machines IoT 2 GeoEvent Server 3 spatiotemporal big data store functional servers & spatiotemporal big data store SHOULD BE on ISOLATED machines

8 ArcGIS Enterprise with real-time capabilities ArcGIS Enterprise 1 OPTIMIZED environment 5 machines IoT GeoEvent Server spatiotemporal big data store functional servers & spatiotemporal big data store SHOULD BE on ISOLATED machines

9 ArcGIS Enterprise with real-time & big data GIS capabilities ArcGIS Enterprise 1 MINIMUM environment 4 machines IoT Big Data GeoEvent Server spatiotemporal big data store GeoAnalytics Server

10 ArcGIS Enterprise with real-time & big data GIS capabilities ArcGIS Enterprise 1 OPTIMIZED environment 10 machines IoT Big Data GeoEvent Server spatiotemporal big data store GeoAnalytics Server

11 ArcGIS Enterprise with real-time & big data GIS capabilities on Microsoft Azure ArcGIS Enterprise Cloud Builder for Microsoft Azure ArcGIS Enterprise 1 IoT Big Data GeoEvent Server spatiotemporal big data store GeoAnalytics Server

12 ArcGIS Enterprise with real-time & big data GIS capabilities on Amazon EC2 ArcGIS Enterprise 1 ArcGIS Enterprise Cloud Builder for Amazon EC2 IoT Big Data GeoEvent Server spatiotemporal big data store GeoAnalytics Server

13 2 Big Data Storage

14 ArcGIS Enterprise with real-time capabilities ArcGIS Enterprise 1 OPTIMIZED environment 5 machines IoT GeoEvent Server spatiotemporal big data store

15 spatiotemporal big data store shards & replication factor spatiotemporal big data store T1 node 1 T3 GeoEvent Server T2 node 5 node 4 r = 1 node 2 T3 node 3 T1 T2

16 spatiotemporal big data store auto-rebalancing of data upon node membership changes, + or -, in the big data store spatiotemporal big data store T1 x T3 GeoEvent Server T2 node 5 T3 T1 node 4 r = 1 T3 node 3 node 2 T1 T1 T2

17 spatiotemporal big data store data retention policies, configured per data source spatiotemporal big data store node 1 r = 1 GeoEvent Server node 5 node 2 node 4 node 3 purge based on data retention

18 spatiotemporal big data store rolling index option, set appropriately to the velocity of your observation data spatiotemporal big data store node 1 GeoEvent Server node 5 indices indices r = 1 node 2 indices node 4 indices node 3 indices purge layer based on data retention policy

19 spatiotemporal big data store automatic data backups using periodic snapshots, including ability to restore from a snapshot spatiotemporal big data store node 1 r = 1 GeoEvent Server node 5 node 2 node 4 node 3 purge layer based on data retention policy snapshot snapshot snapshot snapshot

20 spatiotemporal big data store manual and programmatic export data to cloud stores spatiotemporal big data store node 1 r = 1 GeoEvent Server node 5 node 2 node 4 node 3 delimited text export Azure Blob Amazon S3 purge layer based on data retention policy snapshot snapshot snapshot snapshot

21 spatiotemporal big data store choosing an Object Id option

22 spatiotemporal big data store choosing an Object Id option Max Value # of IDs ArcGIS Clients Int32 2,147,483, billion Pro, Desktop, Ops Dashboard, IoT rate per second per day per week per month per year 1 86, ,800 2,592,000 31,104, ,000 6,048,000 25,920, ,040, ,640,000 60,480, ,200,000 3,110,400,000 1,000 86,400, ,800,000 2,592,000,000 31,104,000,000 10, ,000,000 6,048,000,000 25,920,000, ,040,000, ,000 8,640,000,000 60,480,000, ,200,000,000 3,110,400,000,000 1,000,000 86,400,000, ,800,000,000 2,592,000,000,000 31,104,000,000,000

23 spatiotemporal big data store choosing an Object Id option Max Value # of IDs ArcGIS Clients Int32 2,147,483, billion Pro, Desktop, Ops Dashboard, IoT rate per second per day per week per month per year 1 86, ,800 2,592,000 31,104, ,000 6,048,000 25,920, ,040, ,640,000 60,480, ,200,000 3,110,400,000 1,000 86,400, ,800,000 2,592,000,000 31,104,000,000 10, ,000,000 6,048,000,000 25,920,000, ,040,000, ,000 8,640,000,000 60,480,000, ,200,000,000 3,110,400,000,000 1,000,000 86,400,000, ,800,000,000 2,592,000,000,000 31,104,000,000, days 2.5 days 6 hours 36 minutes

24 spatiotemporal big data store choosing an Object Id option Max Value # of IDs ArcGIS Clients Int32 2,147,483, billion Pro, Desktop, Ops Dashboard, Int64 (signed) 9,223,372,036,854,775, quintillion JavaScript, custom apps IoT rate per second per day per week per month per year 1 86, ,800 2,592,000 31,104, ,000 6,048,000 25,920, ,040, ,640,000 60,480, ,200,000 3,110,400,000 1,000 86,400, ,800,000 2,592,000,000 31,104,000,000 10, ,000,000 6,048,000,000 25,920,000, ,040,000, ,000 8,640,000,000 60,480,000, ,200,000,000 3,110,400,000,000 1,000,000 86,400,000, ,800,000,000 2,592,000,000,000 31,104,000,000,000

25 spatiotemporal big data store choosing an Object Id option Max Value # of IDs ArcGIS Clients Int32 2,147,483, billion Pro, Desktop, Ops Dashboard, Int64 (signed) 9,223,372,036,854,775, quintillion JavaScript, custom apps UniqueStringID n/a unlimited JavaScript, custom apps IoT rate per second per day per week per month per year 1 86, ,800 2,592,000 31,104, ,000 6,048,000 25,920, ,040, ,640,000 60,480, ,200,000 3,110,400,000 1,000 86,400, ,800,000 2,592,000,000 31,104,000,000 10, ,000,000 6,048,000,000 25,920,000, ,040,000, ,000 8,640,000,000 60,480,000, ,200,000,000 3,110,400,000,000 1,000,000 86,400,000, ,800,000,000 2,592,000,000,000 31,104,000,000,000

26 3 Performance, Resiliency, & Scalability

27 Performance, Resiliency & Scalability throughput benchmarks Throughput has increased with every release of ArcGIS GeoEvent Server 10.6 release can support up to 6,000 events per second (e/s) ArcGIS GeoEvent Server Velocity throughput measured in events per second (e/s) up to 500 e/s up to 2,000 e/s up to 3,000 e/s up to 4,000 e/s up to 6,000 e/s

28 Performance, Resiliency & Scalability throughput benchmarks Throughput has increased with every release of ArcGIS GeoEvent Server 10.6 release can support up to 6,000 events per second (e/s) release can support up to 10,000 events per second (e/s) ArcGIS GeoEvent Server Velocity throughput measured in events per second (e/s) up to 500 e/s up to 2,000 e/s up to 3,000 e/s up to 4,000 e/s up to 6,000 e/s up to 10,000 e/s

29 Performance, Resiliency & Scalability multi-machine site support ArcGIS 10.5 Resiliency (high availability) & scalability is only possible if users bring their own gateway. Barrier to entry is HIGH & typically requires a professional services engagement for success. Loses flexibility of input types. ArcGIS 10.6 Provides users with a resilient & scalable Real-Time GIS deployment OUT-OF-THE-BOX. Introduces a gateway process that is automatically configured as part of GeoEvent Server installation. Provides flexibility for all input types. ArcGIS GeoEvent Server Velocity throughput measured in events per second (e/s) up to 500 e/s up to 2,000 e/s up to 3,000 e/s up to 4,000 e/s up to 6,000 e/s up to 10,000 e/s Resiliency & Scalability via multi-machine site no no no no yes yes

30 Performance, Resiliency & Scalability multi-machine site support New tutorial coming with release of 10.7 highlighting multiple deployment patterns

31 Performance, Resiliency & Scalability factors that influence throughput input1 filter (spatial) filter (for input1) buffer output e/s geofences geometry inside Zones/.* 2000 e/s 1000 e/s 1000 e/s 2000 e/s input e/s geotagger geofences 1000 e/s output2 geometry inside Zones/.* 1000 e/s filter (for input2) 2000 e/s motion calculator 1000 e/s output e/s 2000 e/s 4000 e/s 3000 e/s Input event counts don t always tell the whole story

32 Performance, Resiliency & Scalability configuration changes to support larger scale GeoEvent Server by default is only allocated 4GB of RAM for the JVM - If utilizing a large amount of GeoFences it may be necessary to increase this amount This can be modified through the /etc/arcgisgeoevent.cfg up to 32GB (JVM limitation)

33 Performance, Resiliency & Scalability configuration changes to support larger scale GeoEvent Server by default is able to maintain the state of 1000 unique Track_IDs. - This value can be increased by editing /etc/com.esri.ges.manager.servicemanager.cfg You may also need to modify the Incident Manager Setting in the Global Setting Tab if used in conjunction with the Incident Detector Processor

34 4 Stream Services

35 Stream Services why use a stream service instead of a feature service? Stream Services allow for improved rendering detail of moving objects - In a Web Map / App can display up to 10 updates per object per second - Feature Service updates once every 6 seconds Light weight requiring no storage of events Updates are pushed instead of polled, supporting reduced delays With Pro 2.2+ users are able to configure unique feature rendering

36 Stream Services why use a feature service instead of a stream service? Feature Services provide a confirmation that add/updates have been completed - Stream Services do not provide any delivery response (greater risk of missing messages) Data is persisted even when client is not actively receiving data - Stream Service messages are ephemeral, being lost when no client is subscribed Rendering control across all web and desktop clients Aggregation support with the spatiotemporal big data store

37 Stream Services other considerations Can you have both? - Configure a Stream Service with Store Latest - Bootstrap records from a feature service and replaced with new data from the stream Adjust Global Settings for improved throughput

38 5 Service Design Considerations

39 Service Design Considerations which would you choose? Service A Service B

40 Service Design Considerations which would you choose? Each branch in a service contains the same event data. In this example, with three branches, it is creating 3X the volume of data. When possible, pre-filter the input data before ingesting.

41 Service Design Considerations not all components are created equally A Which of these services has the potential to least impact performance? B C

42 Service Design Considerations not all components are created equally B C A The first service only contains components that are utilizing the internal service cache, which can allow for the fastest processing.

43 Service Design Considerations not all components are created equally B C A The second service modifies the incoming event geometry which can be costly. These types of requests are typically very quick but can be impacted by geometry complexity.

44 Service Design Considerations not all components are created equally B C A The third service utilizes Network Analyst to return a drive time polygon which can significantly impact throughput.

45 Service Design Considerations other recommendations Configure Filters and/or Field Reducer Processors as early as possible in a service - This reduces the volume / data size of the events being processed - Potentially simplifies service configuration down stream Avoid Managed GeoEvent Definitions when possible - These are system owned definitions whose lifecycle is entirely controlled by the processors - Editing or Deleting a processor will remove these definitions - If necessary copy generated definition and edit processor to look for it Utilize single data pipelines whenever possible - Branching out or back together of pipelines can have multiplicative impacts - Better to duplicate workflows across multiple services than a single complex workflow

46 6 Troubleshooting

47 Troubleshooting inputs I can t get my data to come in Check the definition - Most input problems are with misconfigured schema (field names, data types, group structure) - Try letting GeoEvent Server create definition for you - but you will likely need to edit the definition and edit the input to use the one you modified

48 Troubleshooting outputs I can t write my data to Check the definition - Most output problems are with misconfigured schema (field names, data types, group structure) - If possible import the definition from the service you are trying to write to - If an Esri Feature Service remove the reserved field names (e.g. ObjectID / OID) - Use the GeoEvent Logger application to verify the expected output data

49 Troubleshooting logging It doesn t work when I Check the logs for errors - For critical problems the cause of the problem will likely display in the log message - You may need to get more detail from a specific logger to isolate a cause

50 Troubleshooting backup Everything was working yesterday but or Someone accidently deleted Did you make a backup of your configuration? - With ArcGIS GeoEvent Server 10.5 or newer, we did for you

51 See Us Here Tuesday: 03:00 PM - 04:00 PM : ArcGIS GeoEvent Server: An Introduction (204 C) 04:15 PM - 05:15 PM : Real-Time and Big Data GIS: Best Practices (204 C) Wednesday: 11:00 AM - 12:00 PM : ArcGIS GeoEvent Server: An Introduction (204 C) 01:30 PM - 02:30 PM : ArcGIS GeoEvent Server: Applying Real-Time Analytics (204 C) 02:45 PM - 03:45 PM : ArcGIS GeoEvent Server: Visualizing Real-Time Data (204 C) 04:00 PM - 05:00 PM : Real-Time and Big Data GIS: Best Practices (204 C) 05:15 PM - 06:15 PM : Real-Time and Big Data GIS: Road Ahead (204 C) Thursday: 11:30 AM - 12:15 PM : ArcGIS GeoEvent Server: A Developer's Guide (209 B) 04:30 PM - 05:15 PM : ArcGIS GeoEvent Server: A Developer's Guide (209 C)

52 Print Your Certificate of Attendance Print Stations Located at L Street Bridge Tuesday 12:30 pm 6:30 pm GIS Solutions Expo Hall D Wednesday 10:45 am 5:15 pm GIS Solutions Expo Hall D 5:15 pm 6:30 pm GIS Solutions Expo Social Hall D 6:30 pm 9:00 pm Networking Reception National Building Museum

53 Please Take Our Survey on the App Download the Esri Events app and find your event Select the session you attended Scroll down to find the feedback section Complete answers and select Submit

54 Questions / Feedback? Josh Joyner ArcGIS GeoEvent Server Product Manager, Esri jjoyner@esri.com Suzanne Foss ArcGIS Analytics for IoT Product Manager, Esri sfoss@esri.com

55