Verarbeitung von Vektor- und Rasterdaten auf der Hadoop Plattform DOAG Spatial and Geodata Day 2016

Transcription

1

2 Verarbeitung von Vektor- und Rasterdaten auf der Hadoop Plattform DOAG Spatial and Geodata Day 2016 Hans Viehmann Product Manager EMEA ORACLE Corporation 12. Mai 2016

3 Safe Harbor Statement The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle s products remains at the sole discretion of Oracle. 3

4 Program Agenda Introduction to Big Data Spatial and Graph Big Data Spatial Features / Vector Analysis Big Data Spatial Features / Raster Analysis More resources Q & A

5 Big Data Spatial and Graph Business Benefits Motivation Big Data often needs an organizing principle Data Harmonization Big Data analysis is often about relationships not aggregation Big Data platform is economically compelling for working with massive data sets found in spatial and graph workflows

6 Graph and Spatial Analysis It is about relationships Are things in the same location? Who is the nearest? What tax zone is this in? Who can reach us in 35 minutes? What is in my sales territory? Is this built in a flood zone? Which supplier am I most dependent upon? Who is the most influential customer? Do my products appeal to certain communities? What patterns are there in fraudulent behavior?

7 Strategic Vision: The platform Database and Big Data Platform support, both On-premise and in the Cloud Oracle Big Data Spatial and Graph Oracle Database Spatial and Graph Spatial and Graph in Cloud Offerings NoSQL Big Data: Single Model Data Store Database 12c: Polyglot (Multi-model) Data Store 7

8 Using Hadoop to Address Big Data Challenges The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models. Hadoop is designed to scale up from single servers to thousands of machines, each offering local computation and storage. Rather than rely on hardware to deliver highavailability, the library itself is designed to detect and handle failures at the application layer, so delivering a highlyavailable service on top of a cluster of computers, each of which may be prone to failures. 8

9 Benefit of Big Data Technologies Low cost due to commodity hardware High horizontal scalability infrastructure Allowing storage of more data, more details over longer time periods Cost-effective way to analyse huge amounts of data Dealing with variable data by means of schema-on-read capability Complementary to existing data warehouse technologies 9

10 Conventional database or Big Data technologies Typical technical decision criteria Straight Through Processing (STP) Tooling maturity Stringent Non-Functionals Ingestion rate Cost effectively store low value data ACID transactional requirement Security Hadoop Relational ETL simplicity Data sparsity Variety of data formats

11 MapReduce Paradigm mostly Batch-oriented Map() Shuffle Reduce() Illustration courtesy of

12 Hadoop Concepts: Map Reduce OReilly Hadoop The Definitive Guide

13 Hadoop Concepts: HDFS Illustration courtesy of 14

14 Hadoop Concepts: HDFS Reading from HDFS Writing to HDFS OReilly Hadoop The Definitive Guide

15 Hadoop Concepts: Job and Task execution OReilly Hadoop The Definitive Guide 16

16 But does this work for geospatial data? Location Intelligence has specific requirements Geometric functions and topological operations have to be ported to MapReduce Taking into account projections, complex operations such as buffering,... More complex processing usually requires spatial indexing eg. spatial joins Spatial data usually comes in specific formats (Shapefiles, GeoJSON,...) Needs to cope with location information which is only included implicitly requires geo-enrichment Visualization is very valuable for inspection of source data and results 17

17 Oracle Big Data Spatial and Graph Spatial Analysis: Location Data Enrichment Proximity and containment analysis, Clustering Spatial data preparation (Vector, Raster) Interactive visualization Property Graph for Analysis of: Social Media Relationships ecommerce Targeted Marketing Cyber-Security, Fraud Detection IoT, Industrial Engineering

18 Program Agenda Introduction to Big Data Spatial and Graph Big Data Spatial Features / Vector Analysis Big Data Spatial Features / Raster Analysis More resources Q & A

19 What problems can Big Data Spatial analysis address? Data Harmonization using any location attribute (address, postal code, lat/long, placename, etc). Categorization and filtering based on location and proximity Preparation, validation and cleansing of Spatial and Raster data Visualizing and displaying results on a map

20 What features does Big Data Spatial have? Data enrichment service API using GeoNames and geometry hierarchy data Spatial processing of data stored in HDFS. Raster processing operations: Mosaic and sub-set operations. Geodetic and Cartesian data MapReduce routines for distance calculations, PointInPolygon, buffer creation, Categorization, KMeansClustering, Binning, etc., Hive integration HTML5 Map Visualization API

21 Spatial Features Technical overview Support for spatial data in 2D or 3D in various formats, geodetic or projected Support for geo-referenced imagery such as satellite images in many formats MapReduce framework for resolution of placenames and determination location hierarchies, including reference dataset Integration with Hive for processing and access from Oracle database Spatial indexing techniques for fast retrieval of spatial data Library of spatial operators for geometric analysis (inside, within distance, K-means clustering, spatial binning,...), including Spatial Joins (new) Library of image processing functions (mosaic, reprojection, format conversion, analysis,...) Console for visual analysis, indexing, processing as sample application

22 Hive integration Table creation CREATE EXTERNAL TABLE IF NOT EXISTS sample_tweets_index (id STRING, geometry STRING, followers_count STRING, friends_count STRING, location STRING) ROW FORMAT SERDE 'oracle.spatial.hadoop.vector.hive.json.geojsonserde' STORED AS INPUTFORMAT 'oracle.spatial.hadoop.vector.mapred.input.spatialindextextinputformat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.hiveignorekeytextoutputformat' LOCATION '/user/oracle/twitter/index'

23 Hive integration HiveQL Query SELECT id, followers_count, friends_count, location FROM sample_tweets WHERE ST_Contains( ST_Polygon('{"type": "Polygon","coordinates": [[[-106, 25], [-106, 30], [-104, 30], [-104, 25], [-106, 25]]]}', 8307), ST_Point(geometry, 8307), 0.5) and followers_count > 50;

24 Spatial Vector Processing Framework Mapper and Reducer Classes Customer Application Sample Application MapReduce Framework, templates Spatial Operators, Functions Java API Spatial Enrichment, Categorization API Customer data Generated data GeoJSON, JGeometry format RecordReader class GeoNames and Hierarchy data Oracle Provided Customer code Geospatial Vector data (any format) HDFS Enrichments, Categorizations results

25 Program Agenda Introduction to Big Data Spatial and Graph Big Data Spatial Features / Vector Analysis Big Data Spatial Features / Raster Analysis More resources Q & A

26 Framework for processing raster data Loading georeferenced raster data into HDFS blocks Single images or in bulk From 100+ source formats using GDAL For parallel processing with minimal data transfer between cluster nodes Pre-processing Re-projection, combination of images into mosaics,... Maintenance of metadata catalog Execution of image processing algorithms on HDFS Raster Algebra, bespoke algorithms On single images or groups Optional distribution into overlapping blocks For algorithms requiring neighbouring pixels during processing

27 Spatial Raster Processing Framework Raster Analysis Application Sample Console Application Raster MapReduce framework Java API Bespoke Algorithms Customer data Subset Mosaic Raster Analysis API Generated data Oracle Provided Customer code Raster Data Files on NFS HDFS Raster catalog Derived Rasters GDAL Loader (GDAL formats) Rasters

28 Examples: Raster data preparation Pyramiding: layers at different resolution Mosaic images Terrains and contours Shaded reliefs

29 Use case: Raster Data Processing in Precision Farming Source: Monsanto, presentation at HadoopWorld 2013

30 SOURCES The Big Picture Oracle Big Data Management System DATA RESERVOIR DATA WAREHOUSE Cloudera Hadoop Oracle Big Data SQL Oracle Event Processing Oracle NoSQL Oracle R Distribution Oracle Big Data Spatial and Graph Big Data Appliance Apache Flume Oracle GoldenGate Oracle Big Data Connectors Oracle Data Integrator Oracle Oracle Database Database In-Memory, Oracle Industry Multi-tenant Models Oracle Industry Models Oracle Advanced Analytics Oracle Advanced Oracle Spatial Analytics & Graph Oracle Spatial & Graph Oracle Data Integrator Exadata Oracle GoldenGate Oracle Event Processing

31 Program Agenda Introduction to Big Data Spatial and Graph Big Data Spatial Features / Vector Analysis Big Data Spatial Features / Raster Analysis More resources Q & A

32 Further Information General information on oracle.com Oracle Technology Network Big Data Spatial and Graph blog Oracle Spatial and Graph (!) group on LinkedIn... or try it out yourself using the latest BigDataLite VM on OTN available for download here

33 Big Data Spatial and Graph Hands-on Labs Load common log data like twitter feeds to HDFS, and perform spatial analysis on that data. Develop a RecordReader for this sample data so that spatial analysis functions can be applied to records in the data. Create and use a spatial index for such data. Use RecordReaders for additional data formats like Shape and JSON. Use the HTML5 mapping API to create interactive visualization applications 35

34 Program Agenda Introduction to Big Data Spatial and Graph Big Data Spatial Features / Vector Analysis Big Data Spatial Features / Raster Analysis More resources Q & A

35

36