IOUG BIWA and Spa.al SIG TechCast: Introducing Oracle Big Data Spa.al and Graph

Size: px

Start display at page:

Download "IOUG BIWA and Spa.al SIG TechCast: Introducing Oracle Big Data Spa.al and Graph"

Clifford Bennett
5 years ago
Views:

Steve Pierce CEO, Think Huddle & Chair, IOUG

1 IOUG BIWA and Spa.al SIG TechCast: Introducing Oracle Big Data Spa.al and Graph July 21, 2015 Speaker: James Steiner Vice President, Oracle Moderator: Steve Pierce CEO, Think Huddle & Chair, IOUG Spa.al SIG Copyright 2015 Oracle and/or its affiliates. All rights reserved. 1

2 The Spa?al & Graph SIG The SIG promotes interaction and communication that can drive the market for spatial technology and data Members connect and exchange knowledge via online communities and at annual conferences and events Join us online LinkedIn (search for LinkedIn Oracle Spatial ) Google+ (search for Google+ Oracle Spatial ) IOUG SIG (sign up for free membership through OTN Spatial Communities (search for Oracle Spatial and Graph Community ) Contact the Board at oraclespatialsig@gmail.com Copyright 2015 Oracle and/or its affiliates. All rights reserved.

accep.ng speaker abstracts! Apply at: www.

3 BIWA & Spa?al Summit 2016 Jan Oracle HQ Conference Center Business Intelligence, Warehousing, and Analy?cs + Spa?al IOUG Special Interest Groups (SIGs) Now accep.ng speaker abstracts! Apply at: Copyright 2015 Oracle and/or its affiliates. All rights reserved.

Safe Harbor Statement The following is intended to outline our general product direc?on. It is intended for informa?on purposes only, and may not be incorporated into any contract.

5 Safe Harbor Statement The following is intended to outline our general product direc?on. It is intended for informa?on purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or func?onality, and should not be relied upon in making purchasing decisions. The development, release, and?ming of any features or func?onality described for Oracle s products remains at the sole discre?on of Oracle. Copyright 2014 Oracle and/or its aﬃliates. All rights reserved. 5

Spa?al and Graph Oracle Database Spa?al and Graph Spa?

7 Oracle s Spa?al and Graph Strategy Enable Spa.al and Graph use cases on every Big Data pla[orm Oracle Big Data Spa?al and Graph Oracle Database Spa?al and Graph Spa?al and Graph in Cloud Offerings NoSQL Copyright 2015 Oracle and/or its affiliates. All rights reserved. 7

8 Conven?onal database or Big Data technologies Typical technical decision criteria Straight Through Processing (STP) Tooling maturity Stringent Non- Func?onals Inges?on rate Cost effec?vely store low value data ACID transac?onal requirement Security Hadoop Rela?onal ETL simplicity Data sparsity Variety of data formats Copyright 2015 Oracle and/or its affiliates. All rights reserved. 8

9 Oracle s Spa?al and Graph Offerings Best in Class Spa?al Technologies: Oracle Database Most widely deployed Spa?al Database Complete GIS and Loca?on Intelligence plahorm services Supports OGC, ISO standards Big Data Plahorm 2D, 3D Spa?al analy?cs; Raster analy?cs framework Geo- enrichment for Big Data Map Visualiza?on Best in Class Graph Technologies: Oracle Database Highest Performance RDF Triple Store; W3C Standards Compliant Network Data Model for Mul?- modal, dynamic Spa?al graphs Big Data Plahorm Distributed, Scalable Property Graph in HBase and NoSQL DB 35+ In- memory Graph Analysis func?ons; Tinkerpop and Blueprints API support 9 Copyright 2015 Oracle and/or its affiliates. All rights reserved.

The Big Picture Oracle Big Data Management System DATA RESERVOIR DATA WAREHOUSE Cloudera Hadoop Oracle Big Data SQL Oracle Event Processing Oracle NoSQL Oracle R Distribu.on Oracle Big Data Spa.

10 The Big Picture Oracle Big Data Management System DATA RESERVOIR DATA WAREHOUSE Cloudera Hadoop Oracle Big Data SQL Oracle Event Processing Oracle NoSQL Oracle R Distribu.on Oracle Big Data Spa.al and Graph Big Data Appliance Apache Flume B Oracle GoldenGate Oracle Big Data Connectors Oracle Data Integrator Oracle Oracle Database Database In- Memory, Oracle Industry Mul.- tenant Models Oracle Industry Models Oracle Advanced Analy.cs Oracle Advanced Oracle Spa.al Analy.cs & Graph Oracle Spa.al and Graph Oracle Data Integrator Exadata Oracle GoldenGate Oracle Event Processing SOURCES Copyright 2015 Oracle and/or its affiliates. All rights reserved. 10

13 Oracle Big Data Spa?al and Graph Property Graph for Analysis of: Social Media rela?onships Internet of Things interac?ons Cyber- Security Spa?al Analysis Features for: Loca?on Data Enrichment Proximity and containment analysis Prepara?on of digital map and imagery data sets Copyright 2015 Oracle and/or its affiliates. All rights reserved. 13

) Assets (pipe lines, cables, transformers) Sales data (sales territory, customer registra?on, etc.

14 What is Spa?al Data Integral part of almost every database Business data that contains or describes loca?on Geographic features (roads, rivers, parks, etc.) Assets (pipe lines, cables, transformers) Sales data (sales territory, customer registra?on, etc.) Street and postal address (customers, stores, factories, etc.) Anything associated with a physical loca?on Described by coordinates or implicitly as text (place name),... Loca?on is a universal key rela?ng otherwise unrelated en??es Copyright 2015 Oracle and/or its affiliates. All rights reserved. 14

tle: * Text message: Driving on 101 North, just reached border between Marin County and San Francisco

15 Linking informa?on by loca?on Are these data points related? Tweet: sailing by #goldengate Instagram image sub?tle: * Text message: Driving on 101 North, just reached border between Marin County and San Francisco County GPS Sensor: N 37 49ʹ 11 W ʹ 44 Now find all data points around Golden Gate Bridge... * Golden Gate Bridge (in Korean) Copyright 2015 Oracle and/or its affiliates. All rights reserved. 15

Oracle Big Data Spa?al Features Geo- enrichment for Data Harmoniza?on Resolu?on of loca?on- related informa?on Determina?on of loca?on hierarchies Categoriza?

16 Oracle Big Data Spa?al Features Geo- enrichment for Data Harmoniza?on Resolu?on of loca?on- related informa?on Determina?on of loca?on hierarchies Categoriza?on and filtering Tracking, proximity analysis, geo- fencing and categoriza?on based on loca?on Data prepara?on Large scale geoprocessing for cleansing, prepara?on of imagery, sensor data, and raw data input Data visualiza?on Copyright 2015 Oracle and/or its affiliates. All rights reserved. 16

Use Cases: Geo- Enrichment i ii (i) Wireless network performance (dropped calls, u?

on analysis: combine transit card/payment info and other sources to determine where

17 Use Cases: Geo- Enrichment i ii (i) Wireless network performance (dropped calls, u?liza?on): aggregate and display geocoded Call Data Records, sensor data, other sources for analysis (ii) Transporta?on origin- des?na?on analysis: combine transit card/payment info and other sources to determine where (and how many) people travel to, star?ng from any sta?on on a transit network (iii) Geotagged Twiuer: where are the tourists and locals twee?ng iii Copyright 2015 Oracle and/or its aﬃliates. All rights reserved. 17

20 Oracle Big Data Spa.al Overview An Apache Hadoop framework Uses MapReduce programs and analy?cs on Hadoop to store, access, and analyze spa?al data Includes a schema and func.ons that facilitate the storage, retrieval, update, and query of collec?ons of spa?al data A Spa?al Server to load, query and analysis vector data An Image Processing Framework to store and process imagery/raster data Copyright 2015 Oracle and/or its affiliates. All rights reserved. 20

21 Big Data Spa?al and Graph Spa?al Vector Processing Framework Mapper and Reducer Classes Customer Applica.on MapReduce Framework, templates Sample Applica.on Spa.al Operators, Func.ons S&G Java API Spa.al Enrichment, Categoriza.on API Customer data Generated data GeoJSON, JGeometry format RecordReader class GeoNames and Hierarchy data Oracle Provided Customer code Geospa.al Vector data (any format) HDFS Enrichments, Categoriza.ons results Copyright 2015 Oracle and/or its affiliates. All rights reserved. 21

22 Overview of Spa?al Features Vector Data Processing Support spa?al processing of data stored in HDFS Commonly used opera?ons like pointinpolygon, buffer crea?on, distance calcula?ons, anyinteract opera?ons, etc. Supports both Geode?c and Cartesian data Data enrichment services using GeoNames and geometry hierarchy data Map visualiza?on API (HTML5) Raster Data Processing GDAL to load raster data onto HDFS Raster processing opera?ons: Mosaic and sub- set opera?ons MapReduce framework for raster analysis opera?ons (for example, calculate the slope at each pixel based on the DEM) Copyright 2015 Oracle and/or its affiliates. All rights reserved. 22

23 Oracle Big Data Spa?al Features Build Map Visualiza.on using HTML5- based API Includes an HTML5- based map client API for developers Default world boundary data provided as JSON files The map view can display any data stored in GeoJSON files Browser based, rich interac?on Copyright 2015 Oracle and/or its affiliates. All rights reserved. 23

24 Oracle Big Data Spa?al Features HDFS storage for the image or raster files We can support dozens of file formats Images are georeferenced Images can be in different coordinate systems and resolu?ons Three main capabili?es GDAL- based loader to load raster data from NFS to HDFS Mosaic and subset opera?ons Image processing framework for raster analysis Copyright 2015 Oracle and/or its affiliates. All rights reserved. 24

25 Oracle Big Data Spa?al Features Console to view the set of images that are available Map displays all available images using image footprints Users can zoom into different areas of the world to see available images for any region Informa?on about the source and other spa?al informa?on for each image can also be displayed Ability to group sets of images into groups for further processing Images can be given priority in a group based on date, resolu?on, etc. Copyright 2015 Oracle and/or its affiliates. All rights reserved. 25

30 There Are Lots of Answers to That Answers from Aggrega.on Who has the most appearances? Who has the most series? Who has the most movie appearances, toys, etc? Tabular ques.ons: Well- suited to SQL- like tools Answers from Connec.vity Who s most central to teams? Who has the strongest cross- overs? How important is the hero s community? Graph ques.ons: We need something different! Copyright 2015 Oracle and/or its affiliates. All rights reserved. Oracle Confiden?al Internal/Restricted/Highly Restricted 30

Graph Data Models Social Network Analysis Property Graph Model Graph

cs Public Safety Social Media search Marke?ng - Sen?ment Spa?

es Telcoms Linked Data / Metadata Layer RDF Data Model Data federa?

31 Graph Data Models Social Network Analysis Property Graph Model Graph Data Management Social Network Analysis En?ty analy?cs Public Safety Social Media search Marke?ng - Sen?ment Spa?al Network Analysis Network Data Model Network path analysis Transporta?on modeling Logis?cs Transporta?on U?li?es Telcoms Linked Data / Metadata Layer RDF Data Model Data federa?on Knowledge representa?on Seman?c Web Life Sciences Health Care Publishing Finance Use Case Graph Model Industry Domain Copyright 2015 Oracle and/or its affiliates. All rights reserved. 31

32 Graph Data Model C B What is a graph? A set of links and nodes (and op?onally auributes) A graph is simply linked data Why do we care? Rise in Commercial use of Big Data A F E D Web log files, Twiue feeds, sensor readings, IoT Cyber networks, power grids, protein interac?on graphs Knowledge graphs (IBM Watson, Apple SIRI, Google Knowledge Graph) Graphs are intui?ve and flexible Easy to navigate, easy to form a path, natural to visualize Do not require a predefined schema Copyright 2015 Oracle and/or its affiliates. All rights reserved. 32

33 The Property Graph Data Model hups://github.com/?nkerpop/blueprints/wiki/property- Graph- Model A set of ver?ces (or nodes) each vertex has a unique iden?fier. each vertex has a set of in/out edges. each vertex has a collec?on of key- value proper?es. A set of edges (or links) each edge has a unique iden?fier. each edge has a head/tail vertex. each edge has a label deno?ng type of rela?onship between two ver?ces. each edge has a collec?on of key- value proper?es. Copyright 2015 Oracle and/or its affiliates. All rights reserved. 33

Common Graph Analysis Use Cases Recommend the most

people that are central in the given network e.g. influencer marke?

fy group of people that are close to each other e.g. target group marke?

?es that match to the given pauern e.g. fraud detec?

34 Common Graph Analysis Use Cases Recommend the most similar item purchased by similar people Find out people that are central in the given network e.g. influencer marke?ng Iden?fy group of people that are close to each other e.g. target group marke?ng Find out all the sets of en??es that match to the given pauern e.g. fraud detec?on Product Recommenda?on Influencer Iden?fica?on Community Detec?on Graph Pauern Matching customer items Purchase Record Communica?on Stream (e.g. tweets) Copyright 2015 Oracle and/or its affiliates. All rights reserved. 34

35 CyberSecurity Modeling / Internet of Things Property graph model Dynamic construc?on of IP network The graph includes metadata as well as events/enriched data Extensible by other data source (add proper?es, rela?ons) Search Text search on graph DB proprie?es 35 Copyright 2015 Oracle and/or its affiliates. All rights reserved.

37 Mo?va?on for an Oracle Graph Database Customers increasingly seek answers from connec?vity RDBMS vendors support triple- stores Good for inference Bad for structural analysis (communi?es, centrality, etc.) Niche vendors require deep commitment Proprietary query languages Proprietary, models Open- source projects fade away Aurelis acquisi?on Giraph (cumbersome, rarely releases), Golden Orb (dead) Copyright 2015 Oracle and/or its affiliates. All rights reserved. Oracle Confiden?al Internal/Restricted/Highly Restricted 37

38 Oracle Big Data Graph Massively- Scalable Graph Database Scales to trillions edges Apache HBase Oracle NoSQL Database In- Memory Graph Analy?cs More than 30 graph analysis algorithms Simple interfaces Java Tinkerpop: Blueprints, Gremlin, Rexster Python Detec?ng Components and Communi?es Evalua?ng Communi?es Ranking/Walking Path- Finding Copyright 2015 Oracle and/or its affiliates. All rights reserved. Oracle Confiden?al Internal/Restricted/Highly Restricted 38

39 Support for Open Source TinkerPop Graph Tool Stack Oracle Big Data Spa?al and Graph Blueprints API implementa?on provides support for the de- facto graph database standard TinkerPop component stack. These include query language, dataflow, REST APIs, and others. Copyright 2015 Oracle and/or its affiliates. All rights reserved. 39

40 Graph Analysis Can Be Complicated Find the size of the 2- hop network (Gremlin+Python) sum([v.query() \.direction(blueprints.direction.out).count() \ for v in OPGIterator(v0.query() \.direction(blueprints.direction.out) \.vertices().iterator())]) Compu?ng importance is more nebulous and harder to code Copyright 2015 Oracle and/or its affiliates. All rights reserved. Oracle Confiden?al Internal/Restricted/Highly Restricted 40

41 Degree Centrality in Big Data Graph Code Influence = analyst.indegreecentrality() Copyright 2015 Oracle and/or its affiliates. All rights reserved. Oracle Confiden?al Internal/Restricted/Highly Restricted 41

42 Big Data Graph Architecture Lightning- Fast In- Memory Analy?cs YARN Container Standalone Server Embedded Massively Scalable Graph Store Oracle NoSQL HBase Copyright 2015 Oracle and/or its affiliates. All rights reserved. Oracle Confiden?al Highly Restricted 42

43 Data Format Support GML, GraphML, GraphSON Oracle- defined Property Graph flat files Vertex file, Edge file Support basic data types + Date with Timezone + Serializable objects Allow mul?ple data types to be associated with one key UTF8 based 1,name,1,Barack%20Obama,, 1,age,2,,53, 1,likes,1,scrabble,, 1,likes,5,,, T00:00: :00 1,occupation,1,44th%20president%20of%20United%20States%20of%20America,, Copyright 2015 Oracle and/or its affiliates. All rights reserved. 43

44 Text Search through Apache Lucene/Solr Integra?on with Apache Lucene/Solr Support manual and auto indexing of Graph elements Manual index: oraclepropertygraph.createindex( my_index", Vertex.class); indexver?ces = oraclepropertygraph.getindex( my_index, Vertex.class); indexver?ces.put( key, value, myvertex); Auto Index oraclepropertygraph.createkeyindex( name, Edge.class); oraclepropertygraph.getedges( name, *hello*world ); Enables queries to use syntax like *oracle* or *graph* Copyright 2015 Oracle and/or its affiliates. All rights reserved. 44

45 30+ Graph Func?ons Detec?ng Components and Communi?es Tarjan s, Kosaraju s, Weakly Connected Components, Label Propaga?on (w/ variants), Soman and Narang s Ranking and Walking Pagerank, Personalized Pagerank, Betweenness Centrality (w/ variants), Closeness Centrality, Degree Centrality, Eigenvector Centrality, HITS, Random walking and sampling (w/ variants) Evalua?ng Community Structures Conductance, Modularity Clustering Coefficient (Triangle Coun?ng) Adamic- Adar Path- Finding Hop- Distance (BFS) Dijkstra s, Bi- direc?onal Dijkstra s Bellman- Ford s Link Predic?on SALSA (Twiuer s Who- to- follow) Other Classics Vertex Cover Minimum Spanning- Tree(Prim s) Copyright 2015 Oracle and/or its affiliates. All rights reserved. 45

46 Unique Graph Filtering Opera?ons Graph analysis engine reads graph into memory from HBase or NoSQL Can reach memory limit for huge graphs Subgraph mechanism to address this Data Access Layer filtering used to create subgraph for Property Graph Engine analy?cs Persisted graph can s?ll be modified through Java and REST APIs Changes can be propagated to in- memory graph Analy?c Analy?c Request Request Property Graph Engine Analy?c Analy?c Request Request Analy?c Request Analy?c Trans- Request ac?onal Request Oracle Property Graph or RDF (HBase or NoSQL) Copyright 2015 Oracle and/or its affiliates. All rights reserved. 46

Graph Solution Workflows and Characteristics Graph Data

and persisted as HDFS Graph queries on HDFS or Oracle

(in-memory analysis engine) Data scientists try different

production system Fixed, automated, batch-oriented,

47 Graph Solution Workflows and Characteristics Graph Data Management Raw business data is converted to graph format and persisted as HDFS Graph queries on HDFS or Oracle NoSQL using Java REST APIs Analysis and Exploration (in-memory analysis engine) Data scientists try different ideas (algorithms) on the data Flexible, interactive, iterative, small-scale (sampled),. Production phase Important discoveries are applied to the production system Fixed, automated, batch-oriented, large-scale, Ideas about the data Produc?on System Data En??es Graph Persistence (Hbase, NoSQL) Graph Query and Analysis Discoveries on the data Copyright 2015 Oracle and/or its affiliates. All rights reserved. 47

48 Big Data Spa?al and Graph Designed for Hadoop/NoSQL workloads High Performance Big Data Scale Commercial quality Graph to model and analyze connec?vity and rela?onships Loca?on services for data harmoniza?on, categoriza?on, filtering, and ELT Copyright 2015 Oracle and/or its affiliates. All rights reserved. Oracle Confiden?al Internal/Restricted/Highly Restricted 48

49 Upcoming Webcast at BIWA Virtual Summit July 30-31, 9:00-1:00 pm, US Central Graph Data Management and Analy?cs for Big Data July 31, 10:00-11:00 am US CDT A deep dive into the property graph capabili?es of Big Data Spa?al and Graph including discovering communi?es and influencers, finding rela?onships, and genera?ng recommenda?ons. Speakers: Bill Beauregard & Zhe Wu, Oracle View full BIWA Virtual Summit agenda: hup:// Register here: hups://auendee.gotowebinar.com/register/ Copyright 2015 Oracle and/or its affiliates. All rights reserved. 49

Resources Recording and slides from today: hup://www.ioug.org/p/cm/ld/fid=153 - Ques?ons a er the techcast? Email speakers@ioug.org Oracle Big Data Spa?al and Graph on Oracle.

alandgraph Blog (technical examples and?ps): hups://blogs.oracle.com/bigdataspa?algraph/ Big Data Lite Virtual Machine (a free sandbox environment to get started): hup://www.

50 Resources Recording and slides from today: hup:// - Ques?ons a er the techcast? speakers@ioug.org Oracle Big Data Spa?al and Graph on Oracle.com: hups:// data- spa?al- and- graph OTN product page (trial so ware downloads, doc, help forum): hup:// technologies/bigdata- spa?alandgraph Blog (technical examples and?ps): hups://blogs.oracle.com/bigdataspa?algraph/ Big Data Lite Virtual Machine (a free sandbox environment to get started): hup:// appliance/oracle- bigdatalite html Copyright 2015 Oracle and/or its affiliates. All rights reserved. 50

Q&A Oracle Big Data Spa.al and Graph: hfps://www.oracle.com/database/big- data- spa.al- and- graph For more ques?

51 Q&A Oracle Big Data Spa.al and Graph: hfps:// data- spa.al- and- graph For more ques?ons a er today s techcast, speakers@ioug.org Copyright 2015 Oracle and/or its affiliates. All rights reserved. 51

Verarbeitung von Vektor- und Rasterdaten auf der Hadoop Plattform DOAG Spatial and Geodata Day 2016

Verarbeitung von Vektor- und Rasterdaten auf der Hadoop Plattform DOAG Spatial and Geodata Day 2016 Hans Viehmann Product Manager EMEA ORACLE Corporation 12. Mai 2016 Safe Harbor Statement The following