What s New with SpaDal and Graph?

Size: px

Start display at page:

Download "What s New with SpaDal and Graph?"

Austin Wood
5 years ago
Views:

2 What s New with SpaDal and Graph? Technologies to Be.er Understand Complex Rela8onships Jim Steiner Vice President, Product Management Server Technologies January, 2016 Copyright 2014, Oracle and/or its aﬃliates. All rights reserved.

3 Safe Harbor Statement The following is intended to outline our general product direcdon. It is intended for informadon purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or funcdonality, and should not be relied upon in making purchasing decisions. The development, release, and Dming of any features or funcdonality described for Oracle s products remains at the sole discredon of Oracle. 3

SpaDal and Graph Technology Graph for Analysis

large scale integradon and reasoning Internet of

Understanding behavior and performance as it

4 SpaDal and Graph Technology Graph for Analysis of: Social Media reladonships Linked data for large scale integradon and reasoning Internet of Things interacdons SpaDal for Analysis of: Understanding behavior and performance as it relates to locadon Data Enrichment Proximity and containment analysis

5 Graph and SpaDal Analysis It is about reladonships Are things in the same locadon? Who is the nearest? What tax zone is this in? Where can deliver in 35 minutes? What is in my sales territory? Is this built in a flood zone? Which supplier am I most dependent upon? Who is the most influendal customer? Do my products appeal to certain communides? What paaerns are there in fraudulent behavior?

6 New Data, New Plaborms 6

7 The DataficaDon Of Every thing Thoughts Things Activities 7

8 New Technologies Enable New Architectures Big Data Highly Parallel and In- Memory Database Architectures Hadoop NoSQL Database Models Key Value, Columnar, Graph Stores Fast Data Stream Processing In-memory clusters Apache Spark Real time Discovery Real time Analytics Hybrid in-memory search / analytic engine

9 Cloud Architecture ConnecDng a System of Systems 9

10 The Internet of Things 10

11 And changes possibilides (and expectadons) Thing + = + Technology SW/HW/NW What the thing always did Value Added Service 11

12 New Offerings for SpaDal and Graph

13 Oracle Big Data SpaDal And Graph 13

14 Database or Big Data technologies Typical technical decision criteria Straight Through Processing (STP) Tooling maturity Stringent Non-FuncDonals IngesDon rate Cost effecdvely store low value data ACID transacdonal requirement Security Hadoop RelaDonal ETL simplicity Data sparsity Variety of data formats

reladonships not aggregadon Big Data plaborm is economically

15 Big Data SpaDal and Graph Big Data onen needs an Organizing principle Data HarmonizaDon Big Data analysis is onen about reladonships not aggregadon Big Data plaborm is economically compelling for working with massive data sets found in spadal and graph workflows

16 Oracle Big Data SpaDal and Graph (BDSG) SpaDal Analysis Features Property Graph Database 16

17 Who is most important? There Are Lots of Answers. Answers from Aggrega8on Who spends the most? Who buys the highest margin goods? Who is most consistently a top contributor? Tabular ques8ons: Well-suited to SQL-like tools Answers from Connec8vity Who s most influendal? Which supplier do I depend on the most? What is the right product mix for millennials? Graph ques8ons: We need something different!

18 Big Data Graph Architecture Lightning-Fast In-Memory AnalyDcs YARN Container Standalone Server Embedded In-Memory Analyst Massively Scalable Graph Store Oracle NoSQL HBase

19 MulDple Interfaces for Many Kinds of Users In-Memory Analyst Graph Database

20 Scales to Trillions of Edges Leverage HBase or Oracle NoSQL for massive storage scalability Use underlying engine for node/edge access and graph filtering Server 1 Server 2 Server 3

21 In-Memory Analyst 1. Analyst engine runs on one or more servers 2. Users request loads subgraph of interest into engine 3. Users call simple funcdons for complex analyses 4. Write results back or share on-the-fly Analyst A Graph Database Analyst B

22 35+ Graph FuncDons DetecDng Components and CommuniDes Tarjan s, Kosaraju s, Weakly Connected Components, Label PropagaDon (w/ variants), Soman and Narang s Ranking and Walking Pagerank, Personalized Pagerank, Betweenness Centrality (w/ variants), Closeness Centrality, Degree Centrality, Eigenvector Centrality, HITS, Random walking and sampling (w/ variants) EvaluaDng Community Structures Conductance, Modularity Clustering Coefficient (Triangle CounDng) Adamic-Adar Path-Finding Hop-Distance (BFS) Dijkstra s, Bi-direcDonal Dijkstra s Bellman-Ford s Link PredicDon SALSA (Twiaer s Who-to-follow) Other Classics Vertex Cover Minimum Spanning-Tree(Prim s)

23 Degree Centrality in Big Data Graph Code heroinfluence = analyst.indegreecentrality()

24 DifferenDators -- Graph Commercial, supported sonware Best of Both Worlds Graph DB In-memory graph algorithms execudon Like Neo4J Distributed graph database model Like Datastax Titan Dozens of pre-built parallel in-memory graph algorithms 10-50x faster analydcs than compedtors offerings Analyze Billion edge graph in memory on a single BDA node

25 Oracle Big Data SpaDal and Graph (BDSG) SpaDal Analysis Features Property Graph Database 25

ExisDng Hadoop-based SpaDal technologies are

26 MoDvaDon for Oracle Big Data SpaDal Emergence of Hadoop for spadal analysis in business and spadal workflows ExisDng Hadoop-based SpaDal technologies are GeoSpaDal-centric not ApplicaDon-centric No significant commercial offerings

27 What problems can Big Data SpaDal analysis address? Data HarmonizaDon using any locadon aaribute (address, postal code, lat/ long, placename, etc). CategorizaDon and filtering based on locadon and proximity PreparaDon, validadon and cleansing of SpaDal and Raster data Visualizing and displaying results on a map

North, just reached border between Marin County and San Francisco County GPS Sensor: N 37

28 Data HarmonizaDon: Linking informadon by locadon Are these data points related? Tweet: sailing by #goldengate Instagram image subdtle: * Text message: Driving on 101 North, just reached border between Marin County and San Francisco County GPS Sensor: N 37 49ʹ11 W ʹ44 Now find all data points around Golden Gate Bridge... * Golden Gate Bridge (in Korean)

29 What features does Big Data SpaDal have? Data enrichment service API using GeoNames and geometry hierarchy data SpaDal processing of data stored in HDFS. Raster processing operadons: Mosaic and sub-set operadons. GeodeDc and Cartesian data MapReduce roudnes for distance calculadons, PointInPolygon, buffer creadon, CategorizaDon, KMeansClustering, Binning, etc. HTML5 Map VisualizaDon API

30 Store any business data with spadal informadon in HDFS Oracle JGeometry USER- PROVIDED InputFormat/ RecordReader Class LOAD ANY FORMAT ANY BUSINESS DATA HDFS

31 Vector Data Processing API FuncDons Single Geometry Length Area Buffer Simplify Geometry Pairs Range Queries Point in Polygon Touch, Overlap, Intersect, Contains, Any InteracDon Join Queries InteracDons on sets of data E.g.: Find all the dropped cell calls in all coverage areas CategorizaDon and Enrichment Associate a data set with a known geometry or named hierarchy Process all Tweets for a period of Dme and count how many are associated with each city, county, state, etc.

Data CategorizaDon Services Any hierarchical geometry data set for reference Customers choose a set of layers For example, they can select (condnents,

32 Data CategorizaDon Services Any hierarchical geometry data set for reference Customers choose a set of layers For example, they can select (condnents, countries, cides) or (countries, states, coundes) as the hierarchy Big Data SpaDal map-reduce job processes the customer data and produces a result file

33 SpaDal Index for SpaDal Queries Spa8al Index stored as a MapFile MapReduce Job with Index Copy the index to distributed cache Mapper reads the index data for the corresponding HDFS block Data Blocks Data Blocks HDFS Data Blocks Data Blocks Process only those records that return hits from the index search 33

34 Big Data SpaDal and Graph SpaDal Vector Processing Framework Mapper and Reducer Classes Customer Applica8on MapReduce Framework, templates Sample Console Applica8on Spa8al Operators, Func8ons S&G Java API Spa8al Enrichment, Categoriza8on API Customer data Generated data GeoJSON, JGeometry format RecordReader class GeoNames and Hierarchy data Oracle Provided Customer code Spa8al Content (any format) HDFS Enrichments, Categoriza8ons results

35 Image Server HDFS storage for the image or raster files We can support dozens of file formats (GDAL supported formats) Images are geo-referenced Images can be in different coordinate systems and resoludons Three main capabilides Loader to load raster data from NFS to HDFS Mosaic and subset operadons based on a virtual mosaic Image processing framework for raster analysis

36 Big Data SpaDal and Graph SpaDal Raster Processing Framework Raster Analysis Applica8on Sample Console Applica8on Raster MapReduce framework S&G Java API Analysis Algorithms Customer data Subset Mosaic Raster Analysis API Generated data Oracle Provided Customer code Raster Data Files on NFS HDFS Raster catalog Derived Rasters GDAL Loader (GDAL formats) Rasters

37 DifferenDators -- SpaDal Commercial, supported sonware ApplicaDon-centric approach vs. GIS-centric Works on any data that includes locadon info, datatype and file format GeoEnrichment services including global geographic hierarchy Supports both spadal processing and spadal analydcs Very rich set (~50) of spadal operators and funcdons Both Vector and Raster services Includes map visualizadon

38 Oracle ConfidenDal 38

Automated Database Cloud Service Full Instance EdiDons Enterprise EdiDon

EncrypDon (TDE) Data Guard All standard EE features Standard EdiDon Full

Compression Advanced Security, Label Security, Database Vault Real

ApplicaDon Clusters (RAC) In Memory AcDve Data Guard Reference: hap://www.

39 Automated Database Cloud Service Full Instance EdiDons Enterprise EdiDon (EE) High Performance Extreme Performance adds adds adds Transparent Data EncrypDon (TDE) Data Guard All standard EE features Standard EdiDon Full database instance Up to 16 OCPUs MulDtenant ParDDoning Advanced Compression Advanced Security, Label Security, Database Vault Real ApplicaDon TesDng OLAP, AnalyDcs, SpaDal and Graph Management Packs Real ApplicaDon Clusters (RAC) In Memory AcDve Data Guard Reference: hap:// Greater CapabiliDes 39

Appliance Fully automated service for Hadoop and Spark

40 Oracle Big Data Cloud Service Now ProducDon! Fast, reliable and secure service on Oracle Big Data Appliance Fully automated service for Hadoop and Spark Available in scalable units of: 216 cores ¾ TB memory 96 TB disk storage 40

41 Oracle Big Data Cloud Service Big Data Key Features Big Data (Hadoop, Spark) as an Automated Service Cloudera Enterprise Data Hub EdiDon 5.4 Oracle Big Data Connectors Oracle Big Data SpaDal and Graph Oracle Data Integrator with Advanced Big Data OpDon Database Cloud Service integradon (via Connectors) Big Data SQL service add-on [Coming Soon] Unified query across Big Data and Exadata Cloud Services Plaborm for new Big Data Services Big Data Discovery [Coming Soon] Benefits Consistently high performance IntegraDon with Exadata Cloud Service means one fast SQL query on all your data 41

Data SpaDal and Graph Oracle Database SpaDal and

42 Oracle s SpaDal and Graph Strategy Enable Spa8al and Graph use cases on every plavorm Oracle Big Data SpaDal and Graph Oracle Database SpaDal and Graph SpaDal and Graph in Cloud Offerings NoSQL 42

The SpaDal and Graph SIG The SIG promotes interacdon and communicadon that can drive the market for spadal technology and data Members connect and exchange knowledge via online communides and at

43 The SpaDal and Graph SIG The SIG promotes interacdon and communicadon that can drive the market for spadal technology and data Members connect and exchange knowledge via online communides and at annual conferences and events Meet us at the Summit Morning RecepDons Tuesday and Wednesday / 7:45 to 8:30 a.m. / RegistraDon Area Birds of a Feather Session Wednesday / 12 to 1 p.m. / Auditorium Look for SpaDal and Graph SIG table Join us online LinkedIn (search for LinkedIn Oracle SpaDal ) Google+ (search for Google+ Oracle SpaDal ) IOUG SIG (sign up for free membership through OTN SpaDal CommuniDes (search for Oracle SpaDal and Graph Community ) Contact the Board oraclespadalsig@gmail.com

Verarbeitung von Vektor- und Rasterdaten auf der Hadoop Plattform DOAG Spatial and Geodata Day 2016

Verarbeitung von Vektor- und Rasterdaten auf der Hadoop Plattform DOAG Spatial and Geodata Day 2016 Hans Viehmann Product Manager EMEA ORACLE Corporation 12. Mai 2016 Safe Harbor Statement The following