Graph and Timeseries Databases

Transcription

1 Graph and Timeseries Databases Roman Kern ISDS, TU Graz Roman Kern (ISDS, TU Graz) Dbase / 31

2 Graph Databases Graph Databases Motivation and Basics of Graph Databases? Roman Kern (ISDS, TU Graz) Dbase / 31

3 Graph Databases Introduction - Graph Databases What is a graph database? Datastorage optimised for graph data structure Ie efficient storage and access Scales gracefully with the amount of data Additional index for look-ups Roman Kern (ISDS, TU Graz) Dbase / 31

4 Graph Databases Introduction - Graph Databases Why should graph databases work? Networks usually have certain properties Small world phenomena even in big networks only a few hops are on average required to reach even distant nodes Access to data follows certain patterns Locality of reference operations are focused on certain areas Roman Kern (ISDS, TU Graz) Dbase / 31

5 Graph Databases Introduction - Graph Databases Why should one not use a graph database? if all the data is updated at once Eg operation applied on all nodes if the query cannot easily be expressed as a graph traversal operation Eg lot of random access, or aggregate functions Roman Kern (ISDS, TU Graz) Dbase / 31

6 Graph Databases Introduction Graph database vs relational database In principle a graph database can be implemented via a relational database using joins or consecutive queries But, relational databases are not optimised for such graph models ie lot of sparse (semi-empty) rows Additionally, relational databases are not designed for changes in the schema, eg dynamic types of relations Roman Kern (ISDS, TU Graz) Dbase / 31

7 Graph Databases Introduction Graph database vs relational database Figure: Comparison of import time for 2 graph databases vs a relational DB Roman Kern (ISDS, TU Graz) Dbase / 31

8 Graph Databases Introduction - Graph Databases Main concepts Many contemporary graph databases are based on property graphs Ie each node and edge are associated with a set of key/values where edges are directed (and often carry a label) Often support ACID properties Ie each modification takes place within a transaction Roman Kern (ISDS, TU Graz) Dbase / 31

9 Graph Databases Introduction - Graph Databases Query types for graph databases Lookup of nodes Traversal of a graph Start at a node continue following edges until a stopping criteria has been reached Breadth-first vs depth-first Path finding Find a path between two nodes (eg Dijkstra, A*) Path matching Matching patterns in graph Roman Kern (ISDS, TU Graz) Dbase / 31

10 Modelling of Graph Databases Modelling of Graph Databases How to represent the data and how to model Roman Kern (ISDS, TU Graz) Dbase / 31

11 Modelling of Graph Databases Modelling of Graph Databases Approach How should a graph database schema look like? Ie how is the data represented as nodes and edges Many different ways of modelling Graphs are a very flexible data structure capable of capturing many domains models In many cases a direct mapping is possible Domain model and graph model Need to review the model Validate that the graph is suited for the queries being used Eg don t mix entities with relations Roman Kern (ISDS, TU Graz) Dbase / 31

12 Modelling of Graph Databases Modelling of Graph Databases How to model for graph databases Roman Kern (ISDS, TU Graz) Dbase / 31

13 Application of Graph Databases Application Practical aspects of graph databases? Roman Kern (ISDS, TU Graz) Dbase / 31

14 Application of Graph Databases Application of Graph Databases Main software tools for graph databases Neo4j OrientDB TitanDB and many others Roman Kern (ISDS, TU Graz) Dbase / 31

15 Application of Graph Databases Application of Graph Databases Panama Papers - Intro Leaked documents from a firm in Panama (26TB of data) about offshore activities Journalist (around the world) were working on analysing the data a graph database was the back-end of this activities Neo4j (plus Apache Solr and Tika) Roman Kern (ISDS, TU Graz) Dbase / 31

16 Application of Graph Databases Application of Graph Databases Panama Papers - Steps Populate the database Analyse the documents eg entity extraction (detect names) entities in the graph Entity types: company, officer, client, address Extract meta-data of documents properties for nodes Detect relationships eg using the connection of sender/receiver of s Refinement of graph Manual work conducted by the journalists More entity types, eg money flow, document types Roman Kern (ISDS, TU Graz) Dbase / 31

17 Application of Graph Databases Application of Graph Databases Roman Kern (ISDS, TU Graz) Dbase / 31

18 Time Series Databases Time Series Databases Motivation and Basics of Time Series Databases? Roman Kern (ISDS, TU Graz) Dbase / 31

19 Time Series Databases Introduction Time Series Databases What is a time series database? Data storage optimised for temporal data Endless stream of incoming data Often accompanied by Tools to acquire time series data Tools to visualise and analyse such data Roman Kern (ISDS, TU Graz) Dbase / 31

20 Time Series Databases Introduction Time Series Databases Typical characteristics of time series databases Fast write/append operations Slow update/delete operations Scales well to huge amount of data Retention policy, ie to forget old data Access restrictions Provide (SQL-like) query languages Optimised for time range queries Specialised queries for aggregates Often rely on other storage mechanism eg key-value store, wide-column storage Roman Kern (ISDS, TU Graz) Dbase / 31

21 Time Series Databases Introduction Time Series Databases Sources of time series Observations, eg weather data, CO2 Stock exchange data Sensor data Log files Roman Kern (ISDS, TU Graz) Dbase / 31

22 Time Series Databases Introduction Time Series Databases Types of time series databases Limited type of payload Eg limited to just timestamp + number least amount of memory needed Flexible payload Allows for richer representation Eg timestamp + document Wide-tables Each row consists of many columns often hundreds of columns sparse rows Roman Kern (ISDS, TU Graz) Dbase / 31

23 Modelling of Time Series Databases Modelling of Time Series Databases How to represent the data and how to model Roman Kern (ISDS, TU Graz) Dbase / 31

24 Modelling of Time Series Databases Modelling of Time Series Databases Approach Often based on single samples (observations) Single vs multiple Eg sensor readouts of multiple sensors (temperature, air pressure) Example: Mesaurement consists of Timestamp, metric name, value, list of filters Eg 10:32, cpu-usage, 087, host=examplecom, cpu=01 Flat file Generic vs specific Store the name of the time series with each observation (generic) Needed in case of dynamic systems eg different sensors become available or disappear Have dedicated time series (specific) Roman Kern (ISDS, TU Graz) Dbase / 31

25 Modelling of Time Series Databases Modelling of Time Series Databases Approach Windowed storage Each row represent a time window Columns for a more fine grained resolution Typically between 100 and 1000 observations per row Alternatively, multiple observations are stored in a single columns Using a custom (compressed) format Special case: temporal and spatial data Requires specialised look-up methods Roman Kern (ISDS, TU Graz) Dbase / 31

26 Time Series Databases Example Example for Time Series Databases Practical Aspects of Time Series Databases? Roman Kern (ISDS, TU Graz) Dbase / 31

27 Time Series Databases Example Time Series Databases Example TICK Stack Collection of tools: Telegraf: server agent for collecting and reporting metrics (stream or batch processing) to write data into the DB InfluxDB: the time series database component Chronograf: Graphing and visualisation frontend for exploration Kapacitor: Data processing engine, can process stream and batch data https: //wwwinfluxdatacom/wp-content/themes/influx/images/tick-stackpng Roman Kern (ISDS, TU Graz) Dbase / 31

28 Time Series Databases Example Time Series Databases Example InfluxDB Features Tags Tags are indexed store commonly-queried meta data if GROUP BY should be used on the data Fields Fields are not indexed Everything that should not be stored as string If aggregation functions should be used on the data (COUNT, MAX, PERCENTILE, CUMSUM) Roman Kern (ISDS, TU Graz) Dbase / 31

29 Time Series Databases Example Time Series Databases Example Figure: Screenshot of example data stored in InfluxDB Roman Kern (ISDS, TU Graz) Dbase / 31

30 Time Series Databases Example The End Next: Map/Reduce Roman Kern (ISDS, TU Graz) Dbase / 31