The Internet of Things:

Transcription

1 The Internet of Things: Sensor Data Management Course website: h8p:// Prof. Luciano Bononi Prof. Marco Di Felice MASTER DEGREE IN COMPUTER SCIENCE DEPARTMENT OF COMPUTER SCIENCE AND ENGINEERING, UNIVERSITY OF BOLOGNA, ITALY

2 2

3 The word big-data is currently used to idensfy: (i) datasources with specific characterisscs, as well as (ii) novel technologies to manage the data. CHARACTERISTICS OF BIG-DATA q Cannot be managed using convensonal technologies of informason systems q Volume à order of Petabytes q Velocity à data produced at high rate q Variety à heterogeneous data (text, image, video, etc) 3

4 ² IoT & Big-data are two sides of the same coin! q Large-scale IoT deployments can produce huge amounts of data q IoT is about data, services, connecsvity: data are gathered by objects, transfered, analyzed, and traslated into services 4

5 ² IoT & Big-data are two sides of the same coin! (DIGITAL) SYSTEM MODEL SENSING DATA FILTERING DATA ANALYSIS MODEL BUILDING REAL SYSTEM Raw sensor data measure how the system is working UNDERSTANDING CorrelaSon among system parameters and between Inputs/outputs PREDICTING System behaviour at Sme t+1 PLANNING What-If analysis 5

6 ² IoT & Big-data are two sides of the same coin! EXAMPLE: TELECOM ITALIA (research project) h8p:// ClassificaRon (i.e. idensfy the appliance) ² 5 smart plugs for house ² Around 200 installasons in private houses ² 1 sampling every 2 minutes RAW SENSOR DATA TRACES Profiling (i.e. idensfy users habbits) PredicRng (i.e. predict energy consumpson) Scheduling (i.e. intelligent ON/OFF schedule) 6

7 ² Time-series à Sequence of Rmestamp plus values 7

8 ² Time-series à Sequence of Rmestamp plus values q Data are immutable. q WriSng in append. q Reading conrguous sequence of samples data. q Highly compressible data. q DeleRng usually across large Sme period q High precision for short period of Sme. q Single value is not so important 8

9 ² RelaRonal Database Management Systems (RDBMS) q Based on the relasonal model first proposed by Edgar F. Codd (1970) q Employ SQL language q Support ACID properses q Scheme-based (structured) database. q Components: Tables (relasons), primary keys, foreign keys, NULL values. 9

10 ² Time-series implemented on RDMBS PROBLEMS ² Scalability (i.e. need to store large amount of Sme-series data) ² Performance (e.g. support for rangebased operasons) ² Aymmetric CRUD operasons 10

11 ² NOSQL Database Management Systems q Set of tools and logic models, alternasve or complementary to RDBMS (norel). q Support BASE properses. q Do not employ the SQL language. q Scheme-less (un/semi-structured) database. q Families: Key-values DB, Document-based DB, Column-based DB, Graph-based DB. 11

12 ² Time-series Database 12

13 ² Time-series Database à Dedicated DBMS opsmized for managing large volumes of Sme-series data q OpSmized data storage and sharding q OperaSonal support (e.g. range-based queries) q Time-granularity management q Time-series analyscs and mining 13

14 ² InfluxDB (hyps:// q Open-source Sme-series database (InfluxData) q Wri8en in GO language q SQL-like query language (InfluxQL) q Command Line Interface (CLI) and HTTP APIs q Support for distributed deployments q IntegraSon with Sme-series tools for data analyscs and visualizason (e.g. Grafana) 14

15 ² InfluxDB (hyps:// q Time-Structured Merge Tree (TSM) à data structure used to contain sorted, compressed series data. q Time Series Index (TSI) à address millions of unique Sme series, regardless of the amount of memory on the server hardware. q AutomaSc downsampling and data rentenson procedures. 15

16 ² InfluxDB (hyps:// q Timestamp à RFC3339 UTC format (yyyy-mm-ddthh:mm:ssz) q Field keys à string metadata, similar to column name q Field values à actual measured data (any type) q Tag-sets à opsmal, extra-informason about the measurements q Tag keys à string meta-data, similar to field keys q Tag values à string values q Measurement à Container to hold the Smestamps, fields and tags (similar to a table in a RDBMS) 16

17 Source: h8ps:// ² InfluxDB (hyps:// q Series à collecson of data-points, containing: ² Measurement ² Tag-sets ² RetenRon policy à period that datapoints are being stored in InfluxDB, which is called DURATION but also the number of versions that should be kept on the cluster, as REPLICATION. 17

18 Source: h8ps:// ² InfluxDB (hyps:// 18

19 Source: h8ps:// ² InfluxDB (hyps:// q Command-Line Interface (CLI) user@mypc:$influx Connected to version InfluxDB shell version: q HTTP-based API curl -i -XPOST --data-urlencode "q=create DATABASE mydb curl -i -XPOST ' --data-binary 'cpu_load_short,host=server01,region=us-west value= ' 19

20 Source: h8ps://docs.influxdata.com/influxdb/v1.5/tools/shell/#write-data-to-influxdb-with-insert ² InfluxDB (hyps:// q Basic Commands ² CREATE DATABASE name ² USE DATABASE name ² SHOW DATABASES ² CLEAR DATABASE name ² CREATE RETENTION POLICY name ON measurement DURATION 1d REPLICATION 1 ² INSERT treasures,captain_id=pirate_king value=2 20

21 Source: h8ps://docs.influxdata.com/influxdb/v1.5/query_language/data_explorason/ ² InfluxDB (hyps:// q InfluxQL language for data exploraron SELECT <field_key>[,<field_key>,<tag_key>] FROM <measurement_name>[,<measurement_name>] SELECT * FROM "h2o_feet" SELECT_clause FROM_clause WHERE <conditional_expression> [(AND OR) <conditional_expression> [...]] SELECT * FROM "h2o_feet" WHERE "water_level" > 8 SELECT_clause FROM_clause [WHERE_clause] GROUP BY [* <tag_key>[,<tag_key]] SELECT MEAN("water_level") FROM "h2o_feet" GROUP BY "location" 21

22 ² Grafana (hyps://grafana.com) 22