Ghislain Fourny. Big Data 2. Lessons learnt from the past

Transcription

1 Ghislain Fourny Big Data 2. Lessons learnt from the past

2 Mr. Databases: Edgar Codd Wikipedia

3 Data Independence (Edgar Codd) Logical data model Lorem Ipsum Dolor sit amet Physical storage Consectetur Adipiscing Elit. In Imperdiet Ipsum ante

4 Data Shapes Lorem ipsum dolor sit amet, consectetur adipiscing elit. Etiam vel erat nec dui aliquet vulputate sed quis nulla. Donec eget ultricies magna, eu dignissim elit. Nullam sed urna nec nisl rhoncus ullamcorper placerat et enim. Integer varius ornare libero quis consequat. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Aenean eu efficitur orci. Aenean ac posuere tellus. Ut id commodo turpis. Praesent nec libero metus. Praesent at turpis placerat, congue ipsum eget, scelerisque justo. Ut volutpat, massa ac lacinia cursus, nisl dui volutpat arcu, quis interdum sapien turpis in tellus. Suspendisse potenti. Vestibulum pharetra justo massa, ac venenatis mi condimentum nec. Proin viverra tortor non orci suscipit rutrum. Phasellus sit amet euismod diam. Nullam convallis nunc sit amet diam suscipit dapibus. Integer porta hendrerit nunc. Quisque pharetra congue porta. Suspendisse vestibulum sed mi in euismod. Etiam a purus suscipit, accumsan nibh vel, posuere ipsum. Nulla nec tempor nibh, id venenatis lectus. Duis lobortis id urna eget tincidunt.

5 Data Shapes Lorem ipsum dolor sit amet, consectetur adipiscing elit. Etiam vel erat nec dui aliquet vulputate sed quis nulla. Donec eget ultricies magna, eu dignissim elit. Nullam sed urna nec nisl rhoncus ullamcorper placerat et enim. Integer varius ornare libero quis consequat. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Aenean eu efficitur orci. Aenean ac posuere tellus. Ut id commodo turpis. Praesent nec libero metus. Praesent at turpis placerat, congue ipsum eget, scelerisque justo. Ut volutpat, massa ac lacinia cursus, nisl dui volutpat arcu, quis interdum sapien turpis in tellus. Suspendisse potenti. Vestibulum pharetra justo massa, ac venenatis mi condimentum nec. Proin viverra tortor non orci suscipit rutrum. Phasellus sit amet euismod diam. Nullam convallis nunc sit amet diam suscipit dapibus. Integer porta hendrerit nunc. Quisque pharetra congue porta. Suspendisse vestibulum sed mi in euismod. Etiam a purus suscipit, accumsan nibh vel, posuere ipsum. Nulla nec tempor nibh, id venenatis lectus. Duis lobortis id urna eget tincidunt.

6 Take-away Concepts

7 Take-away Concepts Table Collection

8 Take-away Concepts Attribute Column Field Property

9 Take-away Concepts Primary Key Row ID Name

10 Take-away Concepts Row Business Object Item Entity Document Record

11 Relational Algebra

12 Function

13 Relation

14 Relation as a list

15 Relation as a list, n-way

16 Relations seen from set theory Given a family of sets A " #$"$% A relation is a subset of their Cartesian product % R ( A ) = A # A, ).# A %

17 Relation as a list, n-way

18 Relational table A B C

19 Relational Algebra x

20 Selection

21 Projection

22 Grouping A B C D

23 Sorting

24 Cartesian Product

25 Joining A B A B A A A B

26 Normal Forms

27 From your Bachelor: Normal Forms Update anomaly Consistency Delete anomaly Insert anomaly

28 1st Normal Form (tabular) The Key

29 1st Normal Form: counter-example Legi Name Lecture City State PLZ Alan Turing Bletchley Park UK MK3 6EB Lecture ID Lecture Name xxx-xxxx-xxx Cryptography L Big Data Georg Cantor Pfäffikon SZ 8808 Lecture ID L Lecture Name Big Data L Set theory Felix Bloch Pfäffikon ZH 8330 Lecture ID Lecture Name L Set theory

30 1st Normal Form Legi Name Lecture ID Lecture Name City State PLZ Alan Turing xxx-xxxx-xxx Cryptography Bletchley Park UK MK3 6EB Alan Turing L Big Data Bletchley Park UK MK3 6EB Georg Cantor L Big Data Pfäffikon SZ Georg Cantor L Set theory Pfäffikon SZ Felix Bloch L Set theory Pfäffikon ZH 8330

31 2nd Normal Form (not joined) The Whole Key

32 2nd Normal Form: Counter-example Legi Name Lecture ID Lecture Name City State PLZ Alan Turing xxx-xxxx-xxx Cryptography Bletchley Park UK MK3 6EB Alan Turing L Big Data Bletchley Park UK MK3 6EB Georg Cantor L Big Data Pfäffikon SZ Georg Cantor L Set theory Pfäffikon SZ Felix Bloch L Set theory Pfäffikon ZH 8330

33 2nd Normal Form: Example Legi Name City State PLZ Alan Turing Bletchley Park UK MK3 6EB Alan Turing Bletchley Park UK MK3 6EB Georg Cantor Pfäffikon SZ Georg Cantor Pfäffikon SZ 8808 Legi Lecture ID xxx-xxxx-xxx L L L L Lecture ID Lecture Name xxx-xxxx-xxx Cryptography L Big Data Felix Bloch Pfäffikon ZH L Set theory

34 3rd Normal Form Nothing But The Key

35 3rd Normal Form: Counter-Example Legi Name City State PLZ Alan Turing Bletchley Park UK MK3 6EB Alan Turing Bletchley Park UK MK3 6EB Georg Cantor Pfäffikon SZ Georg Cantor Pfäffikon SZ Felix Bloch Pfäffikon ZH 8330

36 3rd Normal Form: Example Legi Name City State Alan Turing Bletchley Park UK City State PLZ Bletchley Park UK MK3 6EB Alan Turing Bletchley Park UK Bletchley Park UK MK3 6EB Georg Cantor Pfäffikon SZ Pfäffikon SZ Georg Cantor Pfäffikon SZ Pfäffikon SZ Felix Bloch Pfäffikon ZH Pfäffikon ZH 8330

37 Data Denormalization 3NF 2NF

38 Data Denormalization 3NF 1NF

39 Data Denormalization 3NF 0NF

40 SQL Brush-Up

41 SQL is a declarative language Logical model "What, not how" Physical execution

42 SQL is a declarative language Logical model "What, not how" Query Plan

43 SQL is a declarative language Logical model "What, not how" Parallelism

44 SQL is a functional language "Expressions that nest"

45 Getting a table SELECT * FROM customers

46 Selection SELECT * FROM customers WHERE city = "Zurich"

47 Projection SELECT name, city FROM customers

48 Grouping SELECT city, count(name) FROM customers GROUP BY city

49 Selecting groups SELECT city, count(name) FROM customers GROUP BY city HAVING count(name) > 2

50 Join SELECT cities.country, customers.name FROM customers JOIN cities ON customers.city = cities.name

51 Nesting SELECT country, count(name) FROM SELECT cities.country AS country, customers.name AS name FROM customers JOIN cities ON customers.city = cities.name GROUP BY country HAVING count(name) > 10

52 Some terminology Data Schema DML: Data Manipulation Language (Query, insert, remove rows) DDL: Data Definition Language (Create or table/schema, drop it)

53 Some terminology Create Read Update Delete

54 Transactions

55 The good old times of databases: ACID Atomicity Consistency Isolation Durability

56 Atomicity Either the entire transaction is applied, or none of it (rollback).

57 Consistency After a transaction, the database is in a consistent state again.

58 Isolation A transaction feels like nobody else is writing to the database.

59 Durability Updates made do not disappear again.

60 The new era: the CAP theorem Yet another impossibility triangle

61 The new era: the CAP theorem Consistency Availability Partition tolerance

62 (Atomic) Consistency All nodes see the same data.

63 Availability It is possible to query the database at all times.

64 Partition tolerance The database continues to function even if the network gets partitioned.

65 Performance

66 No such thing as "one size fits all" Data shapes matter

67 Optimize for read vs. write intensive OnLine Transaction Processing OnLine Analytical Processing OLTP OLAP Write-intensive Read-intensive

68 Indices

69 Data Scale-Up

70 Data can have... Lots of rows

71 Data can have... Lots of columns

72 Data can have... Lots of nesting

73 The rest of the lecture: Scaling up Data in the Large Lots of rows Object Storage Lots of rows Distributed File Systems Lots of rows/columns Column storage Lots of rows Massive Parallel Processing Data in the Small Lots of nesting Syntax Lots of nesting Data Models Lots of nesting Querying Lots of nesting Document Stores Data in the Very Small Lots of rows Graph Databases Lots of columns Data Warehousing Lots of columns Business reporting