Hands-on immersion on Big Data tools

Transcription

1 Hands-on immersion on Big Data tools NoSQL Databases Donato Summa THE CONTRACTOR IS ACTING UNDER A FRAMEWORK CONTRACT CONCLUDED WITH THE COMMISSION

2 Summary : Definition Main features NoSQL DBs classification Document store DBs Key-values DBs Column oriented DBs Graph DBs Conclusions 2

3 Summary : Definition Main features NoSQL DBs classification Document store DBs Key-values DBs Column oriented DBs Graph DBs Conclusions 3

4 NoSQL definition NoSQL databases is an approach to data management that is useful for very large sets of distributed data A NoSQL database provides a mechanism for storage and retrieval of data which is modeled in means other than the tabular relations used in relational databases. Actually NoSQL should be called NoRel because the approach does not prohibit SQL In fact NoSQL means NotOnlySQL 4

5 NoSQL DBs Such DBs have existed since the late 1960s but they became popular only in the last 20 years triggered by the needs of : Web 2.0 companies (Facebook, Google, Amazon) Cloud computing Mobile applications NoSQL databases are increasingly used in big data and real-time web applications due to their flexibility and scalability features 5

6 Summary : Definition Main features NoSQL DBs classification Document store DBs Key-values DBs Column oriented DBs Graph DBs Conclusions 6

7 NoSQL: Main Features We will discuss the NoSQL main features by making a comparison with the corresponding Relational DBs ones 7

8 Key characteristics comparison RDBMS NoSQL Structured data (schema) Tuple orientation Atomic transactions Scale UP Impedence mismatch SQL Semi structured/unstructured data (schemaless) Aggregate orientation Eventual consistency Scale OUT Program data organization reflection API, SQL 8

9 Key characteristics comparison RDBMS Structured data (schema) Tuple orientation Atomic transactions Scale UP Impedence mismatch NoSQL Semi structured/unstructured data (schemaless) Aggregate orientation Eventual consistency Scale OUT Program data organization reflection SQL API, SQL 9

10 Schemaless DB A database schema is the definition that describes the entire configuration of the database, its structure, including all of its tables, relations, index, constraints, etc. Specific rigid rules to follow It has various advantages but you have to know it exactly and in advance 10

11 Schemaless DB NoSQL databases are schemaless: A key-value store allows you to store any data you like under a key A document database effectively does the same thing, since it makes no restrictions on the structure of the documents you store Column-family databases allow you to store any data under any column you like Graph databases allow you to freely add new edges and freely add properties to nodes and edges as you wish 11

12 Schemaless DB This has various advantages: Without a schema binding you, you can easily store whatever you need, and change your data storage as you learn more about your project You can easily add new things as you discover them A schemaless store also makes it easier to deal with nonuniform data: data where each record has a different set of fields (limiting sparse data storage) 12

13 Schemaless DB But also some problems Indeed, whenever we write a program that accesses data, that program almost always relies on some form of implicit schema: it will assume that certain field names are present and carry data with a certain meaning, and assume something about the type of data stored within that field Having the implicit schema in the application means that in order to understand what data is present you have to dig into the application code 13

14 Schemaless DB Furthermore, the database remains ignorant of the schema: it cannot use the schema to support the decision on how to store and retrieve data efficiently Also, it cannot impose integrity constraints to maintain information coherent 14

15 Key characteristics comparison RDBMS NoSQL Structured data (schema) Tuple orientation Atomic transactions Scale UP Impedence mismatch SQL Semi structured/unstructured data (schemaless) Aggregate orientation Eventual consistency Scale OUT Program data organization reflection API, SQL 15

16 Aggregate orientation We will talk later about NoSQL classification but it is important to notice a common characteristic Key-values Document store Column Oriented Graph databases The first three share a common characteristic of their data models which we will call aggregate orientation. 16

17 NoSQL: Aggregate An aggregate is a collection of related objects that we wish to treat as a unit The relational model divides the information that we want to store into tuples (rows): this is a very simple structure for data Aggregate orientation takes a different approach. It recognizes that often you want to operate on data in units that have a more complex structure 17

18 NoSQL: Aggregate It can be handy to think in terms of a complex record that allows lists and other record structures to be nested inside it As we will see, key-value, document, and column-family databases all make use of this more complex record However, there is no common term for this complex record; we use here the term aggregate 18

19 NoSQL: Aggregate 19

20 Key characteristics comparison RDBMS NoSQL Structured data (schema) Tuple orientation Atomic transactions Scale UP Impedence mismatch SQL Semi structured/unstructured data (schemaless) Aggregate orientation Eventual consistency Scale OUT Program data organization reflection API, SQL 20

21 NoSQL: BASE Consistency Eventually consistent/not ACID Informally guarantees that, if no new updates are made to a given data item, eventually all accesses to that item will return the last updated value BASE (Basically Available, Soft state, Eventual consistency) semantics in contrast to traditional ACID (Atomicity, Consistency, Isolation, Durability) guarantees 21

22 NoSQL: BASE Consistency ACID (Strong Consistency) Atomicity: every transaction is executed in all-ornothing fashion Coherence (or consistency): every transaction preserves the coherence with constraints on data (i.e., at the end of the transaction constraints are satisfied by data) Isolation: transaction does not interfere. Every transaction is executed as it was the only one in the system (every seralization of concurrent transactions is accepted) Durability: after a commit, the updates made are permanent regardless possible failures 22

23 NoSQL: BASE Consistency Where ACID is pessimistic and forces consistency at the end of every operation, BASE is optimistic and accepts that the database consistency will be in a state of flux Basically Available: The availability of BASE is achieved through supporting partial failures without total system failure. Soft state: data are volatile in the sense that their persistence is in the hand of the user that must take care of refresh them Eventual Consistency: the system eventually converge to a consistent state 23

24 NoSQL: BASE Consistency A common claim we hear is that NoSQL databases don t support transactions and thus can t be consistent Any statement about lack of transactions usually only applies to some NoSQL databases, in particular the aggregate-oriented ones, whereas graph databases tend to support ACID transactions 24

25 Key characteristics comparison RDBMS NoSQL Structured data (schema) Tuple orientation Atomic transactions Scale UP Impedence mismatch SQL Semi structured/unstructured data (schemaless) Aggregate orientation Eventual consistency Scale OUT Program data organization reflection API, SQL 25

26 Scaling Apart from some exceptions : - relational DBs are centralized - NoSQL DBs are distributed This basically means and imply that : - the scaling strategy must be different 26

27 Scaling Relational DBs are designed to scale UP If you need more power usually you must get a bigger box (a more powerful machine) Sooner or later you will not be able to go further... 27

28 Scaling NoSQL DBs are designed to scale OUT If you need more power usually you must get more boxes (add machines in a network) You can also obtain an high level of availability at reasonable costs 28

29 Key characteristics comparison RDBMS NoSQL Structured data (schema) Tuple orientation Atomic transactions Scale UP Impedence mismatch SQL Semi structured/unstructured data (schemaless) Aggregate orientation Eventual consistency Scale OUT Program data organization reflection API, SQL 29

30 Impedence mismatch RDBMS store data in tuples and tables This requires further steps of logical data model conversion for programmers that have to build the objects to be used in applications Base types of the data DBMS CHAR, VARCHAR, DATE, TIME, Prog. languages numbers, strings, Type contruction DBMS tables and tuples Prog. languages classes, inheritance, polimorphism 30

31 Impedence mismatch Operation s semanthic DBMS operations on set of tuples Prog. languages operations on single variables Algorithms techniques DBMS joints Prog. languages objects navigation 31

32 Impedence mismatch On the other hand : NoSQL DBs usually store data in a way that typically reflects the way they are used in Object Oriented Programming (classes with fields) You just have to load an aggregate Using NoSQL databases allows developers to develop without having to convert in-memory structures to relational structures 32

33 Key characteristics comparison RDBMS NoSQL Structured data (schema) Tuple orientation Atomic transactions Scale UP Impedence mismatch SQL Semi structured/unstructured data (schemaless) Aggregate orientation Eventual consistency Scale OUT Program data organization reflection API, SQL 33

34 API & SQL It is always better to have 2 chances instead of only 1 From a developer point of view: Dealing with SQL and related matters (stored procedures, persistence frameworks,...) is annoying It would be better to use an API but You have also to consider that some people could be reluctant in changing their way of doing things (they will always prefer SQL because they do not want to learn other stuff) 34

35 Summary : Definition Main features NoSQL DBs classification Document store DBs Key-values DBs Column oriented DBs Graph DBs Conclusions 35

36 NoSQL classification The NoSQL DBs implementations can be categorized on the base of the adopted data model We can classify them as follow: Document store DBs Key-value DBs Column oriented DBs Graph DBs 36

37 Document Store Basically you can store documents in it A document is a file usually in XML or JSON format We got an ID and a range of data representing a document { } id: 123, name: Bill, surname: Gates, color: blue 37

38 Document Store: Strenght point More flexibility when accessing data: for example, you may want a query that retrieves all the documents with a certain field set to a certain value. SELECT * FROM Users WHERE color = blue SQL statement, very specialized language for DB people db.users.find[{color: blue }] Notation very familiar and comfortable for programmers, more object oriented38

39 Document Store: Example (JSON) 39

40 Document Store: Issues You could have a definition of allowable structures and types there could be some limits on what we can place in it Indexing: necessary for speed-up accesses Indexes can be very big Semantics problem still there: Need for semantic extraction techniques 40

41 Document Store: Examples #5 #11 #14 Source = 41

42 Key-Values DBs Key value databases allow applications to store data in a schema-less way The data could be stored in a datatype of a programming language or an object No fixed data model Example Tool: Riak, Redis, Amazon Dynamo DB 42

43 Key-Values DBs Like Document stores DBs, Key value DBs associate a content to an Id (in this case is a key of a map) but In Key value DBs you cannot do any query inside any doc without having the key first so You cannot say something like: find me all the records where the name is Bill 43

44 Key-Values DBs Question: Why are they useful if I cannot make analysis on data? Answer: They are extremely fast to access data Ideal solution for speedy inquiry 44

45 Key-Values DBs: Killer use cases Server side customer history (e.g. browsing history), knowing a customer history you can provide a better user experience Social networks, for example Twitter uses Redis to load and present the user history when you access your page containing all your messages 45

46 Key-Values DBs : Example Ref: Domenico Lembo, Master Course on Big Data Management 46

47 Key-Values DBs : Issues You can store whatever you like in Values It is the responsibility of the application to understand what was stored You can experience a great inefficiency if the vast majority of the use cases act just on a part of an object associated to a key 47

48 Key-Values DBs : Examples #9 #32 #22 Source = 48

49 Column-oriented DBs Column family stores are modeled on Google s BigTable. The data model is based on a sparsely populated table whose rows can contain arbitrary columns The column-family model can be seen as a twolevel aggregate structure 49

50 Column-oriented DBs As with key-value stores, the first key is often described as a row identifier, picking up the aggregate of interest This row aggregate is itself formed of a map of more detailed values. These second-level values are referred to as columns, each being a keyvalue pair Columns can be organized into column families 50

51 Column-oriented DBs : Example Ref: Domenico Lembo, Master Course on Big Data Management 51

52 Column-oriented DBs : Structure Row-oriented Each row is an aggregate (for example, customer with the ID of 1234) of values column families are useful chunks of data (profile, order history) within that aggregate 52

53 Column-oriented DBs : Structure Column-oriented: Each column family defines a record type (e.g., customer profiles) with rows for each of the records. You then think of a row as the join of records in all column families Column Families can be then to some extent considered as tables in RDBMSs (but a Column Family can have different columns for each row it contains) 53

54 Column-oriented DBs : Examples Google Cloud BigTable #175 Apache Cassandra #7 Source = 54

55 Graph DBs A graph database is a database that uses graph structures with nodes, edges, and properties to represent and store data A management systems for graph databases offers Create, Read, Update, and Delete (CRUD) methods to access and manipulate data Graph databases can be used for both OLAP (since are naturally multidimensional structures ) and OLTP 55

56 Graph DBs Systems tailored to OLTP (e.g., Neo4j) are generally optimized for transactional performance, and tend to guarantee ACID properties Stores natural data relationships between data elements to reveal networks like social networks 56

57 Graph DBs : Relationships Obviously, graph databases are particulary suited to model situations in which the information is somehow natively in the form of a graph The real world provide us with a lot of application domains: social networks, recommendation systems, geospatial applications, computer network and data center management, authorization and access control, etc. 57

58 Graph DBs : Relationships The success key of graph databases in these contexts is the fact that they provide native means to represent relationships Relational databases instead lacks relationships: they have to be simulated through the help of foreign keys, thus adding additional development and maintenance overhead, and discover them require costly join operations 58

59 Graph DBs : Querying Querying = traversing the graph, i.e., following paths/relationships Navigational paradigm: online discovery of resources 59

60 Graph DBs vs Relational DBs - Example Modeling friends and friends-of-friends in a relational database Notice that PersonFriend has not to be considered simmetric: Bob may consider Zach as friend, but the converse does not necessarily hold 60

61 Graph DBs vs Relational DBs - Example Asking who are Bob s friends? (i.e., those that Bob considers as friend) is easy SELECT p1.person FROM Person p1 JOIN PersonFriend ON PersonFriend.FriendID = p1.id JOIN Person p2 ON PersonFriend.PersonID = p2.id WHERE p2.person = 'Bob' 61

62 Graph DBs vs Relational DBs - Example Things become more problematic when we ask, who are the Alice s friends-of-friends? SELECT p1.person AS PERSON, p2.person AS FRIEND_OF_FRIEND FROM PersonFriend pf1 JOIN Person p1 ON pf1.personid = p1.id JOIN PersonFriend pf2 ON pf2.personid = pf1.friendid JOIN Person p2 ON pf2.friendid = p2.id WHERE p1.person = 'Alice' AND pf2.friendid <> p1.id Performances highly deteriorate when we go more in depth into the network of friends 62

63 Graph DBs vs Relational DBs - Example Modeling friends and friends-of-friends in a graph database Relationships in a graph naturally form paths. Querying means actually traversing the graph, i.e., following paths. Because of the fundamentally path-oriented nature of the data model, the majority of path-based graph database operations are extremely efficient. 63

64 Graph DBs : Tipical usage Mary Bob Is friend of visited Japan likes pizza Who are the friends of Mary's friends who like the food that Mary's friends like but haven't visited the places that Mary's friends have visited? 64

65 Graph DBs : Killer use cases Does the previous query sound stupid or unrealistic? What about these other two? People who likes this product are also likely to like that product (Amazon context) We think that you are likely to be friends with this person because of your other connections (social media context) Answering questions like these you can discover information in your data ( knowledge is power! ) 65

66 Graph Database: Examples #21 #198 Source = 66

67 Summary : Definition Main features NoSQL DBs classification Document store DBs Key-values DBs Column oriented DBs Graph DBs Conclusions 67

68 Some considerations All of the different DBs saw serve different purposes It is up to you to use the one that best fit your needs You can also store the data in a combination of these as well as SQL DBs We have a polyglot when we use multiple data stores together 68

69 NoSQL DBs pros and cons PROS CONS Often they don t require schema Much less join operations needed (self contained objects) Initial training period (API) No universal standard like SQL (changing DB could be difficult) Simplicity and flexibility No impedence mismatch (easy object-relational mapping) No controls on data integrity (application responsibility) Maybe not the better choiche for standard needs No size limitations (horizontal scalability) Easier distribution of data (aggregates mean no relations) 69

70 Does it make sense to still use RDBMS? The particular suitability of a given NoSQL database depends on the problem it must solve For traditional requirements the RDBMS solution has proven many times to be a good choice The final decision is yours! 70

71 Thank you for your attention! 71