Motivation Overview of NoSQL space Comparing technologies used Getting hands dirty tutorial section

Transcription

1 NOSQL 1

2 GOAL Motivation Overview of NoSQL space Comparing technologies used Getting hands dirty tutorial section 2

3 MOTIVATION Assume we have a product that becomes popular 3. 1

4 TYPICAL WEBSERVER ARCHITECTURE 3. 2

5 TYPICAL WEBSERVER ARCHITECTURE 3. 3

6 TYPICAL WEBSERVER ARCHITECTURE 3. 4

7 MOTIVATION We are storing more data now than ever before Connections between our data grows We might not know the structure from the beginning 3. 5

8 MOTIVATION Increasing use in industry Big Data Technologies include some of these Disclaimer: There are > 200 different NoSQL databases available, noone is an expert in all of them! Source:

9 MOTIVATION

10 Source:

11 HISTORY OF NOSQL MOVEMENT 4. 1

12 NOSQL TERM NoSQL was used by Carlo Strozzi in 1998 to name his lightweight, Strozzi NoSQL open-source relational database Did not expose SQL interface Still relational Not NoSQL as we know today 4. 2

13 NOSQL TERM Johan Oskarsson, a developer at Last.fm, reintroduced the term NoSQL in early 2009 He organized an event to discuss "open source distributed, non relational databases". Needed a short Twitter Hashtag - used #nosql 4. 3

14 TYPES OF NOSQL DATABASES 4. 4

15 NOSQL TERM The name attempted to label the emergence of an increasing number of non-relational, distributed data stores. Most of the early NoSQL systems did not attempt to provide ACID guarantees, contrary to the prevailing practice among relational database systems. Strozzi suggests, because the current NoSQL movement "departs from the relational model altogether, it should therefore have been called more appropriately 'NoREL' referring to 'No Relational'. 4. 5

16 NOSQL TERM The term NoSQL has absolutely natural origin and has no universally accepted de nition or scienti c institution behind. This title is rather characterized by the vector of development of IT away from relational databases. 4. 6

17 NOSQL TERM De nition of NoSQL: No de nition - but characteristics non-relational Open source Cluster friendly Spawned by 21st Century Website culture Schema-less Most have these characteristics 4. 7

18 NOTABLE MILESTONES Graph database Neo4j is started in Memcached is started in 2003 to power Livejournal. Memcached isn t really a database since it s memory-only but there is soon a version with le storage called memcachedb. CouchDB is started in 2005 a document database. The project moves to the Apache Foundation in Google BigTable is started in 2004, research paper is released in Research paper on Amazon Dynamo is released in

19 NOTABLE MILESTONES Document database MongoDB is started in 2007 as a part of a open source cloud computing stack and rst standalone release in Facebooks open sources the Cassandra project in 2008 Redis is persistent key-value store started in 2009 Riak Another dynamo-inspired database started in

20 NOSQL AND CONSISTENCY 5. 1

21 RDBMS ACID Single unit of information split across tables Don t wan t to write half of the data Must have atomic updates. 5. 2

22 NOSQL BASE Graph db ACID. Before BASE - we should understand CAP 5. 3

23 CAP THEOREM Brewer s CAP theorem states that, in a Distributed Computer System, we can guarantee only two of the following simultaneously: Consistency (all nodes see the same data at the same time) Availability (a guarantee that every request receives a response about whether it was successful or failed) Partition tolerance (the system continues to operate despite arbitrary message loss or failure of part of the system) 5. 4

24 CAP THEOREM 5. 5

25 CAP THEOREM CA data should be consistent between all nodes. As long as all nodes are online, users can read/write from any node and be sure that the data is the same on all nodes. CP data is consistent between all nodes and maintains partition tolerance by becoming unavailable when a node goes down. AP nodes remain online even if they can t communicate with each other and will re-sync data once the partition is resolved, but you aren t guaranteed that all nodes will have the same data (either during or after the partition) 5. 6

26 CAP THEOREM In distributed systems: Partition Tolerance is mandatory, You cannot not choose it. You can trade between consistency and availability. Really: trading Consistency and response times 5. 7

27 CAP AND RDBMS As we have ACID in RDBMS, we have chosen Consistency and Availability Hard to distribute Hard to scale! 5. 8

28 BASE A BASE system gives up on consistency so as to have greater Availability and Partition tolerance. De ned as following: 5. 9

29 CONSISTENCY Consistency: Logical or Replication Sharding data - each piece only exist on one node of cluster. Replication: Clusters have replica of data

30 CONSISTENCE VS AVAILABILITY It is a choice, that should be taken knowing the business model. Can be handled by domain choice 5. 11

31 CONSISTENCE VS AVAILABILITY Hotel Booking example: Can we allow consistency issues, providing high availability? Does hotel have spare rooms for emergencies if booking systems allows inconsistencies? 5. 12

32 CONSISTENCE VS AVAILABILITY Amazon Webshop: As long as you can continue shopping and put things into your shopping basked, all is well. For Amazon/DynamoDB: Make sure that the shopping basked is never down. Can live with consistency issues - by apologizing

33 REAL WORLD TRANSACTIONS 5. 14

34 REAL WORLD TRANSACTIONS 5. 15

35 REAL WORLD TRANSACTIONS 5. 16

36 REAL WORLD TRANSACTIONS 5. 17

37 REAL WORLD TRANSACTIONS 5. 18

38 NEED FOR TRANSACTIONS Aggregate-oriented databases don t need transactions as much. Transaction boundaries should guide in the modelling. Keep transaction within single aggregate. An update for a single document is transactional

39 MODELLING Order has line-items. But we think of it as one order! 5. 20

40 MODELLING In RDBMS we have to split this all over the db. In NoSQL, we keep it as one item Database know the aggregate boundaries (aggregate is the order). Aggregate ids can easier be split over clusters 5. 21

41 MODELLING Can also have issues: If we need information by a product, not by the order easy in RDBMS, harder in NoSQL Advantage if you always use aggregate (the order) - disadvantage if you query in many different ways. For this Store redundant so lookups can be ef cient. But remember updates must happen multiple places

42 NO SCHEMA Preaches how easy it is to have no schema, but in reality, this is not correct. You often assume implicit schema (price, quantity)

43 TYPES OF NOSQL DATABASES First a short introduction to each of the 4 broad chunks of NoSQL Databases 6. 1

44 KEY-VALUE STORES The main idea here is using a hash table where there is a unique key and a pointer to a particular item of data. The key-value model is the simplest and easiest to implement. But it is inef cient when you are only interested in querying or updating part of a value, among other disadvantages. DB does not know what the value is (image, boolean, string etc.) Think of HashMap 6. 2

45 DOCUMENT STORES The model is basically versioned documents that are collections of other key-value collections. The semi-structured documents are stored in formats like JSON. Document databases are essentially the next level of key-value, allowing nested values associated with each key. Document databases support querying more ef ciently. 6. 3

46 COLUMN BASED These were created to store and process very large amounts of data distributed over many machines. There are still keys but they point to multiple columns. The columns are arranged by column family. 6. 4

47 GRAPH DB Instead of tables of rows and columns and the rigid structure of SQL, a exible graph model is used which, again, can scale across multiple machines. Nodes and arches. Great for querying relationships Graph oriented queries that would be hard to write in SQL. Breaks data into even smaller parts 6. 5

48 KEY-VALUE AND DOCUMENT Fuzzy between key-value and document: Some key-value databases allows storage of metadata, and document dbs often have keys, that you can lookup by. In general: You take some data (big complex structure) and save it 6. 6

49 KEY-VALUE STORES 7. 1

50 KEY-VALUE STORES The most exible type of NoSQL database. In a key-value store, there is no schema and the value of the data is opaque. Values are identi ed and accessed via a key, and stored values can be numbers, strings, counters, JSON, XML, HTML, binaries, images, short videos, and more. It is the most exible NoSQL model because the application has complete control over what is stored in the value. 7. 2

51 KEY-VALUE STORES 7. 3

52 EXAMPLE TECHNOLOGIES 7. 4

53 TYPICAL BENEFITS Flexible data modeling No structure on data High performance no need to perform lock, join, union, or other operations when working with objects Think hashing for lookup 7. 5

54 TYPICAL BENEFITS Massive scalability Easy to add new node to cluster High availability Replicate data to n-nodes Operational simplicity As data is on multiple nodes, any failure should be easy to recover from 7. 6

55 TYPICAL USECASES Key-value stores handle size well and are good at processing a constant stream of read/write operations with low latency making them perfect for: 7. 7

56 EXAMPLE CODE DynamoDB dynamodb = new DynamoDB(client) Table table = dynamodb.gettable(tablename); GetItemSpec spec = new GetItemSpec().withPrimaryKey('id', 42) Item outcome = table.getitem(spec) 7. 8

57 DOCUMENT STORES 8. 1

58 DOCUMENT STORE The central concept of a document-oriented database is the notion of a document. While each document-oriented database implementation differs on the details of this de nition, in general, they all assume documents encapsulate and encode data (or information) in some standard format or encoding. Encodings in use include XML, YAML, JSON, and BSON, as well as binary forms like PDF and Microsoft Of ce documents (MS Word, Excel, and so on). Documents in a document store are roughly equivalent to the programming concept of an object. 8. 2

59 EXAMPLE TECHNOLOGIES 8. 3

60 KEYS Documents are addressed in the database via a unique key that represents that document. This key is a simple identi er (or ID), typically a string, a URI, or a path. 8. 4

61 EXAMPLE DOCUMENT 1 { } "FirstName": "Bob", "Address": "5 Oak St.", "Hobby": "sailing" 8. 5

62 EXAMPLE DOCUMENT 2 <contact> <firstname>bob</firstname> <lastname>smith</lastname> <phone type="cell">(123) </phone> <phone type="work">(890) </phone> <address> <type>home</type> <street1>123 Back St.</street1> <city>boys</city> <state>ar</state> <zip>32225</zip> <country>us</country> </address> </contact> 8. 6

63 MONGODB EXAMPLE INSERTION Send JSON to database { } "_id": ObjectID(), "name": "Jacob", "profession": "Nerd" 8. 7

64 MONGODB QUERYING (JAVA) MongoDatabase db = client().getdatabase(databasename) db.getcollection(collectionname).find(new Document("age", 42)) 8. 8

65 MONGODB QUERYING (JSON) Querying in a set { } Name: { $in: [ "Lugia", "Charmander" ] } 8. 9

66 MONGODB QUERYING (JSON) { Type1: "Fire", Legendary: "True" } Querying with AND 8. 10

67 MONGODB QUERYING (JSON) Querying with OR { $or: [{Type1: "Fire"}, {Legendary: "True"}] } 8. 11

68 SEARCH ENGINES Some search engines (aka information retrieval) systems like Elasticsearch provide enough of the core operations on documents to t the de nition of a document-oriented database

69 DATAMODELLING BASICS 9. 1

70 DESIGN TIP You should be thinking by default on how you re going to be reading the data you re about to store in the database. Statistically speaking, the average user is going to spend more time reading data from the database than writing it, so you ll want to make sure reading the data is as painless and uses as little resources as possible. READ > WRITE 9. 2

71 RUNNING EXAMPLE Lets imagine we shall make an event management app 9. 3

72 FIRST ATTEMPT { } users: { "ebrattjorge": { "first_name": "Jorge", "last_name": "Vergara", " ": "j@javebratt.com" } }, events: { "natlpizzaday": { "eventname": "National Pizza Day Celebration", "eventdate": "03/21/2017", } } 9. 4

73 ISSUES How do we relate the event attendees with the event? Let s say that you need to list everyone who went to the "National Pizza Day Celebration" 9. 5

74 IDEA events: { "natlpizzaday": { "eventname": "National Pizza Day Celebration", "eventdate": "03/21/2017", "guestlist": { "ebrattjorge": true } } } 9. 6

75 ISSUES What if we need to displaymore information on the users? How do we avoid that extra query? 9. 7

76 DE-NORMALIZATION You all know the Relational Background about normalization (avoid duplicates, structure tables, etc). NoSQL has a different ow, and it s DE-Normalization, where it is actually OK (in fact, a good practice) to duplicate data. How much data do I duplicate? 9. 8

77 DE-NORMALIZATION You have an event view, where you re seeing the information about that event, inside the view, you have a guest list, that shows just that, a list of the guests who attended the event. You re probably not going to need much information about the guests at that point, so instead of doing an extra query, you could do something like this: 9. 9

78 DE-NORMALIZATION (1) events: { "natlpizzaday": { "eventname": "National Pizza Day Celebration", "eventdate": "03/21/2017", "guestlist": { "ebrattjorge": { "first_name": "Jorge" "last_name": "Vergara" } } } } 9. 10

79 DE-NORMALIZATION (2) users: { "ebrattjorge": { "first_name": "Jorge", "last_name": "Vergara", " ": "j@javebratt.com", "events": { "natlpizzaday": { "name": "National Pizza Day", } } } } 9. 11

80 KEEPING YOUR DATA FLAT Imagine only needing the event name and pulling instead the name + a list of 5,000 people that attended to the event (and National Pizza Day? More like 5,000 million people) From SQL, imagine this as a "Many to Many" relationship, where you create an extra table to store the relationship

81 EVENTS EXAMPLE For example, let s go back to our event/user relationship, what if instead of storing the guestlist inside the event node we create a separate table for it? 9. 13

82 EVENTS EXAMPLE events: { "natlpizzaday": { "eventname": "National Pizza Day Celebration", "eventdate": "03/21/2017" } }, eventguestlist: { "natlpizzaday": { "ebrattjorge": { "first_name": "Jorge" "last_name": "Vergara" } } } 9. 14

83 EVENT DETAIL PAGE Do 2 lookups 1. The event detail 2. The event guestlist 9. 15

84 FINAL STRUCTURE events: { "natlpizzaday": { "eventname": "National Pizza Day Celebration", "eventdate": "03/21/2017" } }, eventguestlist: { "natlpizzaday": { "ebrattjorge": { "first_name": "Jorge" "last_name": "Vergara" } } }, users: { "ebrattjorge": { "first_name": "Jorge", "last_name": "Vergara", " ": "j@javebratt.com", } } 9. 16

85 SMALL ISSUE (OR NOT SO SMALL) What happens if you get a name wrong? What if you need to update someone s name? If you go into users/ebrattjorge/ and change " rst_name": "Andres" it won t affect anything other than that node, meaning you ll end up with something like this: 9. 17

86 SMALL ISSUE (OR NOT SO SMALL) eventguestlist: { "natlpizzaday": { "ebrattjorge": { "first_name": "Jorge" } } }, users: { "ebrattjorge": { "first_name": "Andres", } } 9. 18

87 QUESTIONS 10