class 17 updates prof. Stratos Idreos

Transcription

1 class 17 updates prof. Stratos Idreos

2 UPDATE table_name SET column1=value1,column2=value2,... WHERE some_column=some_value INSERT INTO table_name VALUES (value1,value2,value3,...) updates Stratos Idreos 2 /50

3 traditional applications e.g., banking how many times per day do you send update queries to your bank account Stratos Idreos 3 /50

4 the world has changed a little bit by now updates Stratos Idreos 4 /50

5 still we spy Facebook more than the # of photos we upload CS165, Fall or 2017 # of our twitter posts, etc Stratos Idreos 5 /50

6 not just about user data: everything is data! Stratos Idreos 6 /50

7 monitor CPU utilization monitor memory hierarchy utilization monitor clicks (frequency, locations, specific links, sequences) Stratos Idreos 7 /50

8 Three things are important in the database world: performance, performance, and performance Bruce Lindsay, IBM ACM SIGMOD Edgar F. Codd Innovations award 2012 true for both reads & writes Stratos Idreos 8 /50

9 GREAT PERFORMANCE FOR READS AND WRITES & VARIABLE READ/WRITE RATIO Stratos Idreos 9 /50

10 insert new entry (a,b,c,d, ) on table x update N columns, K trees, statistics, table x A B C D A B D Stratos Idreos 10/50

11 TWO CHALLENGES EFFICIENCY & ROBUSTNESS Stratos Idreos 11/50

12 A A.deletes A.inserts Project: 1 query at a time, (id) (id,value) no failures Stratos Idreos 12/50

13 A A.deletes A.inserts Project: 1 query at a time, (id) (id,value) no failures select(a,v1,v2) A pos scan Stratos Idreos 12/50

14 A A.deletes A.inserts Project: 1 query at a time, (id) (id,value) no failures select(a,v1,v2) A pos pos2 A.del diff scan Stratos Idreos 12/50

15 A A.deletes A.inserts Project: 1 query at a time, (id) (id,value) no failures select(a,v1,v2) A pos pos2 diff A.del A.ins ins scan scan Stratos Idreos 12/50

16 A A.deletes A.inserts Project: 1 query at a time, (id) (id,value) no failures select(a,v1,v2) A pos pos2 diff res scan A.del A.ins ins union scan Stratos Idreos 12/50

17 transaction any database query that should be seen as a single task parse optimize plan( read X write Z ) Stratos Idreos 13/50

18 Atomicity Consistency Isolation Durability Jim Gray, IBM, Tandem, DEC, Microsoft ACM Turing award ACM SIGMOD Edgar F. Codd Innovations award Stratos Idreos 14/50

19 all or nothing Atomicity Consistency Isolation Durability correct >1 queries persistent Stratos Idreos 15/50

20 update all rows where A=v1 & B=v2 to (a=a/2,b=b/4,c=c-3,d=d+2) Stratos Idreos 16/50

21 update all rows where A=v1 & B=v2 to (a=a/2,b=b/4,c=c-3,d=d+2) A B C D search (scan/index) to find row to update select+project actions Stratos Idreos 16/50

22 update all rows where A=v1 & B=v2 to (a=a/2,b=b/4,c=c-3,d=d+2) A B C D search (scan/index) to find row to update select+project actions A B C D list of rowids (positions) (sort) Stratos Idreos 16/50

23 update all rows where A=v1 & B=v2 to (a=a/2,b=b/4,c=c-3,d=d+2) A B C D search (scan/index) to find row to update select+project actions A B C D list of rowids (positions) (sort) we know what to update but nothing happened yet Stratos Idreos 16/50

24 update all rows where A=v1 & B=v2 to (a=a/2,b=b/4,c=c-3,d=d+2) CPU level 1 level 2 A B C D read page in L1 update persist to L2 if problem (power/abort) before we write all pages we are left with an inconsistent state Stratos Idreos 17/50

25 recovery classic example joe owes mike 100$ both joe and mike have a Bank of Bla account possible actions joe -100 mike mike joe what if there is a failure here? Stratos Idreos 18/50

26 recovery classic example joe owes mike 100$ both joe and mike have a Bank of Bla account possible actions actually 1)read mike; 2) mike+100; 3) write new mike; joe -100 mike mike joe what if there is a failure here? Stratos Idreos 18/50

27 update all rows where A=v1 & B=v2 to (a=a/2,b=b/4,c=c-3,d=d+2) CPU level 1 read page in L1 update persist to L2 level 2 A B C D WAL: write ahead log keep persistent notes as we go so we can resume or undo Stratos Idreos 19/50

28 update all rows where A=v1 & B=v2 to (a=a/2,b=b/4,c=c-3,d=d+2) CPU level 1 read page in L1 update persist to L2 level 2 A B C D WAL: write ahead log keep persistent notes as we go so we can resume or undo what do we need to remember (log) Stratos Idreos 19/50

29 update all rows where A=v1 & B=v2 to (a=a/2,b=b/4,c=c-3,d=d+2) CPU level 1 read page in L1 update persist to L2 level 2 A B C D WAL: write ahead log keep persistent notes as we go so we can resume or undo what do we need to remember (log) restart: get set of positions again, and either resume from last written page or undo all previously written pages Stratos Idreos 19/50

30 update all rows where A=v1 & B=v2 to (a=a/2,b=b/4,c=c-3,d=d+2) CPU level 1 read page in L1 update persist to L2 level 2 A B C D (update in place/out of place) problems when failure Stratos Idreos 20/50

31 update all rows where A=v1 & B=v2 to (a=a/2,b=b/4,c=c-3,d=d+2) CPU level 1 in-place updates: the cardinal sin read page in L1 update persist to L2 The Transaction Concept: Virtues and Limitations Jim Gray, Tandem TR 81.3, 1981 level 2 A B C D (update in place/out of place) problems when failure Stratos Idreos 20/50

32 update all rows where A=v1 & B=v2 to (a=a/2,b=b/4,c=c-3,d=d+2) CPU level 1 in-place updates: the cardinal sin read page in L1 update persist to L2 The Transaction Concept: Virtues and Limitations Jim Gray, Tandem TR 81.3, 1981 level 2 A B C D (update in place/out of place) RUMA has it: Rewired User-space problems Memory when failure Access is Possible! Felix Martin Schuhknecht, Jens Dittrich, Ankur Sharma Proceedings of the Very Large Databases Endowment, 9(10): , 2016 Stratos Idreos 20/50

33 memory log non volatile memory SSD disk other machines Stratos Idreos 21/50

34 TWO CHALLENGES EFFICIENCY & ROBUSTNESS Stratos Idreos 22/50

35 C Mohan, IBM Research ACM SIGMOD Edgar F. Codd Innovations award 1993 (rumor has it he has his own Facebook server) (also check his nosql and blockchain lectures) ARIES: A Transaction Recovery Method Supporting Fine-Granularity Locking and Partial Rollbacks Using Write-Ahead Logging C. Mohan, Donald J. Haderle, Bruce G. Lindsay, Hamid Pirahesh, Peter M. Schwarz ACM Transactions on Database Systems, 1992 Stratos Idreos 23/50

36 query 1 parse optimize plan( read X write Z ) query 2 parse optimize plan( read Z write Z ) Stratos Idreos 24/50

37 locks wait until lock is free read X get_rlock(x) read X release_rlock(x) R W R yes no W no no write X get_wlock(x) write X release_wlock(x) commit Stratos Idreos 25/50

38 database table1 lock granularity table2 C1 C2 A survey of B-tree logging and recovery techniques Goetz Graefe ACM Trans. Database Systems, 2012 Stratos Idreos 26/50

39 2 phase locking 1. get all locks 2. do all tasks 3. commit 4. release all locks (and variations) Stratos Idreos 27/50

40 2 phase locking 1. get all locks 2. do all tasks HT 3. commit 4. release all locks (and variations) be positive! optimistic concurrency control Stratos Idreos 27/50

41 Χ classic example joe owes mike 100$ both joe and mike have a Bank of Bla account recovery what about logging and recovery during e-shopping e.g., shopping cart, wish list, check out Quantifying eventual consistency with PBS Peter Bailis, Shivaram Venkataraman, Michael J. Franklin, Joseph M. Hellerstein, Ion Stoica Communications of the ACM, 2014 Stratos Idreos 28/50

42 TWO CHALLENGES EFFICIENCY & ROBUSTNESS Stratos Idreos 29/50

43 remember: it all starts with how we store the data IDEAL a data structure that is best for reads, best for writes and does not require too much metadata Stratos Idreos 30/50

44 100Kx disk Pluto 2 years Jim Gray, IBM, Tandem, DEC, Microsoft ACM Turing award ACM SIGMOD Edgar F. Codd Innovations award 100x memory 10x on board cache 2x on chip cache registers New York 1.5 hours this building 10 min this room 1 min my head ~0 Stratos Idreos 31/50

45 random access & page-based access same for writes! need to only read x but have to read all of page 1 CPU registers data value x on chip cache page1 page2 page3 data move on board cache memory disk Stratos Idreos 32/50

46 update value x to y in page p of array z update Level 1 Level 2 page to update cost what if >1 updates (no locking for now) Stratos Idreos 33/50

47 buffer >>1 updates to this page before pushing to L2 update Level 1 Level 2 page to update Stratos Idreos 34/50

48 1. read input into stream buffer, hash and write to respective partition buffer 2. when input buffer is consumed, bring the next one 3. when a partition buffer is full, write to L2 stream input partition p1 p2 p3 p4 Level 1 p1 p2 p4 p3 Level 2 Stratos Idreos 35/50

49 e.g., from disk to flash ideal write granularity is different what do you think changed in algorithms? Stratos Idreos 36/50

50 Reads Updates Memory you can t always get what you want Designing Access Methods: The RUM Conjecture M. Athanassoulis, M. Kester, L. Maas, R. Stoica, Radu, S. Idreos, A. Ailamaki, M Callaghan International Conference on Extending Database Technology (EDBT), 2016 Stratos Idreos 37/50

51 a perfect access method for reads (point queries) find(x) reads oracle updates x memory CS165, Fall 2016 Stratos Idreos 38/33

52 a perfect access method for reads (point queries) but with no memory overhead reads binary search to find(x) sorted updates memory CS165, Fall 2016 Stratos Idreos 39/33

53 a perfect access method for writes (point writes) reads update(x) x x x update log updates memory CS165, Fall 2016 Stratos Idreos 40/33

54 to structure or not to structure insert v, delete v, update v to v no order fixed-width & dense sorted fixed-width & dense sorted fixed-width with holes CS165, Fall 2016 Stratos Idreos 41/33

55 NoSQL key-value store & CS265 research/systems projects great for inserts/updates great for recent reads optimize as much as possible querying old data Stratos Idreos 42/50

56 all writes go to a small buffer updates are fast reads can find recent updates fast L1 L2 Stratos Idreos 43/50

57 all writes go to a small buffer updates are fast reads can find recent updates fast L1 L2 data is flushed to L2 when buffer is full Stratos Idreos 43/50

58 all writes go to a small buffer updates are fast reads can find recent updates fast L1 L2 data is flushed to L2 when buffer is full data is ordered on ingestion order no in place updates getting old data is a full scan we may have duplicates to clean up Stratos Idreos 43/50

59 L1 L2 multi-level L2 structure push to next level when full remove duplicates with every merge Stratos Idreos 44/50

60 L1 L2 to improve reads: each level may be sorted, it may be a b-tree have a bloom filter, etc worst case scenario: have to scan the whole thing: empty queries Stratos Idreos 45/50

61 L1 L2 to improve reads: each level may be sorted, it may be a b-tree have a bloom filter, etc worst case scenario: have to scan the whole thing: empty queries Stratos Idreos 45/50

62 db complex legacy tuning expensive nosql simple clean just enough why nosql as apps become more complex as apps need to be more scalable newsql Mesa: Geo-Replicated, Near Real-Time, Scalable Data Warehousing Ashish Gupta et al. International Conference on Very Large Databases (VLDB), 2014 F1: A Distributed SQL Database That Scales Jeff Shute et al. International Conference on Very Large Databases (VLDB), 2013 Stratos Idreos 46/50

63 daslab.seas.harvard.edu/crimsondb

64 10x faster daslab.seas.harvard.edu/crimsondb

65 10x faster CS265 research and systems projects daslab.seas.harvard.edu/crimsondb

66 & check out sections 11,12,13 textbook: chapters 16, 17, 18 Positional update handling in column stores Sándor Héman, Marcin Zukowski, Niels J. Nes, Lefteris Sidirourgos, Peter A. Boncz In Proc. of the ACM SIGMOD Inter. Conference on Management of Data, 2010 Updating a cracked database Stratos Idreos, Martin Kersten, Stefan Manegold In Proc. of the ACM SIGMOD Inter. Conference on Management of Data, 2007 Stratos Idreos 48/50

67 class 17 updates DATA SYSTEMS prof. Stratos Idreos