class 17 updates prof. Stratos Idreos

Transcription

1 class 17 updates prof. Stratos Idreos

2 early/late tuple reconstruction, tuple-at-a-time, vectorized or bulk processing, intermediates format, pushing selects down, etc cpu memory algorithms/operators database kernel so far scan, binary search, tuple reconstruction, min, max, search/update b-tree, join, etc. arrays, columns, matrixes, rows,trees disk data data data Stratos Idreos 2 /38

3 UPDATE table_name SET column1=value1,column2=value2,... WHERE some_column=some_value INSERT INTO table_name VALUES (value1,value2,value3,...) updates Stratos Idreos 3 /38

4 traditional applications e.g., banking how many times per day do you send update queries to your bank account Stratos Idreos 4 /38

5 the world has changed a little bit by now updates Stratos Idreos 5 /38

6 still we spy Facebook more than the # of photos we upload CS165, Fall or 2016 # of our twitter posts, etc Stratos Idreos 6 /38

7 so systems should be tuned for more reads yet we have way more writes than before and a variable read/write ratio Stratos Idreos 7 /38

8 which kind of update is more common update, insert, delete Stratos Idreos 8 /38

9 which kind of update is more common update, insert, delete so our new challenge is: reads and inserts + variable read/write ratio Stratos Idreos 8 /38

10 not just about user data: everything is data! Stratos Idreos 9 /38

11 monitor CPU utilization monitor memory hierarchy utilization monitor clicks (frequency, locations, specific links, sequences) what & how Stratos Idreos 10/38

12 today data grows dai y [IB data systems are nearly everywhere continuous need for new and tailored data systems Stratos Idreos 11/38

13 today data grows dai y [IB data systems are nearly everywhere continuous need for new and tailored data systems tomorrow Stratos Idreos 11/38

14 more applications more data more h/w Stratos Idreos 12/38

15 analyze data as it arrives and react (standing queries) merge incoming data with already archived data new data see the correct up-to-date values do not lose any updates (software/hardware failures) data system >>1 updates concurrently Stratos Idreos 13/38

16 conflicting goals moving target (hardware and requirements change continuously and rapidly) application requirements performance budget hardware energy profile Stratos Idreos 14/38

17 Three things are important in the database world: performance, performance, and performance Bruce Lindsay, IBM ACM SIGMOD Edgar F. Codd Innovations award 2012 true for both reads & writes Stratos Idreos 15/38

18 how to do fast (& correct) updates? (more or less same way we do fast reads) locking transactions logging lazy vs eager updates fractured mirrors in-place or not layout vs scheduling Stratos Idreos 16/38

19 name name, address age data structure vs application updates student{name, age, address, telephone, GPA, } Stratos Idreos 17/38

20 insert new entry (a,b,c,d, ) on table x update N columns, K trees, statistics, table x A B C D A B D Stratos Idreos 18/38

21 table x A B C D to index or not to index what info do we need how do we make decisions when to do create indices secondary index on D Stratos Idreos 19/38

22 data (array) value Stratos Idreos 20/38

23 insertion data (array) value Stratos Idreos 20/38

24 delete data (array) value Stratos Idreos 20/38

25 delete data (array) value Stratos Idreos 20/38

26 update data (array) value Stratos Idreos 20/38

27 data (array) value inserts, deletes, updates=deletes followed by inserts Stratos Idreos 20/38

28 100Kx disk Pluto 2 years Jim Gray, IBM, Tandem, DEC, Microsoft ACM Turing award ACM SIGMOD Edgar F. Codd Innovations award 100x memory 10x on board cache 2x on chip cache registers New York 1.5 hours this building 10 min this room 1 min my head ~0 Stratos Idreos 21/38

29 random access & page-based access same for writes! need to only read x but have to read all of page 1 CPU registers data value x on chip cache page1 page2 page3 data move on board cache memory disk Stratos Idreos 22/38

30 1. read input into stream buffer, hash and write to respective partition buffer 2. when input buffer is consumed, bring the next one 3. when a partition buffer is full, write to L2 stream input partition p1 p2 p3 p4 Level 1 p1 p2 p4 p3 Level 2 Stratos Idreos 23/38

31 update value x to y in page p of array z update Level 1 Level 2 page to update cost what if >1 updates (no locking for now) Stratos Idreos 24/38

32 buffer >>1 updates to this page before pushing to L2 update Level 1 Level 2 page to update Stratos Idreos 25/38

33 e.g., from disk to flash ideal write granularity is different what do you think changed in update algorithms? Stratos Idreos 26/38

34 content vs structure update insert tuple(a1, b1, c1, ) insert(a,a1), insert(b, b1), say there is a secondary index on A (1) append a1 anywhere to index (any node/buffer) (2) reorganize index to maintain structure Stratos Idreos 27/38

35 row-store column-store A B C D A B C D vs costs update row7=(a=a,b=b,c=c,d=d) Stratos Idreos 28/38

36 updates reads A B C D A B C D periodic merge and/or on-the-fly merge write optimized-store read optimized-store A case for fractured mirrors Ravishankar Ramamurthy, David J. DeWitt, Qi Su Very Large Databases Journal (VLDBJ), 2003 Stratos Idreos 29/38

37 A A.deletes A.inserts (id) (id,value) select(a,v1,v2) A pos pos2 res A.del diff A.ins ins union scan scan Stratos Idreos 30/38

38 A B C D update all rows where A=v1 & B=v2 to (a=a/2,b=b/4,c=c-3,d=d+2) CPU level 1 level 2 how to perform updates efficiently and correctly? correctly=all or nothing problems to worry about (?): what if user/applications aborts? what if power goes down? what if there is an earthquake in our city? what if aliens come to earth? (assume simplified memory hierarchy) all data fit in L2, not all data fit in L1 L2 is non-volatile, L1 is volatile Stratos Idreos 31/38

39 update all rows where A=v1 & B=v2 to (a=a/2,b=b/4,c=c-3,d=d+2) A B C D search (scan/index) to find row to update select+project actions Stratos Idreos 32/38

40 update all rows where A=v1 & B=v2 to (a=a/2,b=b/4,c=c-3,d=d+2) A B C D search (scan/index) to find row to update select+project actions A B C D list of rowids (positions) Stratos Idreos 32/38

41 update all rows where A=v1 & B=v2 to (a=a/2,b=b/4,c=c-3,d=d+2) A B C D search (scan/index) to find row to update select+project actions A B C D list of rowids (positions) we know what to update but nothing happened yet Stratos Idreos 32/38

42 CPU level 1 level 2 A B C D read page in L1 update persist to L2 if problem (power/abort) before we write all pages we are left with an inconsistent state WAL: keep persistent notes as we go so we can resume or undo Stratos Idreos 33/38

43 when is the log or an update persistent? disk persistent memory, e.g., disk? Stratos Idreos 34/38

44 when is the log or an update persistent? disk persistent memory, e.g., disk? machine 1 machine 2 machine 3 replicate to multiple machines? Stratos Idreos 34/38

45 when is the log or an update persistent? disk persistent memory, e.g., disk? machine 1 machine 2 machine 3 replicate to multiple machines? city 1 city 2 city 3 replicate to multiple machines >1 clusters in >1 cities? Stratos Idreos 34/38

46 more details about all these next class: transactions, ACID what if >>1 update queries at the same time WAL & replication Stratos Idreos 35/38

47 continuous data stream queries wait for data stream system Aurora: a new model and architecture for data stream management Daniel J. Abadi, Donald Carney, Ugur Çetintemel, Mitch Cherniack, Christian Convey, Sangdon Lee, Michael Stonebraker, Nesime Tatbul, Stanley B. Zdonik Very Large Databases Journal (VLDBJ), 2003 Enhanced stream processing in a DBMS kernel Erietta Liarou, Stratos Idreos, Stefan Manegold, Martin Kersten In Proc. of the International Conf. on Extending Database Technology (EDBT), 2013 Stratos Idreos 36/38

48 (also for next class) textbook: chapters 16, 17, 18 Positional update handling in column stores Sándor Héman, Marcin Zukowski, Niels J. Nes, Lefteris Sidirourgos, Peter A. Boncz In Proc. of the ACM SIGMOD Inter. Conference on Management of Data, 2010 Updating a cracked database Stratos Idreos, Martin Kersten, Stefan Manegold In Proc. of the ACM SIGMOD Inter. Conference on Management of Data, 2007 Stratos Idreos 37/38

49 class 17 updates DATA SYSTEMS prof. Stratos Idreos