Frits Hoogland - Oracle Usergroup Norway 2013 EXADATA AND OLTP. Thursday, April 18, 13

Transcription

1 Frits Hoogland - Oracle Usergroup Norway 2013 EXADATA AND OLTP

2 Who am I? Frits Hoogland Working with Oracle products since 1996 Blog: Oracle ACE Director OakTable Member 2

3 Note!! This presentation has been made benchmarking an older version of the Oracle appliances. The versions are explicitly stated. The message of the presentation doesn t change! 3

4 What is Exadata? Engineered system specifically for oracle database. Ability to reach high number of read IOPS and huge bandwidth. Has it s own patchbundles. Validated versions and patch versions across database, clusterware, o/s and storage, firmware. Dedicated, private storage for databases. ASM. Recent hardware & recent CPU. No virtualisation. 4

5 Exadata versions Oracle database 64 bit version >= 11 (.2) ASM 64 bit version >= 11 (.2) Exadata communication is using in skgxp code Linux OL5 x64 No UEK kernel used (except X2-8) Update! Exadata uses el5uek 5

6 Exadata hardware Intel Xeon server hardware Infiniband 40Gb/s Oracle cell (storage) server Flash to mimic SAN cache High performance disks or high capacity disks 600GB 15k RPM / ~ 5ms latency 2/3TB 7.2k RPM / ~ 8ms latency 6

7 Flash Flashcards are in every storage server Total of 384GB per storage server* Do not confuse exadata STORAGE server flashcache with oracle database flashcache! 7

8 Flash Flash can be configured either as cache or as diskgroup or both. cache: flashlog, flashcache* When flash is used as diskgroup latency is ~ 1 ms Faster than disk My guess was < 400µs 1µs infiniband 200µs flash IO time some time for storage server 8

9 Flash Flash is restricted* to 4x96GB = 384GB per storage server. Totals: Q:1152GB, H:2688GB, F:5376GB Net (ASM Normal redundancy): Q: 576GB, H:1344GB, F:2688GB That is a very limited amount of storage. 9

10 Exadata specific features The secret sauce of exadata: the storage server smartscan storage indexes EHCC * IO Resource manager 10

11 Exadata specific features The secret sauce of exadata: the storage server smartscan storage indexes EHCC * IO Resource manager 10

12 Exadata specific features The secret sauce of exadata: the storage server smartscan storage indexes EHCC * IO Resource manager 10

13 Exadata specific features The secret sauce of exadata: the storage server smartscan storage indexes EHCC * IO Resource manager 10

14 OLTP How does OLTP look like (in general simplistic) Fetch small amount of data Invoice numbers, client id, product id select single values or small ranges via index Create or update rows Sold items on invoice, payments, order status insert or update values 11

15 A great way to mimic or measure OLTP performance is SLOB Silly Little Oracle Benchmark Author: Kevin Closson 12

16 SLOB It can do reading: FOR i IN LOOP v_r := dbms_random.value(257, 10000) ; SELECT COUNT(c2) into x FROM cf1 where custid > v_r AND custid < v_r; END LOOP; 13

17 SLOB And writing: FOR i IN LOOP v_r := dbms_random.value(257, 10000) ; UPDATE cf1 set c2 = 'AAAAAAAABBBBBBBBAAAAAAAABBBBBBBBAAAAAAAABBBBBBBBAAAAAAAABBBBBBBBAAAAAAAABBBBBBBBAA AAAAAABBBBBBBBAAAAAAAABBBBBBBBAAAAAAAABBBBBBBB',...up to column 20 (c20)... where custid > v_r AND custid < v_r; COMMIT; END LOOP; 14

18 SLOB Lets run SLOB with 0 writers and 1 reader at: Single instance database, 10G SGA, 9G cache. >> Cold cache << Exadata V2 / Oracle / HP Half rack / 7 storage servers / 84 disks (15k rpm) Exadata X2 / Oracle / HC Quarter rack / 3 storage servers / 36 disks (7.2k rpm) 15

19 SLOB - 1 reader Response time V2 / Half rack X2 / Quarter 0 rack

20 SLOB - 1 reader Response time V2 / Half rack X2 / Quarter rack

21 SLOB - 1 reader Response time V2 / Half rack X2 / Quarter rack ODA 0 18

22 SLOB - 1 reader Response time V2 / Half rack X2 / Quarter rack ODA 19

23 SLOB - 1 reader V2 time: 5 sec - CPU: 84.3% PIO: (0.8%) -- IO time 0.8 sec LIO: X2 time: 4 sec - CPU: 75.7% PIO: (0.8%) -- IO time 0.9 sec LIO: ODA tm: 4 sec - CPU: 55.2% (20 disks 15k rpm) PIO: (0.8%) -- IO time 2.2 sec LIO:

24 SLOB - 1 reader - conclusion The time spend on PIO is 15% - 45% Majority of time spend on LIO/CPU time Because main portion is CPU, fastest CPU wins Actually: fastest CPU, memory bus and memory. 21

25 SLOB - 1 reader - LIO Let s do a pre-warmed cache run Pre-warmed means: no PIO, data already in BC This means ONLY LIO speed is measured 22

26 SLOB - 1 reader - LIO ODA: 1 sec X2: 2 sec V2: 3 sec 23

27 SLOB - 1 reader - LIO Reason: LIO essentially means Reading memory CPU processing Use dmidecode to look to the system s components! 24

28 V2 Intel Xeon 2.53GHz (2s8c16t) L1:128kB, L2:1MB, L3:8MB Memory: Type DDR3, speed 800 Mhz X2: Intel Xeon 2.93GHz (2s12c24t) L1:384kB, L2:1.5MB, L3:12MB Memory: Type DDR3, speed 1333 Mhz ODA Intel Xeon 3.07GHz (2s12c24t) L1:384kB, L2:1.5MB, L3:12MB Memory: Type DDR3, speed 1333 Mhz 25

29 SLOB - LIO - multiple readers Let s take the 1 reader figures, and scale up. What would ODA, Exadata V2 and Exadata X2 do? 26

30 SLOB - LIO - multiple readers 30.0 LIO read response time oda x2 v2 27

31 SLOB - LIO - multiple readers 30.0 LIO read response time oda x2 v2 28

32 SLOB - LIO - multiple readers 30.0 LIO read response time oda x2 v2 29

33 SLOB - LIO - multiple readers 30.0 LIO read response time oda x2 v2 30

34 SLOB - LIO - multiple readers 30.0 LIO read response time 22.5 Core difference and slower memory shows when # readers exceeds core count oda x2 v2 30

35 SLOB - LIO - multiple readers 30.0 LIO read response time 22.5 Core difference and slower memory shows when # readers exceeds core count Same memory speed: CPU speed matters less with more concurrency oda x2 v2 30

36 SLOB - LIO - multiple readers 30.0 LIO read response time oda x2 v2 31

37 SLOB - LIO - multiple readers 30.0 LIO read response time Lesser core s and slower memory make LIO processing increasingly slower with more concurrency oda x2 v2 31

38 SLOB - LIO - multiple readers 30.0 LIO read response time Lesser core s and slower memory make LIO processing increasingly slower with more concurrency For LIO processing ODA (non-exadata) and Exadata does not matter oda x2 v2 31

39 SLOB - LIO - multiple readers Conclusion: LIO performance is impacted by: CPU speed Number of sockets and core s L1/2/3 cache sizes Memory speed Exadata does not matter here! 32

40 SLOB - LIO - multiple readers When comparing entirely different systems also consider: Oracle version O/S and version (scheduling) Hyper threading / CPU architecture NUMA (Exadata/ODA: no NUMA!) 33

41 SLOB - PIO - 1 reader But how about physical IO? Lower the buffercache to 4M sga_max_size to 1g cpu_count to 1 db_cache_size to 1M (results in 4M) Slob run with 1 reader 34

42 110 SLOB - PIO - 1 reader PIO reader response time oda x2 v2 35

43 110 SLOB - PIO - 1 reader PIO reader response time Surprise! ODA is the fastest here with 73 seconds oda x2 v2 35

44 110 SLOB - PIO - 1 reader PIO reader response time 83 The X2 is only a little slower with 76 seconds. 55 Surprise! ODA is the fastest here with 73 seconds oda x2 v2 35

45 110 SLOB - PIO - 1 reader PIO reader response time The V2 is the slowest with 106 seconds. 83 The X2 is only a little slower with 76 seconds. 55 Surprise! ODA is the fastest here with 73 seconds oda x2 v2 35

46 SLOB - PIO - 1 reader Total time (s) CPU time (s) IO time (s) ODA X V IO ODA: 60/ = ms IO X2: 55/ = ms IO V2: 52/ = ms 36

47 SLOB - PIO - 1 reader This is not random disk IO! Average latency of random IO 15k rpm disk ~ 5ms Average latency of random IO 7.2k rpm disk ~ 8ms So this must come from a cache or is not random disk IO Exadata has flashcache. On ODA, data probably very nearby on disk. 37

48 SLOB - PIO - 1 reader Total time (s) CPU time (s) IO time (s) ODA X V Exadata IO takes (way) more CPU. Roughly the same time is spend on doing IO s. 38

49 10 readers 39

50 4000 SLOB - PIO - 10 readers PIO reader response time oda x2 v2 40

51 4000 SLOB - PIO - 10 readers PIO reader response time 3000 Now IO responsetime on ODA is way higher than Exadata (3008s) oda x2 v2 40

52 4000 SLOB - PIO - 10 readers PIO reader response time 3000 Now IO responsetime on ODA is way higher than Exadata (3008s) Both Exadata s perform alike: X2 581s, V2 588s oda x2 v2 40

53 SLOB - PIO - 10 readers Total time (s) CPU time (s) IO time (s) ODA X V IO ODA: 29428/ = ms IO X2: 5213/ = ms IO V2: 4866/ = ms 41

54 20 readers 42

55 5000 SLOB - PIO - 20 readers PIO reader response time oda x2 v2 43

56 SLOB - PIO - 20 readers Total time (s) CPU time (s) IO time (s) ODA X V IO ODA: 88756/ = = ms IO X2: 13010/ = = ms IO V2: 12405/ = = ms 44

57 6000 SLOB - PIO - 30 readers PIO reader response time oda x2 v2 45

58 6000 SLOB - PIO - 30 readers PIO reader response time 4500 ODA (20 x 15krpm HDD) disk capacity is saturated so response time increases with more readers oda x2 v2 45

59 6000 SLOB - PIO - 30 readers PIO reader response time 4500 ODA (20 x 15krpm HDD) disk capacity is saturated so response time increases with more readers Flashcache is not saturated, so response time of IO of readers increases very little oda x2 v2 45

60 ...up to 80 readers 46

61 10000 SLOB - PIO - 80 readers PIO reader response time oda x2 v2 47

62 10000 SLOB - PIO - 80 readers PIO reader response time ODA response time more or less increases linear oda x2 v2 47

63 10000 SLOB - PIO - 80 readers PIO reader response time ODA response time more or less increases linear X2 flashcache (3x96GB) is not saturated, so little increase in response time oda x2 v2 47

64 10000 SLOB - PIO - 80 readers PIO reader response time ODA response time more or less increases linear The V2 response time (with more flashcache!) starts increasing at 70 readers. A bottleneck is showing up! (7x96GB!!) 2500 X2 flashcache (3x96GB) is not saturated, so little increase in response time oda x2 v2 47

65 SLOB - PIO - 80 readers PIO readers IOPS IOPS view instead of responsetime oda x2 v2 48

66 SLOB - PIO - 80 readers PIO readers IOPS IOPS view instead of responsetime x96GB Flashcache and IB can serve > read IOPS! oda x2 v2 48

67 SLOB - PIO - 80 readers PIO readers IOPS IOPS view instead of responsetime x96GB Flashcache and IB can serve > read IOPS! This V2 has more flashcache, so decline in read IOPS probably due to something else! oda x2 v2 48

68 SLOB - PIO - 80 readers PIO readers IOPS IOPS view instead of responsetime x96GB Flashcache and IB can serve > read IOPS! This V2 has more flashcache, so decline in read IOPS probably due to something else! ODA maxed out at ~ read IOPS oda x2 v2 48

69 SLOB - PIO - 80 readers V2 top 5 timed events with 80 readers: Event Waits Time(s) (ms) time Wait Class cell single block physical rea 102,345,354 56, User I/O latch: cache buffers lru chain 27,187,317 33, Other latch: cache buffers chains 14,736,819 19, Concurrenc DB CPU 13, wait list latch free 932, Other X2 top 5 timed events with 80 readers: Event Waits Time(s) (ms) time Wait Class cell single block physical rea 102,899,953 68, User I/O DB CPU 9, latch: cache buffers lru chain 10,917,303 1, Other latch: cache buffers chains 2,048, Concurrenc cell list of blocks physical r 368, User I/O 49

70 SLOB - PIO - 80 readers V2 top 5 timed events with 80 readers: Event Waits Time(s) (ms) time Wait Class cell single block physical rea 102,345,354 56, User I/O latch: cache buffers lru chain 27,187,317 33, Other latch: cache buffers chains 14,736,819 19, Concurrenc DB CPU 13, wait list latch free 932, Other X2 top 5 timed events with 80 readers: 44.1% Event Waits Time(s) (ms) time Wait Class cell single block physical rea 102,899,953 68, User I/O DB CPU 9, latch: cache buffers lru chain 10,917,303 1, Other latch: cache buffers chains 2,048, Concurrenc cell list of blocks physical r 368, User I/O 49

71 SLOB - PIO - 80 readers V2 top 5 timed events with 80 readers: Event Waits Time(s) (ms) time Wait Class cell single block physical rea 102,345,354 56, User I/O latch: cache buffers lru chain 27,187,317 33, Other latch: cache buffers chains 14,736,819 19, Concurrenc DB CPU 13, wait list latch free 932, Other X2 top 5 timed events with 80 readers: 44.1% Event Waits Time(s) (ms) time Wait Class cell single block physical rea 102,899,953 68, User I/O DB CPU 9, latch: cache buffers lru chain 10,917,303 1, Other2.9% latch: cache buffers chains 2,048, Concurrenc cell list of blocks physical r 368, User I/O 49

72 SLOB - PIO - 80 readers On the V2, cache concurrency control throttles throughput On the X2, this happens only very minimal - V2: CPU: Intel Xeon 2.53GHz (2s8c16t) - X2: CPU: Intel Xeon 2.93GHz (2s12c24t) V2 Event Waits <1ms <2ms <4ms <8ms <16ms <32ms <=1s >1s latch: cache buffers chain 14.7M latch: cache buffers lru c 27.2M X2 latch: cache buffers chain latch: cache buffers lru c 10.9M

73 SLOB - Read IO 1 LIO 80 LIO 1 PIO 80 PIO ODA X2 (HC disks) V2 (HP disks)

74 SLOB - Read IO 1 LIO 80 LIO 1 PIO 80 PIO ODA X2 (HC disks) V2 (HP disks) PIO w/o flashcache 80 PIO w/o flashcache ODA X2 167? (> 18000) V

75 SLOB - Read IO For scalability, OLTP needs buffered IO (LIO) Flashcache is EXTREMELY important physial IO scalability Probably should not let flash be used for something else Unless you can always keep all your small reads in cache 52

76 SLOB - Read IO Flashcache mimics a SAN/NAS cache So nothing groundbreaking here, it does what current, normal infra should do too... The bandwidth needed to deliver the data to the database is provided by Infiniband 1 Gb ethernet = 120MB/s, 4 Gb fiber = 400MB/s Infiniband is generally available outside of Oracle 53

77 SLOB - Read IO How much IOPS can a single cell do? According to entry/inside_the_sun_oracle_database A single cell can do IOPS from flash (8kB) Personal calculation: IOPS with 8kB Flashcache cache Caches small reads* (8kB and less) mostly Large multiblock reads are not cached, unless segment property cell_flash_cache is set to keep. 54

78 SLOB - Read IO Is Exadata a good idea for OLTP? From a strictly technical point of view, there is no benefit. But... Exadata gives you IORM Exadata gives you reasonably up to date hardware Exadata gives a system engineered for performance Exadata gives you dedicated disks Exadata gives a validated combination of database, clusterware, operating system, hardware, firmware. 55

79 Writing and Exadata Exadata storage servers provide NO redundancy for data That s a function of ASM Exadata is configured with either Normal redundancy (mirroring) or High redundancy (triple mirroring) to provide data redundancy. 56

80 Writing and Exadata Reading has no problem with normal/high redundancy. During writes, all two or three AU s need to be written. This means when you calculate write throughput, you need to double all physical writes if using normal redundancy. 57

81 Writing on Exadata But we got flash! Right? Yes, you got flash. But it probably doesn t do what you think it does: 58

82 Writing on Exadata This is on the half rack V2 HP: [] stuff]$ dcli -l celladmin -g cell_group cellcli -e "list metriccurrent where name like \'FL_.*_FIRST\'" dm01cel01: FL_DISK_FIRST dm01cel01: FL_FLASH_FIRST dm01cel02: FL_DISK_FIRST dm01cel02: FL_FLASH_FIRST dm01cel03: FL_DISK_FIRST dm01cel03: FL_FLASH_FIRST dm01cel04: FL_DISK_FIRST dm01cel04: FL_FLASH_FIRST dm01cel05: FL_DISK_FIRST dm01cel05: FL_FLASH_FIRST dm01cel06: FL_DISK_FIRST dm01cel06: FL_FLASH_FIRST dm01cel07: FL_DISK_FIRST dm01cel07: FL_FLASH_FIRST FLASHLOG 316,563 IO requests FLASHLOG 9,143 IO requests FLASHLOG 305,891 IO requests FLASHLOG 7,435 IO requests FLASHLOG 307,634 IO requests FLASHLOG 10,577 IO requests FLASHLOG 299,547 IO requests FLASHLOG 10,381 IO requests FLASHLOG 311,978 IO requests FLASHLOG 10,888 IO requests FLASHLOG 315,084 IO requests FLASHLOG 10,022 IO requests FLASHLOG 323,454 IO requests FLASHLOG 8,807 IO requests 59

83 Writing on Exadata This is on the quarter rack X2 HC: ~]# dcli -l root -g cell_group cellcli -e "list metriccurrent where name like \'FL_.*_FIRST\'" xxxxcel01: FL_DISK_FIRST xxxxcel01: FL_FLASH_FIRST xxxxcel02: FL_DISK_FIRST xxxxcel02: FL_FLASH_FIRST xxxxcel03: FL_DISK_FIRST xxxxcel03: FL_FLASH_FIRST FLASHLOG 68,475,141 IO requests FLASHLOG 9,109,142 IO requests FLASHLOG 68,640,951 IO requests FLASHLOG 9,229,226 IO requests FLASHLOG 68,388,238 IO requests FLASHLOG 9,072,660 IO requests 60

84 Writing on Exadata Please mind these are cumulative numbers! The half-rack is a POC machine, no heavy usage between POC s. The quarter-rack has had some load, but definately not heavy OLTP. 61

85 Writing on Exadata I can imagine flashlog can prevent long write times if disk IO s queue. A normal configured database on Exadata has online redo in DATA and in RECO diskgroup Normal redundancy means every log write must be done 4 times Log writer wait times: V2 min: 16ms (1 writer), max: 41ms (20 writers) X2 min: 39ms (10 writers), max: 110ms (40 writers) 62

86 Writing on Exadata: conclusion Log file write response time on Exadata is not in the same range as reads. There s the flashlog feature, but it might work different than the whitepaper explains Be careful with heavy writing on Exadata. There s no Exadata specific improvement for writes*. 63

87 Thank you for attending! Questions? 64

88 Thanks, Links, etc. VX Company Klaas-Jan Jongsma Martin Bach Kevin Closson 65