Performance by combining different log information. Daniel Stein Nürnberg,

Transcription

1 Performance by combining different log information Daniel Stein Nürnberg,

2 agenda about me introduction four examples conclusion 2

3 about me

4 about me 32 years 10+ years experience Java / JDBC 4+ years experience Oracle focusing on CBO / performance at Debeka since 2004 link between developers, DBAs and sysadmins DB application troubleshooter database design application code (COBOL / Java) database 4

5 about my employer founded in 1905 Koblenz / Rheinland-Pfalz Germany s largest private health insurance company more than employees (more than 500 in IT) looking for a job? 5

6 introduction

7 introduction from time to time our batch night overruns, so we wanted to figure out why and how to fix it who is responsible? DEV: my local processing is fast. The DB is slow DBA : most of the time, the DB waits for the applications (SQL* Net Message from client). It's the DEVs fault. ADMIN: packets/sec are sent. All very small. What am I supposed to do? Spoiler! They're all together responsible... and IBM too. 7

8 introduction we needed to understand where the batch jobs spend their time three main areas: application, system, database we had to log the right detail information per batch run without a noticeable overhead on the production system focusing only on database information is not enough 8

9 collected application information all applications are started by a self developed runner also controlling the Oracle session real:45m 22s sys: 1m 15s User 3m 34s this allows us to collect: timing information (unix time command) profiling information (only turned on when needed) too much overhead by parsing the SQLs we know : transaction information: commit / rollback count accessed database objects count (Select / Insert / Update / Delete ) DB-STATISTICS COMMIT 1 KV_TABLE1:INSERT 5 LV_TABLE36:SELECT

10 collected database information Database: V$MYSTAT / V$STATNAME general session information: roundtrip count V$SESSION_EVENT idle / non-idle wait times V$SESS_TIME_MODEL distribution of session on cpu time Session Stats: SQL*Net roundtrips to/from client : Session Wait Stats (total / time waited ): db file sequential read SQL*Net message from client Session Time Model: DB Time CPU Time parse time sql execute elapsed time 10

11 combining the informations Application Time Slices (system resources ) profiling information (only when necessary) application (user + sys) access count DB-objects system further logging information ( SQL*Net Message from client user sys ) database (DB-Time): segmentation: - non-idle wait events - sql-executions -... real real ( SQL*Net Message from client + database) 11

12 examples

13 first example Application Time Slices (system resources ) access count DB-Objects application - LV_TABLE69:SELECT (5) (user (2m)+ sys(3m)) - KV_TABLE77:SELECT (1) system 1h 20m ( SQL*Net Message from client user sys ) - roundtrips ( ) - SQL*Net Message from client ( ) database (20m) (DB-Time) real (1h 45m) real ( SQL*Net Message from client + database) (1h 45m) 13

14 high sys time Tracing System-Calls statx("/usr/share/lib/zoneinfo//europe/berlin", 0x2FF18758, 76, 0) = 0 statx("/usr/share/lib/zoneinfo//europe/berlin", 0x2FF18760, 76, 0) = 0 statx("/usr/share/lib/zoneinfo//europe/berlin", 0x2FF18760, 76, 0) = 0 statx("/usr/share/lib/zoneinfo//europe/berlin", 0x2FF18760, 76, 0) = 0 kwrite(10, "\015\0\006\0\0\0\0\00305".., 21) = 21 kread(10, "\0 z\0\006\0\0\0\0\01702".., 8208) = 122 statx("/usr/share/lib/zoneinfo//europe/berlin", 0x2FF18760, 76, 0) = 0 statx("/usr/share/lib/zoneinfo//europe/berlin", 0x2FF18760, 76, 0) = 0 statx("/usr/share/lib/zoneinfo//europe/berlin", 0x2FF18760, 76, 0) = 0 statx("/usr/share/lib/zoneinfo//europe/berlin", 0x2FF18760, 76, 0) = 0 millions of same statx calls per program execution occasionally, real work is also done 14

15 high sys time Who is calling this method over and over again and why? ProbeVUE Script (AIX) / DTrace (Linux) (almost the same): #!/bin/probevue int stat(const char *pathname, struct stat { if (strcmp(get_userstring( arg1,-1), "/usr/share/lib/zoneinfo//europe/berlin") == 0) { printf( %t\n, get_stktrace(5)); exit(); } } result: the purpose of the call (stat) is to check if the timezone has changed due to time related calls done by the application (gettimeoftheday). 15

16 1. high sys time Is it necessary to check the timezone over and over again? if the timezone never changes, the expensive check can be avoided by setting the TZ environment variable strangely enough, the TZ environment variable is set in our production system (OLSON Format). bug alert! IBM IV86730: PERFORMANCE REGRESSION WHEN USING OLSON TIMEZONE FORMAT 16

17 2. high number of roundtrips when you fetch from a cursor, more than one row is sent to you by default default fetch sizes : JDBC : 10 Procob (Oracle Driver for COBOL ) : 1 (Yes, it s still alive) we have decided to raise the default fetch size to 10 for our COBOL-Applications moreover, we encouraged our developers to use bigger fetch sizes raising those numbers halved the transaction times in our online system boosted our batch night without having been told, our office workers noticed the system feeling more responsive 17

18 2. high number of roundtrips letting our developers raise the fetch sizes leads to: DEV wait event: SQL*Net more data to client size of rows fetched in one attempt exceed the SDU increasing the SDU (Server + Client) improves network bandwidth utilization reduces system-calls to read (App) / write (DB) the socket. kread(10, "\0 z\0\006, 32768) talking to the SYS-admins : modern networks : MTU > 1,5kb (large send / jumbo frames) this reduces TCP/IP protocoll overhead. by segmentation offloading TCP/IP is done in hardware and does not waste CPU DBA ADMIN 18

19 first example afterwards Application Time Slices (system resources ) Access count DB-objects application - LV_TABLE69:SELECT (5) (user (2m)+ sys(1m)) prev: sys(3m) - KV_TABLE77:SELECT (1) system 35m prev: 1h 20m ( SQL*Net Message from client user sys ) - roundtrips ( ) - SQL*Net Message from client ( ) - SQL*Net more data to client (gone) database (15m) (DB-Time) prev: 20m real (58m) real ( SQL*Net Message from client + database) (58m) prev: 1h 45m : (-45%) 19

20 second example Application Time Slices (system resources ) access count DB-objects application - PERSON_TABLE:SELECT ( ) (user (1m)+ sys(10s) - ADDRESS_TABLE:SELECT (50.000) system 45m - BANKACCOUNT_TABLE:SELECT (50.000) - roundtrips ( ) - SQL*Net Message from client ( ) database (20m) - db file sequential read ++ - sql execute elapsed time ++ real (1h 15m 10s ) real ( SQL*Net Message from client + database) (1h 15m 10s) 20

21 too many modules( microservices") redundant access we realized that these selects came from a central component responsible for retrieving clients information. the selects accessing the details resulted in the sum of the selects accessing the persons this component consists of many modules, each of them responsible for retrieving a client's detail e.g. (address, bank account ) transaction sequence (original): 1. address 2. bank account 3 1. load the person 2. load the detail 1. load the person 2. load the detail 1. load the person 2. load the detail 21

22 too many modules("microservices") redundant access Checking the AWR report reveals: in the investigated batch night were 800 million cursor executions in total load the person -cursor with over 100 million executions detail cursors with around 9 million executions over 90 million unnecessary load the person executions transaction sequence (now) : 1. transaction start 2. address 3. bank account 4 load the person load the detail load the detail load the detail 22

23 second example afterwards Application Time Slices (system resources ) access count DB-objects application - PERSON_TABLE:SELECT (50.000) (user (1m)+ sys(10s) - ADDRESS_TABLE:SELECT (50.000) system 35m prev: 45m - BANKACCOUNT_TABLE:SELECT (50.000) - roundtrips ( ) - SQL*Net Message from client ( ) database (15m) prev: 20m - db file sequential read + - sql execute elapsed time + real (50m 10s ) real ( SQL*Net Message from client + database) (50m 10s) prev: 1h 15m 10 s : (-23%) 23

24 third example Application Time Slices (system resources ) access count DB-objects application - TRANSACTION_TABLE:SELECT ( ) (user (1m)+ sys(10s) system 10m - BANKACCOUNT_TABLE:SELECT ( ) - roundtrips (1.000) - SQL*Net Message from client (1.000) database (45m) - db file sequential read sql execute elapsed time real (56m 10s ) real ( SQL*Net Message from client + database) (56m10s) 24

25 third example accounting application dealing with banking transactions storing / processing data in transactional batches occasionally, the banking transactions of a certain period are moved or deleted office workers working with the data in an online system one giant table holding all banking transactions 90% of the queries deal with banking transactions of a certain time period ( e.g. the last quarter) poor database is dealing with the whole table all the time 25

26 getting it right while modeling one fast growing giant table is holding all the banking transactions partitioning is licensed partition by date would be appropriate simplifying moving / deleting of data by grabbing whole partitions and not doing update / insert / deletes also less resource consumption: data are better clustered a time period can be fetched with few multi-reads to the SAN instead of hundreds of single-reads a lot of indexes / index compression candidates but no compression used e. g. BIC: ~ 10 Mio accounts 10 Mio / 4000 BIC ~ 2500 rows per distinct BIC 26

27 third example afterwards Application Time Slices (system resources ) access count DB-objects application - TRANSACTION_TABLE:SELECT ( ) (user (1m)+ sys(10s) system 10m - BANKACCOUNT_TABLE:SELECT ( ) - roundtrips (1.000) - SQL*Net Message from client (1.000) database (10m) prev: 45m - db file sequential read ++ - db file scattered read ++ - sql execute elapsed time + real (21m 10s ) real ( SQL*Net Message from client + database) (predicted: 21m10s) prev: 56m 10s: (-61%) 27

28 fourth example Application Time Slices (system resources ) profiling information (only when necessary) application (user + sys) access count DB-objects system further logging information ( SQL*Net Message from client user sys ) database (DB-Time): segmentation: - non-idle wait events - sql-executions -... always the largest part real real ( SQL*Net Message from client + database) 28

29 timeline of a roundtrip APP SYSTEM / DATABASE APP 29

30 fourth example despite all the optimizations, the "system time" area is still disproportionately large AIX 7.1 we traced the application with probevue 30

31 timeline of a roundtrip APP SYSTEM / DATABASE APP write read: entry off cpu network / DB TCP /IP packet arival (from DB) on cpu read: exit working mysteriously large gap up to 100µsec 31

32 fourth example we can't explain this big period of time the data have arrived / system resources are available interrupt handling? scheduling problem? we have contacted IBM about this if someone in the audience has a clue, this would be very much appreciated 32

33 one last example we have come across a batch job that regularly takes a very long time to complete its task is to produce a monthly report for an insurance department of our company instead of tuning it, the developer spoke to the insurance department reciving the report and asked if they still need it regulary it turned out that the report is no longer needed at all we removed the job from our enterprise scheduler it will never waste resources again best tuning: don't do anything unnecessarily! 33

34 conclusion

35 lessons learned logging the right information is very important for troubleshooting purposes it is very important that all stakeholders work together talk to each other build a network so that you have a contact person in each department! get the following quotes as a tattoo! If there's something you don't understand - research it. Or it will bite you sooner or later. Consider your work in context! Performance matters! 35

36 contact Any questions? Your contact: Debeka Daniel Stein Ferdinand-Sauerbruch-Str Koblenz daniel_wandern 36