Security and Dependability of Computer Systems. Valentina Casola Domenico Cotroneo

Size: px

Start display at page:

Download "Security and Dependability of Computer Systems. Valentina Casola Domenico Cotroneo"

Annabel Lee
5 years ago
Views:

1 Security and Dependability of Computer Systems Valentina Casola Domenico Cotroneo

2 Log-Based Field Failure Data Analysis: Exercise Prof. Domenico Cotroneo p. 2

3 Exercise The students will perform a log-based FFDA of a complex, large-scale computing system Mercury cluster at the National Center for Supercomputing Application* (NCSA); Objective: to learn the basics in data classification/analysis; determining the coalescence window to group a set of error entries; performing the failure analysis; reliability modelling via well-know distributions; * - University of Illinois at Urbana-Champaign (UIUC) p. 3

4 Mercury cluster: overview Mercury consists of 987 IBM nodes. The cluster has a 3-layer architecture + a management node (tg-master) computation nodes tg-master login nodes storage nodes ID of the nodes in the log (X is an integer) tg-loginx tg-cx tg-sx Myrinet. The nodes run a RedHat 9.0 OS Entries have been collected via the *syslog* daemon p. 4

5 More on the available log The log has been filtered/anonimized (only the error entries collected from 01-Jan-2007 to 26-Mar-2007 are given in the exercise log*: logmercury-exe.txt; 80,854 entries) Error categories DEV (Device): PCI cards/peripherals errors MEM (Memory): memory-related non-correctable err. NET (Network): communication issues I-O (Input-Output): SCSI disk error/damaged sectors PRO (Processor): exceptions/machine-checking issues OTH (Other): remaining errors *NOTE: The data/analysis shown in this presentation has been obtained for a subset of 10,000 entries in the exercise log (logmercury-test.txt) p. 5

6 Examples of error entries [DEV] - PCI Component Error Detail: Component Info: VendorId=*, DeviceId=*, ClassCode=*,Seg/Bus/Dev/Func=* - BEGIN HARDWARE ERROR STATE AT CPE [MEM] - Mem Error Detail: Physical Address *, Address Mask:*, Node:*, Card:*, Module: *, Bank:*, Device:*,Row:*, Column:* [NET] - connection down [I-O] - error, dev *:* (hda), sector * - hda: packet command error: error=* [PRO] - Error Map: * NOTE: the same problem is reported multiple times in the log (redundant content)!!! $ head -n 10 logmercury-test.txt tg-c645 PRO tg-c645 +BEGIN HARDWARE ERROR STATE AT CMC tg-c645 PRO tg-c645 Device Error Info Section [ omissis ] tg-c645 PRO tg-c645 Device Error Info Section tg-c645 PRO tg-c645 Error Map: * tg-c238 DEV tg-c238 +BEGIN HARDWARE ERROR STATE AT CPE tg-c238 DEV tg-c238 +Platform PCI Component Error Info Section [ omissis ] p. 6

7 Tools needed for the analysis A suite of home-made BASH scripts (tuple count, grouping of the error entries; total 4 scripts) the basic scripts can be used sequentially to perform the FFDA of the log; the usage of the scripts is described in the presentation; students are free to modify/improve or to write further scripts to automate the analysis. Matlab & statistics toolbox (TTF estimation, curve fitting, ) p. 7

8 Tuple count as a function of CWIN The script tuplecount_func_cwin.sh analyzes how the tuple count varies for increasing values of CWIN in the interval [1;5000]sec The execution takes several min/hour depending on the number of entries in the log* The output (CWIN,tuple-count pairs) is saved in the tuplecount- <log_filename>.txt file $./tuplecount_func_cwin.sh logmercury-test.txt tentative-cwin.txt $ cat tuplecount-logmercury-test.txt CWIN size count [ omissis ] *around 1hour for the logmercury-test.txt log; estimated around 6/7hours for the exercise log (run with an Intel Core2 processor) p. 8

Determining the optimal CWIN The output file can be imported in Matlab to plot the tuple-count curve as follows >> load tuplecount-logmercury-test.

9 Determining the optimal CWIN The output file can be imported in Matlab to plot the tuple-count curve as follows >> load tuplecount-logmercury-test.txt >> plot(tuplecount_logmercury_test(:,1),tuplecount_logmercury_test(:,2),'-*b'); >> xlabel('cwin [s]');ylabel( count ); >> legend('tuple count'); Rule: a good CWIN is right after the knee of the curve CWIN: 240sec (4min) 276 tuples p. 9

10 Grouping the entries with the selected CWIN The script tupling_with_cwin.sh coalesces the entries with the selected CWIN (the script takes around 2min). Output (summary files + content of each individual tuple) is saved in the directory tupling_<log-filename>-<cwin> $./tupling_with_cwin.sh logmercury-test.txt 240 *** Grouping the entries for CWIN=240 *** - creating tuple # 1 - creating tuple # 2 [ ] $ cd tupling_logmercury-test-240/ $ ls interarrivals.txt tuple_16 tuple_30 tuple_45 lenghts.txt tuple_17 tuple_31 tuple_46 startingpoints.txt tuple_18 tuple_32 tuple_47 tuple_1 tuple_76 tuple_90 [ ] p. 10

txt) I 1, I 2,, I N-1 interarrivals (interarrivals.

11 Analyzing the output files S 1 S 2 S 3 S N L 1 I 1 L 2 I 2 L 3 I N-1 L N time tuple1 tuple2 tuple3 tuplen L 1, L 2,, L N tuple lenghts (lenghts.txt) I 1, I 2,, I N-1 interarrivals (interarrivals.txt) S 1, S 2,, S N timestamps of the init point of the tuples (1 col. of startingpoints.txt) * NOTE: each interarrival is a TTF sample * p. 11

TTF: empirical distribution The interarrivals.txt file is imported in Matlab to obtain the empirical CDF of the TTF of the system. >> load interarrivals.

12 TTF: empirical distribution The interarrivals.txt file is imported in Matlab to obtain the empirical CDF of the TTF of the system. >> load interarrivals.txt >> [y,t]=cdfcalc(interarrivals); >> empttf=y(2:size(y,1)); >> emprel=1-empttf; >> plot(t,empttf,'-*b',t,emprel,'-+r'); >> xlabel('time [s]'); ylabel('p'); >> legend('emp TTF', 'emp Rel'); Given t, TTF(t) is the probability that the system exhibits a failure after tsec since the last fail. occurred. empirical reliability p. 12

13 Fitting the empirical reliability The cftool (statistics toolbox) can be used to fit a theoretical distribution (threl) to the empirical (emprel) one ( Fitting button) >> cftool(t,emprel); Tentative type of distribution: exponential threl(t) = ae bt a 0 empirical reliability b<0 p. 13

14 1) Press New fit 2) Give a name to the fit 3) Select the type of fit (e.g., exponential) 4) Select the equation 5) Press Apply Using the Fit Editor 6) Save the results to Matlab workspace Coefficients (a and b) Quality of the fit (SSE, R-square) p. 14

15 Assessing the quality of the fit Is the theoretical distribution a good fit for the empirical one? Matlab indexes: SSE: Sum of Squares due to Errors (has to be close to 0) R-square: ability of the fit to explain the variation in the data threl1: SSE= R-square= Kolmogorov-Smirnov test (adopted method) p. 15

16 Applying the Kolmogorov-Smirnov test 1.) Estimate the maximum distance between emprel and threl, i.e., D = max( emprel-threl ) 2.) Check D against a critical value * D C = / N (N is the number of samples, i.e., empirical points of the curve we are trying to fit) 3.) Result of the test: - if D D C the test assumes a significance level 95%: we state that threl is NOT a good fit for emprel. - else threl is a reasonable fit for emprel. *The formula can be applied if N 100; otherwise D is checked agaist a table of values p. 16

17 K-S test applied to threl1 >> D=max(abs(empRel-thRel1(t))) D = threl1 has been saved to the workspace via the Fit Editor >> size(emprel) ans = The number of empirical samples can be obtained with size >> Dc=1.358/sqrt(255) Dc = D C is estimated as described before *threl1 is not a goot fit for emprel* p. 17

18 Determining a better fit Equation (same steps as for threl1): threl2(t) = ae bt + ce dt Coefficients (with 95% confidence bounds): a = (0.3531, ) b = ( , ) c = (0.6939, ) d = e-05 (-7.615e-05, e-05) >> D=max(abs(empRel-thRel2(t))) D = Dc is the same as before (0.0850) threl2 is a goot fit for emprel (NOTE: but not the only fit ) p. 18

19 Exercises Repeat the analysis for the logmercury-exe.txt file: tuple count plot + selected CWIN; grouping the entries with CWIN; empirical distributions; fit the empirical reliability (exponential model + other tentative distributions, e.g., weibull); analysis of the fit via the K-S test. NOTE: the analysis has been conducted for the system log (all the nodes together). Additional exercises: analysis of the failure distribution for the most entries-prone nodes & relationship with the system distribution; error categories against architectural layer (do the node in the same layer of the cluster exhibit similar error entries?) *** whatever additional considerations/results!!! *** p. 19

Obtaining the overview of the log The script logstatistics.sh provides an overview of the data in the log (useful to figure out entries-prone categ.

20 Obtaining the overview of the log The script logstatistics.sh provides an overview of the data in the log (useful to figure out entries-prone categ. and/or nodes) distribution of the entries by error category $./logstatistics.sh logmercury-exe.txt == Total error entries == == Breakup by CATEGORY == DEV MEM I-O 5547 NET 3702 PRO 1504 OTH 34 nodes with the largest number of entries == Breakup by NODE* == tg-c tg-master 4098 tg-c tg-s tg-c tg-c tg-c tg-login3 382 tg-c tg-c * only the 10 most occurring nodes are reported p. 20

Filtering the log The script filter.sh is then used to extract the entries (by node or category) from the system log. Output is saved in a file named <filtername>-<systemlog>.txt. 1.

21 Filtering the log The script filter.sh is then used to extract the entries (by node or category) from the system log. Output is saved in a file named <filtername>-<systemlog>.txt. 1.) execute the analysis (tuplecount, reliability, ) for the filtered log. 2.) logstatistics.sh applied to a single node returns the mostoccurring entries/errors for that node logfile $./filter.sh logmercury-exe.txt tg-master Filtering by NODE: tg-master... [DONE] $./logstatistics.sh tg-master-logmercury-exe.txt == Total error entries == 4098 == Breakup by CATEGORY == NET 3639 I-O 452 OTH 3 DEV 3 MEM 1 filtername == Breakup by NODE* == tg-master 4098 * only the 10 most occurring nodes are reported p. 21

Empirical Asset Pricing

Empirical Asset Pricing Department of Mathematics and Statistics, University of Vaasa, Finland Texas A&M University, May June, 2013 As of May 17, 2013 Part I Stata Introduction 1 Stata Introduction Interface Commands Command