Quantitative Vulnerability Assessment of Systems Software

Transcription

1 Quantitative Vulnerability Assessment of Systems Software Omar H. Alhazmi Yashwant K. Malaiya Colorado State University

2 Motivation Vulnerabilities: defect which enables an attacker to bypass security measures [Schultz et al: 7] For defects: Reliability modeling and SRGMs have been around for decades. Assuming that vulnerabilities are a special faults will lead us to this question: To what degree reliability terms and models are applicable to vulnerabilities and security? [Littlewood et al.:14]. The need for quantitative measurements and estimation is becoming more crucial.

3 Outline of Our Goals Developing a quantitative model to estimate vulnerability discovery. Using calendar time. Using equivalent effort. Validate these measurements and models. Testing the models using available data Identify security Assessment metrics Vulnerability density Vulnerability to Total defect ratio

4 Time vulnerability discovery model What factors impact the discovery process? The changing environment The share of installed base. Global internet users. Discovery effort Discoverers: Developer, White hats or black hats. Discovery effort is proportional to the installed base over time. Vulnerability finders reward: greater rewards, higher motivation. Security level desired for the system Server or client

5 Vulnerabilities Time vulnerability discovery model Each vulnerability is recorded. Available [ICAT, Microsoft]. Needs compilation and filtering. Data show three phases for an OS. Assumptions: The discovery is driven by the rewards factor. Influenced by the change of market share. Phase 1 Phase 2 Phase 3 Time

6 Vulnerabilities Time vulnerability Discovery model 3 phase model S-shaped model. Phase 1: Knowledge low. Installed base low. Phase 2: Knowledge high. Installed base high and growing. Phase 3: Knowledge-high. Installed base dropping. dy dt y Ay( B y) BCe Vulnerability time growth model Time B ABt 1

7 Vulnerabilities Time based model: Windows Fitted curve Windows 98 Total vulnerabilites Windows 98 A B C χ χ 2 critial Jan-99 Mar-99 May-99 Jul-99 Sep-99 Nov-99 Jan-00 Mar-00 May-00 Juḻ 00 Sep-00 Nov-00 Jan-01 Mar-01 May-01 Jul-01 Sep-01 Nov-01 Jan-02 Mar-02 May-02 Juḻ 02 Sep-02 P-value 1-7.6x10-11

8 Vulnerabilities Time based model: Windows NT Total vulnerabilities Windows NT 4.0 Fitted curve Windows NT Aug-96 Dec-96 Apr-97 Aug-97 Dec-97 Apr-98 Aug-98 Dec-98 Apr-99 Aug-99 Dec-99 Apr-00 Aug-00 Dec-00 Apr-01 Aug-01 Dec-01 Apr-02 Aug-02 Dec-02 Apr-03 A B 136 C χ χ 2 critial P-value

9 Installed Base Percentage Millions of users Usage vulnerability Discovery model The data: The global internet population. The market share of the system during a period of time. Equivalent effort The real environment performs an intensive testing. Malicious activities is relevant to overall activities. n Defined as E ( U ) i 0 i Pi Dec., 1995 Dec., Dec., 1997 Dec., Internet Growth Dec., 1999 Mar Jul., 2000 Dec., 2000 Mar., 2001 Jun., The percentage of the market share of O.S Aug., 2001 Apr Jul., 2002 Sep., 2002 Mar., 2003 Sep., 2003 Oct., 2003 Dec., 2003 Feb., 2004 May, 2004 Windows 95 Windows 98 Windows XP Windows NT Windows 2000 Others May-99 Aug-99 Nov-99 Feb-00 May-00 Aug-00 Nov-00 Feb-01 May-01 Aug-01 Nov-01 Feb-02 May-02 Aug-02 Nov-02 Feb-03 May-03 Aug-03 Nov-03 Feb-04 May-04

10 Vulnerabilities Usage vulnerability Discovery model The model: y B( 1 e E vu ) Exponential growth with effort. The basic reliability model [Musa:21] Time is eliminated Usage (Million user's months)

11 Vulnerabilities Effort-based model: Windows Windows 98 Actual Vulnerabilities Fitted curve Windows B 37 λ vu χ χ 2 critial P-value 1-3.3x10-11 Usage (Million user's months)

12 Vulnerabilities Effort-based model: Windows NT 4.0 Windows NT 4.0 Actual Vulnerability Fitted Win NT B 108 λ vu ` χ χ 2 critial P-value Usage (Millions users months)

13 Vulnerabilities Discussion Windows Fitted curve Total vulnerabilites Excellent fit for Windows 98 and NT 4.0. Model fits data for all OSs examined. Deviation from the model caused by overlap: Windows 98 and Windows XP Windows NT 4.0 and Windows 2000 Vulnerabilities in shared code may be detected in the newer OS. Need: approach for handling such overlap Jan-99 Mar-99 May-99 Jul-99 Sep-99 Nov-99 Jan-00 Mar-00 May-00 Juḻ 00 Sep-00 Nov-00 Jan-01 Mar-01 May-01 Jul-01 Sep-01 Nov-01 Jan-02 Mar-02 May-02 Juḻ 02 Sep-02

14 Vulnerability density and defect density Defect density Valuable metric for planning test effort Used for setting release quality target Limited defect density data is available Vulnerabilities are a class of defects Vulnerability data is in the public domain. Is vulnerability density a useful measure? Is it related to defect density? Vulnerabilities = 5% of defects [Longstaff: 20]? Vulnerabilities = 1% of defects [Anderson]? Can be a major step in measuring security.

15 Vulnerability density and defect density Vulnerability densities: 95/98: NT/2000: V KD /D KD : % less than 5% System MSLOC Known Defects (1000s) D KD (/Kloc) Known Vulner - abilies V KD (/Kloc) Ratio V KD /D KD Win % NT % Win % Win % Win XP * 2.66* %* * The number of defects for Windows XP is for the beta version.

16 Vulnerability / KLOC Defect/KLOC Vulnerability density and defect density Vulnerability densities: 95/98: NT/2000: V KD /D KD : % Vulnerability Density Defect Density Win 95 NT 4.0 Win 98 Win2000 Win XP 0 Win 95 NT 4.0 Win 98 Win2000 Win XP

17 Vulnerability / KLOC Defect/KLOC Vulnerability density and defect density Vulnerability densities: 95/98: NT/2000: V KD /D KD : % Vulnerability Density Defect Density Win 95 NT 4.0 Win 98 Win2000 Win XP 0 Win 95 NT 4.0 Win 98 Win2000 Win XP

18 Results and conclusions Vulnerability Discovery models Time-based model Effort-based model Data fits the models Vulnerability density evaluation Expected ranges of values Future work: Modeling impact of shared code Validation. Reward analysis Other risk factors: patches, vulnerability exploitation

19 Summary and conclusions We have introduced: Models: Time vulnerability model. Usage vulnerability model. Both models shown acceptable goodness of fit. Chi-square test. Measurements: vulnerability density. Vulnerability density vs. defect density.

20 Vulnerability Discovery in Multi-Version Software Systems Jinyoo Kim, Yashwant K. Malaiya, Indrakshi Ray {jyk6457, malaiya, iray

21 Outline Motivation for this study Related Work & Data Sources Vulnerability Discovery Models (VDMs) Software Evolution Multi-version Software Discovery Model Apache, Mysql and Win XP data Conclusions and Future Work 21

22 Vulnerability Discovery Models Describe vulnerability discovery against time Security Maintenance Management Estimating the number of vulnerabilities Patch development planning Guiding Test Effort Applicable to categories (causes and severity levels) Software Risk evaluation Combine with vulnerability exploitation and attack surface 22

23 Motivation for Multi-version VDMs Superposition effect on vulnerability discovery process due to shared code in successive versions. Examination of software evolution: impact on vulnerability introduction and discovery Other factors impacting vulnerability discovery process not considered before 23

24 Related Work Software reliability growth models Logarithmic-Poisson Reliability Model (Musa 84) Vulnerability Discovery Process Quadratic and Linear Models (Rescorla 05) Thermodynamic Model (Anderson 01) Logistic (AML) and Effort-based models (Alhazmi ) Software Evolution Trend of software Evolution (Eick 01) Application of Reliability Growth Model Reliability growth using OpenBSD (Ozment 06) 24

25 Software Evolution The modification of software during maintenance or development: fixes and feature additions. Influenced by competition Code decay and code addition introduce new vulnerabilities Successive version of a software can share a significant fraction of code. 25

26 LOC (Lines of Code) LOC (Lines of Code) Software Evolution: Apache & Mysql Version Number a a a Version Number Initial Code Added Code Initial Code Added Code Modification: Apache 43%, Mysql 31% 26

27 Jun-98 Jun-99 Jun-00 Jun-01 Jun-02 Jun-03 Jun-04 Jun-05 Jun-06 Oct-01 Feb-02 Jun-02 Oct-02 Feb-03 Jun-03 Oct-03 Feb-04 Jun-04 Oct-04 Feb-05 Jun-05 Oct-05 Feb-06 Jun-06 Oct-06 Percentage Vulnerabilities Vulnerability Discovery & Evolution: Apache & Mysql Apache Mysql DBMS 120% 120% 100% 100% 80% 80% 60% 60% 40% 20% c 40% 20% 0% 0% Release Date Release Date Added Code in Next Version Reliability Growth Code increasing Vulnerability Discovery Some vulnerabilities are in added code, many are inherited from precious versions. 27

28 Vulnerability Discovery rate Code Sharing & Vulnerabilities Observation Vulnerability increases after saturation in AML modeling Multiple Software Vulnerability Discovery Trend Accounting for Superposition Effect Shared components between several versions of software Calendar Time 1st Version 2nd Version Shared part Total Version Trend Total Version Trend 28

29 Vulnerability Discovery rate Multi-version Vulnerability Discovery Model Multiple Software Vulnerability Discovery Trend 1st Version Shared part Total Version Trend Calendar Time 2nd Version Total Version Trend B B' ( t) ABt A' B'( t ) BCe 1 B' C' e 1 Cumulative MVDM : fraction of code used in next Previous version Version Apache Mysql ( ) ( ) Next Version ( ) ( ) Shared Code Ratio α 20.16% 83.52% 29

30 Number of Vulnerability Cumulative Vulnerability One vs Two Humps One-humped Vulnerability Discovery Model Calendar Time Calendar Time Superposition affect 30

31 Vulnerability Rate Vulnerability Number Multi-version Vulnerability Discovery Model One-humped Vulnerability Discovery One-humped Vulnerability Discovery Trend Calendar Time 1st Version 2nd Version Shared Total Calendar Time 1st version Shared Total May result in a single hump with prolonged linear period 31

32 Seasonality in Vulnerability Discovery in Major Software Systems HyunChul Joh Yashwant K. Malaiya Dean2026 Department of Computer Science Colorado State University

33 Background Vulnerability: a defect which enables an attacker to bypass security measures [1] Vulnerability Discovery Model (VDM): a probabilistic methods for modeling the discovery of software vulnerabilities [2] Spans a few years: introduction to replacement Seasonality: periodic variation well known statistical approach quite common in economic time series Biological systems, stock markets etc. Halloween indicator: Low returns in May-Oct. [1]Schultz, Brown, and Longstaff, Responding to Computer Security Incidents [2]Ozment Improving vulnerability discovery models, 2007

34 Number of Vulnerabilities (Cumulative) Motivation (Visual Observation) Windows NT cumulative AML each month 0 Calendar Time 34

35 Examining Seasonality Is the seasonal pattern statistically significant? Periodicity of the pattern Analysis: Seasonal index analysis with test Autocorrelation Function analysis Significance Enhance VDMs predicting ability 35

36 Data Sets Data Sets to be analyzed here Windows NT Internet Information Services (IIS) server Internet Explorer (IE) National Vulnerability Database (NVD) [3] U.S. government repository of vulnerability management data collected and organized using specific standards 36 [3] National Institute of Standards and Technology. National Vulnerability Database.

37 Percentage Prevalence in Month Vulnerabilities Disclosed WinNT 95~ 07 IIS 96~ 07 IE 97~ 07 Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Total Mean s.d Percentage of Vuln. for Month Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Month Win NT I I S Internet Explorer 37

38 Seasonal Index Seasonal Index Values WinNT IIS IE Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec p-value 3.04e e e-6 Seasonal index: measures how much the average for a particular period tends to be above (or below) the expected value H 0 : no seasonality is present. We will evaluate it using the monthly seasonal index values given by [4]: where, s i is the seasonal index for i th month, d i is the mean value of i th month, d is a grand average 38 [4] Hossein Arsham. Time-Critical Decision Making for Business Administration. Available: edu/ntsbarsh/business-stat/stat-data/forecast.htm#rseasonindx

39 Autocorrelation function (ACF) Plot of autocorrelations function values With time series values of z b, z b+1,, z n, the ACF at lag k, denoted by r k, is [5]:, where 39 Measures the linear relationship between time series observations separated by a lag of time units Hence, when an ACF value is located outside of confidence intervals at a lag t, it can be thought that every lag t, there is a relationships along with the time line [5] B. L. Bowerman and R. T. O'connell, Time Series Forecsting: Unified concepts and computer implementation. 2nd Ed., Boston: Duxbury Press, 1987

40 Autocorrelation (ACF):Results Expected lags corresponding to 6 months or its multiple would have their ACF values outside confidence interval Upper/lower dotted lines: 95% confidence intervals. An event occurring at time t + k (k > 0) lags behind an event occurring at time t. Lags are in month. 40

41 Conclusion / Future Work The results show strong seasonality in systems examined, with higher discovery rates in some months. This needs to be taken into account for making accurate projections. Study of diverse software products, commercial and open-source, to identify causes of seasonality and possible variation across software systems. 41

42 Return Halloween Indicator Also known as Sell in May and go away Global ( ): Nov.-April: 12.47% ann., st dev 12.58% 12-months:10.92%, st. dev % 36 of 37 developing/developed nations Data going back to 1694 No convincing explanation January February March April May June July August September October November December Jacobsen, Ben and Bouman, Sven,The Halloween Indicator, 'Sell in May and Go Away': Another Puzzle(July 2001). Available at SSRN:

43 References O. H. Alhazmi, Y. K. Malaiya, I. Ray, " Measuring, Analyzing and Predicting Security Vulnerabilities in Software Systems," Computers and Security Journal, Volume 26, Issue 3, May 2007, Pages J. Kim, Y. K. Malaiya and I. Ray, "Vulnerability Discovery in Multi-Version Software Systems," Proc. 10th IEEE Int. Symp. on High Assurance System Engineering (HASE), Dallas, Nov. 2007, pp H. Joh and Y. K. Malaiya, "Seasonal Variation in the Vulnerability Discovery Process, " Proc. 2nd IEEE Int. Conf. Software Testing, Verification, and Validation, April 2009, pp Guido Schryen, Is open source security a myth? What do vulnerability and patch data say?, Communications of the ACM, May 2011, vol. 54, no. 5, pp