5/20/ Advanced Analysis Methodologies

Transcription

1 5/20/205- Advanced Analyss Methodologes

2 Advanced Methodologes Censored Data Convertng to contnuous data often presents analyss challenges For eample, f we use detecton range, how do we account for nondetects n the analyss Censored data provdes a soluton Generalzed Lnear Models System performance s often best characterzed by non-normal data» Tme» Accuracy» Pass/Fal Generalzed lnear models provde a more fleble analyss framewor to handle these non-normal outcomes. Bayesan Methodologes Allow for the ncorporaton of multple sources of nformaton, when t s approprate Provde methodologes for fndng confdence ntervals when there are zero observatons 5/20/205-2

3 Motvatng Eample: Submarne Detecton Tme System Descrpton Sonar system replca n a laboratory on whch hydrophone-level data, recorded durng real-world nteractons can be played bac n real-tme. System can process the raw hydrophone-level data wth any desred verson of the sonar software. Upgrade every two years; test to determne new verson s better Advanced Processor Buld (APB) 20 contans a potental advancement over APB 2009 (new detecton method capablty) 5/20/205-3 Response Varable: Detecton Tme Tme from frst appearance n recordngs untl operator detecton» Faled operator detectons resulted n rght censored data Factors: Operator profcency (quantfed score based on eperence, tme snce last deployment, etc.) Submarne Type (SSN, SSK) System Software Verson (APB 2009, APB 20) Array Type (A, B) Target Loudness (Quet, Loud)

4 Detecton Tme Dstrbuton Detecton tme does not follow a normal dstrbuton Normal Dstrbuton Lognormal Dstrbuton 5/20/205-4

5 Faled Detecton Opportuntes Not all runs resulted n a successful detectons 5/20/205-5

6 Submarne Detecton Tme: Analyss Advanced statstcal modelng technques ncorporated all of the nformaton across the operatonal space. Generalzed lnear model wth log-normal detecton tmes Censored data analyss accounts for non-detects All factors were sgnfcant predctors of the detecton tme Factor/Model Term Descrpton of Effect P-Value Recognton Factor Increased recognton factors resulted n shortened detecton tmes APB Detecton tme s shorter for APB Target Type Detecton tme s shorter for SSN targets Target Nose Level Detecton tme s shorter for loud targets Array Type Detecton tme s shorter for Array B Type* Nose Type* Array Addtonal model terms mprove predctons. Thrd order nteracton s margnally sgnfcant, Nose*Array therefore all second order terms are retaned Type* Nose*Array /20/205-6

7 Submarne Detecton Tme: Results Shows mpact of censored data Medan detecton tmes show a clear advantage of APB- over the legacy APB Confdence nterval wdths reflect weghtng of data towards APB- Statstcal model provdes nsghts n areas wth lmted data Note medan detecton tme n cases wth heavy censorng s shfted hgher 5/20/205-7

8 Introducton to Censored Data Analyss Censored data = we ddn t observe the detecton drectly, but we epect t wll occur f the test had contnued We cannot mae an eact measurement, but there s nformaton we can use. The no detects are on the tal of the dstrbuton! Same concept as a tme-termnated relablty trals (falure data) Run No. Result Result Code Detected Target 2 Detected Target 3 No detect 0 4 Detected Target 5 Detected Target 6 Detected Target 7 No detect 0 8 No detect 0 9 Detected Target 0 Detected Target Correspondng Tmelnes = Detect = No-Detect Run No. Tme of Detecton (hours after COMEX) > >6.2 8 > /20/205-8

9 Parameterzng Data Assume that the tme data come from an underlyng dstrbuton, such as the log-normal dstrbuton Other dstrbutons may apply you must consder carefully. See slde 4 where we dd t for the submarne detecton data That parameterzaton wll enable us to ln the tme metrc to the probablty of detecton metrc. Probablty Densty /20/205-9 Probablty Densty Functon Log-Normal Probablty Densty Functon Tme-to-Detect (hr) Cummulatve Probablty to Detect Cumulatve Dstrbuton Functon (CDF) Cummulatve Dstrbuton Functon (CDF) Tme-to-Detect (hr) Tme-to-Detect (hours)

10 Parameterzng Data Eample: Arcraft must detect the target wthn t s nomnal tme on staton (6-hours) Bnomal metrc was detect/non-detect wthn tme-on-staton If we determne the shape of ths curve (.e., determne the parameters of the PDF/CDF), we can use the tme metrc to determne the probablty to detect! Probablty Densty /20/205-0 Probablty Densty Functon Log-Normal Probablty Densty Functon Tme-to-Detect (hr) Cummulatve Probablty to Detect Cumulatve Dstrbuton Functon (CDF) Cummulatve Dstrbuton Functon (CDF) Tme-to-Detect (hr) Tme-to-Detect (hours)

11 Conceptualzng the Censored-Data Ft For non-censored measurements, the PDF ft s easy to conceptualze For censored measurements, the data can t defne the PDF, but we now they contrbute to the probablty densty beyond the censor pont Eample event from an OT: No Detects (Detect Tme > 6 hours) le somewhere on the tal of the dstrbuton. Detect wll eventually occur sometme after 6 hours, pushng the dstrbuton curve to the rght Mathematcally, there are ways of calculatng the shfted dstrbuton. Cumulatve Probablty to Detect Tme to Detect (hours) Includng a bunch of censored (Tme > 6 hour) events wll push the CDF to the rght (see how probablty to detect s lower at 6 hours) Cumulatve Probablty to Detect Tme to Detect (hours) 5/20/205-

12 Characterzng Performance wth Censored Data Now let s employ DOE Consder a test wth 6 runs Two factors eamned n the test Run Matr: Target Fast Target Slow Totals Test Locaton Test Locaton Detecton Results: Target Fast Target Slow Totals Test Locaton 3/4 4/4 7/8 (0.875) Test Locaton 2 3/4 /4 4/8 (0.5) 6/8 (0.75) 5/8 (0.63) 5/20/205-2

13 Attempt to Characterze Performance As epected, 4 runs n each condton s nsuffcent to characterze performance wth a bnomal metrc Cannot tell whch factor drves performance or whch condtons wll cause the system to meet/fal requrements Lely wll only report a roll-up of /6 90% confdence nterval: [ 0.45, 0.87 ] 5/20/205-3

14 Characterzng Performance Better Measure tme-to-detect n leu of bnomal metrc, employ censored data analyss Sgnfcant reducton n confdence ntervals! Now can tell sgnfcant dfferences n performance» E.g., system s performng poorly n Locaton 2 aganst slow targets We can confdently conclude performance s above threshold n three condtons» Not possble wth a probablty to detect analyss! 5/20/205-4

15 Censored Data Analyss Summary Many bnary metrcs can be recast usng a contnuous metrcs Care s needed, does not always wor, but Cost savng potental s too great not to consder t! Wth Censored-data analyss methods, we retan the bnary nformaton (non-detects), but gan the benefts of usng a contnuous metrc Better nformaton for the warfghter Mantans a ln to the Probablty of requrements Convertng to the censored-contnuous metrc mamzes test effcency In some cases, as much as 50% reducton n test costs for near dentcal results n percentle estmates Beneft s greatest when the goal s to dentfy sgnfcant factors (characterze performance) 5/20/205-5

16 Generalzed Lnear Models Overvew There are many classes of statstcal models: General lnear models (normal dstrbuton) Generalzed lnear models (Eponental famly)» Provdes a smplfed framewor for numercally mamzng the lelhood Locaton-scale regresson (locaton scale, log-locaton scale) Nonlnear regresson (almost everythng else) These regresson analyses are a logcal etenson of standard statstcal regresson analyss However, methods presented here are more general Data not necessarly normal Data may not have constant varance Lnd between data and response may not be lnear Practcal T&E problems often cannot be solved wth straghtforward regresson analyss 5/20/205-6

17 5/20/205-7 Model Specfcaton: GLM versus Generalzed Lnear Model General Lnear Model (e.g., regresson) Model: Where, s the number of factors and h.o.t. are hgher order terms. Generalzed Lnear Model Model:... ), ( ~ ) ( 2 0 t h o Normal y f j j j h o t g f Y E EponentalFamlyDstrbuton y f j j j.. ), ( ) ( ), ( ~ ) ( 2 0 g - () s the nverse ln functon t lterally lns the factors to the epected value of the response

18 Eponental Famly Logstc Regresson s a Generalzed Lnear Model Class of dstrbutons that provdes the bass for Generalzed Lnear Models Dstrbutons nclude: Contnuous» Normal» Log-normal» Beta» Gamma» Eponental Dscrete:» Bnomal/Bernoull» Posson» Negatve Bnomal And several more! Gamma Dstrbuton Beta Dstrbuton Provde fleble shapes that can be used to descrbe almost any type of data! 5/20/205-8

19 Pass/Fal Analyss: A Second Motvatng Eample System s goal s to mantan a loc on a movng target Response Varable: Mantan trac? (Yes/No) Debatable f a contnuous metrc could have replaced ths bnary response. However, no contnuous metrc was traced durng the test, so we are stuc analyzng pass/fal response. Factors: Target Sze (small/large) Target Speed (slow/fast) Tme of Day (day/nght) Target Aspect (frontal/quarter) Maneuverng (yes/no) Generalzed lnear models can be used to ft logstc and probt regresson under the same framewor! 5/20/205-9

20 5/20/ Generalzed Lnear Model: Brea Loc? Logstc Regresson Model: h o t h o t np p n Bnomal y f j j j j j j.. ep.. ep ), ( ~ ) ( * In JMP: Ft Model Generalzed Lnear Model Bnomal Dstrbuton Logt Ln

21 Summarzng Results Day Nght 80% Confdence Intervals Shown.0 Probablty of Mantanng Trac Large Targets Small Targets Fast Large Targets Slow Large Targets Fast Small Targets Slow Small Targets 5/20/205-2

22 Parametrc Statstcal Model Herarchy There s a model for every stuaton! Normalty Homoscedastcty Independence Lnearty One framewor Normalty Homoscedastcty Independence Lnearty +More Fleble Forms Independence +Random Effect Independence Regresson/ANOVA General Lnear Models Generalzed Lnear Models Locaton-Scale Models, Non-parametrc Models Generalzed Lnear Med Models 5/20/ for Bayesan versons of these model forms, whch can also ncorporate pror nowledge Note, Bayesan methodologes can mae analyss easer by avodng the comple optmzaton problem.

23 Bayesan Methodology Overvew Model for Data Lelhood L(data θ) Classcal Statstcs Inference Data Posteror f(θ data) Pror f(θ) The ncluson of the pror dstrbuton allows us to ncorporate dfferent types of nformaton n the analyss 5/20/205-23

24 Motvatng Eample: Stryer Relablty Analyss Statstcal methods (ncludng DOE) apply to relablty data as well as performance data Stryer Retrospectve Case Study Infantry Carrer Vehcle (ICV) - the nfantry/msson-vehcle type Base vehcle for eght separate confguratons IOT&E Results: Stryer Relablty by Varant usng Operatonal Test Data Vehcle Varant Total Mles Drven System Aborts MMBSA MMBSA 95% LCL Results do not leverage DT data or relatonshps between vehcles MMBSA 95% UCL Anttan Guded Mssle Vehcle (ATGMV) Commander s Vehcle (CV) Engneer Squad Vehcle (ESV) Fre Support Vehcle (FSV) Infantry Carrer Vehcle (ICV) Mortar Carrer Vehcle (MCV) Medcal Evacuaton Vehcle (MEV) Reconnassance Vehcle (RV) Total /20/205-24

25 The Stryer Relablty Data Set Developmental Testng Operatonal Testng Mles Before System Abort Eact Falure Rght Censored Vehcle Type 5/20/205-25

26 Bayesan Analyss for Incorporatng Developmental Test Informatve Prors Based on subject matter epertse (there wll be a degradaton n OT relablty)» Data s already ncluded n model Herarchcal Models Assumes the parameters are related, the data tells us how closely related Herarchcal models for the Stryer case study allow us to estmate MEV relablty based on other data Bayesan Analyss Model: ~ ~ /,2,, 8 (vehcle varants ncludng MEV) ~, ~, ~.00,. 00 ~.00,. 00 5/20/205-26

27 Stryer Relablty Results Mles Operatonal Test MMBSA Estmates (95% Confdence Intervals)? ATGMV CV ESV FSV ICV MCV RV MEV Tradtonal Analyss Eponental Regresson Bayes Non Informatve Bayes Informatve Tradtonal Approach: Mles MMBSA # Falures Etremely wde confdence ntervals Results n unrealstc estmates for the Commander s Vehcle Eponental Regresson Approach & Bayesan Approaches MMBSA f TestPhase Varant Allows for a degradaton n MMBSA from DT to OT (ncreases could occur as well). Leverages all nformaton» Better estmates of MMBSA» Tghter confdence ntervals 5/20/205-27

28 Bayesan Methods Summary Provde very fleble analyss methods Prors allow us to consder other types of data, basng decsons on all avalable nformaton about a system Methods can easly be etended to ncorporate other stuatons: Kll chan analyss Comple system structures relablty analyss Incorporate any relevant pror testng, modelng and smulaton, or engneerng analyss 5/20/205-28