A Measurement Systems Analysis of Total Shutter Open Time (TSOT)

Transcription

1 A Measurement Systems Analysis of Total Shutter Open Time (TSOT) Louis Tijerina Ford Motor Company ISO Workshop Ottawa October 2-3, 2006

2 Measurement System Analysis Test-Retest Reliability Start Split-Group Repeatability R 2.5? Yes Correlation To Driving No Stop No R 2.5? Yes No %.7? Statistical Discriminability Sorting Rules Keep Yes Prior Prediction

3 CAMP Visual-Manual Tasks Coins: Select specified amount (e.g., 65 ) from coins in a center-console cup holder HVAC: Adjust fan speed, temperature and vents to requested settings Radio ( Easy ): Tune radio to specified frequency given radio powered, on correct band Radio ( Hard ): Tune radio to specified frequency given radio off, incorrect band initially Manual Dial: Enter own area code, home phone number, and <Talk> into a flip phone Cassette: Remove a cassette from its case and insert Side B into dash-mounted player CD/Track 7: Remove a colored CD from visor-wallet, insert into car radio, select Track 7 Destination Entry: Enter address (city, state, street, number) into Magellan II Navi Route Tracing: Trace a route from a point of origin to a point of destination in a maze Read ( Easy ): Silently read 30-word text, 4 th - 5 th grade reading level; Cloze Procedure Read ( Hard ): Silently read 60-word text, 7 th - 8 th grade reading level; Cloze Procedure Map ( Easy ): Say relative orientation of two destinations on a paper map with 12 callouts Map ( Hard ): Say relative orientation of two destinations on a paper map with 22 callouts

4 Experimental Approach Phased testing of 234 licensed drivers Males and Females ranging from 21 to 79 years of age Each participant was tested in one and only one of three settings: In the lab (using surrogate methods) (N = 57) On-the-road (using instrumented vehicles) (N = 108) On a test track (using instrumented vehicles) (N = 69) In each setting, participants were asked to perform a variety of in-vehicle tasks, following formal training Participants were asked to perform each task twice

5 Occlusion Surrogate Method Settings: 1.5 s shutter open time/2.0 s closed cycle until task completed ISO standard (at the time testing was initiated) Each participant completed two, non-consecutive, trials for each visual-manual task Metric Total Shutter Open Time (TSOT) Note: Other surrogate methods were also applied during lab testing

6 Driving Test Approach Turn Signal Illumination CHMSL Illumination Deceleration of Lead Vehicle

7 Driving Performance Metrics Visual Allocation Glance Duration No. of Glances Glance Location Vehicle Control Lane Keeping Car Following Speed Control Object & Event Detection % Missed Events Response Times

8 Test-Retest Reliability Dest Entry Task: Rep 2 vs. Rep 1 TSOT OccTSOT_Dest = OccTSOT_Dest S = R-Sq = 50.8 % R-Sq(adj) = 49.7 % OccTSOT_Dest OccTSOT_Dest Observations: Test-Retest Reliability was generally low (R-sq < 0.30 for 10 of 13 tasks) This is consistent with other data in the human factors literature (Lane, Kennedy, & Jones, 1986)

9 Split-Group Repeatability Repeatability: Median Total Shutter Open Time (TSOT) 50 Mdn_TSOT_Grp1 = Mdn_TSOT_Grp0 S = R-Sq = 99.5 % R-Sq(adj) = 99.4 % Mdn_TSOT_Grp Mdn_TSOT_Grp0 Observation: TSOT results are repeatable when reps are averaged per participant and then summarized by task 30 40

10 Mdn_StaticTime Visual-Manual Tasks: (No DestEntry) Draftsman's Plot of Static Time, TSOT and Selected Track Measures Mdn_TSOT MdnTaskDur:Tk Mdn_SDLP:Tk Mdn_SpeedDiff:Tk PctLVDecelMiss:Trk MeanglncesTR:TK MeanduratTR:Tk Note: MeanduratTR:TK = Mean TGT for Track trials Observations: Median TSOT predicts (R 2 ~ 0.50 or more): Task time when driving; Std. Deviation of Lane Position (SDLP); Speed Difference between task start and end; TGT and Counts of Task-Related (TR) Glances away from the road scene

11 Discriminability Results for Visual-Manual Lab Tasks: Median TSOT Example Lower Workload Expected Higher Higher Workload Expected HVAC Radio(Easy) Radio(Hard) Cassette CD/Track 7 Coins ManualDial * * * * * Read(Easy) * * * * * Read(Hard) * * * * * * Map(Easy) * * * Map(Hard) * * * * * * Route Tracing * * * * * * DestEntry * * * * * * Note: * = Row Task value is greater than Column value, p <0.05 by one-tail sign test blank = not significantly different Percent Significance: 100*(37/42) = 88%

12 Prior Prediction: Motivation Problem: Workload can t be measured directly Real driver distraction with CAMP DWM tasks unknown Solution: Predictions of relative task workload that are independent of any CAMP data Predictions based on Literature (both basic and prior driver research, based on the preponderance the findings) and Theory (Multiple Resources Theory) CAMP quantitative task modeling Quantitative models developed from basic cognitive literature (e.g., Card, Moran, and Newell Human Information Processor) and Motion Time Study Literature (e.g., MTM) Quantitative models also included an MRT model developed by CAMP from Sarno & Wickens (1995) Conflict Matrix Staff engineering judgment based on research experience

13 Prior Predictions of Lower vs. Higher Task Workload Task Type Visual-Manual Auditory-Vocal Relative Workload Lower Higher HVAC Radio(Easy) Radio(Hard) Cassette CD/Track 7 Coins ManualDial Read(Easy) Read(Hard) Map(Easy) Map(Hard) Route Tracing DestEntry Just Drive (no load) Sports Broadcast Book-on on-tape Listen Bio Q&A BookOnTape /Summary Route Instruction Route Orient Travel Comp No order within categories Notes: Only Two Categories (No within-category task ordering) OEM acceptance testing is often binary Fine-grained predictions are not within the state-of-the-art Relative Workload Relation to natural driving phenomena is very indirect and poorly understood Only higher vs. lower workload estimation is possible at this time

14 Cluster Analysis of Visual-Manual Tasks: Based on OWL & Multitasking Difficulty Ratings Together Distance Lower Workload Higher Workload 0.00 Coins Radio(Hard) Cassette HVAC Tasks Observations: Test Participants reported subjective workload with Operator Workload (OWL) scale and MultiTasking Difficulty Magnitude Estimation Cluster analysis shows tasks sorted consistent with Higher vs. Lower workload predictions Radio(Easy) CD/Trk 7 Manual Dial Map(Easy) Route Tracing Read(Easy) DestEntry Read(Hard) Map(Hard)

15 Prior Prediction: Classification Errors for 7 TSOT Criteria 13 CAMP Tasks on Test Track Rule 1 Rule 2 Rule 3 Rule 4 Rule 5 Rule 6 Rule 7 A B C D E F Prior Mean 85%-ile Mean 85%-ile Mean 85%-ile A > 15 s B > 15 s C > 15 s D > 15 s C > 7.5 s D > 7.5 s F>1.0 Static Static Prediction Task Type Time Time TSOT TSOT R R Radio("Easy") Lower Lower Lower Lower Lower Lower Lower Higher* HVAC Lower Lower Lower Lower Lower Lower Lower Lower Cassette Lower Lower Higher* Lower Lower Lower Higher* Lower Coins Lower Lower Lower Lower Lower Lower Higher* Higher* Radio("Hard") Lower Lower Lower Lower Lower Lower Higher* Lower CD/Track7 Lower Lower Higher* Lower Lower Higher* Higher* Lower Map("Easy") Higher Lower* Lower* Lower* Lower* Higher Higher Lower* Read("Easy") Higher Lower* Higher Lower* Lower* Higher Higher Lower* Manual Dial Higher Higher Higher Lower* Lower* Higher Higher Lower* Route Tracing Higher Lower* Higher Lower* Lower* Higher Higher Higher Read("Hard") Higher Higher Higher Lower* Higher Higher Higher Lower* Map("Hard") Higher Lower* Higher Lower* Higher Higher Higher Higher DestEntry Higher Higher Higher Higher Higher Higher Higher Lower* Classification 2 FP 2 FP 1 FP 4 FP Errors: 4 FN 1 FN 6 FN 4 FN 5 FN Observations: Rule 5 was best (1 False Positive) This type of analysis is needed in order to develop criteria

16 Summary TSOT: Measurement System Analysis Results* Test-Retest Reliable? No Repeatable? Yes Statistically Discriminating? Yes Predictive? Yes for: Task Completion Time while driving; SDLP; SpeedDiff; Total Glance Time (TGT) away from the road (task-related), Number of Glances away from the road (task-related) Prior Prediction (needed to avoid circularity): Plausible but exploratory Sorting Rules: Categorization Errors with various rules identified * Based on CAMP Driver Workload Metrics Project tasks, methods, data, and analysis. Obtained results may differ for other applications.

17 End of Slides