DETECTING ERRORS AND IMPUTING MISSING DATA FOR SINGLE LOOP SURVEILLANCE SYSTEMS

Transcription

1 DETECTING ERRORS AND IMPUTING MISSING DATA FOR SINGLE LOOP SURVEILLANCE SYSTEMS Chao Chen Department of Electrcal Engneerng and Computer Scence Unversty of Calforna, Berkeley CA Tel: (510) Jamyoung Kwon Department of Statstcs Unversty of Calforna, Berkeley CA Tel: (510) , Fax: (510) John Rce Department of Statstcs Unversty of Calforna, Berkeley CA Tel: (510) , Fax: (510) Alexander Skabardons Insttute of Transportaton Studes Unversty of Calforna, Berkeley CA Tel: (510) , Fax: (510) Pravn Varaya Department of Electrcal Engneerng and Computer Scence Unversty of Calforna, Berkeley CA Tel: (510) For Presentaton and Publcaton 82 nd Annual Meetng Transportaton Research Board January 2003 Washngton, D.C. November 15, 2002

2 Chen/Kwon/Skabardons/Varaya 1 No WORDS: 4627 Plus 7 Fgures (1750) Plus 4 Tables (1000) TOTAL: 7377 Correspondng Author

3 Chen/Kwon/Skabardons/Varaya 2 ABSTRACT Sngle loop detectors provde the most abundant source of traffc data n Calforna, but loop data samples are often mssng or nvald. We descrbe a method that detects bad data samples and mputes mssng or bad samples to form a complete grd of clean data, n real tme. The dagnostcs algorthm and the mputaton algorthm that mplement ths method are operatonal on 14,871 loops n sx Dstrcts of the Calforna Department of Transportaton. The dagnostcs algorthm detects bad (malfunctonng) sngle loop detectors from ther volume and occupancy measurements. Its novelty s ts use of tme seres of many samples, rather than basng decsons on sngle samples, as n prevous approaches. The mputaton algorthm models the relatonshp between neghborng loops as lnear, and uses lnear regresson to estmate the value of mssng or bad samples. Ths gves a better estmate than prevous methods because t uses hstorcal data to learn how pars of neghborng loops behave. Detecton of bad loops and mputaton of loop data are mportant because they allow algorthms that use loop data to perform analyss wthout requrng them to compensate for mssng or ncorrect data samples.

4 Chen/Kwon/Skabardons/Varaya 3 INTRODUCTION Loop detectors are the best source of real tme freeway traffc data today. In Calforna, these detectors cover most urban freeways. Loop data provde a powerful means to study and montor traffc (2). But the data contan many holes (mssng values) or bad (ncorrec values and requre careful cleanng to produce relable results. Bad or mssng samples present problems for any algorthm that uses the data for analyss. Therefore, we need both to detect when data are bad and throw them out, and to fll holes n the data wth mputed values. The goal s to produce a complete grd of relable data. We can trust analyses that use such a complete data set. We need to detect bad data from the measurements themselves. The problem was studed by the FHWA, Washngton DOT, and others. Exstng algorthms usually work on the raw 20-second or 30- second data, and produce a dagnoss for each sample. But t s very hard to tell f a sngle 20-second sample s good or bad unless t s very abnormal. Fortunately, loop detectors don t just gve random errors some loops produce reasonable data all the tme, whle others produce suspect data all the tme. By examnng a tme seres of measurements one can readly dstngush bad behavor from good. Our dagnostcs algorthm examnes a day s worth of samples together, producng convncng results. Once bad samples are thrown out, the resultng holes n the data must be flled wth mputed values. Imputaton usng tme seres analyss has been suggested before, but these mputatons are only effectve for short perods of mssng data; lnear nterpolaton and neghborhood averages are natural mputaton methods, but they don t use all the relevant data that are avalable. Our mputaton algorthm estmates values at a detector usng data from ts neghbors. The algorthm models each par of neghbors lnearly, and fts ts parameters on hstorcal data. It s robust, and performs better than other methods. We frst descrbe the data and types of errors that are observed. We then survey current methods of error detecton, whch operate on sngle 20-second samples. Then we present our dagnostc algorthm, and show that t performs better. We then present our mputaton algorthm, and show that ths method s better than other mputaton methods such as lnear nterpolaton. DESCRIPTION OF DATA The freeway Performance Measurement System (PeMS) (1,2) collects, stores, and analyzes data from thousands of loop detectors n sx dstrcts of the Calforna Department of Transportaton (Caltrans). The PeMS database currently has 1 terabyte of data onlne, and collects more than 1GB per day. PeMS uses the data to compute freeway usage and congeston delays, measure and predct travel tme, evaluate ramp-meterng methods, and valdate traffc theores. There are 14,871 man lne (ML) loops n the PeMS database from sx Caltrans dstrcts. The results presented here are for man lne loops. Each loop reports the volume q( the number of vehcles crossng the loop detector durng a 30-second tme nterval t, and occupancy k( the fracton of ths nterval durng whch there s a vehcle above the loop. We call each par of volume and occupancy observatons a sample. The number of total possble samples n one day from ML loops n PeMS s therefore (14871 loops) x (2880 sample per loop per day) = 42 mllon samples. In realty, however, PeMS never receves all the samples. For example, Los Angeles has a mssng sample rate of about 15%. Whle t s clear when we mss samples, t s harder to tell when a receved sample s bad or ncorrect. A dagnostcs test needs to accept or reject samples based on our assumpton of what good and bad samples look lke.

5 Chen/Kwon/Skabardons/Varaya 4 EXISTING DATA RELIABILITY TESTS Loop data error has plagued ther effectve use for a long tme. In 1976, Payne (3) dentfed fve types of detector errors and presented several methods to detect them from 20-second and 5-mnute volume and occupancy measurements. These methods place thresholds on mnmum and maxmum flow, densty, and speed, and declare a sample to be nvald f they fal any of the tests. Later, Jacobsen and Nhan at the Unversty of Washngton defned an acceptable regon n the k-q plane, and declared samples to be good only f they fell nsde the regon (4). We call ths the Washngton Algorthm. The boundares of the acceptable regon are defned by a set of parameters, whch are calbrated from hstorcal data, or derved from traffc theory. Exstng detecton algorthms (3,4,5) try to catch the errors descrbed n (3). For example, chatterng and pulse break up cause q to be hgh, so a threshold on q can catch these errors. But some errors cannot be caught ths way, such as a detector stuck n the off (q=0, k=0) poston. Payne s algorthm would dentfy ths as a bad pont, but good detectors wll also report (0,0) when there are no vehcles n the detecton perod. Elmnatng all (0,0) ponts ntroduces a postve bas n the data. On the other hand, the Washngton Algorthm accepts the (0,0) pont, but dong so makes t unable to detect the stuck type of error. A threshold on occupancy s smlarly hard to set. An occupancy value of 0.5 for one 30-second perod should not ndcate an error, but a large number of 30-second samples wth occupances of 0.5, especally durng non-rush hours, ponts to a malfuncton. We mplemented the Washngton Algorthm n Matlab and tested t on 30-second data from 2 loops n Los Angeles, for one day. The acceptable regon s taken from (4). The data and ther dagnoses are shown n Fgure 1. Vsually, loop 1 looks good (Fgure 1b), and loop 2 looks bad (Fgure 1d). Loop 2 looks bad because there are many samples wth k=70% and q=0, as well as many samples wth occupances that appear too hgh, even durng non-rush hours, and when loop 1 shows low occupancy. The Washngton Algorthm, however, does not make the correct dagnoss. Out of 2875 samples, t declared 1138 samples to be bad for loop 1 and 883 bad for loop 2. In both loops, there were many false alarms. Ths s because the maxmum acceptable slope of q/k was exceeded by many samples n free flow. Ths suggests that the algorthm s very senstve to thresholds and needs to be calbrated for Calforna. Calbraton s mpractcal because each loop wll need a separate acceptable regon, and ground truth would be dffcult to get. There are also false negatves many samples from loop 2 appear to be bad because they have hgh occupances durng off peak tmes, but they were not detected by the Washngton Algorthm. Ths llustrates a dffculty wth the threshold method the acceptable regon has to be very large, because there are many possble traffc states wthn a 30-second perod. On the other hand, a lot more nformaton can be ganed by lookng at how a detector behaves over many sample tmes. Ths s why we easly recognze loop 1 to be good and loop 2 to be bad by lookng at ther k( plots, and ths s a key nsght that led to our dagnostcs algorthm. PROPOSED DETECTOR DIAGNOSTICS ALGORITHM Desgn The algorthm for loop error detecton uses the tme seres of flow and occupancy measurements, nstead of makng a decson based on an ndvdual sample. It s based on the emprcal observaton that good and bad detectors behave very dfferently over tme. For example, at any gven nstant, the flow and occupancy at a detector locaton can have a wde range of values, and one cannot rule most

6 Chen/Kwon/Skabardons/Varaya 5 of them out; but over a day, most detectors show a smlar pattern flow and occupancy are hgh n the rush hours and low late at nght. Fgure 2a and 2b show typcal 30-second flow and occupancy measurements. Most loops have outputs that look lke ths, but some loops behave very dfferently. Fgure 2c and 2d show an example of a bad loop. Ths loop has zero flow and an occupancy value of 0.7 for several hours durng the evenng rush hour clearly, these values must be ncorrect. We found 4 types of abnormal tme seres behavor, and lst them n Table 1. Types 1 and 4 are selfexplanatory; types 2 and 3 are llustrated n Fgure 2c, 2d, and Fgure 1b. The errors n Table 1 are not mutually exclusve. For example, a loop wth all zero occupancy values exhbts both type 1 and type 4 errors. A loop s declared bad f t s n any of these categores. We dd not fnd a sgnfcant number of loops that have chatter or pulse break up, whch would produce abnormally hgh volumes. Therefore the current form of the detecton algorthm does not check for ths condton. However, a ffth error type and error check can easly be added to the algorthm to flag loops wth consstently hgh counts. We developed the Daly Statstcs Algorthm (DSA) to recognze error types 1-4 above. The nput to the algorthm s the tme seres of 30-second measurements q(d, and k(d,, where d s the ndex of the day, and t=0,1,2,...,2879 s the 30-second sample number; the output s the dagnoss (d) for the dth day: (d)=0 f the loop s good, and (d)=1 f the loop s bad. In contrast to exstng algorthms that operate on each sample, the DSA produces one dagnoss for all the samples of a loop on each day. We use only samples between 5am and 10pm to do the dagnostcs, because outsde of ths perod, t s more dffcult to tell the dfference between good and bad loops. There are second samples n ths perod, therefore the algorthm s a functon of 2041x2=4082 varables. Thus the dagnostc (d) on day d s a functon, (d) = f(q(d,a), q(d,a+1),..., q(d,b), k(d,a), k(d,a+1),..., k(d,b)), where a=5120=600 s the sample number at 5am, and b=22120=2640 s the last sample number, at 10pm. To deal wth the large number of varables, we frst reduce them to four statstcs, S 1,...,S 4, whch are approprate summares of the tme seres. Ther defntons are gven n Table 2, where S j (,d) s the jth statstc computed for the th loop on the dth day. The decson becomes a functon of these four varables. For the th loop and dth day, the decson whether the loop s bad or good s determned accordng to the rule where s j ( d ) = 1 0 f S1 (, d ) > s1 or S 2 (, d ) > s 2 or S 3 (, d ) > s3 or S 4 (, d ) < s 4 otherwse are thresholds on each statstc. These four statstcs summarze the daly measurements well because they are good ndcators of the four types of loop falure lsted n Table 1. Ths s seen n the hstogram of each of statstc dsplayed n Fgure 3. The data are collected from Los Angeles on 4/24/2002. The dstrbuton of each statstc shows two dstnct populatons. In S 1, for example, there are two peaks at 0 and Ths shows that there are two groups of loops one group of about 4700 loops have very few samples that report zero occupancy, and another group of about 300 that report almost all zeros. The second group s bad, because they have type 1 error. Snce all four 1

7 Chen/Kwon/Skabardons/Varaya 6 dstrbutons are strongly b-modal, Equaton 1 s not very senstve to the thresholds s j whch just have to be able to separate the two peaks n the four hstograms n Fgure 3. The default thresholds are gven n Table 2. The only other parameters of ths model are the tme ranges, and the defnton of S 3, where an occupancy threshold of 0.35 s specfed. The DSA uses a total of 7 parameters, lsted n Table 3. They work well n all 6 Caltrans dstrcts. Performance The DSA algorthm s mplemented and run on PeMS data. The last column n Table 1 shows the dstrbuton of the 4 types of errors n Dstrct 12 (Orange County) for 31 days n October, Because we don t have the ground truth of whch detectors are actually bad, we must verfy the performance of ths algorthm vsually. Fortunately, ths s easy for n most cases, because the tme seres show dstnctly dfferent patterns for good and bad detectors. A vsual test was performed on loops n Los Angeles, on data from 8/7/2001. There are 662 loops on Interstate 5 and Interstate 210, out of whch 142 (21%) were declared to be bad by the algorthm. We then manually checked the plots of occupancy to verfy these results. We found 14 loops that were declared good, but ther plots suggested they could be bad. Ths suggests a false negatve rate of 14/( ) = 2.7%. There were no false postves. Ths suggests that the algorthm performs very well. Real-Tme Operaton The detecton algorthm descrbed above gves a dagnoss on samples from an entre day. But we are also nterested n real-tme detecton the valdty of each sample as t s receved. Therefore what we want s a decson ˆ ( d,, where d s the current day, and t s the current sample tme. We use the smple approxmaton: ˆ ( d, = ( d 1) 2 where s defned n Equaton 1. Equaton 2 has two consequences. Frst, a loop s declared good or bad for an entre day. As a result, we lose some flexblty because we may be throwng away good data from a partally bad loop ths pont s dscussed n the concluson secton. Second, there s a one-day lag n the dagnoss, whch ntroduces a small error. We estmated the probablty of loop falure gven the loop status on the prevous day, and found Equaton 2 to be true for 98% of the tme. Therefore, t s a good approxmaton. IMPUTATION OF MISSING AND BAD SAMPLES The Need for Imputaton We model the measurement of each detector as ether the actual value or an error value, dependng on the status : qmeas, ( d, = qreal, ( d, (1 ( d)) + ε ( d, ( d), 3 k ( d, = k ( d, (1 ( d)) + φ ( d, ( d), 0 t 2879 meas, real, where q meas, and k meas, are the measured values, q real, and k real, are the true values, and ε and φ are error values that are ndependent of q real, and k real,. We obtaned an estmate of the loop status n Equaton 2. It tells us to dscard the samples from detectors that are declared bad. Ths leaves holes n the data, n addton to the orgnally mssng samples. Ths s a common problem at each sample

8 Chen/Kwon/Skabardons/Varaya 7 tme, the user must determne whether t s a good sample or not. An applcaton that analyzes the data must deal wth both possbltes. One approach to mssng data s to predct them usng tme seres analyss. Nhan modeled occupancy and flow tme seres as ARMA processes and predcted values n the near future (6); Daley presented a method of predcton from neghbor loops usng a Kalman flter (9). In our case, the errors do not occur randomly, but persst for many hours and days. Tme seres predctons become nvald very quckly and are napproprate n such stuatons. We developed an mputaton scheme that uses nformaton from good neghbor loops at only the current sample tme. Ths s a natural way of dealng wth mssng data and s used by tradtonal mputaton methods. For example, to fnd the total volume of a freeway locaton wth 4 lanes and only 3 workng loops, one may reasonably use the average of the 3 lanes and multply t by 4. Ths mputes the mssng value usng the average of ts neghbors. Lnear nterpolaton s another example. Suppose detector s bad, and s located between detectors j and k whch are good. Let x, x j, x k be ther locatons, and x j < x < x k, then ( x x ) q ( + ( x x ) q ( j k k j qˆ ( = 8 xk x j s the lnear nterpolaton mputaton. Whle these tradtonal mputaton methods are ntutve, they make nave assumptons about the data. Our proposed algorthm, on the other hand, models the behavor of neghbor loops better because t uses hstorcal data. Lnear Model Of Neghbor Detectors We propose a lnear regresson algorthm for mputaton that models the behavor of neghbor loops usng hstorcal data. We fnd that occupances and volumes of detectors n nearby locatons are hghly correlated. Therefore, measurements from one locaton can be used to estmate quanttes at other locatons, and a more accurate estmate can be formed f all the neghborng loops are used n the estmaton. We defne two loops to be neghbors f they are n the same locaton n dfferent lanes, or f they are n adjacent locatons. Fgure 4 shows a typcal neghborhood. We fnd that both volume and occupancy from neghborng locatons are strongly correlated. Fgure 5 shows two pars of neghbors wth lnearly related flow and occupances. Fgure 6 plots the dstrbuton of the correlaton coeffcents between all neghbors n Los Angeles. It shows that most neghbor pars have hgh correlatons n both flow and occupancy. The hgh correlaton among neghbor loop measurements means that lnear regresson s a good way to predct one from the other. It s also easy to mplement and fast to run. We use the followng parwse lnear model to relate the measurements from neghbor loops: q ( = α (, j) + α (, j) q ( + nose k ( = β (, j) + β (, j) k ( + nose For each par of neghbors (,j), the parameters α 0 (,j), α 1 (,j), β 0 (,j), β 1 (,j) are estmated usng fve days of hstorcal data. Let q (,q j (, t=1,2,...,n be the hstorcal measurements of volume, then j j 1 n 2 α 0 (, j), α1 (, j) = arg max [ q ( α0 α1 q j ( ] 10 α ', α ' n t =

9 Chen/Kwon/Skabardons/Varaya 8 The parameters for densty are ftted the same way. We can fnd parameters for all pars of loops that report data n our hstorcal database, but some loops never report any data. For them, we use a set of global parameters α 0 (δ,l 1,l 2 ), α 1 (δ,l 1,l 2 ), β 0 (δ,l 1,l 2 ), β 1 (δ,l 1,l 2 ) that generalze the relatonshp between pars of loop n dfferent confguratons. For each combnaton of (relatve locaton, lane of loop 1, lane of loop 2), we have a lnear model as follows. q ( = α 0 ( δ, l, l j ) + α 0 ( δ, l, l j ) q j ( + nose 11 k ( = β ( δ, l, l ) + β ( δ, l, l ) k ( + nose where 0 δ = 0 f and j are n the same locaton on the freeway, 1 otherwse l = lane number of loop l j = lane number of loop j l,l j = 1,2,3,...,8 j 1 The global parameters are ftted to data smlar to the local parameters. In Los Angeles, there are 60,760 pars of neghbors (,j) for 5377 loops; n San Bernardno, there are 3,896 pars for 466 loops. The four parameters for each par are computed for these two dstrcts and stored n database tables. When mputng values for loop usng ts neghbors, each neghbor provdes an estmate, and the fnal estmate s taken as the medan of the par wse estmates. Both volume and occupancy mputaton are performed the same way. The mputaton for volume s j j qˆ j 0 1 ( d, = α (, j) + α (, j) q qˆ ( d, = medan{ qˆ j j ( d, ( d,, j : neghbor of, ˆ j ( d, = 0} 12 Here ˆ j ( d, obtaned from Equaton 2 s the dagnoss of the jth loop only estmates from good neghbors are used n the mputaton. Equaton 12 s a way to combne nformaton from multple neghbors. Whle ths method s suboptmal compared to those wth jont probablty models, such as multple regresson, t s more robust. Multple regresson models all neghbors jontly, as opposed to the par-wse model adopted here. Daley also presented an estmaton method based on all neghbors jontly usng a Kalman flter (9). But we chose the par-wse model for ts robustness--t generates an estmate as long as there s one good neghbor. In contrast, multple regresson needs values at each sample tme from all the neghbors. Robustness s also ncreased by use of the medan of q s nstead of the mean, whch s affected by outlers and errors n j. After one teraton, the mputaton algorthm generates estmates for all the bad loops that have at least one good neghbor. We stll need to do somethng for the bad loops that don t have good neghbors. We have not decded on a scheme for how to do ths, but there are several alternatves. The current mplementaton smply terates the mputaton process. After the frst teraton, a subset of the bad loops s flled wth mputed values these are the loops wth good neghbors. In the second teraton, the set of good loops grows to nclude those that have been mputed n the prevous teraton, so some of the remanng bad loops now have good neghbors. Ths process contnues untl all loops are flled, or untl all the remanng bad loops don t have any good neghbors. The problem wth ths method s that the mputaton becomes less accurate wth each succeedng teraton. Fortunately, most of the bad loops are flled n the frst teraton. In Dstrct 7 on 4/24/2002, for example, the percentages of flled loops n the frst 4 teratons are 90%, 5%, 1%, 1%; the entre grd s flled after 8 teratons. Another alternatve s to use the current mputaton only for the frst n mputatons. After that, f there are stll loops wthout values, we can use another method such as ˆj

10 Chen/Kwon/Skabardons/Varaya 9 hstorcal mean. In any case, an alternatve mputaton scheme s requred for sample tmes when there are no good data for any loop. Performance We evaluate the performance of ths algorthm on data from 4/24/2002. To run ths test, we found 189 loops that are themselves good and also had good neghbors. From each loop, we collected the measured flows and occupances q ( and k (; we then ran the algorthm to compute the estmated values qˆ ( and kˆ ( based on neghbors. From these, we found the root mean squared errors for each loop, see Table 4. Ths table shows that the estmates are unbased as they should be. The standard devaton of mputaton error s small compared to the mean and standard devaton of the measurements. Fgure 7 compares the estmated and orgnal values for one loop. They show good agreement. We also compared the performance of our algorthm aganst that of lnear nterpolaton. Ffteen trplets of good loops were chosen for ths test. Ten of the trplets are loops n the same lane, dfferent locatons, whle 5 other trplets have ther loops n the same locaton, across 3 lanes. In each trplet, we used two loops to predct the volume and occupancy of the thrd loop usng lnear nterpolaton. In every case, the neghborhood method produced a lower error n occupancy estmates; t produced smaller errors n flow estmates n 10 of 15 locatons. Overall, the neghborhood method performed better n the mean and medan, as expected. CONCLUSION We presented algorthms to detect bad loop detectors from ther outputs, and to mpute mssng data from neghborng good loops. Exstng methods of detecton evaluate each 20-second sample to determne f t represents a plausble traffc state, but we found that there s much more nformaton n how detectors behave over tme. Our algorthm makes dagnoses based on the sequence of measurements from each detector over a whole day. Vsually, bad data s much easer to detect when vewed as a tme seres. We found that our algorthm found almost all of the bad detectors that could be found vsually. Our mputaton algorthm estmates the true values at locatons wth bad or mssng data. Ths s an mportant functonalty, because almost any algorthm that uses the data needs a complete grd of data. Tradtonally, the way to deal wth mssng data s to nterpolate from near-by loops. Our algorthm performs better than nterpolaton because t uses hstorcal nformaton on how the measurements from neghbor detectors are related. We model the volume and occupancy between neghbor loops lnearly, and fnd the lnear regresson coeffcents of each neghbor par from hstorcal data. Ths algorthm s smple and robust. There reman many possbltes for mprovements to the algorthms descrbed here. The detecton algorthm descrbed here has a tme lag. To address ths, we are developng a truly real tme detecton algorthm that ncorporates neghbor loop measurements as well as the past day s statstcs. Whle the lnear model descrbes most neghbor pars, some pars have non-lnear relatonshps, so a more general model may be better. Another area of mprovement s the handlng of entre blocks of mssng data. The current mputaton algorthm needs a large number of good loops to mpute the rest, but t doesn t work f most or all the loops are bad for a sample tme. We need a method to handle ths stuaton.

11 Chen/Kwon/Skabardons/Varaya 10 Sngle loop data dagnostcs s an mportant area of research. Whle loop detectors are the most abundant source of traffc nformaton, the data are sometmes bad or mssng. The algorthms we presented construct a complete grd of clean data n real tme. They smplfy the desgn of upper level algorthms and mprove the accuracy of analyss based on loop data.

12 Chen/Kwon/Skabardons/Varaya 11 ACKNOWLEDGEMENTS Ths study s part of the PeMS project, whch s supported by grants from Caltrans to the Calforna PATH Program. We are very grateful to engneers from Caltrans Dstrcts 3, 4, 7, 8 and 12 and Headquarters for ther encouragement, understandng, and patence. They contnue to shape the evoluton of the PeMS vson, to montor ts progress, to champon PeMS wthn Caltrans. Wthout ther generous support ths project would not have reached such a mature stage. The contents of ths paper reflect the vews of the authors who are responsble for the facts and the accuracy of the data presented heren. The contents do not necessarly reflect the offcal vews of or polcy of the Calforna Department of Transportaton. Ths paper does not consttute a standard, specfcaton or regulaton.

13 Chen/Kwon/Skabardons/Varaya 12 REFERENCES Chen, C., K. Petty, A. Skabardons, and P. Varaya. Freeway Performance Measurement System: Mnng Loop Detector Data. Transportaton Research Record No.1748, Transportaton Research Board, Washngton, D.C., 2001, pp Payne, H.J., E.D. Helfenben and H.C. Knobel. Development and testng of ncdent detecton algorthms, FHWA-RD-76-20, Federal Hghway Admnstraton, Washngton DC Jacobson, L., Nhan, N., and J. Bender, Detectng Erroneous Loop Detector Data n a Freeway Traffc Management System. Transportaton Research Record 1287, Transportaton Research Board, Washngton, D.C., 1990, pp Cleghorn, D., F. Hall, and D. Garbuo. Improved Data Screenng Technques for Freeway Traffc Management Systems. Transportaton Research Record 1320, Transportaton Research Board, Washngton, D.C., 1991, pp Nhan, N. (1997) Ad to Determnng Freeway Meterng Rates and Detectng Loop Errors. Journal of Transportaton Engneerng, Vol 123, No 6, November/December 1997, pp Turochy, R.E. and B.L. Smth. A New Procedure for Detector Data Screenng n Traffc Management Systems. Transportaton Research Record 1727, Transportaton Research Board, Washngton, D.C., 2000, pp Daley, D.J. Improved error detecton for nductve loop sensors, WA-RD 3001 Washngton State Dept. of Transportaton, May Davs, G. and N. Nhan. Usng Tme-Seres Desgns to Estmate Changes n Freeway Level of Servce, Despte Mssng Data. Transportaton Research. Part A, Vol 18A, no. 5/6, Oct/Dec 1984, pp

14 Chen/Kwon/Skabardons/Varaya 13 LIST OF TABLES Table 1 Error Types...14 Table 2 Statstcs for dagnostcs...15 Table 3 Parameters of the Daly Statstcs Algorthm, and ther default settngs...16 Table 4 Performance of mputaton...17 LIST OF FIGURES Fgure 1 The Washngton Algorthm on two loops. Loop 1 and 2 are n Los Angeles, I-5 North, postmle 7.8, lanes 1 and 2; data collected on 8/7/ Fgure 2 Typcal and abnormal 30-sec flow (lef and occupancy measurements Fgure 3 Hstograms of S 1 - S Fgure 4 Example of neghborng loops Fgure 5 Scatter plot of occupances and flows from two pars of neghbors Fgure 6 Cumulatve dstrbuton of the correlaton coeffcents between neghbors...23 Fgure 7 Orgnal and estmated occupances and flows for a good loop...24

15 Chen/Kwon/Skabardons/Varaya 14 Table 1 Error Types. Error Type Descrpton Lkely Cause Fracton of loops n Dstrct 12 1 Occupancy and flow are mostly zero Stuck off 5.6% 2 Non-zero occupancy and zero flow, see Hangng on 5.5% Fgure 2c and 2d. 3 Very hgh occupancy, see Fgure 1d Hangng on 9.6% 4 Constant occupancy and flow Stuck on or 11.2% off All Errors 16%

16 Chen/Kwon/Skabardons/Varaya 15 Table 2 Statstcs for dagnostcs. Name Defnton Descrpton S 1 (,d) 1 ( k ( d, = 0) number of samples that have occupancy = 0. S 2 (,d) S 3 (,d: S 4 (,d) a t b a t b a t b ( 1) 1 ( k ( d, > 0)1( q ( d, = 0) number of samples that have occupancy>0 and flow=0 1( k ( d, > k ), k = number of samples that have occupancy > k (=0.35) pˆ( x) = x: p( x) > 0 pˆ( x) log( pˆ( x)), a t b 1( k ( d, = x) 1 a t b entropy of occupancy samples a well-known measure of the randomness of a random varable. If k (d, s constant n t, for example, ts entropy s zero.

17 Chen/Kwon/Skabardons/Varaya 16 Table 3 Parameters of the Daly Statstcs Algorthm, and ther default settngs Parameter Value k 0.35 s s 2 50 s s 4 4 a 5am b 10pm

18 Chen/Kwon/Skabardons/Varaya 17 Quantty Mean Standard Devaton Table 4 Performance of mputaton. Mean Absolute Error Standard Devaton of Error Mean Error Occupancy Volume (vph)

19 Chen/Kwon/Skabardons/Varaya 18 a b c d Fgure 1 The Washngton Algorthm on two loops. Loop 1 and 2 are n Los Angeles, I-5 North, postmle 7.8, lanes 1 and 2; data collected on 8/7/2001. Occupancy s n percent.

20 Chen/Kwon/Skabardons/Varaya 19 a b c d Fgure 2 Typcal and abnormal 30-sec flow (lef and occupancy measurements.

21 Chen/Kwon/Skabardons/Varaya 20 Fgure 3 Hstograms of S 1 - S 4.

22 Chen/Kwon/Skabardons/Varaya 21 Fgure 4 Example of neghborng loops.

23 Chen/Kwon/Skabardons/Varaya 22 Fgure 5 Scatter plot of occupances and flows from two pars of neghbors.

24 Chen/Kwon/Skabardons/Varaya 23 Fgure 6 Cumulatve dstrbuton of the correlaton coeffcents between neghbors.

25 Chen/Kwon/Skabardons/Varaya 24 Fgure 7 Orgnal and estmated occupances and flows for a good loop.