Identifying and Tracking Pedestrians Based on Sensor Fusion and Motion Stability Predictions

Transcription

1 Sensors 2010, 10, ; doi: /s OPEN ACCESS sensors ISSN Artice Identifying and Tracking Pedestrians Based on Sensor Fusion and Motion Stabiity Predictions Basam Museh 1, Fernando García 1, Javier Otamendi 2, José Mª Armingo 1 and Arturo de a Escaera 1, * 1 2 Inteigent Systems Laboratory, Universidad Caros III de Madrid/ Avda de a Universidad 30, 28911, Leganés, Madrid, Spain; E-Mais: bmuseh@ing.uc3m.es (B.M.); fegarcia@ing.uc3m.es (F.G.); armingo@ing.uc3m.es (J.M.A.) Universidad Rey Juan Caros/ Paseo Artieros s/n , Vicávaro, Madrid, Spain, E-Mai: franciscojavier.otamendi@urjc.es (J.O.) * Author to whom correspondence shoud be addressed; E-Mai: escaera@ing.uc3m.es; Te.: ; Fax: Received: 1 Juy 2010; in revised form: 9 August 2010 / Accepted: 26 August 2010 / Pubished: 27 August 2010 Abstract: The ack of trustworthy sensors makes deveopment of Advanced Driver Assistance System (ADAS) appications a tough task. It is necessary to deveop inteigent systems by combining reiabe sensors and rea-time agorithms to send the proper, accurate messages to the drivers. In this artice, an appication to detect and predict the movement of pedestrians in order to prevent an imminent coision has been deveoped and tested under rea conditions. The proposed appication, first, accuratey measures the position of obstaces using a two-sensor hybrid fusion approach: a stereo camera vision system and a aser scanner. Second, it correcty identifies pedestrians using inteigent agorithms based on poyines and pattern recognition reated to eg positions (aser subsystem) and dense disparity maps and u-v disparity (vision subsystem). Third, it uses statistica vaidation gates and confidence regions to track the pedestrian within the detection zones of the sensors and predict their position in the upcoming frames. The inteigent sensor appication has been experimentay tested with success whie tracking pedestrians that cross and move in zigzag fashion in front of a vehice. Keywords: pedestrian detection; advanced driver assistance systems; stereo vision; aser technoogy; confidence intervas; sensor fusion

2 Sensors 2010, Introduction Trustworthy sensors are key eements regarding current road safety appications. In recent years, advances in information technoogies have ead to more inteigent and compex appications which are abe to dea with a arge variety of situations. These new appications are known as ADAS (Advance Driver Assistance Systems). In order to provide reiabe ADAS appications, one of the principa tasks invoved is obstace detection, especiay for those obstaces that represent the most vunerabe road users: pedestrians. In terms of quantifying the accident rate depending on the type of transportation, pedestrians, who account for 41% of the tota number of victims, represent the argest number of traffic accident victims in terms of deaths. It is we known that human errors are the cause of most traffic accidents. The two main errors are drivers inattention and wrong driving decisions. Governments are trying to reduce accidents with infrastructure improvements and educationa campaigns, but they cannot be competey eiminated due to the human factor. That is why ADAS can reduce the number, danger and severity of traffic accidents. Severa ADAS, which nowadays are being researched for Inteigent Vehices, are based on Artificia Inteigence and Robotics technoogies. On-board perception systems are essentia to estimate the degree of safety in a given situation and to aow the contro system to make a suitabe decision. Traffic safety research, deveoped around the word, shows that it is not possibe to use ony one sensor to get a reevant information from the road environment, making data fusion from different kinds of sensors necessary. In this artice a nove fusion method is proposed. The method combines the information provided by a 2D aser range finder and a stereo camera to detect pedestrians in urban environments. By combining both sensors, imitations inherent to each one can be overcome. Laser range sensors provide a reiabe distance to the cosest obstaces, thus giving trustabe information of the surrounding, but this information is imited due to the ow amount of data provided by the device and occusions. With this ack of information, estimation of the type of obstaces found in a road environment is a tough task. On the other hand, data provided by computer vision systems have more information but ess structured. This information can be very usefu when trying o estimate the type of obstace i.e., pedestrian detection, but ess precise to give a robust ocaization. A fusion system can be hepfu to fufi the requirements of such exigent appications as vehice safety systems. It aso can assure than in situations when one of the sensors is not avaiabe the other one can be used to aow the appication to work under the hardest conditions The objectives that are addressed are: 1. Identification of pedestrians and tracking of their trajectories. The focus is to detect the objects that are in the environment, cassify the pedestrians and track them modeing their trajectory and identify possibe coisions. 2. Instaation of an inteigent system in the vehice that tes the driver of potentia dangers. The toos that are going to be used are: 1. The sensors that aow for the acquisition of data from the environment. 2. Statistica inference or decision making to perform a probabiity cacuation on the prediction of the trajectories.

3 Sensors 2010, Agorithms that wi match the measurements and the predictions so as to cassify the objects and determine their exact ocation, and send aarms in case the object is too cose to the vehice. 2. State of the Art Statistics show that more than a haf of accidents resuting in fata victims happened in urban environments, in other words, where the active safety vehice s systems, e.g., ABS, ESP, have ower infuence. Because of that, ADAS for front-side coisions, pedestrian run-over or automatic emergency braking are attracting an increasing interest. In addition, systems aiming to protect the most vunerabe users of these infrastructures such as pedestrians, cycists, etc., are difficut to deveop due to the great variety of shapes, sizes and appearances invoved [1]. Sensor data fusion [2] has been proposed in order to improve the performance, both in ocaization and robustness, of agorithms deveoped for detecting obstaces in urban environments. Making use of sensoria fusion techniques the perception of the environment can be improved as we as making up for the incompeteness of sensors which have partia fauts or provide imited information. Current perception systems are designed based on muti-sensor design using computer vision (monocuar or stereoscopic) in the visibe spectrum and infrared and aser sensors, idar or radar [3,4]. There are some constraints reated to perception systems design that have to do with coverage, precision, uncertainty, etc. One of these probems is the imitation in spatia coverage. Usuay a unique sensor is used to cover a reduced area; perhaps a higher coverage can be achieved doing data fusion from severa sensors [5]. Limited tempora coverage is produced by the time needed to obtain and transmit a measurement by the sensor. Increasing the number of sensors used when making data fusion wi reduce these imitations. Another aspect to consider is the sensoria imprecision, inherent to the nature of sensors. Measurements obtained by each sensor are imited by the precision of the sensor used. The higher the number of sensors is, the higher the eve of precision that is achieved in data fusion [6]. There is a new probem when designing perception systems to be appied to Inteigent Transportation Systems uncertainty which depends on the object observed instead of the sensor. It is produced when some specia characteristics (such as occusions) may appear when the sensor is not abe to measure a attributes reevant to perception or when the observation is ambiguous [7]. A unique sensor may be unabe to reduce the uncertainty in its perception due to its imited vision of the object [8]. This kind of situations comes up frequenty in urban environments, where pedestrians, streetights, etc, constanty appear bocked by parked vehices, stopped in the street, etc. Fusion methods are typicay divided into two types according to the eve in which fusion is performed: ow eve fusion, so caed centraized fusion schemes [9], and high eve fusion, so caed decentraized schemes. Low eve schemes perform fusion using a set of features extracted from both sensors. High eve fusion performs different cassifications with data provided by each sensor separatey. A fina stage combines information from a cassifications. Each configuration has its own advantages and disadvantages [10]. Low eve fusion combines information from both sensors creating a new set of data with more information, but probems reated to data association arise. Low eve approaches that take advantage

4 Sensors 2010, of statistica knowedge [9,11] obtain information from a sensors and combine the information using Bayes formua, Support Vector Machines (SVM), Neura Networks, etc. High eve fusion schemes aow fusion in an easier and more scaabe way; new sensors can be added more easiy but with ess information to do the cassification. They can be differentiated in track based fusion and ce based fusion schemes. The first one tries to associate the different objects found in each sensor [12]. The second one [13] uses occupation grids, adding confidence according to the type of sensor that detects the obstace, but osing the geometrica structure. Other works reated to fusion schemes take advantage of aser scanner trustworthiness to seect regions of interest where vision-based systems try to detect pedestrians [6,14]. In [2] detection of especiay dangerous zones is done using aser scanner information integrated aong time. In [7], information from different sensors creating a feature vector is used to perform an unique cassification (caed medium eve schemes). 3. The IvvI Project IvvI (Inteigent Vehice based on Visua Information, Figure 1a) is a research patform for the impementation of systems based on computer vision and aser technoogy, with the goa of deveoping ADASs. The purpose of the IvvI patform is to test perception agorithms under rea conditions, and five sensing capabiities are being researched for Lane Keeping System, Adaptive Cruise Contro, Pedestrian Detection, Traffic Sign Recognition and Driver Drowsiness Detection. Figure 1. IvvI research patform. Research resuts are being currenty impemented in a Nissan Note (Figure 1a). There is a DC/AC power converter connected to an auxiiary vehice's battery. Through it, the eectrica power needed for the computers, cameras and aser is obtained. There is a CMOS coor camera for the detection of traffic signs and other vertica signs (Figure 1d) and another CMOS coor camera inside the car to detect drowsiness situations. A binocuar stereo vision system (Figure 1d) is used for pedestrian and ane detection during day driving, whereas an infrared camera paced on the wing mirror is used for pedestrian detection during night driving. A aser paced on the front bumper is used for pedestrian and

5 NON-PARAMETRICVALIDATIONGATES/CONFIDENCEREGIONS NON-PARAMETRICVALIDATIONGATES/CONFIDENCEREGIONS Sensors 2010, vehice detection (Figure 1a). There are three PCs (Figure 1b) in the vehice's boot which are used for processing the information captured by the sensors. The aser and the stereo vision system are the input sensors to the fusion based tracking agorithm, which is the main ine of research of this artice. The agorithm provides information about the environment to the driver. This is done through a monitor (Figure 1c), simiar to a GPS device that many vehices carry nowadays, where the pedestrians are graphicay represented and, above a, through the vehice oudspeakers that pay severa warning messages depending on the ocation of the pedestrian and the seriousness of the possibe run over. 4. Genera Description of the Agorithm The inteigent tracking agorithm (Figure 2) ooks for the correct cassification of objects as we as for their exact ocation. Its main step is the matching of the data or measurements obtained by the different fused sensors and the predictions on the tracked ocation of the pedestrian. These predictions are based on continuousy monitoring the stabiity of the sensor measurements for the very near past, that is, for the ast set of frames and for different time increments, so that changes in directions and speeds are accounted for. Figure 2. Tracking framework. SENSOR DATA VISION LASER PEDESTRIAN IDENTIFICATION AND LOCATION BY FUSION NON-PARAMETRIC VALIDATION GATES/ CONFIDENCE REGIONS COMPOSITE MATCHING MEASUREMENT STABILITY AND MOTION PREDICTION The individua-sensor data is to be jointy fused by the proper caibration and coordination agorithms. It is necessary that each and every sensor perform a measurement exacty at the same moment in time so that a composite fused measurement might be obtained. The fusion step has to account for an absence of measurements by any or a of the devices, so that the trajectories and the stabiity of the time movements are statisticay tracked. Raw data is recorded at time t in mutipe dimensions [in this case, two (x t, y t )]; then data is converted into movements that the pedestrian has performed in a time increment ( Δ x,δ y ) so they are used as the basis for the predictions. t t Two sets of statistica inference procedures are to be performed. The first procedure is the anaysis of the stabiity of the dispacements, that is, the anaysis of the consistency or the homogeneity of the

6 Sensors 2010, current measurement with the previous movements of the same time increment. The stabiity hypothesis is usuay tested using confidence intervas or vaidation gates in one dimension and simutaneous confidence regions in mutipe dimensions [15,16]. The second procedure is the prediction of the motion of the pedestrian or his/her ocation at a particuar future time. Based on the current ocation and the stabe movements for different time increments, it is possibe to set confidence regions for the ocation at future times t+ [17]. These predictions are to be made for each pedestrian independenty. The matching agorithm confronts then the fused stabe measurements for different time increments with a the ocation predictions that have been made in previous moments of time. If within the vaidation gates, that is, with the occurrence of proper mutipe matches, the known pedestrians are iabe to be continuousy tracked. If no match is achieved, new pedestrians may be avaiabe for tracking. After each successfu cassification or tracking stage, the predictions must be updated, because changes in directions or veocity may very ikey occur. By performing moving predictions, that is, taking into account ony measurements for past short time intervas, these changes wi not negativey affect the predictions and ruin the tracking of the proper trajectories. What foows is a detaied expanation of each of the stages of the agorithm. Section 5 expains the aser subsystem incuding its detection and cassification stages. Section 6 detais the computer vision system and its pedestrian identification step. Section 7 is then used to address the inteigent tracking agorithm, which is tested in Section 8 with rea experiments in an urban outdoor environment. Section 9 is finay used to present the concusions and future work. 5. Laser Subsystem The aim of the aser subsystems is to detect pedestrians based on the data received from the aser scanner device. The aser, a SICK LMS 291-S05, has a measurement range of 80 meters and a frequency up to 19 frames per second. The detection process is composed of two stages. In the first stage, the data is received and obstaces shapes are estimated. A second stage performs obstace cassification according to the previousy estimated shape. In the present research, pedestrian cassification is performed by searching through specific patterns reated to eg positions Obstace segmentation The aser scanner provides a fixed amount of points that represents the distance to the obstaces for a given ange, from the coordinate origin situated in the bumper of the vehice. This measurement is taken from a singe aser that performs a 180 rotation. Thus, there is a time difference between each distance measured. Due to the vehice movement and aser scanner rotation there is a variation aong time incuded in the measures, therefore vehice egomotion correction is mandatory before processing the data; this is done thanks to an on-board GPS-IMU system. The resuting points are joined according to the Eucidean distance among them (Figure 3). After the custering agorithm, poyines are created, which join the points contained within segments. These poyines give information about shape and distance of the obstace to the vehice.

7 Sensors 2010, Figure 3. Environment information given by the agorithm after obstace segmentation. Left: shape estimation using poyines; detected pedestrian is highighted. Center: Rea image captured by a camera mounted in the vehice. Right: raw data captured by the aser after egomotion correction Pedestrian cassification Cassification is performed according to obstaces shape [18]. A study was performed to observe the pedestrian pattern during the waking process. Specific patterns were searched to identify a singe pedestrian ony using the information provided by the aser radar. Observations showed the movements patterns described in Figure 4. Figure 4. Pattern given by a pedestrian, according to eg situation. This pattern may appear rotated. Observation showed that most of the patterns shared a common feature, consisting of two different 90 degrees anges. This pattern was checked under different conditions and movements incuding test for standing pedestrians facing the aser and atera standing pedestrians. Regarding to atera standing pedestrians test showed that the pattern given by the aser incudes the two mentioned

8 Sensors 2010, anges by getting the whoe shape of a eg. Taking advantage of such behavior a static mode was created. The process foowed to match the found pattern, incuding rotation, consists on a first segmentation according to obstace s size and a fina matching based on poyines shape. Segmentation computes the size of the poyine and checks that the detected obstace has a size proportiona to a human being. An obstace that fufis the size requirements is marked as candidate to be a pedestrian. An additiona stage compares it with the mode. The comparison stage inks every two consecutive anges (Figure 5) with poyines and gives a simiarity percentage according to equations (6) to (8): π θ S 1 2 θ =, π (6) 2 π α S 1 2 α = π (7) 2 S = S S α θ (8) Figure 5. Mode for pedestrian where the two anges of interest are detaied. A singe simiarity score is computed for each of the two anges separatey by comparing their vaue with the idea [equations (6) and (7)]. Then, the tota aggregated vaue, which is cacuated by mutipying both singe scores, measures the simiarity of the measurements to the mode. If the case arises where more than three poyines are present, the agorithm is appied to every pair of consecutive anges and those with the highest vaues are chosen as the poyine simiarity vaue. A threshod is used to cassify the obstace as a pedestrian. 6. Vision Subsystem The purpose of this subsystem is aso to detect and cassify pedestrians; the detection range is 30 meters and a frequency up to 10 frames per second. In order to have depth information in computer vision it is necessary to set two cameras: in the IvvI is a stereo Bumbebee camera by Pointgrey. This system automaticay performs the necessary rectification step [19-21]. Once the two images are correcty rectified, our proposed agorithm deveops the dense disparity map and u-v disparity [22] to perform the anaysis of the environment and the pedestrians. These

9 Sensors 2010, tasks have got a high computationa cost; therefore NVIDIA CUDA framework [23,24] is used to process in the GPU (Graphics Processing Unit) Dense disparity map The disparity map represents the depth W of every image point. The depth is cacuated as foows: W = f B /(u u ) = f B d (9) L R / where d is the disparity, f is the foca ength and B is the baseine distance. (u R,v R ) and (u L,v L ) are the projection in the camera panes for the right and eft cameras respectivey of the point P = (U,V,W) T of the word. For this cacuation to be vaid, the two image panes must be copanar, their optica axes must be parae and their intrinsic parameters must be the same. It is therefore necessary to find the correspondence between points of the right and eft images to determinate the disparity d (known as the stereo matching probem), using the foowing rectification: v L / f = V / Z;v R / f = V / Z v L = v R (10) There are severa possibe soutions to this stereo matching probem in order to obtain the dense disparity map. Our agorithm foows the taxonomy presented by Scharstein and Szeiski in [25], where they propose that stereo agorithms are performed by the foowing four steps: A. Matching cost computation: Athough there are more accurate cost functions [26], squared differences (SD) is preferred because it is faster and easier to impement in GPU processing. SD assumes equa gain in both cameras; that is why both images are pre-processed by Lapacian of Gaussian (LoG). B. Cost (support) aggregation: There are different kinds of support regions, and their choice infuences in the resuting disparity map [27]. The agorithm impemented is based on squarewindows support regions for cost aggregation because it is better in reation to GPU performance and the resuting disparity map is accurate enough. C. Disparity computation: There are mainy two methods for disparity computation: oca [25] and goba agorithms [28]. The oca method WTA (Winner-take-a) is chosen. For a posterior disparity refinement task, the disparity map for the eft image (eft disparity map) and for the right one (right disparity map) are constructed. To avoid redundant computations, it is possibe to use computations from the eft disparity map to construct the right disparity map. D. Disparity refinement: This step tries to reduce the possibe errors in the disparity map, which are usuay produced in areas where texture does not exist, in areas near depth discontinuity boundaries [29], or in areas where there are repeated patterns, for exampe, on was of buidings. For instance, enough texture does not exist either in the sky or in the road for images of driving environments, as figure 6a shows. The errors in the disparity map are ikey to appear in these areas, see Figures 6c and 6d. To reduce these possibe errors, a cross-check is performed; the resut is shown in Figure 6b.

10 Sensors 2010, Figure 6. (a) Left image. (b) Cross-Checking image. (c) Left disparity map image. (d) Right disparity map image. (e) Disparity map and its corresponding v-disparity on the right and u-disparity beow U-V disparity Once the disparity map has been generated, it is possibe to obtain the u-v disparity. As there is a univoca reationship between disparity and distance, the v-disparity expresses the histogram over the disparity vaues for every image row (v coordinate), whie the u-disparity does the same but for every coumn (u coordinate). In short, the u-disparity is buit by accumuating the pixes of each coumn with the same (u, d) and the v-disparity by accumuating the pixes of each row which the same (v, d). An exampe is iustrated in Figure 6e. If it is assumed that obstaces have panar surfaces, every one appears in the u-disparity image as pixes whose intensity is the height of that obstace. As the u-disparity image dimensions are the width of the origina image and the height is the disparity range, those pixes have the same horizonta coordinate than the obstace in the disparity map and the vertica coordinate is the disparity vaue of the obstace. Regarding v-disparity, as its image dimensions are the disparity range and the height of the origina image, the obstaces appear as vertica ines in its corresponding disparity vaue [30] as they are at the same disparity or distance. Another interesting feature is that the ground appears as an obique ine. This feature is very usefu because the pitch, θ, and height, h, of the cameras can be measured for each frame [31]. This information wi be used to determine accuratey the obstace ocaization in the word coordinates Obstace detection The main goa of this system is to determine the regions of interest (ROI), which wi be ater on used to concude if the obstaces are pedestrians or not. In order to do that, the road profie is estimated by means of the v-disparity [31]. This is why panar road geometry is assumed, which is reasonabe at cose areas in front of the vehice. There are other obstaces detection systems which use the u-v disparity, such as the proposed in [32,33]. Our obstace system is divided into the foowing three steps: 1) The first step is a preiminary detection over u-disparity. This task consists in threshoding the u- disparity image to detect obstaces which have a height greater than a threshod. This way the

11 Sensors 2010, threshoded u-disparity is constructed at the bottom of Figure 7a. Bobs anaysis is made on the binary image to determine the tota number of obstaces and their horizonta position and width. 2) In the disparity image, the subimages defined by the horizonta obstace position and width, red squares in Figure 7a, are threshoded using the disparity ranges obtained before, Figure 7b. They are the obstaces in front of the vehice. This binary image is used as a mask to obtain a disparity map without obstaces and a partia v-disparity is constructed, where the road profie is extracted as a ine, corresponding to equation (11), by means of the Hough transform (see Figure 7c): v = m d +b (11) The pixes of the obstaces are eiminated because in the event of appearing numerous obstaces, the profie road may be incorrect [34]. The equation (11) represents the road profie as a function of the v image coordinate, the disparity d and the constant b and the road sope m. 3) Finay, a second bob anaysis is performed to determine obstaces features, area and position, on the threshoded disparity map, Figure 7b. On the basis of this features, regions of interest are constructed on the visibe eft image for a posterior processing. Figure 7. (a) Disparity map and threshoded u-disparity with the threshoding areas. (b) Threshoded disparity map where appear the obstaces in the study region. (c) Disparity map for a study region without obstaces and the road profie as red ine obtained by means of the Hough transform Obstaces ocaization (a) (b) (c) The obstaces ocaization in word coordinates (U, V) can be obtained, and it is a function of the image coordinates (u, v) of the contact point between the obstaces and the ground. In order to do that, equations (13) is obtained from equations (9) (11) and (12), where the parameter Cu corresponds to u coordinate of the optica center and θ is the pitch of the stereo rig. In this way, the obstaces ocaization is computed with more resoution than if the disparity vaues are used excusivey. (u Cu)V U= f fbm W= cos( θ ) v b (u Cu)B m cos ( θ ) U= v b (12) (13)

12 Sensors 2010, Obstace cassification The cassification divides the obstaces into two groups: pedestrians and non-pedestrians. The resut of the cassification agorithm is a confidence score for the fact that the obstace is a pedestrian; it is compared with a threshod and if it is greater, the obstace is cassified as a pedestrian. This cassification is based on the simiarity between the vertica projection of the sihouette and the histogram of a norma distribution. Figure 8 iustrates two exampes of the vertica projection of a pedestrian sihouette from two different viewpoints, where both vertica projections are simiar to the histogram of the norma distribution. The vertica projection for each obstace is computed by means of the ROIs in the threshoded disparity map, which are resuts of the obstaces detection agorithms. In order to characterize the vertica projection, the standard deviation, σ, is computed as if the vertica projection was the histogram of a norma distribution. In order not to make the standard deviation be a function of the obstace dimension or independent on the obstace ocaization, the standard deviation is divided by the width of the ROI getting σ w. This standard deviation wi be used to compute the score. Severa vertica projections of pedestrian have been processed to obtain their standard deviations; these standard deviations foow a norma distribution N(μ σw,σ σw ). In order to compute the score for an obstace, its standard deviation is used to obtain the vaue of the probabiity density function, where the maximum score 100% is produced if the standard deviation is equa to μ σw and gets worse when the standard deviation is different from μ σw ( Figure 9). Figure 8. Two exampes of pedestrians, their sihouette and their vertica projection. Figure 9. Process scheme to obtain a pedestrian score. (a) Pedestrian image and his sihouette. (b) The vertica projection of the pedestrian sihouette. (c) Norma distribution of the standard deviations and the score for the σ corresponding with the vertica projection of the pedestrian sihouette.

13 Sensors 2010, Measurement Stabiity and Motion Prediction Once the sensors and their corresponding agorithms have taken measurements individuay and processed them in order to identify and cassify objects as pedestrians, it is necessary to provide the inteigent system with toos that track the pedestrians and aert the driver to possibe imminent coisions. In this section statistica modes are deveoped to robusty infer the possibe routes based on the current position as we as near past ocations Background on errors in mutipe statistica inference Inference is the part of statistics that reates sampe information with probabiity theory in order to estimate or predict the vaue of one or severa unknown parameters or compare two hypotheses about their vaue. The hypotheses are caed nu (which is the one statisticay proven to be true or fase according to a pre-specified confidence eve γ) and aternative (which is chosen whenever the nu is rejected). In individua hypothesis testing about a singe parameter φ, an observed vaue x is compared against a threshod vaue that resut of the appication of the confidence eve γ, and a decision is taken by deciding to reject or not reject (accept) the nu hypothesis. Tabe 1. Test-reated decision-making probem. Nu hypothesis Decision made Accept nu Reject nu True CORRECT DECISION ω - Fase positive or reevance Fase β - Fase negative or standard CORRECT DECISION It is we known, however, that two errors can occur when a decision of this kind is made about the vaue of one parameter: the nu hypothesis is rejected when it shoud have been accepted (fase positive or fase detection, significance eve = ω =1 γ) or accepted when it shoud have been rejected (fase negative or fase standard, probabiity = β, testing power = 1 β). Tabe 1 depicts this decision-making probem. In mutipe testing (Tabe 2), the number of tests is arge (M), as many as parameters, and the process shoud distinguish between nu hypotheses which are reay true (O) and those which are reay fase (A). Tabe 2. Decision-making probem in mutipe testing s. ACCEPTED H 0,m REJECTED H 0,m TOTAL TRUE NULLS P F Fase detections O FALSE NULLS N Fase acceptance T A TOTAL W R M If ω is used in each individua test, the probabiity of fase positive errors increases consideraby: the probabiity of accepting ony and a nu hypotheses when they are true is ony (1 ω) O. Goba confidence is reduced, and is therefore not 1 ω but 1 Ω, where Ω is the goba eve of significance, much higher (worse) than the theoreticay desired ω. On the other hand, when a arge number of nu

14 Sensors 2010, hypotheses are rejected by this procedure, the error of faiing to discover reevant aternative hypotheses is practicay zero. In other words, as the reevant hypotheses are overestimated, practicay a non-rejected nu hypotheses are reay standard. This approach therefore favours the determination of a significant and some other hypotheses (fase positives, which coud be numerous) as reevant, in exchange for having no fase negatives. Therefore, in order to make a correct decision for an aggregate eve of significance Ω, and prevent too many fase positives from occurring, the form in which individua testing is performed has to be adjusted. Individua tests have traditionay been maintained, athough the eve ω has been adjusted. Usuay, ω is adjusted and controed in two different ways. The first is known as the Bonferroni correction [16]. The eve of significance of each test is individuay reduced from ω to ω/m, so the p-vaues must be much ower for a nu hypothesis to be individuay rejected. As the number of significanty reevant tests is reduced, the number of fase positives aso diminishes. The number of fase negatives, however, or the number of nu hypotheses which shoud have been rejected, increases consideraby. The correction, therefore, by attempting to avoid fase positives, gives rise to too many fase negative errors. The same occurs with the Sidak adjustment [35], in which ω is reduced to: = 1 (1 Ω) 1/ M ω, (14) maintaining the goba eve of significance Ω. If M is high and Ω is ow, ω wi be very cose to zero and it wi be difficut to reject (by very sma p-vaues) any individua nu hypotheses, and no fase nu hypotheses wi be rejected. These two traditiona corrections, then, favour the determination of a non-significant or standard and some significant (fase negatives) hypotheses as reevant Background on robust confidence intervas on correated parameters Robust mutivariate hypothesis testing invoves the simutaneous comparison of sampe vaues with threshods. One possibe way to set the threshods is to use confidence intervas, whose main purpose is the estimation of the vaue of one or severa unknown parameter. Confidence intervas come in the form of confidence imits or prediction threshods. If a new sampe vaue ies within the confidence imits, the vaue is said to be homogeneous with the previous vaues, accepting the nu hypothesis of beonging to the same underying distribution. Otherwise, the vaue is said to beong to a different distribution, rejecting the nu and accepting the aternative. Confidence intervas (CI) on a singe unknown parameter are a means to set threshods on its vaues and have the form ϕ [ϕ -,ϕ + ] or CI ϕ or aternativey ϕ [ϕ * -k 1 (V(ϕ * )) 1/2,ϕ * +k 2 (V(ϕ * ) 1/2 )], where φ* is the point estimator of the parameter and V(φ*) the variance of the estimator. The estimation probem reates therefore to the proper identification of the distribution of the contro statistic and the associated probabiity cacuation based on the confidence eve that resuts in the k i vaues. If the distribution is not known, an upper threshod on the vaue of the k i might be however readiy cacuated using Chebishev s inequaity, k = k 1 = k 2 = non-parametric confidence interva on one parameter being ϕ [ϕ * (V(ϕ * )) 1/2 ]. 1 ω, with the corresponding 1 ω (V(ϕ* )) 1/2, ϕ * + 1 ω

15 Sensors 2010, However, in most rea situations, there are more than one variabe or parameter invoved in the decision making or inference process. In other words, confidence intervas are to be set for severa parameters, which are usuay correated. The first possibiity is to set individua and independent confidence intervas for each and every variabe, but adjusting the confidence percentage to account for the mutivariate situation according to Bonferroni s or Sidak s corrections, as we as the setting bounds using Chebishev s inequaity. The corresponding confidence region comes in the form of a rectange in two dimensions, a prism is three dimensions and a poyhedron in more dimensions [36]. However, it has been shown that if the variabes are correated, the fase negative rate is very high, since the response area covered by the combination of individua CI s is much arger than what it shoud be. The corresponding area shoud come in the form of an eipse in two dimensions and an eipsoid in higher dimensions. Figure 10 expains the probem in two dimensions. Figure 10. Confidence regions in two dimensions. CI - 2 CI - 1 The equation of the eipsoid, or the eipse in two dimensions, in terms of the Mahaanobis standardized distances of each point to the center of the eipse, E, is as foows [36]: x x s x 2 y y + s y 2 ( 1 R) x x y y 2R s x s y = D = E 2 2 max, (15) where the constant D is the mutivariate equivaent to the vaue k in one dimension, that is, the maximum distance measured in standard deviations from the center of the confidence region to the fringe of the eipse. D is approximated as D = k * 7/1.5 [37] Appication to the tracking of pedestrians Mutivariate statistica modes are ready to be particuarized in this artice to the movement of pedestrians. The data is measured at each time t individuay from the sensors: (x ; y ) where s = 1,, S sensors and s = f for the fused vaues with as many measurements as objects i = 1, I are detected. The vaues in absoute units are aso transformed into movements or dispacements Δ x and Δ y, where = 1, L accounts for the time interva used to cacuate the dispacements: Δ x s,t,i = x s,t,i x s,t-,i Δ y = y s,t,i y s,t-,i s, i, s, i,

16 Sensors 2010, The first set of modes reate to the contro of the stabiity of the dispacements. For each object i, the ast C vaues are used to cacuate the averages on the moves Δ, xs,t, i Δ y s, i,, as we as the standard deviations Δ, sxs,t, i Δ sy s, i, and the correation among dimensions Δ. Rs,t, i The confidence intervas might readiy be cacuated using Chebishev s inequaity and Sidak s corrections: Δ x [ Δ y [ Δ x 1 1-( 1- Ω) Δ sx Δ x,δ x + ]= s,t, i ( 1/M ) ± 1 1-( 1- Ω) Δ y,δ y + ] = Δ y ± Δ sys,t, i ( 1/M ) Simiary, the confidence regions or eipsoids ( Δ x ; Δ y ), Eipse, are: s, i, s, i, (16) Δ x Δ x Δ sx s, i, 2 Δ y Δ y + Δ sy 2 2 ( Δ R ) ( 1 Δ R ) 2 Δ x Δ x Δ sx Δ y Δ y Δ sy = D 2 (17) The second set of modes are used to determine where the object is going to be at t +, by just adding the average observed move to the current position: x = x + = y = Δ x 1 1-( 1- Ω) Δ x ( 1/M ) + Δ x 1 1-( 1- Ω) Δ x + ( 1/M ) Δ 1 1-( 1- Ω) Δ y y ( 1/M ) Δ sx Δ sx s,t, i s,t, i Δ sy s,t, i (18) y + = ( ; y ) + Δ 1 1-( 1- Ω) Δ y y + ( 1/M ) x Eipse * s, i, and s=f Δ sy The tota number of prediction modes, M, is M = (S + 1) * I * L * Composite matching and cassification The ast stage in this mutivariate assessment is the ocation of the pedestrians. After the discussion in the previous sections, the information avaiabe at each time t is: The measurements from the sensors. The confidence bounds or vaidation gates for the prediction of moves for each object and ag, for each sensor and dimension as we as for the fused data, and in combined confidence regions s,t, i

17 Sensors 2010, The vaidation gates for the prediction of ocation for each object and ag, for each sensor and dimension as we as for the fused data, and in combined confidence regions. The agorithm then must confront the raw data, the agged data and the fused data with the vaidation gates and prediction regions so as to assign the measurements to an existing or new object. There exist severa possibe resuts: A the vaidation gates and confidence regions are positivey met for one of the existing objects. The measurement is assigned to that object, which continues to be a pedestrian or another object, fixed or not. None of the vaidation gates or confidence regions are met. A new object is created and starts to be tracked. If either the gates for the stabiity of moves or the position gates are met, due to a no-read or a sudden change in direction or veocity, the measurement is assigned to same object which continues to be tracked. 8. Experimenta Resuts The foowing experiments have been carried with the IvvI vehice outdoors in order to evauate the robustness and reiabiity of the proposed detection and tracking agorithm. Figure 11 shows the capabiity of the perception system to detect mutipe objects and identify them as pedestrians. The figure shows four pedestrians crossing in front of the vehice, two in each direction. The vision image shows boxes around the identified pedestrians. The aser frame shows possibe pedestrians surrounded by boxes after processing the raw data. The data obtained out of the sensoria system has been used to test the performance of the fusion agorithm under different rea conditions: crossings of pedestrians whie moving in zigzag and changes of speeds. Figure 11. Pedestrian detection by the IvvI vehice Pedestrians crossing and changing directions The IvvI vehice is first set on the road to test the proposed inteigent fusion-based tracking system outdoors, where pedestrians wander foowing both inear and non-inear paths.

18 Sensors 2010, Definition of the experiment Two pedestrians move for 29.2 seconds (292 frames) in front of the vehice foowing the paths incuded in Figure 12. The trajectories are highighted by the crossing of the two pedestrians and a singe pedestrian changing direction in a zig-zag fashion. Figure 12. Paths foowed by pedestrians Parameterization of the tracking agorithm The parameter S, the number of sensors, is set to S = 2, as a camera and a aser are used to obtain data from the environment. The parameter I, the count of objects, is set to I = 2, as two are the pedestrians being tracked. The parameter L, the number of time intervas, is set to L = 3 to aow for a quick execution of the agorithm. The parameter M, or the number of simutaneous tests that are performed at each t is M = (S + 1) * I * L * 3 = 54. The parameter C, or the number of past data used to cacuate the trajectories and the moves, is set to C = 10, since that is the vaue corresponding to the number of maximum frame rate of the camera. The parameter Ω, or the overa significance eve, is set to 5%. 1 1 Therefore: k = = = ( 1/M ) ( 1/54 ) 1-( 1- Ω ) 1-( ) D = k * 7/1.5 = Anaysis of the crossing The cross happens in between frame number 70 to 90, or 2 second. The information provided by the two sensor systems, as we as the resut of the tracking agorithm are incuded in Figure 13.

19 Sensors 2010, Figure 13. (a) Sequence of the crossing resuting from the vision subsystem. (b) Sequence of the crossing resuting from the aser subsystem. (c) Sequence of the crossing resuting from the tracking agorithm. (a) (b) (c)

20 Sensors 2010, The tracking images show the eipses corresponding to a time interva of 1 frame. At the time of the crossing, the agorithm is ony abe to cassify one pedestrian. The resut is a arger prediction region that covers both pedestrians. It aso aows for the tracking of both as depicted in the figures corresponding to frames 95 and Anaysis of the Zigzag movement The pedestrian changes directions between frames 205 and 270 for more than 6 seconds (Figure 14). The changes are propery picked but with the penaty of carrying arger eipses, due to the increase in the vaue of the cacuated standard deviations. Figure 14. (a) Images sequence of a pedestrian with zigzag trajectory resuting of the vision subsystem. (b) Images sequence of a pedestrian with zigzag trajectory resuting of the aser subsystem. (a) (b)

21 Sensors 2010, Figure 14. Cont Reiabiity resuts Tabe 3 shows the absoute frequency distribution of measurements by each of the sensors (V = vision, L = aser, F = fusion) for each of the two pedestrians as we as the fase positives. The first pedestrian shoud be detected in a of the 292 frames, whereas the second one ony for the first 192 frames. The camera shows an additiona sti object for 288 of the 292 frames and the aser shows aso the presence of another sti object. By fusing the sensors, the 2 pedestrians are ceary picked in 257 frames, with the other 35 frames picking just one of the two. The number of fase positives reaches 3, since in one frame 5 objects are picked other than the two sti objects. (c) Tabe 3. Distribution of detections per frame. DETECTIONS MEASUREMENTS TOTAL PEDESTRIAN 1 PEDESTRIAN 2 OTHER V L F V L F V L F The hit rate per pedestrian (Tabe 4) is above 86% for the fused data. The hit rate is cacuated as the percentage of reads over the count of frames in which the pedestrians were correcty situated in front of the vehice. If C consecutive reads were not avaiabe, then it is not possibe to cacuate neither an average of the data nor a fitered vaue, and thus, the hit rate diminishes.

22 Sensors 2010, Tabe 4. Hit rates per pedestrian. HIT RATE (%) PEDESTRIAN 1 PEDESTRIAN 2 VISION % % LASER % % FUSION % % Tabe 5 incudes the nu acceptance ratio or the percentage of time in which the new vaue fas within the a the vaidation gates and confidence regions, both individuay and jointy for the stabiity of the movements and the tracking of the trajectories. Tabe 5. Hit rate per mode. STABILITY % % TRACKING % % AT LEAST ONE % % BOTH % % The percentage is over 85% and shoud increase in uncontroed environments whenever the variabiity of the measures is higher. In controed environments, the pedestrian knows that it is being tracked and acts consistenty so that the cacuated standard deviation is smaer than it is when variabiity is speed and direction is more ikey to occur. In fact, the hit rate is higher for the first pedestrian who at one point foows a zigzag pattern Changes in speeds of both pedestrian and vehice An additiona experiment is performed to assess the performance of the agorithm when the pedestrian is changing speeds whie the vehice is moving (Figures 15a and 15b). Figure 15. (a) Visuaization from the vehice. (b) Zenitha map of the situation. (c) Sequence of the crossing resuting from the tracking agorithm. (a) (b) (c)

23 Sensors 2010, The vehice starts detecting the pedestrian about 18 meters before the zebra crossing (Figure 15c). The vehice keeps on approaching the pedestrian up unti a distance of 8 meters, as shown by the portion the graph that has a negative sope. Then, both the pedestrian and the vehice stop. When the pedestrian sees that the vehice has come to a fu stop, he starts waking again at a higher pace. The vehice does not start moving unti the pedestrian has competey crossed the road, about three metres to the right of the vehice; the graphs during these frames shows no sope. The vehice starts its movement again, with the graph presenting a negative sope. The graph shows as we the prediction eipses every 50 frames. 9. Concusions and Future Work Sensors are ubiquitous in today s word, athough it is necessary to give them with autonomy to process the information they get from the environment. Our research aims at deveoping inteigent sensors in a demanding fied ike Inteigent Transportation Systems. More specificay this artice addresses the probem of the identifying and cassifying objects and pedestrian so as to hep drivers to avoid accidents. Deveopments in both hardware and software are necessary to create robust and inteigent appication in Advanced Driver Assistance Systems. The sensoria fusion of a aser and a computer vision system as we as a cassification agorithm has proven successfu for the tracking of pedestrians that cross and wander in zigzag in front of the vehice. Origina agorithms have been deveoped for cassification and tracking. A new approach to pedestrian detections based on a aser and variabe modes has been presented, giving an estimation of how cose they are to the idea pattern for a pedestrian. Regarding the stereo-vision subsystems two origina contributions are worth mentioning. First, the impementation of the disparity map construction with the cross-checking and the u-v disparity using CUDA in order to obtain a rea time system. Second, a nove and fast procedure for pedestrian identification using the sihouette of the stereo image has been presented. The success of the matching procedure is based on the appication of non parametric mutivariate statistics to the ocaization probem whie tracking pedestrians. More specificay, the Sidak correction has been appied to cacuate the proper mutivariate significance eve, the Chebishev inequaity has been used to account for non-normaity and confidence regions have been cacuated to determine the positioning of the pedestrians in the upcoming frames. Two other contributions have made for the robustness of the agorithm. The use of movements and not raw measurements has aowed for the proper contro and dimensioning of the confidence regions. The check for stabiity of the measurements prior to the cacuation of the predictions has aso increased the hit ratio whie recognizing and cassifying pedestrians. A experiments have been performed in rea environments using the IvvI research patform, where a the agorithms have been impemented and tested. Improvements shoud be made on the perception as we as the tracking systems to improve the hit rate. The cassification of the obstaces detected by the stereo system can have more features into account. Once the obstaces have been detected and their size and distanced to vehice found, methods that use severa image features ike [38] can be appied and sti work in rea-time. Experiments with more pedestrians are aso currenty being carried out. Future works wi be focused on other kind of