Development and Evaluation of a Procedure for the Calibration of Simulation Models

Transcription

1 Development and Evaluation of a Procedure for the Calibration of Simulation Models Byungkyu (Brian) Park and Hongtu (Maggie) Qi Microscopic traffic simulation models have been playing an important role in the evaluation of transportation engineering and planning practices for the past few decades, particularly in cases in which field implementation is difficult or expensive to conduct. To achieve high fidelity and credibility for a traffic simulation model, model calibration and validation are of utmost importance. Most calibration efforts reported in the literature have focused on the informal practice, and they have seldom proposed a systematic procedure or guideline for the calibration and validation of simulation models. This paper proposes a procedure for microscopic simulation model calibration. The validity of the proposed procedure was demonstrated by use of a case study of an actuated signalized intersection by using a widely used microscopic traffic simulation model, Verkehr in Staedten Simulation (VISSIM). The simulation results were compared with multiple days of field data to determine the performance of the calibrated model. It was found that the calibrated parameters obtained by the proposed procedure generated performance measures that were representative of the field conditions, while the simulation results obtained with the default and best-guess parameters were significantly different from the field data. Microscopic traffic simulation models have been widely and increasingly applied for the evaluation of transportation system design, traffic operations, and management alternatives for the past few decades because simulation is inexpensive, risk-free, and fast. As recognized by the Highway Capacity Manual (HCM), simulation becomes a valuable aid in assessing the performance of transportation systems (1). Furthermore, traffic simulation models sometimes provide significant advantages over traditional planning or analytical models, such as Highway Capacity Software (HCS). For example, the estimation of delay from HCS is often inappropriate when the impacts of a large-scale system are considered or oversaturated conditions are prevalent (2). Other common reasons for the popularity of simulation models include their attractive animations; their stochastic variability for the capture of real-world traffic conditions; and their capabilities to model complex roadway geometries, such as combined systems of urban streets and freeways. When more and more people begin to rely on simulation as an important evaluation tool, an essential concern is its credibility. Potential issues are whether these models accurately replicate realistic driver B. Park, Department of Civil Engineering, University of Virginia, and Virginia Transportation Research Council, P.O. Box 4742, Charlottesville, VA H. Qi, BMI-SG, 833 Boone Boulevard, Suite 4, Vienna, VA Transportation Research Record: Journal of the Transportation Research Board, No. 1934, Transportation Research Board of the National Academies, Washington, D.C., 25, pp behavior and whether one can trust the decisions made on the basis of the simulations. Microscopic simulation models contain numerous independent parameters that can be used to describe traffic flow characteristics, driver behavior, and traffic control operations. These models provide a default value for each parameter, but they also allow users to change the values to represent local traffic conditions (3). The process of adjusting and fine-tuning model parameters by using realworld data to reflect local traffic conditions is model calibration. Simulation model-based analyses are often performed with default parameter values or manually adjusted values. Rigorous calibration procedures are often omitted because doing so requires a great deal of time as well as a huge amount of field data. The findings and conclusions based on the uncalibrated or inappropriately calibrated models could be misleading and even erroneous. Thus, proper calibration is a crucial step in simulation applications. To achieve adequate reliability of the simulation models, it is important that a rigorous calibration procedure be applied before any further study and analysis are conducted. Changes to the parameters during calibration should be justified and defensible by the users. Most of the calibration efforts were performed to achieve a reasonable correspondence between the field data and the output of the simulation model. Recently, more and more transportation researchers and practitioners have realized the importance of model calibration and have spent significant time and effort to demonstrate the validities of their models. However, it was found that simulation model calibration was often informally practiced among researchers and had seldom been formally proposed as a procedure or guideline (3). Therefore, proposing a general and practical procedure for simulation model calibration is an urgent task. Several researchers have discussed the general requirements of a simulation model calibration procedure (3 9). Park and Schneeberger proposed a general calibration procedure based on a linear regression model (3). However, they did not consider the correlations among the parameters. Sacks et al. demonstrated an informal validation process and emphasized the importance of data quality and visualization (4). Hellinga proposed general requirements for model calibration (5). However, a formal procedure for conducting calibration was not provided. The objective of this paper is to propose and evaluate a general procedure for calibration of microscopic simulation models. The validity of the proposed procedure was evaluated and demonstrated by use of a case study with an actuated signalized intersection by using a widely used microscopic traffic simulation model, Verkehr in Staedten Simulation (VISSIM). The remainder of the paper is organized as follows: a brief background of the test site and simulation model, VISSIM, is provided; the methodology of the proposed procedure is described and is followed by the implantation and evaluation of the procedure 28

2 Park and Qi 29 by use of a case study; lastly, conclusions and recommendations are provided. TEST SITE AND SIMULATION MODEL Test Site An actuated signalized intersection was modeled in VISSIM to test the proposed procedure for simulation model calibration. The intersection is located at the junction of U.S. Route 15 and U.S. Route 25 in Virginia. The test site is referred to as Site 15 throughout the remainder of this paper. As shown in Figure 1, it has four legs with a single-lane approach for all directions, and there are exclusive rightturn bays 11 ft long on all approaches. The southbound approach carries a relatively heavy traffic volume and has a large proportion of left-turning vehicles during peak hours. This isolated actuated intersection has two phases and functions, such as minimum and maximum green times, added initial, passage time, and minimum gap. Simulation Model VISSIM, a microscopic and stochastic simulation model, was chosen in this research for evaluation of the proposed procedure. VISSIM is capable of simulating traffic operations on urban streets and freeways, with a special emphasis on public transportation and multimodal transportation (1). VISSIM consists of two different programs: the traffic simulator and the signal state generator. The traffic simulator is a microscopic simulation model that comprises car-following and lane-changing logics, and it provides a simulation environment. The simulator is capable of simulation at a resolution of up to.1 s. The signal state generator allows users to define their own signal control logic, including actuated signal control. VISSIM uses the psychophysical driver behavior model developed by Wiedemann (11), which uses dynamic speeds and stochastic car-following models and, hence, models driving behavior close to that detected in the field. In this research, all simulation work was carried out with Version 3.7 of VISSIM on a personal computer with a Pentium 2.53-GHz central processing unit and 1. GB of random access memory. PROPOSED PROCEDURE The proposed procedure for simulation model calibration consists of simulation model setup, initial calibration, feasibility test, parameter calibration by use of a genetic algorithm (GA), and model evaluation, as shown in Figure 2. Simulation Model Setup Simulation model setup comprises those tasks that are conducted before the commencement of model calibration. These tasks include the definition of the study scope and purpose, site selection, determination of measures of effectiveness (MOEs), field data collection, and network coding. Initial Evaluation Once a simulation model is correctly set up, the simulation performance data based on the default parameter are obtained and compared with the field data. If a close match between them is found, the default model could be deemed appropriate for use for further analysis. Otherwise, the subsequent steps must be performed. Route 15 Route 25 FIGURE 1 Site 15 in VISSIM.

3 21 Transportation Research Record 1934 Unsatisfied Simulation Model Setup - Study scope and purpose - Site selection - Determination of measures of effectiveness - Field data collection - Network coding Initial Evaluation on Default Parameters Feasibility Test Passed? Parameter Calibration Using Genetic Algorithm Satisfied End Unsatisfied Initial Calibration - Identification of calibration parameters - Experimental design for calibration (Latin Hypercube Design) - Multiple runs Feasibility Test - Statistical plots - Analysis of variance - Identification of key parameters Yes Evaluation of the Parameter Sets - Distribution of simulation output - Statistical test - Visualization check No Satisfied Adjust key parameter ranges number of combinations to a practical amount while still reasonably covering the entire parameter surface. LHD provides an orthogonal array that randomly samples the entire design space broken into regions of equal probability. This type of sampling can be regarded as a stratified Monte Carlo sampling method in which the pairwise correlations can be minimized to a small value, which is essential for uncorrelated parameter estimates (12). LHD is especially useful for exploring the interior of the parameter space and for limiting the experiment to a fixed or a user-defined number of combinations. This technique ensures that the entire range of each parameter is sampled. In this study, a Latin Hypercube Sampling toolbox in MatLab (13) was used to generate a predetermined number of scenarios by using the calibration parameters of each simulation model. Multiple Runs To reduce stochastic variability, multiple runs are conducted for each scenario from the experimental design. The number of replications should be a function of the variability of the system, the acceptable error level, and computation time constraints. The average performance measure and standard deviation are recorded for each of the runs. The simulation results are used in the feasibility test. Feasibility Test The purpose of the feasibility test is to check whether the field data are reasonably covered by the distribution of simulation results and, hence, to determine whether the current parameter ranges are feasible. If field data fall within the middle 9% of the distribution of the simulation results, defined as the acceptable range, the current parameter ranges are deemed sufficient to generate the field condition. Otherwise, the ranges of key parameters are adjusted to achieve the desired results. However, the adjusted value should be reasonable and realistic. With the adjusted parameters, another set of a predetermined number of scenarios is generated and evaluated. This process is repeated until the field data fall within the acceptable range. FIGURE 2 Methodology flowchart. Statistical Plots Initial Calibration Identification of Calibration Parameters All calibration parameters within the simulation model except those that do not have any relevant impact on the simulation results are identified. For example, for a simple network without a route diversion possibility, the parameters related to the route changes could be eliminated from the calibration parameter list. The acceptable ranges of selected calibration parameters are determined. Experimental Design for Calibration The number of combinations among feasible controllable parameters is extremely large, such that all possible scenarios cannot be evaluated in a reasonable amount of time. A Latin hypercube design (LHD) algorithm (12), an experimental design method, is used to reduce the With the performance measure recorded in each of the runs, statistical plots such as scatter plots and histograms are drawn to help interpret the results. The plots include the distribution of performance measures from multiple runs and the performance measure versus each calibration parameter. For the key parameters, the trend is usually easy to observe. Analysis of Variance Analysis of variance (ANOVA), a more rigorous statistical method, is applied to identify key parameters. ANOVA tests the null hypothesis that the means for several groups in the population are equal by comparing the sample variance estimated from the group means with that estimated within the groups (14). In this study each pair of calibration parameter and simulation results was examined by one-way ANOVA by use of a statistical package, SPSS (15). It can identify statistically significant parameters on the basis of the significance value of the F-test and can quantify the

4 Park and Qi 211 degree to which each parameter explains the performance measure on the basis of the sum of squares (SSRs) between the groups. When the P-value is smaller than the user-defined confidence level, the null hypothesis is rejected, thereby indicating that group means are statistically different. Therefore, the parameters with small P values and large SSRs are identified as key parameters. Parameter Calibration by Use of GA After the identification of appropriate parameters and their acceptable ranges, a GA is applied to find the optimal parameter values. Analysis by use of a GA is a heuristic optimization technique based on the mechanics of natural selection and evolution (16 ). It works with a population of individuals, each of which represents a possible solution to a given problem. A fitness value is assigned to each individual according to how good its represented solution is. Selection of the best individuals from the current generation and mating of the best individuals to produce a new set of individuals produce a new population of possible solutions. Three basic operators used in GA analysis are reproduction, crossover, and mutation. The reproduction operator selects individuals with higher fitness. The crossover operator creates the next population from the intermediate population, whereas the mutation operator is used to explore some areas that have not been searched. Schema theorem and building blocks hypothesis are rigorous explanations of how the highly fit individuals survive to the next generations (17 ). In this study the relative error of the average travel time between the simulation output and the field data was used as the fitness value for the GA. The fitness function takes the form evaluations of travel time distributions, statistical tests, and animations are satisfactory. Otherwise, either the previous step, which is parameter calibration by use of a GA, or the initial calibration stage must be reperformed. IMPLEMENTATION OF PROPOSED PROCEDURE The proposed procedure was implemented and evaluated in a case study by using VISSIM. This section describes the main steps in detail and highlights the strategies of fine-tuning of the key parameters. Simulation Model Setup The tasks under the first step, simulation model setup, consist of the definition of the study scope and purpose, site selection, determination of MOEs, field data collection, and network coding. The data needed in this case study included the simulation input data and MOEs for calibration purposes. To code Site 15 in the simulation environment, input data such as traffic counts, the length of each approach lane, intersection geometric characteristics, detector locations, and the posted speed limit were required. To account for dayto-day variability, data were collected during the evening peak hour between 5: and 6: p.m. on multiple days, April 15 and 22 and May 13, 23. The MOE used in this network was the average travel time on the southbound through and left-turn movements because it directly reflected the level of service of the signalized intersection. Furthermore, the collection of data from both the field and the simulation was relatively easy. TTfield TT FV = TT field sim where FV = fitness value, TT field = average travel time from the field, and TT sim = average travel time from the simulation. () 1 Initial Evaluation Once the simulation model was correctly set up, multiple simulations based on the default parameters were conducted and the average travel time was recorded. It was found that the average travel time of 23.4 s from the simulation was far less than the field travel time of s. Thus, it was deemed that calibration was necessary and that the following steps needed to be practiced. Evaluation of Parameter Sets Once GA finishes parameter calibration, multiple runs are separately conducted for the default parameters and the GA-based parameters for comparison. The best-guess parameter set is another parameter set that could be included in the comparison. Here, the best guess is derived by assignment of the most adequate value to each parameter on the basis of engineering judgment and knowledge of local traffic conditions. For each parameter set, a distribution of performance measures is developed and compared with the field measures. Statistical testing, for example, by the t-test and visualization checking, is conducted. The t-test is used to verify whether the calibrated parameter set from GA can generate statistically significant results and perform better than the default parameter set. In this study, the performance measure used in the t-test was the average travel time within the study period. Visualization is used to evaluate the calibrated models. A model cannot be considered calibrated if the animations are not realistic. Animations from multiple runs of each model are viewed to ensure that the animations are acceptable. The model is regarded as properly calibrated only if the results of the Initial Calibration Identification of Calibration Parameters In the initial phase, users may not be able to identify correctly all the critical parameters or the reasonable range for each parameter. Two criteria could potentially help users select the parameter candidates to be calibrated: 1. Conduct a trial-and-error test for each parameter. If the parameter does not affect the simulation result at all, it can be excluded from the parameter candidate list. 2. Judge from a traffic engineer s point of view. For instance, because the study network has only one lane for each approach and because the origin destination was simple without any diversion possibility, parameters related to lane change or route choice behaviors should not have an impact on the simulation result. Thus, the minimum headway (front and rear) and the waiting time before diffusion were eliminated from the parameter list.

5 212 Transportation Research Record 1934 Users can determine the acceptable range for each parameter selected on the basis of a review of the literature, including the manuals for simulation models and other existing research findings. So that the calibration direction is not misled, the use of unrealistic values in the range should be avoided. The feasibility of selected parameters and their ranges was later verified through a feasibility test. The following is the list of parameters and acceptable ranges determined initially. A detailed description of each parameter can be found in the VISSIM manual (1). 1. Simulation resolution: 1 to 9 time steps per simulation second 2. Number of observed preceding vehicles: 1 to 4 3. Average standstill distance: 1 to 5 m 4. Saturation flow rate: 1,756, 1,8, 1,846, 1,895, and 1,946 vehicles per hour Additive part of desired safety distance: 2. to 2.5 Multiple part of desired safety distance: 3. to Priority rules for minimum headway: 5 to 2 m 6. Priority rules for minimum gap time: 3 to 6 s 7. Desired speed distribution: 4 to 5, 3 to 4, and 2 to 3 mph Both the additive and the multiple parts of the desired safety distances in the Wiedemann 74 car-following model determine the saturation flow rate for VISSIM. The values were based on Table (flow rate, 25 s of through green time) in the VISSIM manual. The priority rules and the two associated parameters determine the priorities of conflicting traffic movements in the intersection and designate the right-of-way. Experimental Design for Calibration The Latin Hypercube Sampling toolbox in MATLAB was used to generate 2 scenarios by using the first set of parameters and their ranges, as defined above. In this case, for eight controllable input parameters and five different values for each parameter, it would require 5 8 (i.e., 39,625) possible combinations. Any number of scenarios within this value could have been used. However, a significant amount of time would be required to conduct such a large number of experiments. Furthermore, the large number of simulation runs would be increased if multiple simulation runs for each scenario were required to reduce stochastic variability. Initial tests indicated that the use of 2 scenarios was adequate to cover the entire parameter surface and for computational simulation and calculation. Multiple Runs Five random seeded runs were conducted in VISSIM for each of the 2 cases, for a total of 1, runs. The average travel time was recorded for each of the 1, runs. The results from the five multiple runs of each scenario were then averaged to represent each of the 2 parameter sets. Feasibility Test With the simulation results based on the initial parameter set, it is important to answer the following key questions by applying statistical methods and analyzing plots. Are the simulation results based on the current parameters, and do their ranges include the field conditions? If not, some parameters and their ranges need to be adjusted or new parameters should be explored. Histogram analysis is appropriate for this purpose. Figure 3 shows that the average field travel time, s, falls outside of the simulated distribution generated from the initial ranges of parameters. It indicates that the current parameters and their ranges were not satisfactory and needed adjustment. Is the simulation result sensitive to the different values of each parameter? Also, which parameters are critical to the simulation results? The use of plots and the ANOVA test is appropriate. Scatter plots of each parameter versus the travel time from the simulations were investigated. It was found that the minimum gap and the desired speed distribution were two parameters important to the results. The travel time became higher when the minimum gap increased or the mean desired speed decreased, or both. Figure 4 provides two examples of the scatter plots. The mean desired speed demonstrates an apparent trend, while the simulation resolution has no effect on the simulation result. However, the identification of the critical parameters through statistical analysis is more rigorous. The values of each parameter were made discrete, separated into several groups, and indexed. A one-way ANOVA procedure was then used to test the null hypothesis that the means for two or more groups were equal. If the parameter was sensitive to the result, the means for different groups should be statistically different. Table 1 shows the ANOVA results and lists the significance value of the F-test, the mean difference between groups, and the SSRs between groups. A small P-value and a large SSR indicate that the null hypothesis is rejected and that this parameter is important to the result. The group pairs were listed if the mean difference was significant at either the.5 or the.5 level. 7 6 Frequency sec 1 FIGURE Travel Time (sec) Feasibility test results for Site 15 with VISSIM.

6 Park and Qi Travel Time (sec) Mean Desired Speed (mph) (a) 6 5 Travel Time (sec) Simulation Resolution (b) FIGURE 4 Calibration parameters versus VISSIM travel time: (a) mean desired speed and (b) simulation resolution. On the basis of the ANOVA results in Table 1, the desired speed distribution and the minimum gap time were identified as critical parameters. The desired speed distribution is an important parameter because it has a significant influence on roadway capacity and achievable speeds. If a driver is not hindered by other vehicles, the driver will travel at a desired speed with small stochastic variation. The minimum gap is an important parameter for the modeling of conflicting traffic movements and determination of when the movement with a lower priority could proceed. To achieve a longer travel time from the simulation, the mean desired speed should be decreased or the minimum gap should be increased. However, because the upper bound of the minimum gap (i.e., 6 s) has already been reached, the use of higher TABLE 1 ANOVA Results for Site 15 with VISSIM Significance Value Significant Mean SSR Between Site 15 VISSIM (P-value) Differences (sig. <.5) Groups Simulation resolution.989 NA 12.8 No. of preceding vehicles , Average standstill distance Saturation flow.91 NA 4.9 Min. headway (m) , Min. gap time (s). 1 3, 1 4*, 2 4*, Desired speed distribution. All * = The significance value is less than.5; NA = not applicable.

7 214 Transportation Research Record 1934 values for the minimum gap is not realistic. It was believed that some other factors that might have affected the result were not yet identified. Therefore, additional tools, like the HCM procedure and field data, were used to help clarify the ranges of some parameters, such as the saturation flow on the southbound approach and the field speed within the intersection. Check Speed Within the Intersection Although a posted speed limit of 45 mph was observed in the field, the actual speed within the intersection could be less than 45 mph because of inadequate sight distance, permitted left turns, a narrow intersection, and other factors, such as the presence of gas stations about 5 ft upstream of the stop line on the southbound approach. It is more reasonable to use the field average speed to set the link speed and turning speed. By viewing the videotapes, the speed data were retrieved as the tail of the vehicle crossed the stop bar on the southbound approach. The first few vehicles in the queue were not considered because they were still accelerating. The result shows that the mean field speed within the intersection is 35.3 mph, with a standard deviation of 14.9 mph. Check Saturation Flow The saturation flow rate is another important factor for determination of the intersection capacity, and this in turn affects the vehicle travel time. It can be obtained from field data and by the HCM procedure (1). The following shows how the saturation flow rate was calculated and estimated by using these approaches: Calculation of queue discharge headway from field data. The field saturation flow rate on the southbound approach was obtained from the videotapes. First, the time stamps of the third and the last vehicles in the queue were recorded as they passed the stop line in each cycle. The saturation flow rate was then calculated by using Equation 2. The average headway was 3.13 s, which resulted in a saturation flow of 1,149 vehicles per hour per lane. SF = 3, 6 ( t t ) ( n 3) 3 n where SF = saturation flow rate (vehicles per hour per lane), t 3 = discharge headway of the third vehicle, and t n = discharge headway of the nth vehicle. Calculation by HCM procedure. A saturation flow rate was calculated by the HCM procedure (1). The ideal saturation flow rate of 1,9 vehicles per hour per lane was adjusted for the prevailing conditions. The adjustment is made by introducing factors such as the number of lanes, the lane width, the heavy vehicle percentage, and right and left turns. The adjusted saturation flow rate on the southbound approach was 1,313 vehicles per hour per lane. Parameter Range Modification ( 2) The parameter ranges were modified on the basis of the field speed data and the estimated saturation flow rates. The modified desired speed ranges were 3 to 4, 32.5 to 37.5, and 27.5 to 42.5 mph; and the new ranges of the additive part of desired safety distance and the multiple part of desired safety distance were 1. to 5. and 1. to 6., respectively. The updated parameter set and its ranges are as follows: 1. Simulation resolution: 1 to 9 time steps per simulation second 2. Number of observed preceding vehicles: 1 to 4 3. Maximum look-ahead distance: 2 to 3 m 4. Average standstill distance: 1 to 5 m 5. Saturation flow rate Additive part of desired safety distance: 1. to 5. Multiple part of desired safety distance: 1. to Priority rules for minimum headway: 5 to 2 m 7. Priority rules for minimum gap time: 3 to 6 s 8. Desired speed distribution: 3 to 4, 32.5 to 37.5, and 27.5 to 42.5 mph With the new parameters, another 2 cases were generated by the LHD method. The result shows that the new simulated distribution covers the field data and indicates that the new parameter ranges were able to capture the field condition. Parameter Calibration by Use of GA A GA was integrated with the VISSIM model to calibrate the parameters. First, the GA is started by generating a number of individuals in the population, each of which represents a feasible solution. Second, each individual is decoded into actual parameter values and inserted into the simulation model. Multiple simulations were then carried out for each set of decoded values, with the simulation results channeled into a fitness function. In this case, five simulation runs were conducted for each parameter set to reduce simulation variability. Third, a fitness value is calculated by comparison of the simulation results and field data by using Equation 1. If the comparison result does not meet the stopping criterion, then a new population is generated after the implementation of selection, crossover, mutation, and elitism operations. This process is continued until the stopping criterion is met or the maximum number of generations is reached. Figure 5a shows the convergence of the GA. The results were based on 1 generations and a population size of 2. Other parameter settings included a crossover rate of.8 and a mutation rate of.5. Both the best fitness and the average fitness values decrease as the number of generations increases, which indicates that the simulation result becomes closer to the field value. The small oscillation of fitness values in the last generations is due to the stochastic variability of the VISSIM simulation because only five runs were conducted for each set of parameters. For the parameter set with the best fitness value, VISSIM was run 1 times and the average travel time was recorded for each run. The resulting distribution of travel times is shown in Figure 5b. The field travel times from 3 days also are shown, and they all fall within the simulated distribution. Evaluation of Parameter Sets In addition to the default and GA-optimized parameter sets, a bestguess parameter set was prepared to evaluate the performances of the calibrated simulation models. The best-guess parameter set was prepared because default simulation parameters were often adjusted on the basis of the engineers knowledge of local traffic and other conditions. As both the default and the GA-optimized parameter sets

8 Park and Qi Best Avg..3 Fitness Number of Generations (a) Frequency Field 2: secs Field 3: secs Field 1: 7.43 secs Travel Time (sec) FIGURE 5 Initial calibration result obtained by use of GA: (a) convergence of fitness value with generation and (b) travel time distribution of GA-based parameter set. (b) were already identified, this section describes how the best-guess parameter set was obtained. Best-guess parameters were obtained on the basis of the users experience with the study network and with the help of the analytical tool in HCM. For instance, in this case study, additional information such as saturation flow was obtained from the HCM procedure. According to the tables provided in the VISSIM manual, a linear regression model with an R 2 value of.92 was derived to estimate the saturation flow rate: SF = 2, ( bx_add) ( bx_mult) ( 3) where SF = saturation flow rate (vehicles per hour per lane), bx_add = additive part of desired safety distance, and bx_mult = multiple part of desired safety distance. By use of the Solver function in Microsoft Excel software, an optimal combination of these two parameters was obtained to minimize the difference between the HCM and VISSIM saturation flow. The values for bx_add and bx_mult were 5 and 2.85 m, respectively. For comparison purposes, the parameter values for each set are listed in Table 2. The evaluation of three parameter sets (i.e., default, GA optimized, and best guess) were based on three criteria: (a) comparison of the distributions of 1 VISSIM simulation outputs, (b) paired t-test, and (c) animations. Comparison of these three parameter sets shows the importance of calibration for microscopic simulation models. The travel times along the southbound approach are compared in Figure 6. Figure 6 shows that all three field travel times fall within the distribution of simulation results generated with the GA-optimized parameter set. The two other models (i.e., the default and best-guess parameter sets) generated travel times much less than those observed in the field. A t-test was conducted to compare the simulation results obtained with the GA-optimized parameter set with those obtained with the two other parameter sets. The result shows that the GA-optimized parameter set generated statistically significant results, unlike the two other parameter sets. Animations of each parameter set were viewed to determine whether the animations

9 216 Transportation Research Record 1934 TABLE 2 Three Parameter Sets for Site 15 with VISSIM Site 15 VISSIM Default Best-Guess GA-Based Simulation resolution Number of observed preceding vehicles Maximum look-ahead distance (m) Average standstill distance (m) Additive part of desired safety distance Multiple part of desired safety distance Priority rules minimum headway (m) Priority rules minimum gap time (s) Desired speed distribution (mph) Car: 4 5 Car: 4 5 Car: HV: 3 4 HV: 3 4 HV: Average travel time (s) HV = heavy vehicle. were realistic or unrealistic. For the GA-optimized calibrated parameter set, the animations at several travel time percentiles were found to be acceptable. For the default and the best-guess parameter sets, the majority of vehicles passed the intersection without waiting, which was not realistic. CONCLUSIONS AND RECOMMENDATIONS This paper has proposed and evaluated a general calibration procedure for microscopic simulation models. The validity of the proposed procedure was demonstrated by use of a case study of an actuated signalized intersection with VISSIM. The performance of the GAoptimized calibrated model was evaluated by comparing the distribution of the simulation output with multiple days of field data. The result shows that use of the GA-optimized calibrated parameters yielded a distribution of simulation outputs that matched the field travel times for multiple days, while use of the default parameters and best-guess parameters resulted in significant discrepancies between the simulated and the field data. Therefore, the importance of model calibration is manifested. It is recommended that a simulation model be calibrated before any further simulation-based traffic analyses are conducted. Feasibility testing was found to be useful in identifying the reasonable and appropriate ranges of calibration parameters. It was performed as an initial check of whether or not the selected parameters and their ranges were able to achieve the field condition. In addition, field data and analytical tools could be used to help verify the initial parameter ranges. The impact of key parameters cannot be overstated, and appropriate values need to be assigned. Scatter plots and statistical analysis, such as analysis by the ANOVA test, were effective in identifying key parameters. Because this study used only one MOE, travel time, for model calibration, the performance of other MOEs is uncertain. Further research is recommended to include additional MOEs in the calibration process. However, for some MOEs, such as delay and queue length, it should be noted that different simulation models provide different calculation methods. Furthermore, to validate the model, it is desirable to collect another set of data under conditions different from those used for the calibration. Different conditions could be those that have been sec 35 Frequency sec sec Travel Time (sec) Default Parameters Best-Guess GA-based FIGURE 6 Comparison of travel times by different parameter sets.

10 Park and Qi 217 untried, such as after the implementation of a new signal timing plan, or the use of different days on similar transportation facilities. To account for day-to-day variability, multiple days of field data should be used. The values of the data from only a single day could be higher or lower than the true mean of the field value and may not represent the general field conditions. A comprehensive range of data collected by using advanced equipment as well as intelligent transportation system technologies should be obtained for the full calibration and validation of a simulation model. Although a network with simple geometric alignments was tested in this study, a more complex network should be considered to verify the performance of the proposed procedure. Doing so will provide more insight into the feasibility of the proposed procedure. Finally, with the increased trend of the use of simulation applications for large and complex network-based problems, it is necessary to calibrate not only driver behavior parameters but also dynamic origin destination demand and route choice parameters. The proposed procedure for simulation model calibration could be enhanced to include more functions and evaluated for use with these scenarios. REFERENCES 1. Highway Capacity Manual. TRB, National Research Council, Washington, D.C., White, T. General Overview of Simulation Models. 49th Annual Meeting, Southern District Institute of Transportation Engineers, Williamsburg, Va., Jan Park, B., and J. D. Schneeberger. Microscopic Simulation Model Calibration and Validation: A Case Study of VISSIM for a Coordinated Actuated Signal System. In Transportation Research Record: Journal of the Transportation Research Board, No. 1856, Transportation Research Board of the National Academies, Washington, D.C., 23, pp Sacks, J., N. Rouphail, B. Park, and P. Thakuriah. Statistically Based Validation of Computer Simulation Models in Traffic Operations and Man- agement. Journal of Transportation and Statistics, Vol. 5, No. 1, 22, pp Hellinga, B. R. Requirement for the Calibration of Traffic Simulation Models. Department of Civil Engineering, University of Waterloo. Accessed March Milam, R. T. Recommended Guidelines for the Calibration and Validation of Traffic Simulation Models. Fehr & Peers Associates, Inc. Accessed March Cohen, S. L. An Approach to Calibration and Validation of Traffic Simulation Models. Presented at 83rd Annual Meeting of the Transportation Research Board, Washington, D.C., Dowling, R., A. Skabardonis, J. Halkias, G. M. Hale, and G. Zammit. Guidelines for Calibration of Microsimulation Models: Framework and Applications. Presented at 83rd Annual Meeting of the Transportation Research Board, Washington, D.C., Zhang, Y., and L. E. Owen. Systematic Validation of a Microscopic Traffic Simulation Program. Presented at 83rd Annual Meeting of the Transportation Research Board, Washington, D.C., VISSIM Version 3.7 Manual. PTV Planug Transport Verkehr AG, Innovative Transportation Concepts, Inc., Jan Wiedemann, R. Modeling of RTI-Elements on Multi-Lane Roads. In Advanced Telematics in Road Transport, DG XIII, Commission of the European Community, Brussels, Belgium, McKay, M. D., and R. J. Beckman. A Comparison of Three Methods for Selecting Values of Input Variables in the Analysis of Output from a Computer Code. Technometrics, Vol. 21, No. 2, 1979, pp MATLAB Users Manual. The Mathworks, Inc., Natick, Mass., Milton, J. S., and J. C. Arnold. Introduction to Probability and Statistics. McGraw-Hill, New York, SPSS for Windows Release SPSS Inc., Chicago, Ill., Goldberg, D. E. Genetic Algorithms in Search, Optimization, and Machine Learning. Addison-Wesley Publishing Co., Inc., Reading, Mass., Beasley, D., D. R. Bull, and R. R. Martin. An Overview of Genetic Algorithms. Part I. Fundamentals. University Computing, Vol. 15, No. 2, 1993, pp The Traffic Flow Theory and Characteristics Committee sponsored publication of this paper.