A software tool for the quantitative evaluation of 3D dose calculation algorithms

A software tool for the quantitative evaluation of 3D dose calculation algorithms William B. Harms, Sr., Daniel A. Low, John W. Wong, a) and James A. Purdy Washington University School of Medicine, Mallinckrodt Institute of Radiology, 510 S. Kingshighway Boulevard, St. Louis, Missouri 63110 Received 30 October 1997; accepted for publication 17 July 1998 Current methods for evaluating modern radiation therapy treatment planning RTP systems include the manual superposition of calculated and measured isodose curves and the comparison of a limited number of calculated and measured point doses. Both techniques have significant limitations in providing quantitative evaluations of the large number of dose data generated by modern RTP systems. More sophisticated comparison techniques have been presented in the literature, including dose-difference and distance-to-agreement DTA analyses. A software tool has been developed that uses superimposed isodose plots, dose-difference, and DTA distributions to quantify errors in computed dose distributions. Dose-difference and DTA analyses are overly sensitive in regions of high- and low-dose gradient, respectively. The logical union of locations that fail both dose-difference and DTA acceptance criteria, termed the composite evaluation, is calculated and displayed. The composite evaluation provides a method for the physicist to efficiently identify regions that fail both the dose-difference and DTA acceptance criteria. The tool provides a computer platform for the quantitative comparison of calculated and measured dose distributions. 1998 American Association of Physicists in Medicine. S0094-2405 98 00410-6 Key words: 3D radiation therapy treatment planning, dose calculation, dose calculation verification Dose computation is one of the most important features of radiation therapy treatment planning systems. As the technologies of both computer hardware and software used to model radiation therapy beams improve, it becomes apparent that improved quantitative methods for the evaluation of three-dimensional 3D dose calculation programs are required. The technique used most frequently by radiation oncology physicists to compare measured and calculated dose distributions is the superposition of measured and calculated isodose curves, with a subsequent qualitative assessment of the acceptability of the calculation algorithm. This method does not provide a quantitative assessment of the accuracy of the calculation program. Determination and presentation of the numerical difference between the measured and calculated dose distributions highlight regions of disagreement. Mah et al. 1 used dosedifference distributions to evaluate two-dimensional 2D and 3D electron beam dose calculation algorithms. Others have used color wash presentations and dose-difference histograms. 2 4 One difficulty with dose-difference distributions is that they are very sensitive in high-dose gradient regions. In the beam penumbra region, a small spatial offset between the calculated and measured dose distributions will result in a large, but clinically insignificant, dose difference. To provide a more suitable evaluation of the dose calculation algorithm in regions of high-dose gradient, the distance-to-agreement DTA concept was developed. 5 8 The DTA is the distance between a measured dose point and the nearest point in the calculated distribution containing the same dose value. The DTA distribution provides an excellent measure of the calculation quality in regions of high-dose gradient but is overly sensitive in regions of low-dose gradient, a slight difference in dose e.g., due to a difference in normalization or block transmission yields a large DTA value. Van Dyk et al. 9 provided guidelines for the evaluation of treatment planning systems. They recommended that the percent dose difference and DTA be less than 3% and 4 mm for photon beams in regions of low- and high-dose gradient 30% cm 1, respectively. For electrons, the recommended criteria were 4% and 4 mm, respectively, with the same gradient specification and definition. In each case, these criteria can be applied to the dose-difference and DTA distributions to identify regions that fail the criteria. These are consistent with the aforementioned regions the dose-difference and DTA distributions are not overly sensitive. If these guidelines are followed, a logical extension to the dose difference and DTA is a simultaneous examination of the dosedifference and DTA criteria. 8,10 A software tool has been developed to support 3D radiation therapy treatment planning system testing 11 by implementing the dose-distribution superposition, dose-difference, and DTA analyses described above. In addition, a composite distribution is developed based on the concepts of Shiu et al. 8 as applied by Cheng et al. 10 The dose-difference and DTA tests are overly sensitive in regions of high- and lowdose gradients, respectively. The composite distribution analysis shows only regions that fail both criteria, and is therefore is not overly sensitive in these regions. For example, in regions of high-dose gradient, the dose-difference 1830 Med. Phys. 25 10, October 1998 0094-2405/98/25 10 /1830/7/$10.00 1998 Am. Assoc. Phys. Med. 1830

1831 Harms et al.: Tool for the evaluation of algorithms 1831 distribution yields large values for small spatial offsets between the tested dose distributions. The DTA analysis will return the approximate spatial difference between the two distributions. If the spatial offset is less than the criteria, the DTA analysis will pass, and the algorithm will have passed the composite analysis. The software tool compares measured and calculated planar dose distributions for identical beam and phantom geometries. Because of limitations of dose-distribution measurement and display devices, comparisons are made on a planar basis and calculated dose distributions are not examined beyond the comparison plane. Because the measured distribution will, in general, have fewer data points, comparisons are made for each measurement point, and the comparison quantities are determined on the measurement coordinate system. The dose-difference, (r m ), at position r m on the measured distribution is equal to r m r m,r c r m r c, r m,r c D m r m D c r c is the general difference between dose values on the measured, D m (r m ), and calculated, D c (r c ), distributions at r m and r c, respectively. The DTA, d(r m ), is given by d r m min r 0 r m,r c r c, r 0 r m,r c r r m,r c r m,r c 0 is the distance between the set of points that have a difference of zero between calculated and measured doses and r r m,r c r m r c. The composite distribution c(r m ) is a binary distribution formed by the points that fail both the dose-difference and DTA criteria, D M and d M, respectively. c r m f r m d f r m, and f r m d f r m 1 2 3 4 5 6 0 r m D M 7 1 r m D M 0 d r m d M 8 1 d r m d M are the points that fail the dose-difference and DTA criteria, respectively. The criteria D M and d M selected for the examples are 3% and 3 mm, respectively, the dose difference is in percent of a selected normalization value e.g., the dose at d max. The selection of d M 3mm is based on our internal quality assurance criteria. A binary distribution does not easily lend itself to interpretation, so the dose-difference distribution is presented in the regions c(r m ) 1. The analysis programs are written within the confines of a commercial scanning dosimetry system software platform DynaScan, Computerized Medical Systems, Inc., St. Louis, MO. It provides the ability to develop only the tools necessary for computing the required distributions while allowing the use of existing software to handle the tasks of displaying and plotting the isodose, isodose difference, iso-dta, and composite distributions. An additional program is written on a UNIX platform to extract the desired planar dose matrices from computed 3D dose distributions. Ideally, the entire 3D calculated dose distribution would be used to generate the comparisons. However, software limitations restrict the comparison to 2D planar extractions. The calculated dose-distribution planar extraction routines include software that 1 allows the user to specify the dose plane to be extracted from the 3D dose matrix, 2 extracts the dose for the specified plane, and 3 writes an intermediate text data file containing the planar dose array for import by the dosimetry system software. The data files are transferred from the 3D RTP system 11 to the dosimetry system by means of a computer network. To facilitate the dose-difference computations, the measured and calculated dose-distribution datasets are resampled onto a common rectilinear grid. This common grid is constructed from the spatial intersection of the grids from both datasets. For transverse photon distributions, the spacing of the grid points is 0.2 cm across the beam and 0.5 cm along the beam. A grid spacing of 0.2 cm is used along both axes for electron beams and beam s eye view photon distributions, and will be used in subsequent tests of intensity modulated dose distributions. The coarser spacing for transverse photon distributions is used due to the relatively small dose gradient along the depth direction. To ensure correct divergence for transverse distributions, the resampling is performed along ray lines intersecting the rectilinear matrix points. An additional feature of this resampling is that both matrices are rescaled so that their dose values are a percentage of the normalization dose. The dose-difference distribution is calculated for each point on the common grid. Software programs are added to the dosimetry system software to facilitate implementation of the quantitative dose analysis. These routines include programs to 1 convert the common text file format into the internal file format of the dosimetry system, 2 resample the measured and calculated planar dose matrices onto a common, regularly spaced grid, 3 compute dose-difference distributions (r m ), 4 compute DTA distributions d(r m ), and 5 generate composite distributions c(r m ) to demonstrate regions outside the selected acceptance limits. The DTA calculation is made point-by-point on the common grid. The DTA value for a specific measurement dose point is determined by searching the calculated dose matrix for the same dose value. In general, the calculated dose points bracket the measured dose value. A method is used that rapidly interpolates the location of the calculated dose value on the calculation matrix. Figure 1 shows an example of the DTA calculation, the DTA is to be determined for point r m (x m,y m ) with dose value D m (x m,y m ). The

1832 Harms et al.: Tool for the evaluation of algorithms 1832 FIG. 1. Drawing of the distance-to-agreement calculation geometry. The ideal isodose line is shown, along with the intersections, I x and I y, used to calculate the list of distances, r 0 (r m,r c ), from which the minimum, d(r m ), is selected. FIG. 2. Beam s eye view of mantle portal blocking geometry. The tic marks indicate 1 cm spacing projected to the plane of isocenter. The dashed cut line shows the intersection with the transverse plane evaluated in Figs. 3 and 4. The dark and light lung blocks are 1 HVL and 5 HVL, respectively. calculation points r c under investigation in this example are (x c,y c ), (x c x,y c ), and (x c,y c y) with doses equal to D c (x c,y c ), D c (x c x,y c ), and D c (x c,y c y), respectively, x and y are the matrix spacing in the x and y directions, respectively. The interpolated intersections of the D m (x m,y m ) isodose line with the calculation matrix are identified in Fig. 1 by I x and I y. The positions of I x and I y are determined by linear interpolation of the calculation matrix and I x x c D m D c x c,y c M x I y y c D m D c x c,y c M y, 10 and M x D c x c x,y c D c x c,y c x 9 11 M y D c x c,y c y D c x c,y c. 12 y It is possible that the intersection coordinates I x and I y of the isodose lines with the matrix segments do not lie between the points (x c,y c ), (x c x,y c ), and (x c,y c y). A test is applied to I x and I y to determine if they are suitable for inclusion in the list of values r 0 (r m,r c ) by inspecting their values and determining if they lie between (x c,y c ), (x c x,y c ), and (x c,y c y). The DTA search is not conducted over the entire calculation space. A DTA value greater than 1.0 cm is considered excessive usually associated with a low dose-gradient region, so the search is terminated when no suitable point in the calculation matrix is located within 1.0 cm of the measurement point. The value of 1.0 cm is used as the DTA in these cases. To show the tool s effectiveness, two photon beam dose calculation algorithms are presented. 12,13 The measured dose distribution from a 6 MV photon beam with a mantle field portal shown in Fig. 2 is compared against calculations using the convolution adapted ratio tissue air ratio CAR- TAR algorithm 13 and the ratio-tar RTAR algorithm. The CARTAR algorithm models block penumbras, while the block penumbra of the RTAR algorithm is a discontinuous change in dose level. The mantle block contains both fully attenuating 5 half-value layers HVL and partial transmission 1 HVL lung blocks. The results for the CARTAR mantle calculation are shown in Fig. 3. Regions of interest have been identified by letters A to C. Region A is at the left-most collimator-defined penumbra; region B lies at the left edge of the central open field portion of the beam, and region C lies beneath the 1 HVL lung block. Figure 3 a shows the superimposed isodose distributions. Quantitative assessment of the dose calculation quality is difficult using only this information. The dose-difference distribution is shown in Fig. 3 b and highlights regions the dose distributions disagree by more than 3%, principally within the block and collimator penumbra regions A and B. The DTA distribution shown in Fig. 3 c appears more complex than the dose-difference distribution. Region A exhibits a DTA of more than 3 mm, while region B does not, indicating that a discrepancy of more than 3 mm exists between the calculated and measured collimator penumbra positions, but that the block positions and penumbra are accurately modeled. Region C contains DTA values of up to the 1.0 cm limit the software is written to display the results in mm due to the relatively small dose gradient in this region. The composite distribution is shown in Fig. 3 d. As previously mentioned, the dose difference is selected as the plotted quantity. The composite distribution clearly shows that the dose calculation fails in region A, but passes in regions B and C. This identifies the need to further investigate the collimator positions and penumbra. Additional regions of disagreement lie near the surface within the build-up region.

1833 Harms et al.: Tool for the evaluation of algorithms 1833 Figure 4 shows the RTAR algorithm results. Figure 4 a shows the superimposed isodose distributions. Region A appears similar to that for the CARTAR algorithm, due to the collimator penumbra model in the RTAR algorithm. The isodose lines near the block penumbra, however, have an unrealistic square shape. It is clear that the RTAR algorithm does not model the block penumbra as well as the CARTAR algorithm. As in Fig. 3, quantitative assessment is difficult. FIG. 3. a Superimposed measured and calculated isodoses for the transverse, off-axis plane shown in Fig. 2. The convolution-adapted ratio-tar algorithm is used, and doses are shown as a percent of the central axis dose at d max. The measurement is plotted with a solid line and the calculation with a dashed line. b Dosedifference plot for distributions shown in a. The curves indicate the percentage error relative to the normalization dose value of 100%. c Distance-toagreement distribution in mm for dose distributions shown in a. d Composite distribution demonstrating regions in which the dose difference and distance-to-agreement both exceed acceptance limits of 3% and 0.3 cm, respectively.

1834 Harms et al.: Tool for the evaluation of algorithms 1834 FIG. 3 Continued. Figure 4 b shows the composite distribution. Region A appears similar to that of the CARTAR algorithm see Fig. 3 d. The CARTAR algorithm passes in region B, but the RTAR algorithm does not pass, a fact made clear by the composite distribution. The RTAR algorithm passes in region C due to the relatively low dose level and subsequent small dose difference. This software tool has been extremely effective for plane-

1835 Harms et al.: Tool for the evaluation of algorithms 1835 FIG. 4. a Superimposed measured and calculated isodoses, for the transverse, off-axis plane shown in Fig. 2. The calculation uses the ratio-tar algorithm. The measurement is plotted with a solid line and the calculation with a dashed line. b Composite distribution demonstrating areas in which the dose difference and distance-toagreement both exceed the acceptance limits of 3% and 0.3 cm, respectively. The curves indicate percent dose difference relative to the normalization dose value of 100%. by-plane analysis of 3D dose calculation software. The traditional method of dose calculation software review, in which the evaluation is limited to a subjective review of the superposition of isodoses, is inadequate for commissioning 3D treatment planning systems. This tool provides an effective means for the physicist to clearly identify and document the commissioning of dose calculation software. While the current software compares only calculation planes that coin-

1836 Harms et al.: Tool for the evaluation of algorithms 1836 cide with the measured planes, in the future, the DTA distribution will consider calculation points within the entire 3D dose calculation volume. Manufacturers of isodose scanning systems are encouraged to work with 3D treatment planning manufacturers to implement the type of quantitative analysis tools described in this report. a Also at William Beaumont Hospital, Royal Oak, MI. 1 E. Mah, J. Antolak, J. W. Scrimger, and J. J. Battista, Experimental evaluation of a 2D and 3D electron pencil beam algorithm, Phys. Med. Biol. 34, 1179 1194 1989. 2 B. A. Fraass, Quality assurance for 3D treatment planning, in Teletherapy: Present and Future Advanced Medical Publishing, Vancouver, B.C., 1996. 3 B. A. Fraass and D. McShan, Three-dimensional photon beam treatment planning, in Radiation Therapy Physics, edited by A. R. Smith Springer-Verlag, New York, 1995. 4 B. A. Fraass, M. K. Martel, and D. L. McShan, Tools for dose calculation verification and QA for conformal therapy treatment techniques, in XIth International Conference on the use of Computers in Radiation Therapy Medical Physics Publishing, Madison, WI, 1994. 5 K. R. Hogstrom, M. D. Mills, J. A. Meyer, J. R. Palta, D. E. Mellenberg, R. T. Moez, and R. S. Fields, Dosimetric evaluation of a pencil-beam algorithm for electrons employing a two-dimensional heterogeneity correction, Int. J. Radiat. Oncol., Biol., Phys. 10, 561 569 1984. 6 ICRU, Report 42, Use of Computers in External Beam Radiotherapy Procedures with High-Energy Photons and Electrons ICRU, 1987. 7 H. Dahlin, I. L. Lamm, T. Landberg, S. Levernes, and N. Ulso, User requirements on CT based computerized dose planning systems in radiotherapy, Acta Radiol.: Oncol. 22, 398 141 1983. 8 A. S. Shiu, S. Tung, K. R. Hogstrom, J. W. Wong, R. L. Gerber, W. B. Harms, J. A. Purdy, R. K. TenHaken, D. L. McShan, and B. A. Fraass, Verification data for electron beam dose algorithms, Med. Phys. 19, 623 636 1992. 9 J. Van Dyk, R. B. Barnett, J. E. Cygler, and P. C. Shragge, Commissioning and quality assurance of treatment planning computers, Med. Phys. 26, 261 273 1993. 10 A. Cheng, W. B. Harms, R. L. Gerber, J. W. Wong, and J. A. Purdy, Systematic verification of a three-dimensional electron beam dose calculation algorithm, Med. Phys. 23, 685 693 1996. 11 J. A. Purdy, W. B. Harms, J. W. Matthews, R. E. Drzymala, B. Emami, J. R. Simpson, J. M. Manolis, F. U. Rosenburger, Advances in 3- dimensional radiation treatment planning systems: Room-view display with real-time interactivity, Int. J. Radiat. Oncol., Biol., Phys. 27, 933 944 1993. 12 D. A. Low, X. R. Zhu, W. B. Harms, J. W. Matthews, J. A. Purdy, A. Cheng, and R. L. Gerber, Evaluation of ratio-tar photon and electron pencil beam dose calculation algorithms for 3D treatment planning, in XI International Conference on the Use of Computers in Radiation Therapy Manchester, England, 1994. 13 X. R. Zhu, D. A. Low, W. B. Harms, and J. A. Purdy, A convolution adapted ratio-tar algorithm for 3D photon beam treatment planning, Med. Phys. 22, 1315 1327 1995.