Multi-Target Tracking In Clutter

Transcription

1 Multi-Target Tracking In Clutter John N. Sander-Reed, Mary Jo Duncan, W.B. Boucher, W. Michael Dimmler, Shawn O Keefe ABSTRACT A high frame rate (0 Hz), multi-target, video tracker ha been developed and ued in an operational environment. The tracker i able to track multiple target imultaneouly, while providing fine-track track error on a uer elected target. The ytem i able to accommodate real-world iue uch a enor bad pixel and natural background clutter. Several fine track mode are available, including centroid, leading edge, and correlation. The tracker provide ub-pixel track accuracy againt both reolved and unreolved target. The ytem ha been extenively teted againt radiometrically correct cloud cene and ha been uccefully integrated in a tracking ytem to provide multi-target tracking of real target viewed againt terrain and cloud background. Keyword: tracking, multi-target, clutter, electro-optic. INTRODUCTION A high frame rate, preciion, electro-optical image-baed tracker ha been developed to perform general purpoe, multitarget, ground-to-air tracking. The tracker accept target cueing from a wide area urveillance ytem and perform target acquiition and tracking to provide high tability, ub-pixel tracking. The tracker build on previou experience with imaging tracker for preciion tracking ytem and preciion pointing and tracking work -4. The tracker can maintain track on everal object in the FOV at one time. The ytem i deigned to track both reolved and unreolved target againt natural cloud background 5. While target mut have ome local contrat, the target Signal to Noie Ratio (SNR) over the entire image may be much le than. The tracker algorithm include proceing for background clutter uppreion, enor defect compenation, and ub-pixel, fine track proceing on the primary target. The fine track proceing can provide a centroid, edge, or correlation track. The tracker accept -bit digital imagery at 0 frame per econd with a latency of le than one frame time.. ALGORITHM DESCRIPTION.. Proceing Stage The tracker ha three primary proceing tage: Data input and caling, Target detection and gate izing, and Fine track poition etimation. The input tage accept -bit digital imagery and cale it to 8-bit for proceing. An optional bad pixel removal algorithm can be run on thi data. Next, an edge enhancement operation i applied to the entire image. Thi tend to highlight harp edge of a target, while uppreing bright background region, uch a cloud. Cloud edge are alo enhanced but thee are uually not a harp a man-made target and hence are not enhanced a much. The mean and tandard deviation of the edge enhanced image are computed and ued to et a threhold. The reult i a lit of pixel above the threhold. An alternative bad pixel removal algorithm can be applied at thi point (only one or the other can be applied, not both). The pixel are now paed through a multi-frame coincidence check. The coincidence check aume that the target i moving relative to the background, and hence any background edge which paed the threhold operation will tend to move acro the image at a fater rate than edge pixel from a target. Since the target i being tracked (either by the wide area urveillance ytem or by the tracker), it hould remain relatively tationary. Pixel which pa the coincidence check are declared target pixel and clumped to form dicrete target. A minimum and maximum target ize threhold i applied to the dicrete target and target which are either too large or too mall are rejected. Once the target have been identified, track gate are placed around the individual target. The track gate are Jack.Sander-Reed@boeing.com; phone (505) ; fax (505) ; Boeing-SVS, Inc. 44 The 5 Way NE, Albuquerque, NM 8709 SPIE 474 April, 00

2 dynamically ized to the target. At thi point, multiple target have been identified and located to approximately pixel accuracy. Fine track proceing i performed only on the primary target. The poition of the remaining target i etimated a the center of the bounding box for each target, baed on the edge pixel which paed the coincidence check. Fine track proceing ue the original 8-bit grey cale pixel (before edge enhancement) located within the primary target track gate. The mean and tandard deviation of thee pixel i computed and ued to et a threhold. The centroid of the pixel above threhold i computed. If centroid fine track i the elected fine track mode, thi i the etimated target location. If leading edge fine track i elected, a leading edge algorithm i ued to find the edge of the target in the direction of gimbal motion. If correlation tracking i elected, a template baed on a temporally weighted average of the target from the previou frame i ued a a correlation map to find the target location in the current frame. Whichever method i ued to determine the fine track poition, the difference between thi poition and the center of the image i computed a the track error which i ent to the ret of the control ytem, external to the tracker. A block diagram of the algorithm i hown in Figure. Image 8 Bit Scaling Edge Enhancement Threhold Hitory Map Coincidence Check Clump & Size Threhold Place Track Gate Fine Track - Centroid - Leading Edge - Correlation Track Error Figure. Track algorithm block diagram The following ection decribe each tep of the algorithm in more detail. There i alo a dicuion of the tracker mode logic... Data Input and Scaling... Data Scaling The tracker accept -bit grey cale digital imagery, but procee only 8-bit imagery. Therefore the input tage ha to cale the imagery from to 8 bit. A Look-Up Table (LUT) having 4096 entrie (indexed 0 to 4095) with value between 0 and 55 i ued. Each incoming -bit pixel ha a value between 0 and Thi value i ued a the index into the table. The pixel value i replaced with the value in the LUT at that location, reulting in a value between 0 and 55, thu giving an 8-bit reult. The LUT begin a a divide by 6 LUT (hift right 4 bit), picking the mot ignificant 8 bit of the incoming image. The LUT i adjuted baed on the tatitic of the current 8-bit image. Once the -bit image ha paed through the current LUT, the image mean i computed to determine if it i too large or too mall. If the mean i greater than 5, the SPIE 474 April, 00

3 LUT hift i increaed by (equivalent to divide by more than the current LUT). If the image mean i le than 3, the LUT hift i decreaed by (equivalent to divide by le than the current LUT). For thi reaon the algorithm i called a bit-picker. The new LUT will be applied to the next incoming -bit image.... Bad Pixel Removal Two different bad pixel rejection cheme were implemented in the tracker, one in the data input ection and one, decribed later, applied after edge enhancement and threholding. Thee two technique are excluive, either one or the other i elected at compile time. The camera ha built-in bad pixel rejection, which eliminate pixel which are alway on and pixel which are alway off. The camera alo ha a two point flat field correction which compenate for pixel-to-pixel non-uniformitie in the dark current and reponivity. If the camera view a uniform cene, immediately after the two-point flat field calibration i performed the camera output a uniform image in which all pixel are randomly cattered about the image mean with a mall tandard deviation. However, due to the manufacturing proce, ome pixel are untable over time and the output from thee pixel drift higher or lower than the other pixel in the cene. The tracker need to be able to identify and eliminate thee pixel o they do not lead to fale detection. The input tage bad pixel removal ue a Rank Value Filter (RVF) combined with a lit of bad pixel location. Two copie of the unproceed 8-bit image are generated. A rank value filter i applied to one copy of the image. The RVF generate a lit of pixel value for the current pixel, and the urrounding pixel (typically a 3x3 or 5x5 region), and order the lit from brightet to darket. The value at the current pixel i replaced with the econd brightet value in the lit. The effect i imilar to a median filter and can be ued to remove impule noie, uch a bad pixel. The hardware RVF can only be applied to the entire image, not to individual pixel or region. However, if the RVF were applied to the entire image, it would alo remove ingle pixel target. Therefore, a bad pixel map i ued to copy value only at bad pixel location from the rank value filtered image to the unproceed 8-bit image. Thi unproceed image in which only bad pixel location have been replaced by RVF value i the image paed to the ret of the proceing chain. The bad pixel map ued to decide which location to copy i generated a decribed for the alternative bad pixel removal technique, in the threholding ection..3. Target Detection and Gate Sizing Target detection require uppreion of background clutter (cloud) and ignal enhancement of the target. In order to perform thi, we rely on two phenomena: Firt, the (man made) target will tend to have harper edge than the natural cloud background, and Second, in mot cae, the target will be moving relative to the background. Note that thi applie even in the cae of a rocket launched at the tracker ince the rocket will fly a parabolic trajectory. In thi cae the target will be een to rie vertically againt the background, and only in the terminal tage of flight will it have no motion relative to the background. The proceing tage conit of edge enhancement, threholding, a multi-frame coincidence check, clutering to form dicrete target, application of target ize threhold, and track gate placement and izing..3.. Edge Enhancement A modified Sobel edge enhancement i ued. The tandard Sobel edge enhancement ue the two 3x3 convolution kernel given in equation (). 0 S = 0 S = () 0 The reult are added in quadrature o that the Sobel image S, may be written a hown in equation (): ( I S ) + ( I ) S =, () S where I Si repreent the convolution of the image I with the Sobel kernel S i. The tandard Sobel kernel reult in pixel at a tep function edge in an image (note that thi reult in an unbiaed indication of the edge location). If an image with identical row coniting of pixel value ( ) i paed through the Sobel edge enhancement operation, the reult will be an image with identical row coniting of pixel value ( ). SPIE 474 April, 00 3

4 In order to reduce the number of edge pixel ued in later proceing, a modified pair of convolution kernel were developed. 0 K = 0 0 K =. (3) Thee kernel have the property of producing only a ingle pixel in the edge enhanced image, rather than two, thu reducing the number of edge pixel ued in later proceing. The price i a half pixel bia in the edge location. Rather than performing the quadrature addition of the tandard Sobel enhancement, we imply add the two convolution image and take the abolute value: K = I K + I. (4) K For the example image above, the new kernel reult in an edge enhanced image with identical row coniting of pixel value ( )..3.. Threholding The mean (µ ) and tandard deviation (σ ) of the edge enhanced image are computed. A threhold (T ) i computed with a uer upplied parameter (α ): T = α σ + µ. (5) Alternatively, an abolute threhold value can be elected by the uer. The threhold i applied to the edge enhanced image and a lit of the location of all pixel above the threhold i generated. An alternative bad pixel removal algorithm can be applied at thi point. A bad pixel map i generated by viewing a uniform cene. The tandard proceing chain i applied through the threhold operation. Any detection are placed in the bad pixel map. Thi bad pixel map i then ued during normal tracking operation. During normal proceing, the lit of detection i compared with the bad pixel map. Any detection at the ame pixel coordinate a an entry in the bad pixel map are removed from the detection lit. The choice of which bad pixel removal algorithm, to ue (thi one or the input tage algorithm) i a compile time option Coincidence Check The coincidence check i ued to differentiate any cloud edge which pa the threhold detection, from legitimate target edge, baed on motion in the image. The aumption i that the target i being tracked (either by the wide area urveillance ytem or by the tracker), and hence i relatively tationary in the image, while the background (cloud) appear to lew acro the image. The maximum allowable motion for a target edge i pixel per frame. An edge ha to pa the coincidence check for even conecutive frame to be declared a target. The coincidence check i implemented uing a hitory map, which ha the ame dimenion a the image. Each pixel location in the hitory map contain the number of conecutive frame in which an edge ha been oberved at that location. Initially, the hitory map conit of all zero. A each ucceive frame i proceed, the current lit of detection i ued to create a new hitory map. A value of one i placed at the location of each of the current detection. Now the old hitory map i canned in a pixel radiu about each of the current detection. If a non-zero entry i found in the old hitory map within thi pixel earch radiu, the value i added to the detection pixel in the new hitory map. The effect i to move the hitory count to the current detection location. Since the new hitory map only ha entrie at the location of current detection, a econd effect i to eliminate any entrie in the old hitory map that are not mapped to a current detection. Thee effect are hown in Figure with the earch area hown in grey. SPIE 474 April, 00 4

5 = Old Hitory Current Detection New Hitory Figure. Coincidence check hitory map update Target detection pixel are only declared at location for which the hitory map contain a value of 7 or greater. Thu it take at leat 7 frame for a new target to be declared. The lit of hitory map location having a value of 7 or greater i paed to the next tage of proceing for clumping into dicrete target. Note that while there i a 7 frame latency in acquiring a new target, track error are generated uing data from the current frame and thu there are no multi-frame latency or filtering effect in the track error. While the coincidence check work well once a table track ha been etablihed on a target, the proce of bringing a target to the center of the image can caue a problem for the coincidence check. The external control ytem attempt to minimize the difference between the detected target poition and the center of the image. When a target i initially detected it may be anywhere in the image. Upon detecting the target, the control ytem i capable of rapidly moving the gimbal to bring the target to the center of the image. Thi may reult in target motion much greater than pixel per frame. The tracker accept an input from the control ytem which indicate how many pixel the target i expected to move in the next frame. Thi i ued to hift the hitory map, prior to creating a new hitory map baed on current edge detection. Since the gimbal i moving at approximately a contant rate to keep the target tationary during acquiition, thi input ignal i an acceleration term Clumping and Size Threhold The lit of pixel which pa the coincidence check are now clutered to form a few dicrete target. Pixel are clumped in rectangular region. Pixel which are within 4 pixel of each other are combined into a ingle target. The minimum and maximum extent of each clump are computed, in both the x and y direction. Minimum and maximum ize threhold are applied to the clump and ued to reject clump which are either too large or too mall. The minimum and maximum ize threhold pecify the x and y extent of the target, a oppoed to total number of pixel Gate Placement and Sizing A track-gate ize i computed for each of the target. The track gate i uppoed to be ized to leave a 5 pixel buffer around the target, but due to hardware limitation, only 9 preet track gate ize are available. Baed on the computed track gate ize, the next larger track gate i ued. If multiple target are initially preent, the larget i elected a the primary target. It i maintained a the primary target in ubequent frame, even if it i no longer the larget target. If the primary target diappear (i not detected) the ytem enter coat mode (decribed later) until the target i again detected, the operator command the ytem out of coat mode, or the ytem time out of coat mode. If a target other than the primary target i not detected, it track gate i immediately available for reaignment to other target a they appear..4. Fine Track Proceing Fine track proceing i performed on the primary target only. For centroid and leading edge fine track proceing, the original 8-bit imagery (without edge enhancement) i ued. The mean (µ) and tandard deviation (σ) of the pixel within SPIE 474 April, 00 5

6 the track gate are computed. A threhold (T) i determined in a manner imilar to that with the edge enhanced image. The uer can pecify an abolute grey cale threhold, or a threhold baed on the image tatitic and a uer parameter (α): T = ασ + µ. (6) In either cae, the centroid of the pixel above threhold i computed. Thi i the target fine track centroid poition. Leading edge fine track proceing i an extenion of the centroid fine track. A line i computed from the centroid location, in the direction of gimbal motion (or alternatively, in an angle pecified by the operator). Now, beginning where thi line interect the track gate, a econd line i computed, perpendicular to the firt. Thi econd line i moved backward, toward the centroid location, until ome point on the line firt interect a target pixel. The point where the econd (perpendicular) line interect the firt (gimbal motion) line, when thi occur, i declared the leading edge. Note (figure 3) that thi point may not correpond to a phyical point on the target at all. Gimbal Motion Target Centroid Etimated Leading Edge Poition Figure 3. Leading edge poition etimation Unlike the centroid and leading edge fine track algorithm, the correlation algorithm ue the lit of edge pixel ued to form the target. The correlation template i a tatic template, generated when the ytem firt enter correlation fine track mode. The template i the lit of edge pixel in the primary target at that time. Thi template i correlated with the lit of edge pixel for the primary target in each ubequent frame in order to determine the correlation fine track poition..5. Mode Logic There are four primary mode for the tracker (in addition to tartup and hutdown mode). Thee are Standby, Acquire, Track, and Coat. During Standby mode, input i ignored and no track error are generated. The ytem i commanded from Standby to Acquire mode. In Acquire, the ytem earche for target. Once at leat one target ha been identified, the ytem enter Track mode and attempt to fine track the primary target. The ytem remain in Track mode until commanded to a different mode (Standby or Acquire), or until the primary target i lot. If the primary target i lot, the ytem enter Coat mode. In the tandard verion of the tracker, the ytem remain in Coat mode until either the target i reacquired, or the operator command the tracker back to Standby or Acquire mode. However, other verion of the tracker eventually time-out of Coat mode and automatically re-enter Acquire mode. If multiple target are preent, the operator can command the tracker to change primary target from the current target to another operator deignated target. Statu output indicate the mode (Standby, Acquire, Track, or Coat), and if the ytem i in track mode, whether the track error i from the edge enhanced image (coare track) or from fine track proceing..6. Improvement Several improvement to the current algorithm may be conidered. The current coincidence check reject edge pixel moving at more than pixel per frame. A more robut implementation might determine the overall background motion, SPIE 474 April, 00 6

7 and reject edge pixel moving within pixel per frame of thi motion. Thi would provide more background pecific clutter uppreion while allowing greater target motion during acquiition, or greater relative target motion. Currently the track gate are contrained to be rectangle aligned with the image frame. The ability to ue rotated, rectangular track gate would allow the gate ize to be reduced for extended target, and hence provide both better background uppreion in fine track mode and allow object to be more cloely paced and till ditinguihed. The leading edge fine track algorithm hould be modified uch that it declare either the interection point of the perpendicular line with the object, or the firt target pixel on the gimbal motion vector to be the leading edge. Ideally, the operator could chooe which point would be reported. 3. REAL-TIME IMPLEMENTATION The tracker accept bit digital imagery and a frame rate of 0 Hz over a digital data interface. An alternative verion of the tracker ha been developed which accept analog video at a 60 Hz frame rate. Actual gimbal pointing and rate information i tranmitted to the tracker over a erial link. Track error output are ent over a eparate erial link. Another bi-directional erial link provide the command and control interface. The hardware conit primarily of an image proceing board and a general purpoe CPU board. Mot of the algorithm i written in general purpoe C code, with ome hardware pecific code. 4. SUMMARY A high frame rate (0 Hz), multi-target, video tracker ha been developed and ued in an operational environment. The tracker i able to track multiple target imultaneouly, while providing fine-track track error on a uer elected target. The ytem i able to accommodate real-world iue uch a enor bad pixel and natural background clutter. Several fine track mode are available, including centroid, leading edge, and correlation. The tracker provide ub-pixel track accuracy againt both reolved and unreolved target. The ytem ha been extenively teted againt radiometrically correct cloud cene and ha been uccefully integrated into a ytem tracking multiple live target. REFERENCES. P. Merritt, M. Kramer, Field Tet of active tracking of a ballitic miile in the boot phae, Proc SPIE, 3086, April M. Schulthe, S. Baugh, T. Schneeberger, Control Sytem Conideration for a Preciion Pointing and Tracking Experiment on a High Altitude Balloon 3th International Federation of Automatic Control, September -6, 994, Palo Alto California. 3. W.B. Boucher, M.R. Michnovicz, Dynamic Range Selection in Image Proceing Hardware to Maximize SNR while Avoiding Image Saturation, Proc SPIE, 369, April, S. Chodo, Track loop bandwidth, enor ample frequency, and track loop delay, Proc SPIE, 3365, April "Scene generation for tracking mall target againt cloud background", J.N. Sander-Reed, W.M. Dimmler, Proc. SPIE, 4376, April 00. SPIE 474 April, 00 7