Examination of Hybrid Image Feature Trackers

Transcription

1 Examination of Hybrid Image Feature Trackers Peter Abeles Robotic Inception Abstract. Typical image feature trackers employ a detect-describe-associate (DDA) or detect-track (DT) paradigm. Intuitively, a hybrid of the two approaches inherits the benefits of each approach and possibly their defects, however this has never been demonstrated formally in a more general setting. In this paper, the stability and speed of DDA, DT, and hybrid trackers are compared and discussed using a diverse set of real-world video sequences. Introduction Image point feature tracking in video sequences is a key component in many computer vision problems, such as structure from motion and object detection/recognition. Specific applications include: scene recognition for loop-closure, face recognition, video stabilization, visual odometry, and augmented reality. The most popular feature trackers can be categorized as employing detect-describe-associate (DDA) or detect-track (DT) paradigms. DDA-based trackers operate by ) detecting feature locations and characteristics, 2) describing each feature with local image information, and 3) associating features between images. Numerous feature detectors (Harris [], Shi-Tomasi [2], FAST [3], MSER [4]) and local feature descriptors (SIFT [5], SURF [6], BRIEF [7]) are available for DDA trackers. Association is done by selecting pairs of feature descriptors between two images that minimize an error metric. DDA trackers are robust to temporary obstructions and abrupt changes in view, but are computationally expensive. DT trackers operate by first detecting features, then updating each feature track location by performing a local search initialized using the last known location. Several template based DT trackers have been proposed in the past, with Kanade-Lucas-Tomasi (KLT) [8,2] being the most popular. Under nominal conditions, DT trackers are faster and more stable than DDA trackers, but if the scene changes abruptly, a DT tracker will fail and cannot recover. Despite DT trackers having superior nominal performance, their usage is limited to a subset of problems that DDA is applied to. The reason for DDA s wider usage is its resilience and ability to recover tracks. To overcome this issue, hybrid approaches have been proposed in the past in an attempt to improve runtime speed and/or stability. Past hybrid approaches were application specific, making their results difficult to generalize. Most practitioners in the field accept or assume these qualitative statements about the relative merits of each type of tracker. However, a direct comparison of these three More commonly referred to as detect then track.

2 2 Peter Abeles approaches, which validates and quantifies this knowledge, does not exist in the literature. In this work, a performance study is presented that compares the three types of trackers. To facilitate this comparison a generic hybrid tracker is described. Real-world video sequences are used to evaluate performance, each of which is designed to stress the trackers in different ways. 2 Related Work and Background The first step for both DDA and DT trackers is feature detection. Corner (e.g. Harris and Shi-Tomasi) and blob (e.g. Hessian Determinant [9]) detectors are commonly used to identify salient features within images for tracking purposes. After a feature has been detected, a description of that feature is extracted from its local region. DT trackers can use simple templates as descriptors due to their assumption that the scene is slowly changing. DDA make a weaker assumptions and require descriptors that can handle greater changes in appearance. The major philosophical difference between the two approaches involves how tracks are updated. DDA perform an expensive global search for features, followed by association. DT performs a local search for each track independently. Feature association is an expensive operation for DDA trackers. The simplest solution has a complexity of O(N 2 ), although faster approximate alternatives are available (e.g. k-d trees []). An optimal solution is defined as the two features that minimize a distance metric, often the L 2 (Euclidean) norm: a i = min j F i F j () where F and F are sets of feature descriptors from two different images. All DT trackers work by minimizing the difference between a template and the image with a local search around feature s previous location. Besides KLT, several others have been proposed [,2]. It was pointed out by Baker and Matthews [3] that these approaches are equivalent to a first order approximation. The primary difference between each approaches lies in the types of motion models used and if the template and/or image is warped. While mathematically similar, these differences significantly affect computation performance. KLT updates track locations using a translational motion model and minimizes the difference between the track s description and the image using the following cost function: ɛ = [I(x d) J(x)] 2 w dx (2) W where W is a local region around location x, w is an optional weight, d is the translation parameter being optimized, and I and J are the first and second images in the sequence. Pyramidal approaches are used to handle larger displacements. Unlike DDA trackers, DT trackers runtime speed is dependent on the number of tracks and not image size. Track stability is often improved by updating the description after each frame. The downside to updating the descriptor is that image noise will cause tracks to perform a random walk.

3 Lecture Notes in Computer Science 3 Several hybrid trackers that employ both DT and a DDA tracker have been proposed in the past. In Uemura and Mikolajczyk [4] diverged KLT tracks are detected using a SIFT descriptor when detecting human actions. In Pilet and Saito [5] robustness is added to a region based normalized-cross-correlation (NCC) tracker by switching to KLT when association fails to find a match. To reduce image processing overhead for visual loop closing in SLAM, Pradeep et. al [6] proposed to attach SIFT descriptions to KLT tracks with periodically associating all tracks to newly detected features. Ladikos et. al [7] track the pose of known objects using a combination of template (DT type) and feature (DDA type) trackers. In all of the just mentioned works, a direct evaluation of track performance is lacking. In those works, performance is measured indirectly as a function of the target application, e.g. localization accuracy or detection rate. Test environments are limited and focused on specific applications. They also lack a more generalized discussion of design issues and failure conditions unique to hybrid tracking. 3 Generalized Hybrid Tracker For purposes of comparison, a generalized hybrid DDA-DT tracker with a modular design is described below. For sake of brevity this specific tracker will be referred to as the hybrid tracker. The previously mentioned hybrid trackers were designed for specific applications. The generalized hybrid tracker only specifies a high-level design and requires that specific implementations for feature detection, feature description, and DT tracking be provided to it. In the hybrid tracker, tracking starts by spawning new tracks from detected image features. A feature track is defined by a 2D location and has two types of descriptions, namely, DDA description and DT description. The DDA description is immutable and the DT description is updated after each image is processed. When the next image in the sequence arrives, tracks are first updated using the DT tracker. If a track fails a DT update, the track is placed in storage. Tracks in Drop Unused Storage Failure Detect Describe Associate Matched Track Tracker Success Unmatched Feature Spawn Fig. : Flow chart showing track life cycle. Track locations are updated using DT tracker. Tracks in storage are periodically reactivated from detected features. storage are not updated as new images arrive. When triggered, new tracks can be spawned and old tracks in storage reactivated. A trigger can be the number of active tracks dropping below a threshold and/or periodically after N frames are processed. The purpose of using a trigger is for computational efficiency alone.

4 4 Peter Abeles The first step in spawning new tracks or reactivating tracks in storage is detecting image features. After image features have been detected they are associated against all active tracks and all tracks in storage. If an image feature is unassociated, then a new track is spawned from it. If a track in storage is associated with an image feature then it is reactivated by removing it from storage and setting its location to the feature and updating its DT description. A flow diagram of the track life cycle is shown in Figure. A greedy nearest-neighbor algorithm is used for association in this implementation. 3. Design Issues When selecting algorithms to use inside the hybrid tracker, there are several issues that need to be considered. Can the DT tracker track features found by the detector? Can the DT tracker detect track faults? Texture is used by DT trackers to track features. Unlike corner features, blob features contain all their texture information along the blob s outside edges, with little information inside. In theory, if a large blob was detected, the DT tracker s template could lack texture and fail, even with a pyramidal approach. In practice, this was found not to be an issue with real-world data due to sufficient texture across scales. In the hybrid algorithm, a feature only switches to using the DDA approach when the DT tracker detects a fault. DT trackers typically employ several methods for detecting faults, but are susceptible to gradual drift. As will be shown below, it is possible for a slowly changing video sequence to produce worse performance than one with abrupt changes. This issue of drift can be mitigated in applications where geometric information is available. For example, when estimating the camera s ego-motion, features which drift are outliers and are dropped. 4 Experimental Setup Performance is evaluated using video sequences for which a homography describes the relationship between each video frame. This approach is similar in spirit to the approach taken by Mikolajczyk et. al [8], where sequences of still images are provided along with corresponding homographies. Videos were collected using a consumer grade hand held point-and-shoot camera. Specifically, a Sony Cyber-Shot DSC-Hx5V camera capturing 64x48 MP4 video. A homography provides a unique mapping between pixels in two image, x H x where H R 3 3 is the homography and x is a homogeneous 2D pixel coordinate. A homographic relationship comes about when the scene is planar, or the camera s motion is purely rotational. Thus, the evaluated video sequences are either of planar objects, panoramic, or taken with a stationary camera. An automated algorithm is used to reconstruct the true homography between the first image and each subsequent image. Procedure: ) Remove lens distortion. 2) Track point features. 3) Estimate homography using points. 4) Non-linear refinement using inlier point set. 5) Non-linear refinement that minimizes average squared difference of pixel intensity.

5 5 Carpet Bricks Lecture Notes in Computer Science Move In Move Out Rotate Skew Fig. 2: Four different 6-DOF camera motions are evaluated for planar scenes. Bricks is more uniform in appearance and level of texture, while has clearly defined objects with less interior texture. Illumination Panoramic Compressed Fig. 3: Changes in ambient light are evaluated in illumination. In panoramic the camera is approximately d about its focal point in an urban environment. For compressed, the skewed sequence is highly compressed using JPEG. Compression artifacts are difficult to see in this figure due the reduced image size. Track stability is measured using F-measure Eq. 3, which is a function of precision and recall. precision recall F =2 (3) precision + recall Precision is defined as the number of true positive tracks divided by the total number of active tracks. Recall is defined as the number of true positives divided by the initial number of tracks. A true positive track is defined as a track that is within tolerance of the true feature location. The choice of tolerance for defining true positive tracks is some what arbitrary, and a value 5 pixels was selected. The same conclusions are reached when other tolerances are used. Four scenes are used to evaluate performance (see Figures 2 and 3). Two scenes of planar objects with four different 6-DOF camera motions, one stationary camera view of a bookshelf with variable illumination, and one panoramic of an urban environment. The two 6-DOF scenes, and, are views of planar objects with different

6 6 Peter Abeles textures intended to stress trackers differently. The effect of frame rate/object speed on tracker performance is examined by reprocessing a sequence with axial camera rotation at different frame rates. Table : Tracker Summary Type Detector Descriptor KLT DT Shi-Tomasi 5x5 Pyramid FAST-BRIEF DDA FAST BRIEF FH-BRIEF DDA FH BRIEF FH-SURF DDA FH SURF-64 FH-BRIEF-KLT Hybrid FH Multiple FH-SURF-KLT Hybrid FH Multiple Evaluated trackers are summarized in Table. Specific implementations of the hybrid tracker are named using a three word pattern, (DETECTOR) - (DESCRIPTOR) - (DT). DDA trackers are named using two words (DETECTOR) - (DESCRIPTOR). Each word signifies an internal algorithm. SURF is a popular state-of-the-art descriptor designed for speed and stability. BRIEF is a more recently proposed descriptor that uses binary encoding. Fast Hessian (FH) is the scale-space blob detector proposed with SURF. Both Shi-Tomasi and FAST are corner detectors, with the former using the image gradient and the latter using pixel intensity values. FAST is popular in applications with constrained computation resources due to its speed, but less stable than other detectors. If provided, all descriptors, detectors, and trackers use recommended parameters from the original authors. Algorithms are tuned for stability across across all scenarios. Runtime performance is evaluated on an Intel Quad Core 2 Q66 at 2.4 GHz running Ubuntu Linux with kernel Algorithms are provided by BoofCV, see project home page ( for a discussion on correctness and performance. All trackers use the first frame as their key-frame and never spawn new tracks. 5 Results Stability as an average across each sequence is shown in Figure 4. The complementary nature of KLT and the DDA trackers is clearly evident. During move out scenarios KLT excels while DDA trackers perform poorly and the reverse is true for rotation scenarios. Ability of feature detectors to estimate scale is limited by the camera s resolution, while KLT continuously updates each feature s description. KLT has more problems tracking sequences with heavy compression than DDA, likely caused by the lack of smoothness between frames due to compression artifacts. The FAST detector is not invariant to illumination or rotation, causing FAST-BRIEF to perform relatively poor for scenarios with changes in illumination and rotation. Other behaviors are more evident when examining performance as a function of frame number, Figures 6 and 7. When the camera is rotating, DDA s stability is cyclical because of its ability to recover tracks, while KLT gradually decays. Hybrid trackers

7 Mean F-Statistic Lecture Notes in Computer Science move in move out Average Track Stability For Each Scenario skew 4 8 move in move out skew 4 8 lighting Compressed Panoramic KLT FAST-BRIEF FH-BRIEF FH-SURF FH-BRIEF-KLT FH-SURF-KLT Fig. 4: Average F-statistic across each scenario, higher is better. Hybrid trackers are top performer in almost every scenario. Other trackers exhibit greater variability, with poor performance in one or more scenarios. switching between the two techniques can also be seen. When tracking first begins it tends to behave like a DT tracker, but if tracks are dropped it recovers them by switching over to DDA. The hybrid tracker s stability has a non-linear relationship with frame rate. This is illustrated by Figure 6 where stability can either improve or degrade as more frames are dropped. The hybrid tracker relies on KLT detecting its own faults, which can be unreliable at certain frame rates. Once a fault is detected, it switches to become more DDA tracker like and its performance recovers. Runtime performance across all scenarios is shown in Figure 5. KLT has the fastest performance followed by the two hybrid trackers. FAST-BRIEF is a close fourth place, but has poor stability. FH-BRIEF and FH-SURF are many times slower than other trackers. 6 Future Work While a popular choice for evaluation in the literature, using homographic scenes to evaluate performance has significant disadvantages. The use of homographies is primary out of necessity, it provides a way to uniquely map pixels between two images. It does not capture the complexities found in arbitrary 3D motion. One such complexity involves objects overlapping each other and moving at different speeds. Evaluating trackers under arbitrary 3D motion requires complete knowledge of the scene s 3D structure and the camera s pose. Recently, data sets have been made available [9]

8 Mean Time (ms) 8 Peter Abeles 3 25 Runtime Performance Fig. 5: Runtime performance box and whisker plot for all scenarios, lower is better. Y-axis is the average time to process a frame in milliseconds. Statistics shown are minimum, 25%, median, 75%, and maximum. Hybrid trackers are about 7.5 times faster than comparable DDA trackers and.5 times slower than KLT on average. that claim high accuracy vehicle pose and capture the scene s structure using LADAR. These could be used to evaluate tracking performance under a less restricted environment. 7 Conclusions For the first time, track stability of DDA, DT, and hybrid trackers have been compared against each other for track stability and runtime performance. This comparison allows practitioners to make more informed decisions about the relative merits of each strategy in different scenarios. Hybrid trackers exhibit many of the advantages of DDA and DT tracking and to a lesser extent their disadvantages. The primary weakness arises from the need to detect faults in the DT tracker, which is unreliable in specific situations. While hybrid trackers have received little attention in the literature, they were among the top performers across all scenarios and never experienced catastrophic failures. In some cases, they out performed both DDA and DT trackers. Performance of DDA and DT trackers fell along predictable lines and with each experiencing catastrophic failures. The results of this study suggest that a larger emphasis should be placed on the development of hybrid trackers. References. Harris, C., Stephens, M.: A combined corner and edge detector. In: Proceedings of the 4th Alvey Vision Conference. (988) 2. Shi, J., Tomasi, C.: Good features to track. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR 94). (994)

9 Lecture Notes in Computer Science 9 3. Rosten, E., Drummond, T.: Machine learning for high-speed corner detection. In: European Conference on Computer Vision. Volume. (26) Matas, J., Chum, O., Urban, M., Pajdla, T.: Robust wide-baseline stereo from maximally stable extremal regions. Image and Vision Computing 22 (24) Lowe, D.: Distinctive image features from scale-invariant keypoints, cascade filtering approach. International Journal of Computer Vision (IJCV) 6 (24) 9 6. Bay, H., Ess, A., Tuytelaars, T., Gool, L.V.: Surf: Speeded up robust features. Computer Vision and Image Understanding (CVIU) (28) Calonder, M., Lepetit, V., Ozuysal, M., Trzcinski, T., Strecha, C., Fua, P.: Brief: Computing a local binary descriptor very fast. IEEE Transactions on Pattern Analysis and Machine Intelligence 34 (22) Lucas, B.D., Kanade, T.: An iterative image registration technique with an application to stereo vision. In: International Joint Conference on Artificial Intelligence. (98) 9. Lindeberg, T.: Feature detection with automatic scale selection. International Journal of Computer Vision 3 (998) Silpa-Anan, C., Hartley, R.: Optimised kd-trees for fast image descriptor matching. In: Conference on Computer Vision and Pattern Recognition. (28). Hager, G.D., Belhumeur, P.N.: Efficient region tracking with parametric models of geometry and illumination. IEEE Trans. Pattern Anal. Mach. Intell. 2 (998) Baker, S., Matthews, I.: Lucas-kanade 2 years on: A unifying framework: Part. Technical report, Robotics Institute (22) 3. Baker, S., Matthews, I.: Equivalence and efficiency of image alignment algorithms. In: Proceedings of the 2 IEEE Conference on Computer Vision and Pattern Recognition. Volume. (2) Uemura, H., Mikolajczyk, S.I.K.: Feature tracking and motion compensation for action recognition. In: British Machine Vision Conference (BMVC). (28) 5. Pilet, J., Saito, H.: Virtual augmenting hundreds of real pictures: An approach based on learning, retrieval, and tracking. In: IEEE Virtual Reality 2. (2) 6. Pradeep, V., Medioni, G., Weiland, J.: Visual loop closing using multi-resolution sift grids in metric-topological slam. In: IEEE Conference on Computer Vision and Pattern Recognition. (29) 7. Ladikos, A., Benhimane, S., Navab, N.: A realtime tracking system combining templatebased and feature-based approaches. In: VISAPP 27. (27) 8. Mikolajczyk, K., Schmid, C.: A performance evaluation of local descriptors. IEEE Trans. Pattern Anal. Mach. Intell. 27 (25) Geiger, A., Lenz, P., Urtasun, R.: Are we ready for autonomous driving? the kitti vision benchmark suite. In: Conference on Computer Vision and Pattern Recognition (CVPR). (22)

10 Peter Abeles Bricks Skip Bricks Skip 4 Bricks Skip KLT FAST-BRIEF FH-BRIEF FH-SURF FH-BRIEF-KLT FH-SURF-KLT Carpet Skip Carpet Skip 4 Carpet Skip Fig. 6: Track stability by video frame for axial camera rotation when N frames are skipped, higher is better. Bricks Move In Bricks Move Out Bricks Skew KLT FAST-BRIEF FH-BRIEF FH-SURF FH-BRIEF-KLT FH-SURF-KLT Carpet Move In Carpet Move Out Carpet Skew Illumination Panoramic Compressed Fig. 7: Track stability by video frame for several scenarios, higher is better.