Fast and Robust Endoscopic Motion Estimation in High-Speed Laryngoscopy

Size: px

Start display at page:

Download "Fast and Robust Endoscopic Motion Estimation in High-Speed Laryngoscopy"

James Green
6 years ago
Views:

1 Fast and Robust Endoscopic Motion Estimation in High-Speed Laryngoscopy Dimitar Deliyski, Szymon Cieciwa *, Tomasz Zielinski * Communication Sciences and Disorders University of South Carolina, Columbia, SC, USA * Instrumentation and Measurement Department AGH University of Science and Technology, Krakow, Poland ddeliyski@sc.edu Seven methods for endoscopic motion compensation for laryngeal high-speed videoendoscopy (HSV) are compared. Two of them are based on tracking the maximum of the crosscorrelation function of two images; two are based, respectively, on the minimization of the L norm and the magnitude difference distance between two images; and the other three utilize properties of the FFT-based cross-power spectrum of two images. All seven methods were applied to compensate the motion, at the sub-pixel level, of the endoscopic lens relative to the vocal folds in HSV recordings. The motion compensation methods based on FFT crosspower spectrum demonstrated remarkable computational speed and satisfactory accuracy, while also offered wider motion-tracking range. The proposed two-step least square fitting of the FFT cross-spectrum phase plane was found to be the fastest among all seven approaches. Keywords: high-speed videoendoscopy, motion compensation, voice evaluation, kymography 1. Introduction The endoscopic (camera lens) motion in high-speed videoendoscopy (HSV) affects the time alignment of the laryngeal anatomic structures in the image. Sub-pixel endoscopic motion compensation (MC) is an important preprocessing operation in certain visual-perceptual and automated techniques for evaluation of vocal fold movement [1]. The problem of endoscopic MC for HSV is complex due to the dynamics of the vocal folds during phonation (Fig. 1a). Laryngeal HSV is essentially different from any other medical image because it registers the motion of an organ that moves very fast (7-1 Hz), affecting practically all connected tissues and creating motion across the whole image. The motion of the connected tissues contains a fast component, comparable in speed with the vocal folds, but also slower components, some of which are comparable with the speed of the endoscopic motion (less than 15 Hz). No clear spatial outlier can show the motion relative to the camera lens located on the tip of the endoscope. Fortunately, the endoscopic motion and the changes

in the glottis during phonation have different dynamics. This dynamic difference has been successfully used [1] for building the missing outlier.

1b) has been found to be very effective when building the missing spatial outlier for endoscopic motion tracking [1]. Fig. 1.

The global displacement between two consecutive frames in video sequences have been typically estimated using direct image correlation or image difference minimization (L -norm or magnitude

However, the FFT method in its original version is limited to integer shifts only and requires enhancement for sub-pixel image registration [3-7].

2 in the glottis during phonation have different dynamics. This dynamic difference has been successfully used [1] for building the missing outlier. In order to dynamically separate the fast vocal fold movements from the slow camera lens motion it is necessary to smooth the HSV. Smoothing of the time differential of the HSV image (Fig. 1b) has been found to be very effective when building the missing spatial outlier for endoscopic motion tracking [1]. Fig. 1. a) Open and closed phase of the vocal folds in two different x-y positions. Fig. 1. b) Smoothed time-differential images of vocal folds in two different x-y positions. The global displacement between two consecutive frames in video sequences have been typically estimated using direct image correlation or image difference minimization (L -norm or magnitude difference) techniques. Fast motion estimation has been realized using crosspower spectrum (CPS) algorithms based on the D fast Fourier transform (FFT) []. However, the FFT method in its original version is limited to integer shifts only and requires enhancement for sub-pixel image registration [3-7]. The accuracy of different well-known sub-pixel motion estimation methods is compared in [8], including for the polyphase decomposition approach of Foroosh [3, 4], the frequency domain masking technique of Stone [5] and the sub-space identification extension algorithm of Hoge [6, 7]. The latter singular value decomposition (SVD)-based method was shown to be the most robust in almost-noisefree environment. A novel fast sub-pixel algorithm using a two-step least-squares (LS) approximation of the CPS phase plane was recently developed [9]. In the present work of the same team, that method is described in more detail, additionally optimized in respect to parameters values and compared not only to the classical image alignment techniques such as the image correlation and difference minimization but also to the SVD-based method [6], which was shown to be the best in [8]. All techniques were implemented in Matlab environment and their efficiency was tested on laryngeal HSV images with natural and artificially added displacement. This study aimed the estimation of translations. The methods estimating image rotation and scaling [1, 11], are outside the scope of this work.

3 . Methodology.1. Known Motion Compensation Methods Given that f 1 (x, y) and f (x, y) are two continuous functions, in this case two images, where the second function is a shifted in space version of the first one: f (x, y) = f 1 (x x, y-y ), we can find the displacement {x, y } making use of one of the following methods. a) b) c) x 1 7 x 1 5 z z z.4. 1 dy dx dy -1-1 dx 1 1 y x 5 1 x 1 7 x 1 5 z z z dy - - dx dy - - dx dy - - dx Fig.. Detection matrices (similarity measures) for different methods used for HSV motion detection: up higher searching range; down lower searching range after interpolation (dx = a, dy = b); a) maximum of correlation similarity; b) minimum of L -like similarity; c) peak for cross-power spectrum similarity. Correlation function method. The classic method for {x, y } detection relies on the properties of the cross-correlation function of f 1 (x, y) and f (x, y), which is defined as follows: corr (, ) = 1 +, + ) - - D a b f x y f x a y b dxdy. (1) The function D corr (a, b) reaches its maximum for a = x and b = y. However, the maximum is flat (as shown in Fig. a) and computation is time consuming. We can observe that f (x, y) is shifted back in x and y dimensions by a and b, to fit the original image f 1 (x, y). Speed optimization can be achieved by defining equation (1) for a limited range of a and b. Such approach is appropriate for HSV recordings when estimating small shifts within ± pixels [9]. The functional (1) denotes also the D convolution D conv (a, b) = D corr ( a, b) of the images f 1 (x, y) and f (x, y). The method described in [1] utilized the FFT-based implementation of D

4 convolution function, namely conv in Matlab environment. That computational approach is further termed the Convolution method and will serve as a baseline for evaluating the other methods. The Convolution approach is algorithmically fast, but not optimal for HSV images because the function D conv (a, b) is calculated in this case superfluously for all possible integer shifts a and b (depending on images size), not only for the their small values. This makes the method unnecessarily computationally intensive when shifts are small, but also very robust in detecting large shifts. L norm and magnitude difference minimization methods. The spatial shifts can be determined simply by minimizing the difference between two images while artificially shifting one of them and computing a similarity measure of their difference, such as: D ( a, b) = f ( x, y) f ( x+ a, y+ b) dxdy Δ D ( a, b) = f ( x, y) f ( x+ a, y+ b) dxdy Δ () (3) where () represents the L -norm measure of the image difference, and (3) is the magnitude difference (MD) measure. The minima of such functions are flat in the range of ± pixels as shown in Fig. b. FFT-based cross-power spectrum method. It is known that the Fourier spectra of two images f 1 (x, y) and f (x, y) = f 1 (x x, y y ) are related as: j(. x. y ) x y F (, ) F (, ) e ω +ω ω ω = ω ω (4) x y 1 x y where F 1 (ω x, ω y ) and F (ω x, ω y ) denote their Fourier transforms. The normalized cross-power spectrum of these images is equal to: * F ωx ωy F1 ωx ωy 1 ( x, y) * F ωx ωy F1 ωx ωy (, ) (, ) G ω ω = = e = e (, ) (, ) jφ ( ω, ω ) j( ω. x +ω. y ) 1 x y x y, (5) where the operation * denotes complex conjugation. Therefore, the inverse Fourier transform of G 1 (ω x, ω y ) results in: 1 1 j ( ) ( ) ( ω x,,. x +ωy. y = ω ω = ) =δ(, ) g x y Fourier G Fourier e x x y y 1 1 x y, (6) and it is characterized by a sharp Dirac delta function centered at (x, y ). This property is very useful for motion detection. In the discrete case, f 1 (m, n) and f (m, n) = f 1 (m m, n n ), the above property still holds when fast FFT algorithms of direct and inverse discrete Fourier transform (DFT) are applied. Now (4) and (5) can be rewritten in a discrete form as follows:

5 F ( k, l) F ( k, l) e 1 j π( k m / M+ l n / N) =, (7) * 1 1 * 1 F ( k, l) F ( k, l) jφ ( k, l) j π( k m / M+ l n / N) j( α k+βl) G ( k, l) = = e 1 = e = e F ( k, l) F ( k, l), (8) where: M 1N 1 j π ( mk/ M+ nl/ N) = ( ) =, (9) F( k, l) DFT f ( m, n) f ( m, n) e m= n= M, N denote number of pixels in rows and columns, m, n designate real-value row and column shifts, and km, =,1,,..., M 1, ln, =,1,,..., N 1. Now, the Dirac impulse takes a form of a D sinc function, centered at (m, n ): ( m m ) ( n n ) 1 sin π ( + ) sin π ( + ) g1 ( m, n) = DFT [ G1( k, l) ] = π ( m+ m ) π ( n+ n ) shown in Fig. c-up, the spline interpolation of which is presented in Fig. c-down. Sub-pixel extensions. Motion detection matrices D corr (a, b) (1), D Δ (a, b) () and D Δ (a, b) (3) can be easily interpolated as it is shown in Fig. ab-down for sub-ranges ( a max, a max ) and ( b max, b max ) around its maximum or minimum. Adaptive strategies for changing the values of a and b can be applied. The same is true for g 1 (m, n) (1) that can be interpolated near its maximum (m, n ), as shown in Fig. c-down. LS fitting of cross-power spectrum phase plane. Further, only the discrete case will be discussed. Instead of calculation of g 1 (m, n) from (1) and its interpolation near maximum, it is also possible to estimate directly the shifts m and n between the images f 1 (m, n) and f (m, n) = f 1 (m m, n n ) using two points (samples) Φ 1 (k 1, l 1 ) and Φ 1 (k, l ) of the phase plane Φ 1 (k, l) (8) and solving the following set of two equations with two unknowns m and n : where Φ 1( k1, l1 ) = π( k1 m / M + l1 n / N) =α k1 +βl1 Φ ( k, l ) = π ( k m / M + l n / N) =α k +β l 1 ( G1 k l ) ( G k l ) 1 Im (, ) Re 1(, ) ( ( ) ( )) Φ ( kl, ) = tan = tan Im G ( kl, ),Re G ( kl, ) α= π m / M, n (1) (11) (1) β= π / N (13)

6 Examples of a phase function Φ 1 (k, l), representing a plane in 3D space with characteristic angles α and β, is shown in Fig. 3. Due to effect of phase wrapping, as illustrated in Fig. 4, it is necessary to solve Φ 1 (k, l) for small values of k and l. α β k l Fig. 3. Example of a phase function Φ 1 (k, l) (8). a) b) fi(k,l) fi(k,l) k l k l Fig. 4. Phase plane Φ 1 (k, l) (8) for preprocessed HSV images of vocal folds for: a) small, and b) large values of their relative displacement (only part of the matrix Φ 1 (k, l) is shown). Since the operations involved in (1) are very noise sensitive, using more samples of the phase plane and least squares estimation is recommended: k1 l1 Φ1( k1, l1) k l α Φ1( k, l) = M M β M Ax = b (14) kk lk Φ1( kk, lk)

7 T 1 T x= ( A A) A b= pinv( A) b (15) In Matlab language, the operation (15) corresponds to x = A\b. The shifts m and n are computed from (13) by finding the optimal (in LS sense) values of α and β. Subspace estimation of cross-power spectrum phase plane [3]. Another interesting approach of exploiting of cross-power spectrum equation spectrum for calculation of displacement of two images f 1 (m, n) and f (m, n) is based on the singular value decomposition (SVD) of the matrix G 1 (k, l) (8) that is of rank one: ( ) ( ) 1 (, ) j α k j β l H G k l = e e = uv (16) where H denotes complex-conjugate transpose. For the given matrix G 1 (k, l) one should find the dominant complex vectors u and v associated with the maximal singular value, calculate and unwrap their phases, and perform 1D least squares fitting of these phases to strait lines, in a similar manner as shown above, in order to calculate the values of α and β (see Fig. 5). Only part of the G 1 (k, l) matrix can be used for calculation to speed up the computations (e.g pixels were used in the example presented in Fig. 5). For laryngeal HSV recordings the best results have been obtained for submatrices with dimensions M /1 N /1 where means rounding down to the nearest integer value. unwrap(angle(u)) [rd].4 unwrap(angle(v)) [rd] α -. β index index Fig. 5. Calculation of motion parameters α and β from dominant complex vectors u and v of the matrix G 1 (k, l) (16)... Proposed Motion Compensation Method The block diagram of the proposed two-step hierarchical LS motion estimation algorithm is presented in Fig. 6. Modules marked with dashed lines are only used when relative displacements greater then 1 pixel occur, which is often the cases with laryngeal HSV recordings. Thus, in the proposed method, the integer shift is first computed and subtracted

8 and then the phase of the cross-power spectrum is calculated. Next, the first LS iteration (LS #1) is performed making use of (13)-(15), when the Φ 1 (k, l) samples fulfill the condition:.5 rd Φ1( kl, ).5 rd, k<.5 M, l<.5n. (17) The computed shifts m and n are used for calculating the reference phase plane (denoted as Ref) and the standard deviation of the Φ 1 (k, l) around it (denoted as Std). Finally, the second LS iteration (LS #) is executed, this time using Φ 1 (k, l) samples fulfilling the condition: Ref 3 Std Φ1( kl, ) Ref + 3 Std, k<.1 M, l<.1n, (18) allowing higher precision in finding the image shifts m and n. The optimal sizes of the Φ 1 (k, l) submatrices, used in (17) and (18), assuring both minimum mean square error and fast computation were found experimentally. The reason for choosing scaling a factor of.1 in (18) is explained by the data shown in Fig. 7, which is based on results from conducted tests on artificially shifted smoothed differential HSV images, as in [1]. Even though the minimum MSE motion estimation error is obtained for scaling factor equal to., a more conservative, safe value of.1 is chosen in the LS # due to observations made when applying the proposed method to noisy laryngeal HSV images. Image 1 FFT D Cross-power spectrum Angle(.) LS Image FFT D Subtract integer shift FFT D (.) * LS #1 Threshold LS # LS #1 Find Ref phase plane Dimensions:.5M.5N Choosing samples of Angle:.5 rd < Angle <.5 rd Threshold Compute standard deviation Std of (Angle Ref) Dimensions:.1M.1N LS # Find final phase plane Dimensions:.1M.1N Choosing samples of Angle: Ref 3*Std < Angle < Ref + 3*Std Fig. 6. Block diagram of the proposed two-step LS motion estimation & compensation method applied to laryngeal HSV recordings.

9 a) b). MSE error [pixels] y axis.4 mean computation time [s].1.5 x axis part of Φ 1 (k,l) axes [%] part of Φ 1 (k,l) axes [%] Fig. 7. a) Mean square error and b) mean computation time of image shift estimation as a function of dimensions of Φ 1 (k, l) submatrix used in the second LS iteration (18)..3. Experimental Design Initial Testing of Basic Methods: The first step toward developing a robust MC algorithm was to test the accuracy and speed performance of the basic functions for detection of spatial shifts described in the previous section. The performance of the following basic computational methods was compared on artificially shifted by x and y images at different shift directions and magnitudes: Convolution [1], Correlation (1), L-norm (), MD (3), FFT-SplineSinc - spline interpolation of D sinc function (1), FFT-SVD SVD-based decomposition of the cross-power spectrum (14) and, finally, FFT-TwoStepLS the proposed two-step LS hierarchical approach, described above. Three parameters were reported: average execution time (in ms); mean absolute error ε M (in pixels by x and y); and absolute range of error ε R (in pixels by x and y). These measurements do not warrant accuracy of the whole MC implementation. MC Algorithm: The MC algorithm implemented in these experiments was the same algorithm used in [1] with the basic functions replacing the original Convolution approach. It consists of the following main steps: (i) Establishing dynamic vocal fold outliers for MC by computing pixel by pixel the time differentials of the HSV image sequence; (ii) Eliminating the highfrequency components of the vibrating vocal folds via smoothing; (iii) Suppressing the effect of the boundary discontinuities of the image frames (only necessary for Convolution); (iv) Detecting the displacement between adjacent frames by using one of the seven methods; (v) Computing the displacement vectors (motion trajectories); (vi) Subtracting the motion trajectories from the spatial coordinates of the original HSV image using two-dimensional spline interpolation.

10 Performance on Real HSV Data with Simulated Motion: To assess the accuracy in extreme conditions, the MC method was tested on simulated data with known motion trajectories. These data are identical to and described in more detail in [1]. Although the proposed methods are expected to work better for lower motion frequencies, these data allow for a thorough testing in extreme conditions, in which all frequencies of the covered range.1 to 15 Hz are equally represented. The data consisted of -second long (4 frames) HSV movies with exactly known motion trajectories. Motion varied from to 8 pixels by x and y in nine different magnitudes. Two types of motion curves, a random and a cyclic, were added to two real HSV recordings, one of a male and one of a female speaker, totaling 34 HSV recordings with simulated motion. The data were analyzed by each modified technique. The following parameters were reported: average speed of computation S C (in seconds per one second of HSV data); mean absolute error ε M and absolute range of error ε R (in pixels) as defined in [1]. Assessing the MC method on data with simulated motion was important to show whether the detected trajectories really correspond to motion and to assess the accuracy of the method in extreme motion conditions with known characteristics. 3. Results and discussion Initial Testing of Basic Methods: The results obtained from testing the basic MC method are presented in Table 1. The FFT-SVD and FFT-TwoStepLS basic methods were found to be several times faster relative to the Correlation, L -based and MD methods. L -norm, FFT- SVD and FFT-TwoStepLS were found to be the most accurate. Additional observations include: noise was not found to be destructive for tracking shifts, and all methods except for FFT-SplineSinc were more sensitive to horizontal movement, since vocal folds are vertical. The Convolution method [1], i.e. initial implementation of (1) using conv function in Matlab, was found to be significantly slower relative to the other six methods investigated. Table 1: Comparison of accuracy and speed of the newly implemented basic methods for motion compensation. MC method Time ε M ε R [ms] [pixels] [pixels] by x by y by x by y Convolution [1] Correlation L -norm MD FFT-SplineSinc FFT-SVD FFT-TwoStepLS

11 Performance on Real HSV Data with Simulated Motion: The results from testing the MC algorithms on real data with simulated motion are shown in Table. They generally agree with the results from testing the basic methods. All MC methods demonstrated satisfactory sub-pixel accuracy and all new techniques were significantly faster relative to the original Convolution approach. Table : Average computation speed S C, mean absolute error ε M, and absolute range of error ε R for seven MC algorithms as tested on real HSV data with simulated motion. The error ranges are shown in parentheses. MC method S C [s/s] Convolution [1] Correlation L -norm MD FFT-SplineSinc FFT-SVD FFT-TwoStepLS 4.78 ε M [pixels].168 (.-.331).64 (.-.181).64 (.-.181).69 (.-.188).91 (.-.351).18 (.-.61).147 (.-.3) ε R [pixels].38 (.-.931).1 (.-.664).1 (.-.664).9 (.-.677).538 (.-.94).47 ( ).46 (.-1.8) Correlation, L -norm and MD had almost identical performance and best accuracy of all methods. Their mean absolute error was.65 pixels and their speed of computation was 15 times higher relative to Convolution. The serious disadvantage of these three methods is their limited range of shift tracking, which limits their implementation for certain types of HSV material. They would have difficulties with transitional events such as phonatory breaks, vocal offsets and onsets, or intermittent obstructions in the view of the vibrating vocal folds, making it difficult to recover when visible vibration resumes. Convolution, FFT-SplineSinc, FFT-SVD and FFT-TwoStepLS do not have this limitation. The fastest methods were FFT-TwoStepLS, FFT-SVD and FFT-SplineSinc, outperforming Convolution 1, 63 and 37 times, respectively. The accuracy of these three methods was lower but still acceptable at the sub-pixel level. The increased error was mainly due to the extreme frequency testing conditions to which the methods were subjected. Considering the exceptional robustness of the FFT-TwoStepLS method, further investigation is necessary to understand and eliminate the sources of errors in order to build a practical tool for motion compensation, which is highly necessary.

12 4. Conclusion The fast FFT-based approach has been applied successfully for the endoscopic motion compensation in HSV recordings of vocal folds. The experimental results demonstrated that the application of the FFT-based cross-power spectrum approach with two-step least square fitting is highly effective. The new method is 1 times faster than the convolution-based approach, almost twice faster than its previous version introduced in [9], and offers acceptable sub-pixel accuracy and wide range of shift tracking. 5. References [1] Deliyski D: Endoscope Motion Compensation for Laryngeal High-Speed Videoendoscopy. Journal of Voice, 19(3): , 5. [] Kuglinand CD, Hines DC: The Phase Correlation Image Alignment Method. Proc IEEE Conference on Cybernetics and Society, , [3] Shekarforoush H, Berthod M, Zerubia J: Subpixel Image Registration by Estimating the Polyphase Decomposition of the Cross Power Spectrum. Proc Int Conf on Computer Vision and Pattern Recognition, [4] Foroosh H, Zerubia J, Berthod M: Extension of Phase Correlation to Sub-Pixel Registration. IEEE Trans on Image Processing, 11(3):188-,. [5] Stone H, Orchard M, Chang E-C, Martucci S: A fast direct Fourier-based algorithm for subpixel registration of images. IEEE Trans on Geoscience and Remote Sensing, 39(1):35-43, 1. [6] Hoge W: A Subspace Identification Extension to the Phase Correlation Method, IEEE Trans Medical Imaging, ():77-8, 3. [7] Foroosh H, Balci M: Sub-Pixel Registration and Estimation of Local Shifts Directly in the Fourier Domain. Proc IEEE Conf Image Processing ICIP-4, , 4. [8] Argyriou V, Vlachos T: Using Gradient Correlation for Sub-Pixel Motion Estimation of Video Sequences. Proc IEEE Int Conf Acoustics, Speech and Signal Processing ICASSP-4, Montreal, III-39-33, 4. [9] Cieciwa S, Deliyski D, Zielinski T: Fast FFT-Based Motion Compensation for Laryngeal High-Speed Videoendoscopy. Proc MAVEBA-5, Firenze, 4:19-13, 5. [1] Reddy BS, Chatterji BN: An FFT-Based Technique for Translation, Rotation, and Scale-Invariant Image Registration. IEEE Trans Image Processing, 5(8): , [11] Vandewalle P, Susstrunk S, Vetterli M: A Frequency Domain Approach to Registration of Aliased Images with Application to Super-resolution. EURASIP Journal on Applied Signal Processing, vol. 6, Article ID 71459, 14 pages, 6.

Robust Phase Correlation based Motion Estimation and Its Applications

Robust Phase Correlation based Motion Estimation and Its Applications Hongshi Yan and Jian Guo Liu Department of Earth Science and Engineering Imperial College London Prince Consort Road, London SW7 2AZ