Endoscopic Motion Compensation of High Speed Videoendoscopy

Transcription

1 Endoscopic Motion Compensation of High Speed Videoendoscopy Bharath avuri Department of Computer Science and Engineering, University of South Caroina, Coumbia, SC ravuri@cse.sc.edu Abstract. High Speed Videoendoscopy offers a promising way to study the voca fod vibration. However, the motion of the endoscope, and noise present in it affects the accuracy of the information extracted from it. The current work focuses on processing the HSV to compensate the endoscope motion. Our experiments empoy methods based on convoution and L -optimization, for detecting such motion. Both methods have been successfuy impemented to isoate the endoscope motion trajectories in the horizonta and vertica directions with precision up to a pixe. 1 Introduction Utra-high-speed photography, first deveoped by Be Teephone Laboratories in the 1930s [1], has been used by many scientists in various studies of voca fod vibration []. The first commercia High-Speed Videoendoscopy (HSV) system was introduced in HSV systems have been highy anticipated by voice researchers and cinicians because they contain a arge amount of physioogica and dynamic information (figure 1). The technoogy for HSV capture is ever improving by aowing increased data rates, higher resoution and better pixe quaity, by introduction of coor imaging. The cinica, and even sometimes the aboratory appication of these systems remain impractica, because there is no estabished methodoogy for extracting the essentia information [3, 4]. Whie, researchers are deveoping methods to extract usefu information from HSV, certain aspects of HSV such as the endoscopic motion compensation, noise remova etc., have to be addressed prior to information extraction. The videoendoscopic images acquired by digita HSV systems are affected by the motion of the endoscope, as is the case with any other type of endoscopic images. The motion of the camera cannot be competey avoided regardess of the experience and the attentiveness of the operator. If eft uncorrected this motion affects the information extracted and thereby reduces the usefuness of the data. The HSV image is essentiay different from any other image used in medicine. Typicay in other areas of imaging the motions of the recording device (camera) are faster compared to the overa change of the image across frames. Aso, there is a

2 noticeabe marker (background, bone, or other object) that is near static, which eases image processing. The ack of a cear marker and the fact that the image coud be changing significanty across frames, make image processing with HSV images chaenging. Fig. 1. Snapshots of the intra-gotta cyce during norma voca fod vibration, captured using a commerciay avaiabe anaog HSV system (Mode 9700, Kay Eemetrics Corp.). However, the endoscope motions and the changes of the gottis have very different dynamics, which can be used as discriminators. It is practicay feasibe to isoate the endoscope motion trajectories, since the frequencies ( Hz) of vibration of the voca fods are much higher than the frequencies of the motion of the endoscope (ess than 0 Hz). The most important components of endoscope motion are: horizonta and vertica shifts, rotation in the vertica pane, and inward/outward motion. Our current work primariy focuses on detecting the horizonta and vertica shifts of the endoscope. This work coud be further extended to detect other components of the motion. The Method A series of experiments were conducted to evauate the feasibiity of HSV motion compensation. The methods empoyed are used in other fieds of computer imaging. This section outines the agorithm that we experimented with good success for soving the motion compensation probem of HSV. The consecutive frames of the videoendoscopic images acquired by the high-speed camera are considered as a sequence of functions in two variabes denoted by f k y), k = 1,... n. Each f k y) is a piecewise constant function on N x M pixes (e.g., N=10, M=56) and it takes integer vaues in a range (typicay 0-55) depending on the sensitivity and dynamic

3 range of the camera. Each frame ( x y) f k, is defined on a rectange, containing a pixes. The proposed method for motion compensation is carried out in three steps: smoothing, preprocessing and motion detection, which are briefed beow..1 Smoothing In this step we aim at eiminating the high-frequency components of the vibrating voca fods resuting in images, which coud be used to approximate the ow frequency endoscope motion. The consecutive frames f k y), k = 1,... p of the video are smoothed pixe-by-pixe with a smooth window function (Hanning) φ (t). Fig.. A typica Hanning window shown in the time and frequency domains. The vaue of p determines the accuracy of information incorporated in to the new frames. If we regard the consecutive frames, as a function of three variabes f y, t) (discrete in time), a simpe convoution of f y, t) with φ ( t) in time produces the desired resut. The resut is denoted by g y), k = 1,... m. k ( f φ ) y, t) = f y, t u) φ( t)du, where ( t) = 1 φ dt. (1). Preprocessing The purpose of this step is to remove the boundary discontinuities between the g k x, y, k = 1,.... To this end, we first subtract from each func- frames, ( ) m tion ( x y) g k,, its average, c k defined as 1 (, ) g, ( ) k x y A dxdy where A() is the area of. ()

4 Secondy, we smooth out each of the consecutive functions near the boundary of. More precisey, we define gˆ y) ( g y) c ) ψ ( x y) = (3) k k k, where ψ y) =ψ ( x) ψ ( y) with ψ ( x) and ψ ( y) form shown in figure 3. smooth Tukey windows of the Fig. 3. A typica Tukey window shown in the time and frequency domains. The equation for computing the coefficients of the Tukey window is [ + 1] w k 1.0, N = k cosπ N ( 1+ r) ( 1 r), 0 k N N ( 1+ r) ( 1+ r) k N (4) The vaue of r determines the shape of the Tukey window produced. For r = 1the window produced is same as the Hanning window, and for r = 0, a rectanguar window is produced..3 Motion detection After the initia enhancement of the HSV, our next goa is to identify the motion. The key idea of our agorithm is that the motion of the frames gˆ k y), k = 1,... m approximates with high accuracy the ow frequency motion of the video frames, f k y), k = 1,... n, due to the motion of the endoscope. To capture the motion of the functions, gˆ k y), k = 1,... m we proceed in consecutive steps in time, determined by a sequence of indices = k < k < k... k. 0 1 < m

5 We recursivey process the frames gˆ k y), as foows. Suppose that we have aready processed the functions gˆ k y), for k k j, we now have to determine a vector ( a i, b i ), which describes the shift of the endoscope after the ˆ ˆ that g k y) g ( x a, y b), j 1 k j distortion between ˆ k y) and ˆ ( x + a, y b) g j +1 F th k j frame. Assuming we find the vector ( a, b) which minimizes the g k j + defined by ( a, b) = gˆ y) gˆ ( x + a, y + b) We now have to minimize ( a b) ( a b) minimum of ( a b) k j+ 1 k j (5) L F,, the east square distortion. The vaues of F, can be quicky cacuated using a standard operation in MATLAB. Then the F, can be found by another such operation. However, an aternate approach coud be empoyed to minimize the distortion be- ˆ k x, y ˆ x + a, y b. We know that the east square distortion tween ( ) and ( ) g j +1 g k j + coud aso be represented in the integra form as F dxdy j 1 k j ( a, b) gˆ y) gˆ ( x + a, y + b) = + It is easy to see that in order to minimize ( a b) G ( a, b) defined by k (6) k + j+ k j, (, b) = gˆ y) gˆ ( x + a y b) G a 1 F, it suffices to maximize the function dxdy (7) 1 Indeed, this foows by the foowing identity (with genera functions f and g, assuming that f dxdy = g dxdy = C ). f g L = = f f = C y) g( x + y, y + b) y) dxdy + g y) dxdy f y) g( x + a, y + b) f dxdy y) g( x + a, y + b) dxdy dxdy G, can be cacuated at a points of interest using a stan- It is readiy seen that G ( a, b) is the convoution of gˆ k y) and gˆ k j ( x, y) + 1 ( a, b). The vaues of ( a b) dard operation in MATLAB. Then the maximum of ( a b) such operation. We use the above procedures to compare ĝk j with (8) at G, can be found by another g ˆ, k j+ 1 g ˆ k j + As soon as a or b becomes bigger than the side of a pixe, we stop the current iteration, and this wi

6 give the next index k j+ 1 and the new base frame g ˆ k j+ 1. We denote the corresponding vector by ( a i, b i ), and proceed further ti we get to the ast frame avaiabe. 3 Experiments x A set of three video footages (in.avi format) was used for testing the empoyed methods. This set incuded test sampes varying in size (n = 549, 401 and 4368 frames) and in the motion of the endoscope. The resoutions of these videoendoscopic images were N x M = 160 x 140, 160 x 140 and 10 x 56 pixes respectivey. In the smoothing phase, we try to incorporate the information present in p consecutive frames of the video sequence, into a singe frame, such that there is a smooth transition between frames. This process is recursivey done, after every s frames, starting with the first frame. These m new frames are now processed with the Hanning window to remove the high frequency components. Hamming window was aso considered for this purpose. However, the Hanning window provides more gradua transitions between the high and ow frequencies, making it a better choice. In the preprocessing stage, Tukey window with r = 0. 3 was used in the experiments. Two 1D-Tukey windows, T and T are convoved to form a D-Tukey window. This window is empoyed for the remova of boundary discontinuities. After the preprocessing phase, we have a smooth video, which ony has the ow frequency components that approximate the motion of the endoscope. The initia smoothing and pre-processing phases are simiar for both methods. The next step is to detect this motion. y Fig. 4. Left: esut of convoving the zero-padded first base frame with the next frame, for vaid points. ight: esut of L -optimization of the first base frame and the next frame. (n=401, m=113, N=160, M=140, z=60, d=10). Our first approach uses convoution to detect the motion. The first frame of the video sequence is zero-padded with z zeros to give the initia base frame for the convoution. The next frame is fipped by 180 o, and convoved with the base frame for vaid points. The vaue of z determines the size of the convoution resut. Typicay, if a frame of size N M, was convoved with the base frame of size,

7 N + z M + z, the convoution resuts woud be of size z z. In our experiments we used z = 60, 50, 40. The centre of the convoution resut (figure 4) is used as a measure to approximate the motion of the endoscope ( a, b) in the horizonta and vertica directions. For e.g., suppose that the centre of the convoution resut (for the first base frame and the second frame and z = 60) is at (31, 31). When, the next frame is convoved with base frame if the centre is at (31, 31), we move to the third frame and do convoution with the same base frame. If the centre moves to (30, 31) now, we have a motion of the endoscope in the horizonta direction. We record this motion and the third frame becomes our new base frame after zero padding. We recursivey process the remaining frames. For, the east square-optimization method, as with convoution, we start with the first frame in the sequence. However, now the size of the base frame is smaer than f x, y were N M pixes, the actua frame size. In other words, if the frame size of ( ) our base frame woud be of size, N d M d pixes, where d is number of pixes discarded in each dimension, for computing the L -distance. In our experiments we used a vaue of 6, 8, 10, 1 for d. The minimum vaue from the L -optimization resut (figure 4) is used for approximating the motion of the endoscope (figure 7). A the frames are processed as described earier, ti we get to the ast frame in the sequence. Fig. 5. The horizonta component of the endoscope motion as detected by the convoution method. Left: Actua Motion; ight: Cumuative motion. (n=549, m=0, N=160, M=140, z=60). The motion of the endoscope was estimated in two vectors. One vector contains the actua motion of the endoscope, i.e., the number of pixe movements that occur from one frame to the next. Another vector was used to record the absoute or cumuative motion. Figure 5 shows these two measurements of motion, in the horizonta direction, computed using convoution. Figure 6 shows the motion recorded in the vertica direction. Note that for this particuar fie, there is no motion in this direction.

8 Fig. 6. The vertica component of the endoscope motion as detected by the convoution method. Left: Actua motion; ight: Cumuative motion. (n=549, m=0, N=160, M=140, z=60). Figure 7 shows the resuts for another fie, which had motion in both the directions. The input fie had 4368 frames, each of size 10 x 56 pixes. These frames were smoothed out into 11 frames. Convoution was performed with z = 60 and L - optimization was done with d = 10. Fig. 7. The different components of the motion of the endoscope as detected by L -distance method. Left: Actua motion; ight: Cumuative motion. Top: Horizonta component; Bottom: Vertica component. (n=4368, m=11, N=10, M=56, d=8).

9 4 Discussions Both methods have been successfuy impemented to isoate the endoscopic motion trajectories in the horizonta and vertica directions with precision up to a pixe. There are a number of parameters that affect the computationa speed of the proposed methods. A such parameters have to be optimized to enhance the performance of the agorithm. The parameters that have a major infuence on the computation speed are the samping parameters, s and p, used during the smoothing phase. Even though oversamping ensures information accuracy and smooth transitions between frames, the computation becomes sow. The number of zero-padding pixes, z, in the convoution part and the number of discarded pixes, d in the L -optimization. requires further investigation. If z is sufficienty arge, the computation of the convoution becomes sower. Whie on the other hand, if it were sma, may resut in unacceptabe resuts. Further experimentation and investigation is necessary to arrive at optima vaues for these parameters. A good trade-off between computation speed and accuracy is to be determined. From our experimenta resuts, convoution is beieved to produce a more reiabe resut, whereas L -optimization appears as a faster approach. Comparativey the operations invoved in L -optimization require esser computationa time than those invoved in convoution. However, the resuts obtained from these two methods are comparabe as is evident from figure 8. The effectiveness of these methods is to be determined, as our present work does not approximate the a the components of the motion. In addition, no prevaent method is known that approximates the motion of the endoscope in a HSV sequence, which coud be used as a benchmark to compare the resuts of our experiments. Fig. 8. The comparative cumuative motion of the endoscope detected using convoution and L -optimization; eft: horizonta component; right: vertica component. (n=401, m=113, N=160, M=140, z=50, d=1). The proposed methods coud be extended further to detect the other components of motion. Once a the components of motion are determined, the endoscopic motion

10 can be compensated competey. Suppose that, the motion of the endoscope has been estimated in the vectors, A and B, which can be defined as A = a j j= 1 and B = b j j= 1 = 1,,...m (9) These vectors ( A, B ), 1,... m, f k y), = 1,... n. Namey, we repace each ( x y) f ( x + A, y + B ). k = woud be used to offset the shifts of the video f k, by its shift 5 Concusions The proposed method aims at detecting the motion of the endoscope in horizonta and vertica directions. The methods devised in the agorithm work we and the resuts are promising. The effectiveness of these methods coud ony be determined once motion compensation is performed. Motion detection up to a pixe has been achieved. More experiments are necessary to determine ways to detect motion, which is ess than a pixe. Our future work woud incude detection of other components of the motion and de-noising of the images. eferences 1. Farnsworth, D.W., High-Speed Motion Pictures of the Human Voca Cords. Be Lab ecord 18, Hirano, M., Cinica Examination of Voice. Wien: Springer-Verag. 3. Kent,.D., Ba, M.J Voice Quaity Measurement. Singuar Pubishing Group, pp MacAusan, J Geometric Signa Processing: The arynx as a noninear dynamica System, esearch Lab for eectronics, MIT. 5. Herze, H., Berry, D., Titze, I.., & Saeh, M Anaysis of voca disorders with methods from noninear dynamics. Journa of Speech and Hearing esearch, 37(5),