UUV DEPTH MEASUREMENT USING CAMERA IMAGES

Transcription

1 ABCM Symposium Series in Mecatronics - Vol. 3 - pp Copyrigt c 2008 by ABCM UUV DEPTH MEASUREMENT USING CAMERA IMAGES Rogerio Yugo Takimoto Graduate Scool of Engineering Yokoama National University 79-1 Tokiwadai, Hodogaya-ku, Yokoama, , JAPAN takimotoyugo@vento.sp.ac.jp Tsugukiyo Hirayama Graduate Scool of Engineering Yokoama National University 79-1 Tokiwadai, Hodogaya-ku, Yokoama, , JAPAN irayama@ynu.ac.jp Fabio Kawaoka Takase Departamento de Engenaria Mecatrônica e de Sistemas Mecânicos Escola Politécnica da Universidade de São Paulo Av. Prof. Mello Moraes, 2231, , São Paulo, Brasil fktakase@usp.br Abstract. In some cases, te control of an unmanned underwater veicle (UUV) running near te free surface is required. For suc purpose, accurate measurement of te dept of te UUV is needed. Tis can be performed troug different tecniques. Among tem, it is possible to mention te pressure sensor and te sonar. Eac tecnique as its own set of unique features, advantages and disadvantages. Te sonar, for example, delivers accurate range but can be disturbed by reflections. Tis work proposes an alternative solution for te accurate dept measurement problem at sort range for control purposes. Troug te use of computer vision tecniques, te dept of an UUV near te free surface is evaluated. First te CAMSHIFT tracking algoritm, a color based tracking algoritm, is applied to determine te displacement in te image plane and ten a projective transformation is applied in order to evaluate te dept of te UUV in te world plane. Te projective transformation is a plane to plane omograpy, implemented using te DLT (Direct Linear Transformation) algoritm equations. Te system was implemented in te C programming language and extensive experimental tests were performed to evaluate te quality of measurements. Keywords: unmanned underwater veicle, dept measurement, computer vision 1. Introduction Nowadays, it is possible to find computer vision applications in many areas because tere is an increased need for te automatic processing of te uge amount of digital images generated by cameras. Te availability of faster computers as opened te way for new applications in te computer vision area. Applications tat provide industrial inspection, traffic monitoring measurement become available and tere is also some application in te naval arcitecture area. Projects using computer vision tecniques for te naval arcitecture area are related to 3D reconstruction of underwater scenes from 2D images or to te underwater veicle position control from real time processing of video data. Tese projects are useful for many undersea applications including scientific, commercial and military applications, suc as te ocean floor exploration, te marine life study and underwater structures inspection and repair. Computer Vision tecniques can also be used to provide accurate dept measurement at sort ranges and control an UUV. Suc a feature is an important resource for underwater navigation. For suc purpose, tis work sows te applicability of computer vision tecniques to recover te dept of an UUV. Te implementation of suc functionalities represents te first step in te development of a perceptual interface system, one of te most promising applications in te computer vision area. Perceptual interfaces are ones in wic te computer is given te ability to sense and produce analogs of te uman senses, suc as allowing computers to perceive and produce localized sound and speec, giving computers a sense of touc and force feedback, and in our case, giving an ability to see and extract useful information from te scene. In order to provide tis feature in advance, tis researc proposes an object tracking and distance measurement application. Te proposed system represents te first step in te development of a complete navigation system for an UUV since it is possible to track and determine te size of a known object in te scene. Tis system makes use of commercial products suc as personal computers and video cameras. In order to recover te UUV dept, tis work uses tracking tecniques to identify te image plane displacement and projective transformation tecniques to estimate te distance in te world plane. In addition, te measurement in te world plane is compared to te real dept to quantify te result of estimation algoritm.

2 Te objective of tis paper is to describe te algoritm used in te software development and te results obtained in te experiments to measure te dept of an AUV near te free surface. 2. Homograpy According to Hartley and Zisserman (2000), a camera is a mapping between te 3D world (object space) and a 2D image and it can be modeled using a pinole camera. Tis model is te simplest, te most general linear camera model and is modeled from a 3D point in space projected onto te image plane. Te camera image formation process can be represented by te intersection of te visual ray connected to camera center (C) and te 3D point (X) on te image plane corresponds to te image point (x) (Fig. 1). Figure 1. Te pinole camera model Tis model can also be used to map points from two 2D planes. Te mapping between two 2D planes (plane to plane) or te 2D projective mapping is also known as projectivity or projective transformation or omograpy. From te camera model for perspective images of planes it is possible to map points on a world plane to points on anoter plane (Semple and Kneebone, 1979). It is possible to notice tat te sape is distorted under perspective imaging because parallel lines in te image tend to converge to a finite point. Te distance in te real world (world plane) can be evaluated after removing te projective distortion. Tis can be done troug a projective transformation or omograpy to cange te visualization plane, to correct te geometric sapes from te objects in te image and to eliminate te projective distortion on te determined points. After tat, te mapping of an image point to a world point is straigtforward troug te use of an Euclidean relationsip between te desired points. Figure 2 sows a workflow illustrating te projective transformation and te process to evaluate te real distance between two points selected on te image plane. Figure 2. Homograpy Transformation Workflow If a coordinate system is defined in eac plane and points are represented in omogeneous coordinates (Fig. 3), ten te omograpy can be expressed by (1). X = Hx (1)

3 or by (2) X 1 11 X = 2 21 X x 13 1 x x 3 (2) Were X and x represent te points coordinates in eac plane and H represents te omograpy matrix and is described by a 3x3 non-singular matrix. Te omograpy matrix (H) as 9 elements and only 8 degrees of freedom because te scale of te matrix does not affect te equation. Figure 3. Homograpy Transformation Tere are two metods to evaluate te omograpy matrix. One metod uses te camera internal parameters and te relative positioning of te two planes and camera center. Te oter one evaluates te omograpy matrix directly from image to world correspondence eliminating te need to calculate te camera s parameters. In tis metod, at least four image to world feature correspondences are necessary because eac point-to-point correspondence is responsible for two constraints and a 2D point as two degrees of freedom (Hartley and Zisserman, 2000). If te number of feature correspondences increase, we ave an over determined solution and H is determined by finding te transformation H tat minimizes some cost function. To evaluate te omograpic matrix it is necessary tat te four points must be in general position, wic means tat no tree points are collinear. Among te available algoritms to estimate te matrix H, it is possible to mention te Direct Linear Transformation (DLT) algoritm, a widely used algoritm because it is accurate and is relatively simple to implement Te Direct Linear Transformation (DLT) Algoritm Taking te Eq. (2) and considering a pair of matcing points (X,Y) and (x,y) in te world and image plane respectively, te following transformation can be written. X ( x + y + ) = x + y Y ( x + y + ) = x + y (3) were X X = 1, X Y = 2, X3 X3 x1 x3 x = and x2 x3 y = correspond to te inomogeneous coordinates in te world and image plane respectively. Expressing te transformation given by te Eq. (3) in terms of vector product ( X Hx = 0 ) enable a simple linear equation to be derived. It is possible to write te Eq. (3) in terms of vector product because, since tis is an equation involving omogeneous vector, te 3-vectors X and Hx are not equal (tey ave te same direction but may differ in magnitude by a non zero scale factor) (Hartley and Zisserman, 2000). From te vector product, it is possible to write te Eq. (3) as A = 0 (4) were for n points A is a 2n x 9 matrix and is a 9-vector made up of te elements of te matrix H. Te Eq. (3) is a omogeneous set of equations and can only be determined up to a non-zero scale factor. After selecting a set of four correspondents points, it is possible to solve tis equation troug te use of te least square

4 metod to find out te solution for, tat is te singular vector corresponding to te smallest singular value of A, or te last column of V in te SVD ( A = UDV T ). Tis solution minimizes A subject to te condition = 1. Te SVD (singular value decomposition) is a factorization of A as A = UDV T, were U and V are ortogonal matrices and D is a diagonal matrix wit non-negative elements (Criminisi, 1999; Hartley and Zisserman, 2000). Tis algoritm known as te basic DLT algoritm can be improved by using a metod of data normalization for te DLT algoritm, consisting of translation and scaling of image coordinates. Tis algoritm is known as DLT algoritm wit data normalization. Te objective of te normalization is to translate te coordinates of te image to new set of points suc tat te centroid of tis new set of points is te origin. Te coordinates are also scaled to adjust te average distance from te origin to te same average magnitude. 3. Tracking Te ability to tracking moving objects in real time is very useful in many different areas for purposes suc as robot navigation, surveillance, video conferencing, etc. In te real time tracking, a process tat follows a moving object (or several ones) in time using a camera is required. Tis process analyses eac video frame and outputs te location of moving targets witin te video frame based on a feature of tis particular object. Among te available tracking algoritms, tis work uses a color based algoritm to output te location of an object to evaluate te AUV dept. To implement tis algoritm a HSV color system is used to separate te ue (color) from te saturation (ow concentrated te color is) and from te brigtness. Tis algoritm creates a probability distribution image of te desired color in te video taking 1D istogram from te ue cannel in HSV space. Using te color istogram as a model te incoming video is converted to a corresponding probability image of te tracked color and te position of te desired image in te video is evaluated. In tis work a color-based tracking algoritm known as Continuous Adaptive Mean Sift (CAMSHIFT) algoritm (Bradski, 1998) was cosen. Since te color distribution canges over time for eac video frame, an algoritm tat searces tis distribution canges is required. Te algoritm tat satisfies tis condition is te modified mean sift algoritm or te CAMSHIFT algoritm because it adapts te probability distribution of te object dynamically to track te object in eac video frame. Te CAMSHIFT algoritm uses a robust non-parametric tecnique for climbing density gradients to find te mode of probability distribution CAMSHIFT and Mean Sift Algoritm Te mean sift algoritm is a non-parametric tecnique developed by Fukunaga and Hostetler (1975) tat climbs te gradient of a probability distribution (Fig. 4) to find te nearest dominant mode peak. Since te objective of tis tecnique is finding te densest region in te image, it is possible to notice tat te algoritm always converges to te local maximum region in te probability density function. Figure 4. Probability Density Function Mapping To analyze te algoritm convergence let s consider n data points x, i = 1,, n on a d-dimensional space i According to Silverman (1986) te multivariate kernel density estimate obtained wit kernel K(x) and window radius, computed in te point x is defined as 1 n x x f ( x) = K i (5) n d i = 1 were te d-variate kernel K(x) is nonnegative and integrates to one. A widely used class of kernels is te radial symmetric kernels. d R.

5 k, d 2 ( ) K ( x) = c k x (6) were te function k(x) is called te profile of te kernel, and integrates to 1 (Wand and Jones, 1995). Te gradient of te density estimator (Eq. (5)) is c, is a normalization constant tat makes K(x) k d f x x 2 n i = x g i 2c 2 1 i k, d n x x = i (7) ( x) g x n d + 2 i= 1 2 x x n i = g i 1 were g( s) = k'( s). Te first term is proportional to te density estimate at x computed wit kernel 2 G ( x) = c g, d g x and te second term 2 n x x i i = 1 x i g m( x) = x (8) 2 n x x i i = 1 g is te mean sift (Ceng, 1995). Te mean sift vector always points toward te direction of te maximum increase in te density and is proportional to te normalized density gradient. Te convergence proof is discussed in Ceng (1995) and is guaranteed to converge to a point were te gradient of density function is zero ( f ( x) = 0 ). Te mean sift algoritm interatively performs computation of te mean sift vector m ( x t ), t+ update te current position 1 t x = x + m ( x t ) Te mean sift algoritm process is illustrated in Fig. 5. Figure 5. Mean Sift Algoritm Since te size and location of te probability distribution canges in time due to te object movement in te video it is necessary to modify te mean sift algoritm because it is designed for static distributions. Te algoritm tat satisfies tis requirement is te CAMSHIFT algoritm because it is designed for dynamically canging distributions since te searc window size is adjusted in te course of its operation. Te CAMSHIFT algoritm relies on te zerot moment information to continuously adapt its window size witin or over eac video frame. Tus, window radius, or eigt and widt, is set to a function of te zerot moment found during searc.

6 4. Experimental Results 4.1. Preliminary Results In order to validate te distance measurement module of te developed software some tests were made before te underwater experiment. Te test was made marking some points on a wite board. Te Fig. 6 sows an image of te sceme used to measure te real distance from two points using te developed software. After selecting four known points in te image, te omograpy matrix can be evaluated and te real distance considering two points on te wite board image can be evaluated. Figure 6. Distance Evaluation Test Te Tab. 1 sows te results of te preliminary experiments. In order to compare te real distance wit te measured distance, te absolute and te relative error are also presented Calibration Table 1. Preliminary Experiment Results. Points Real Distance (cm) Measured Distance (cm) Absolute Error (cm) Relative Error (%) ± ± ± ± ± ± In order to enable te absolute dept measurement of te AUV it is necessary to perform an initial calibration. Te software calibration step can be divided in two parts, te first part involves te calibration to remove te perspective distortion of te measurement plane and te second part involves te calibration to determine te relationsip between te dept of te AUV and te position of te ligt reflection in te camera image. To remove te perspective distortion (omograpy matrix evaluation) of te measurement plane it is necessary to find out te displacement plane of te ligt reflection. In our case tis plane is te perpendicular line to te wave surface passing troug te source of ligt. Te Fig. 7 sows te sceme for te plane calibration. Figure 7. Calibration Plane and Ligt Position

7 After removing te perspective distortion of te measurement plane, te software is ready to measure distance between any two points in te image. Tus it is possible to evaluate te AUV vertical displacement (relative dept) considering an initial dept. To measure te real dept (absolute dept) it is necessary to find out te relationsip between te dept of te AUV and te position of te ligt reflection in te camera image. After some preliminary tests, it was possible to conclude tat eac dept corresponds to a point coordinate in te image because as te dept of te AUV increase te angle of incidence ( θ ) in te camera also increase. Ten point coordinate and te real dept of te AUV was determined from tis relationsip AUV Dept Measurement experiment After te calibration step, underwater tests were performed in still water condition, regular wave condition and irregular wave condition. Taking te most general and real situation case, te results are as follow. Te Fig. 8 sows te dept variation in case of irregular waves. Figure 8. AUV Dept Time History in Irregular Waves Condition. In order to compare te irregular wave influence on te measurement te output measured dept was compared to te real one. Te Tab. 2 sows a comparison between te measurement dept data output and te real value. Table 2. AUV Dept Measurement in Irregular Wave Condition. Time Interval (sec) Real Dept (cm) Mean Dept (cm) Minimum Dept (cm) Maximum Dept (cm) Absolute Error (cm) Relative Error (%) ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± Since te irregular wave is often a combination of many wave components, te definition of te wave caracteristics (eigt, period, etc.) must be statistical or probabilistic, indicating te severity of wave conditions. Among tese statistical parameters it is possible to mention te significant wave eigt (mean of te largest 1/3 of waves measured during te sampling period) and te average period. In our experiment tese parameters are cm for te significant wave eigt and sec for te average period. It was possible to notice tat te presence affects negatively te dept measurement increasing te uncertainty of te measurement. In tis situation tere is a perturbation on te water surface and it is possible to notice two penomena tat affect te AUV dept measurement. First, te wave canges te eigt of te wave surface displacing te reflection ligt coordinate in te image wen compared to te position witout waves. Considering a given instant, te measured dept will depend on te wave eigt in te same instant. Second, te ligt reflection suffers a deformation due to te water surface movement affecting te tracking and te ligt reflection coordinate identification From te experimental results it was possible to analyze te tracking algoritm performance in wave condition (Fig. 8). Te Fig. 9 sows te areas were te tracking algoritm performance is good wen comparing te dept of te AUV and te wave type used in te experiment.

8 Figure 9. Tracking Stability Analysis 5. Conclusion As an alternative solution for te AUV or UV dept measurement problem, te experimental results sow tat te mean measured value is very close to te real dept value. Tis proximity sows tat, altoug te discrepancy of te measured value dept and te real dept is ig for a given instant t, it is possible to evaluate te dept of te AUV wit a good approximation taking te mean value in a coerent interval of time. Te relative error considering te mean measured value is less tan 10%. It was possible to notice also tat two factors affect te tracking performance: te tracked object size and te wave type. Te tracking performance is better for bigger objects tan for small one. In our case since te tracked object size depends on te dept, te tracking performance depends on te dept. Te tracking performance also is better wen te wave eigt and te wave frequency are low (Fig. 9). From te experiments it was possible to conclude tat, te omograpy transformation represents a good approac for te distance measurement wen te displacement plane is known and te tracking performance decrease as te wave eigt and te wave frequency increase. 6. References Bradski G. R., 1998, Computer vision face tracking for use in a perceptual user interface, Intel Tecnology Journal, Q2:1 15. Ceng Y., 1995, Mean sift, mode seeking, and clustering, IEEE Trans. Pattern Anal. Macine Intell, 17: Criminisi A., 1999, Accurate Visual Metrology from Single and Multiple Uncalibrated Images, P.D. tesis., Dept. Engineering Science, University of Oxford, Oxford, UK. Fukunaga K. and Hostetler L. D., 1975, Te estimation of te Gradient of a Density Function, wit Applications in Pattern Recognition, IEEE Trans. Info Teory, vol. IT-21, Hartley R. and Zisserman A., 2000, Multiple View Geometry in Computer Vision, Cap. 1,2,3,4,5. Cambridge University Press. Semple J. and Kneebone G., 1979, Algebraic Projective Geometry, Oxford University Press. Silverman B. W., 1986, Density Estimation for Statistics and Data Analysis, New York: Capman and Hall. Wand M. P. and Jones M. C., 1995, Kernel Smooting, Capman & Hall, London.