" Video Completion using Spline Interpolation

Transcription

1 521 المجلة العراقية لتكنولوجيا المعلومات.. المجلد. - 7 العدد " Video Completion using Spline Interpolation Dr. Abdulameer A. Kareem Sura Hameed Mahdi Computer Science Department /University of Technology Ameeraldelphi@yahoo.com surahameed90@yahoo.com Abstract Completion is the process of rebuilding lost or degraded areas from frames of the video that result from mandatory or optionally removal in a way that does not visually notice. In this paper an efficient algorithm are proposed to fill the missing parts of the frames within the video, and try to remove static object (Logo) from frames of video. the proposed algorithm is Spline Interpolation using Chi square (SPICA) this algorithm is depended on chi square for completing, SPICA algorithm contain steps (texture analysis, horizontal and vertical spline interpolation filling, insertion of interpolated value and gray scale morphological operation) for completing. The results showed that the quality of SPCIA algorithm to fill the missing areas an efficient way, mean squared errors (MSE) of frames and peak signal to noise ratio (PSNR) are used for testing quality of completion.

2 521 المجلة العراقية لتكنولوجيا المعلومات.. المجلد. - 7 العدد Keyword: Video Completion, Image Morphology, Spline Interpolation and Chi-Square Test.. Introduction Video become important in our daily, and in many filed for example surveillance camera system, mobile camera, so that video attracted attention in the field of research from these research is video completion. Image processing indicates to perform operations on the images in order to improve them according to specific criteria or extract information from the image. There are several stages and process applied on image such: Preprocessing which contain conversion and filtering the image, Feature extraction, Classification and Segmentation which contain separate important information like separate object from background. Image restoration (completion) is an interesting subject in the field of image processing. Completion is a method used for rebuilding the lost or deteriorates areas or used for remove the defects from image or frames of video. The defects that result from mandatory or optionally removal in a way that does not visually noticed. Completion technique is also used to remove undesirable object from images or frames of video. The process of completion are include two categories: reparation the new data in missing or damage areas in order to fill it based on the frames itself, the other method may be search in other frames to find the new data for filling. Completion is also called

3 527 المجلة العراقية لتكنولوجيا المعلومات.. المجلد. - 7 العدد restoration or retouching or in painting [1]. Early work converge on expansion of texture [2], and used to elimination text, logo, scratches, spots form frames [3] and solving of a problem of filling missing areas for example ( gaps ) in image completion techniques [4]. Completion is a structure problem (outline of objects) because the filling process didn t distinguish from the shape and texture viewer. There are two methods for image completion are available: in painting and texture synthesis. In painting method is used to fill the small missing or damage area in an image and this method caused blurring when fill large missing area, there for exemplar based method to fill the large missing area are proposed [2], this method depend on diffusion-based method for filling [5]. Texture synthesis method is another method for completion process, this method deal with a large missing, damage and complex area in image and fills the missing region in a visually pleasing way. This method depend on take sample part of it from the unknown region (target) and other from known region (source) these sample are compare with entire image and then choose the best sample that matching the missing or damage region in image. Texture methods is useful for many implementations such as (computer vision, graphics, and image processing) and fill the missing areas with new values depends on information of structure and texture in multiple images or in signal image [6]. Related Work In the field of image completion in video there are several antecedent studies have evolved to solution of problem that represented in missing or damage area in frames of video.

4 521 المجلة العراقية لتكنولوجيا المعلومات.. المجلد. - 7 العدد In [7] suggested method to fill the missing areas in images by copying the small area from source to target area, proposed method are depended on similarity measures (gradient and direction magnitude, distance between patches to be compared, and gray level value) to choose better patches. The filling of the proposed method are visually acceptable. In [8] proposed method based on two step salient structure completion and texture propagation. Wavelet transform are used to detected incomplete salient structures and to complete the salient by using extension and curve fitting. The information of texture is propagated into missing area through Criminisi s patch based method. The method tries to perform well in most circumstances images. The author in [9] proposed method depend on similarity analysis and transformation to solution problems of image completion. In similarity analysis method, analysis source region to find best patch that have same structure and texture with target region, and used random mapping method to find near neighbor for each pixel. Finally confidence is used to calculate priority to choose best matching patch if there are more than one patch are best. The result of algorithm is efficient in speed. In [4] suggested method to fill any missing or damage areas in images by develop a confidence propagation. The result of the proposed method is good and not notices by an observer.

5 521 المجلة العراقية لتكنولوجيا المعلومات.. المجلد. - 7 العدد In [10] presented an innovative method for reconstruct the missing region in image of historical artifacts. This method depends on take similar sample shape for the damage object of image in other images. The proposed method depended on two step: the first step using set point registration to complete the salient shape of the damage object, the second step histogram specification, photometric and geometric transformation are used to fill the target region with the texture sample object. The result of the proposed was promising in many of historical artifacts image. Video Completion The term completion is used in both sides either in image or in video, completion technique appears at the first in images for many years. Completion techniques in the field of image have been developed and this develop is extended to the video. The process of image completion is more quite close to the process of video completion because video is basically a group of still images (frame), but the main difference between them that in video there are more than one frame used as completion sources. Video completion is a process of rebuilding the missing or damage areas of frames that result from removal unwanted object in frames of video in a visually pleasing way. Object removal from video may be static or dynamic, and filling the missing region depends on background information. The process of completion in video are include two categories:

6 531 المجلة العراقية لتكنولوجيا المعلومات.. المجلد. - 7 العدد a) Spatial Completion: used to reparation the data in missing areas by depend on the information of the same frames, when the objects removal is static. b) Temporal Completion: used to fill the missing region depends on the information of the next or previous frames, and it is usually used when the object removal is dynamic. Data Interpolation in image Important techniques in numerical analysis are interpolation techniques. They are commonly used in the field of science and engineering, the term of interpolation refers to estimate the missing data from known surrounded data depending on some algorithms. This estimation is useful in different types of application to restore missing value of that image. and this process also useful in several situations such as to enlarge (zooming) the small image by increasing the number of pixels [11] and to fill some gaps in image or to fill the missing or damage areas in images or in frames of video that results from object removal from image or video. This process depends on some theories and algorithms which are used in one dimension (e.g. nearest neighbor interpolation) to find the relationship between the missing values and the known value, or using twodimensional (e.g. bilinear interpolation) to find relationship between the missing values and the known value. 4.1 Spline Interpolation

7 535 المجلة العراقية لتكنولوجيا المعلومات.. المجلد. - 7 العدد Spline interpolation represent different method to interpolated data by using single formula, to correlate all the data points, a polynomial between each couple of data points that represent in form of interval since these interval are refer to simple function represent in graph is a spline, one of these point is specified non- locally, since nonlocals are used to guarantee a smoothness, there are several formulas of spline each formula represent a low degree polynomial to pass al data point, spline is favored than polynomial interpolation, because the rate of error is low when using low degree polynomials for spline interpolation, kind of spline are: linear, cubic, quadratic, most commonly way of spline interpolation is cubic aim of it that get on form of interpolation continue from the first and second derivative, and simple form of spline is linear interpolation and the quadratic function, the graph became a smoothness when the quadratic function are put in each interval. Spline interpolation in that interval gives the interpolation formula [12]: A=, B=..(1) where x is the interpolation points x-axis coordinate, y is the interpolation point y-axis coordinate, x k is the first data point x-

8 532 المجلة العراقية لتكنولوجيا المعلومات.. المجلد. - 7 العدد axis coordinate, y k is the first data point y-axis coordinate, x k+1 is the second data point x-axis coordinate, y k+1 is the second data point xaxis coordinate. Linear interpolation is quick and easy in interpolated data. Image Morphology Operation The morphology began to evolve as a part from image analysis. Morphology word indicates to branch of biology that transacts with shapes frames for example plants and other. Set of theory that represents objects in image is mathematical morphology; the work of morphology is depending on extension and retraction [13]. Morphology operations are used for the following processing [14], pre or post processing image for example simplify form of object, used to remove noise from image or video, separate the object from the background of the image or frames of video, description area and shape of object in image or video. Morphological processes are work depend on structure have two concept (Dilation and Erosion), this operation is essential for processing as a (close, opening). 5.1 Dilation Operation The maximum value in the window is define as a dilation, operation of dilation are used to expand objects in image, used to fill the missing region, and image look a brighter after process

9 533 المجلة العراقية لتكنولوجيا المعلومات.. المجلد. - 7 العدد dilation, expanding process of dilation are work by replace value zero to one, dilation is represent in equation (2), where I represent image and S represent structuring element.... (2) Where: I represent Image, S represent Structure element 5.2 Erosion Operation The minimum value in the window is define as an erosion, operation of erosion are used to retraction or thin objects in image, and image look a darker after erosion operation apply, retraction process of erosion are work by replace value one to zero. Where: I represent Image, S represent Structure element 5.3 Opening and Close Both of methods are implemented in particular way where opening operation represented at the first erosion process and then dilation process, close operation represent dilation and then erosion operation. Chi-Square Test This test working on the process of measuring the correlation between two variables, Chi-Square uses to measure

10 534 المجلة العراقية لتكنولوجيا المعلومات.. المجلد. - 7 العدد the correlation between the pixels that surround the lost or degraded areas in the frame of video to show if the pixel values are correlation or non- correlation for the purpose of filling these lost or degraded areas. This procedure is in accordance with the equation (4) [15] and the resulting value from the equation are compares with the values determined by the researcher this value are found in a table called chi- square test table, table contains from row and column of fixed values (threshold), column represent number of pixels that are used to measure the amount of correlation and row represent expected value for block of pixels that are selected to measure the correlation...(4) Where: oi represent observed frequency. ei represent expected frequency. The x 2 value obtained from equation is compared with that tabulated value in chisquare table underappreciated degree of freedom. If calculated value is larger than tabulated value then the differences between examined variables are significant which means there is something affecting these variables but in case it is less than tabulated value that means that the differences between examined variable are unreal and due to random mistakes of the test, theoretical or tabulated value underappreciated degree of freedom.

11 531 المجلة العراقية لتكنولوجيا المعلومات.. المجلد. - 7 العدد Proposed Algorithm This paper include a proposed algorithm to remove object (logo) from frames of video and then fill the missing area result from removing process in way cannot detected by user. The AVI video file format is used as digital video format in proposed system. The missing areas are specified using binary mask since the mask was extracted from data set that design to store some specific logos. The proposed algorithm is Spline interpolation using chi square (SPICA), this algorithm is design for completing the cracked pixels results from removing logo in frames of video based on chi square. The chi square test is used for texture analysis that is surrounding the area of logo the pixels upper, lower, left and right the logo to select the best fit region for interpolating. The filling is done on the correlated direction. General view of this algorithm is shown figure (2). This algorithm contain steps to completion process, algorithm (1) explain the SPICA for completion: a) The First Step: Locate logo in Frame The binary mask enters to algorithm for identifying the location of logo by replacing the white pixels (cracked pixels) in

12 531 المجلة العراقية لتكنولوجيا المعلومات.. المجلد. - 7 العدد binary mask with black pixels in original frame to deal with it in a direct manner. b) The Second Step: Texture Analysis After locate the missing area in frame from the previous step, analysis the texture that surrounding the missing area to show the correlation between pixels, the correlation chi square test used for this purpose and apply for each pixel in four direction of logo.

13 537 المجلة العراقية لتكنولوجيا المعلومات.. المجلد. - 7 العدد The Third Step: Horizontal Spline Interpolation Filling In this step find the intermediate point in the horizontal direction for each row to find the total missed point in area of specified logo. c) The Fourth Step: Vertical Spline Interpolation Filling In this step find the intermediate point in the vertical direction for each column to find the total missed point in area of specified logo. The Fifth Step: Insertion of Interpolated Value This step depended on chi square value to insert of interpolated data that take from the third or fourth step. The minimum values are taken to replace in new frame (filling frame). d) The Sixth Step: Gray Scale Morphological Operation The frame result from previous step resize (enlarge) to increase the resolution of frame. The dilation is used on frame as morphological operation in a gray scale status. The dilation used to uniform or smooth the filled region. After dilation the frame resize again to normal size (original size). The area obtained from these steps is cut and replace the missing area of removed logo. Results The mean square errors (MSE) and peak signal to noise ratio (PSNR) are used for testing frames of video after remove logo from it by using the proposed algorithm to evaluate the

14 531 المجلة العراقية لتكنولوجيا المعلومات.. المجلد. - 7 العدد completion method and quality of frames with respect to original frames. The PSNR and MSE are explain in table (1) of some sample of frames in video, while figure (3, 4 and 5) explain some sample of videos, the original and completion frames. Quality of filling depended on the nature of frames and degree of complexity. The time consuming for each frame are explain in table (2). The (SPCIA) algorithm is taken more or less time consuming for implementation depending on the background information and complexity.

15 531 المجلة العراقية لتكنولوجيا المعلومات.. المجلد. - 7 العدد

16 541 المجلة العراقية لتكنولوجيا المعلومات.. المجلد. - 7 العدد Conclusions The proposed algorithm can used for frame completion in video with different characteristics and from the implementation of this

17 545 المجلة العراقية لتكنولوجيا المعلومات.. المجلد. - 7 العدد algorithm some important conclusion will presented as follows: The differences between results of proposed algorithms depend on the background of the logo in frame (simple, semi complex or complex).the completion of frame effected by the animation of the background (static or dynamic background) and The difference size and location of the object in each frame effected on frame completion. The proposed algorithm can be applied for removing dynamic logo (animated logo). The proposed method is flexible for contribution with other method. References [1] Bertalmio, M, Sapiro, G., Caselles and V., Ballester, C., Image Inpainting, pages , SIGGRAPH [2] Alexei A. Efros and Thomas K. Leung, Texture Synthesis by nonparametric Sampling, In Proceedings of the International Conference on Computer Vision Vol. 2, ICCV 99, pages 1033, Washington, DC, USA, IEEE Computer Society, [3] Marcelo Bertalmio, Guillermo Sapiro, Vincent Caselles, and Coloma Ballester, Image Inpainting, In Proceedings of the 27th annual conference on Computer graphics and interactive techniques, SIGGRAPH 00, pages , New York, NY, USA, ACM Press/Addison- Wesley Publishing Co,2000. [4] Xiao Tan, Changming Sun, Kwan-Yee K. Wong and Tuan D. Pham, Guided

18 542 المجلة العراقية لتكنولوجيا المعلومات.. المجلد. - 7 العدد Image Completion by Confidence Propagation, Original Research Article Pattern Recognition, Vol. 50, Elsevier,Pages , February [5] T.F. Chan, S.H. Kang and J. Shen, Euler s, Elastic and Curvature-Based Inpainting, SIAM J. Appl. Math, , [6] M. Xiao, G. Li, L. Xie, Y. Tan and Y. Mao, Contour-Guided Image Completion Using a Sample Image, J. Electron. Imaging vol. 24, page 2, [7] Dr.Abdulameer A.kareem, A Patch Based Image Completion Algorithm with Similarity Criteria, first computer science conference proceeding/uot, [8] Shutao Li and Ming Zhao, Image Inpainting with Salient Structure Completion and Texture Propagation, Elsevier, Volume 32, Issue 9, Pages , 1 July [9] Mang Xiao, Guangyao Li, Yunlan Tan, and Jie Qin, Image Completion Using Similarity Analysis and Transformation, International Journal of Multimedia and Ubiquitous Engineering,Vol. 10, no. 4, [10] Mang Xiao, Guangyao Li, Lei Peng, Yangjian Lv, and Yuhang Mao, Completion of Images of Historical Artifacts Based on Salient Shapes, International Journal for Light and Electron Optics, vol. 127 no [11] Rukundo Olivier and Cao Hanqiang, Nearest Neighbor Value Interpolation, International Journal of Advanced Computer Science and Applications (IJACSA), Vol. 3, No. 4, [12] Mohammed Jamal, Gaps Filling of Landsat EMT Image using Digital

19 543 المجلة العراقية لتكنولوجيا المعلومات.. المجلد. - 7 العدد Filtering Techniques, MSC thesis, Collage of University of Baghdad, [13] Reecha Sharma, Beant Kaur, Detection of Edges Using Mathematical Morphology for X-Ray Images, An International Journal of Engineering Sciences ISSN: Issue Dec. Vol. 5, [14] Gonzalez, Digital Image Processing, 2nd Edition, Prentice Hall, [15] Abdul Ameer Abdulla Kareem, Data Extraction of Digital Images Using Statistical Analysis, A Dissertation submitted to the department of computer science, university of technology, 2006.