Learning and Adaptive Video Processing Methods

Transcription

1 University of Pannonia Information Science and Technology PhD School Learning and Adaptive Video Processing Methods Summary of Doctoral Theses Attila Licsár Department of Image Processing and Neurocomputing Supervisor: Prof. Tamás Szirányi Veszprém, 2007

2 Preliminaries and Objectives of Research Preliminaries and Objectives of Research In the last decades the increase of the computational performance and advances of image acquisition and display devices made possible the processing of high resolution image sequences (e.g. 6000x4000 pixel) and real time analysis of images grabbed by camera systems. The efficient processing of the increasing amount of information is influenced by the speed, quality of methods as well the human factor, which is required in case of manual correction of wrong results. Since the human intervention reduces the speed of processes and raises price, the introduction of efficient and reliable processing methods became more important. The author, during his research work at the Department of Image Processing and Neurocomputing of University of Pannonia, worked in the field of hand gesture recognition by vision based methods and digital restoration of degradations on archive films. In this period he participated in the project called DIMORF ( Digital Motion Picture Restoration System for Film Archives ), sponsored by the National Research and Development Program (NKFP-2/049/2001) of the Ministry of Education, and worked in the development of DIMORF software framework and several film restoration methods. The research work in the dissertation can be grouped into two main areas: improvement of conditions of the human-computer interaction by the efficient and trainable recognition of hand gestures and correction of image vibration and blotch defects on archive films. In all of the examined tasks the problem formulation and the selection of research goals have been done by the analysis of real world situations and environments when the computer has to make decisions and interventions by adaptive and learning methods during the fast analysis of image sequences. The title of the dissertation summarizes these issues. The examined practical problems are in connection with some important tasks of video processing: 1) reduced efficiency of gestures recognition of users who did not train the system by their gesture samples [22,23]; 1

3 Preliminaries and Objectives of Research 2) erroneous results of automatic image stabilization methods if the given scene involves complex, multiple object motions; 3) high false positive error rate of automatic detection of blotches, where image pixels are badly classified as corrupts regions, results in undesirable modifications of the uncorrupt regions during the correction step of defects. In the restoration process of archive films an important rule is that the original image content cannot be modified except when it is inevitable, due to the loss of the original information [24,25]. In the examined methods the inadequate results of the automatic processing are usually corrected manually by human interventions. During the analysis, further difficulties were caused by: The huge amount of data to be processed due to the high resolution images or realtime analysis. Several types of image defects being simultaneously present on archive film sequences (e.g. intensity fluctuations or flickering, grain noise) reduce the reliability of the analysis and processing of local and global image information. Thus, during the analysis and processing of image sequences, further aims were the reduction of the computation cost of various tasks (e.g. classification, motion estimation) and the improvement of their reliability and precision in the presence of several film degradations. The dissertation aims to introduce and investigate learning and adaptive methods in the outlined video processing tasks, to improve the efficiency of the humancomputer interaction, to increase speed and quality of the processing, and to reduce the amount of the necessary human intervention. 2

4 Research methodology Research methodology During the research work, requirements involved the high level of automation of the processing by the analysis of real use cases. For this reason the proposed procedures apply supervised and unsupervised learning, and adaptive methods. During the examination of the methodology for restoration processes and during the problem formulation it has been a great help to be involved in the development of the DIMORF software [14] and the restoration of the feature film Lúdas Matyi ( Mattie, the Gooseboy, 1949) [6]. The objective evaluations of the quality of methods were done by constructing ground truth data sets that involve the corresponding reference data, for example the classes of the gestures or the positions of blotches. The ground truth data set of blotches was generated by a new semi-automatic method that made possible the efficient selection of the numerous defects and the performance evaluation on original archive images. The author developed a software framework for testing and visualization of results and generation of statistical data. The enhanced version of this software was the framework of restoration methods in the DIMORF project. Its tasks were processing and documentation of restoration methods and handling, visualization, comparison of image sequences. The statistical results were stored in XML ( Extended Markup Language ) and CSV ( Comma Separated Values ) file formats that allowed of the further processing and visualization. The development of methods was done in Microsoft Visual C++ environment. The author utilized free and open source libraries for basic image processing tasks and image handling: OpenCV [26] ( Open Source Computer Vision Library ), IPL [27] ( Image Processing Library ) developed by Intel. The SVM ( support vector machine ) based learning method was implemented by the LIBSVM [28] library. 3

5 Thesis Groups Thesis Groups 1. Thesis group: hand gesture recognition with interactive training in a camera-projector environment. 1.1 I introduced a user-adaptive gesture recognition method that improves the recognition efficiency of gestures by continuous user-computer interaction. A typical problem of recognition methods which utilize separate training and recognition phases is that the recognition performance decreases when gestures are performed by users who did not train the system previously with their own gesture samples [22,23]. I showed that even if user does not accomplish preliminary training the falsely recognized gestures can be corrected by a novel interactive training method during the recognition phase. I experimentally proved that it is enough to retrain only the falsely recognized gestures by the proposed method without performing preliminary training of all gestures to ensure the suitable performance. 1.2 I introduced a contour based recognition method of hand gestures by Fourier descriptors, which analyzes several subsequent contours in time and classifies gestures by maximum likelihood estimation. I showed that temporal analysis of gestures improves the performance of the recognition since user usually performs the same gesture for a given time. I experimentally proved that the introduced contour signature function increases the performance of the Fourier descriptor based classification compared to previously applied functions. During the development of training and classification methods I also considered that low complexity operations are needed, due to the real-time interaction between human and computer. 4

6 Thesis Groups 1.3 I introduced an arm and hand segmentation method that extracts hand contours in a camera-projector environment. I showed that the contour of the arm and the hand, which is pointed onto the projected image, can be extracted by the proposed method in case of any static or dynamic background. Its advantage is that the segmentation method does not need special devices such as infra cameras and specific light sources. I experimentally proved that the efficiency of the hand segmentation is not restricted by the clothes of users. Related publications: [2,4,12]. 2. Thesis group: adaptive stabilization of image vibration on archive films by automatic selection of ROF ( region of fixation ). 2.1 I introduced an image stabilization method that adaptively selects an image region (ROF) by spatial segmentation in a quad-tree manner and by temporal analysis of motion information to estimate the vibration of the image sequence. I showed that the improper results of stabilization are usually caused by complex and/or multiple object motions since the motion estimation, measured on the whole image region, is usually not reliable. The proposed method adaptively selects a ROF, depending on the complexity and structure of moving objects, where the motion estimation is reliable and the motion is relatively continuous. I experimentally proved that image sequences, with different complexity of motions, can be efficiently stabilized by motion parameters measured in the adaptively selected ROF. Related publications: [5,13,3]. 5

7 Thesis Groups 3. Thesis group: detection of blotches on high resolution archive films with minimal human interaction. 3.1 I introduced a blotch detection method that reduces the false alarm rate of pixels (wrongly classified as blotches) originating from motion by ROI ( region of interest ) based motion estimation and compensation on the region of blotch candidates. I showed that it is enough to estimate and compensate motions in the region of the preliminary selected blotch candidates (ROI) instead of the computation on the whole image. I experimentally showed that ROI based motion estimation considerably reduced the computation time without decreasing the performance of the detection compared to methods that compensate motions on the whole image region. I introduced a semiautomatic method to generate ground truth data sets of blotches that define positions of artifacts to be made possible the performance evaluation on original archive and high resolution images. 3.2 I introduced a method that reduces false alarm results by classification of the preliminary determined blotch candidates and removing of non-blotch objects. The inaccurate motion estimation, which is caused by artifacts on archive films and complex object motions, results in several false alarms. I experimentally proved that classification of image features, extracted in regions of blotch candidates and their neighbors, significantly reduces false alarm results. I introduced a new evaluation technique of the performance of blotch detection that estimates the amount of the required human work, which is needed for manual classification of blotch candidates, following the automatic process. I experimentally proved that the proposed NN ( neural network ) and SVM ( support vector machine ) based classification of blotch candidates extensively decreased the amount of the necessary human work. Related publications: [1,6,7,9,10]. 6

8 Possible Applications Possible Applications With the introduced hand gesture recognition method and camera-projector environment any user interface can be controlled by the recognized gesture and the detected position of the hand. In the MUSCLE ( Multimedia Understanding through Semantics, Computation and Learning ) project [29] of the European Union there was a cooperation to apply hand recognition method in the BilVideo [30] video database management system. The aim was the gesture based generation of queries that involves spatial relation of objects such as object A is behind object B. The developed framework software of DIMORF and several restoration methods (e.g. flicker correction, image stabilization) were applied in the reconstruction of the first Hungarian color feature film Lúdas Matyi ( Mattie, the Goose-boy, 1949) [6]. The Hungarian National Film Archive, sponsored by the Hungarian Ministry of National Cultural Heritage, accomplished the reconstruction work together with the Hungarian Film Laboratory, University of Pannonia (formerly the University of Veszprém), MTA SZTAKI, and RDI Sound Studio. The DIMORF software and the proposed methods can easily be used for the restoration works of further films. The DIAMANT [31] film restoration software developed and distributed by HS- ART Digital, is a well-known and wide-spread tool utilized in national film archives and in the film industry. In the cooperation between HS-ART Digital and University of Pannonia we have the opportunity to implement and evaluate our methods in the software environment of DIAMANT. 7

9 List of Publications List of Publications International SCI journal paper [1] A. Licsár, T. Szirányi, L. Czúni, Trainable blotch detection on high resolution archive films minimizing the human interaction, Machine Vision and Applications Journal, accepted, (IF: 0.667) [2] A. Licsár, T. Szirányi, User-adaptive hand gesture recognition system with interactive training, Image and Vision Computing, Vol. 23, No.12, pp , (IF: 1.159) [3] L. Czúni, A. Hanis, L. Kovács, B. Kránicz, A. Licsár, T. Szirányi, I. Kas, Gy. Kovács, S. Manno, Digital Motion Picture Restoration System for Film Archives (DIMORF), SMPTE Motion Imaging Journal, Vol. 113, pp , (IF: 0.333) International SCI periodical [4] A. Licsár, T. Szirányi, Hand Gesture Recognition in Camera-Projector System, International Workshop on Human-Computer Interaction, Lecture Notes in Computer Science, Vol. LNCS 3058, pp.83-93, (IF: 0.513) [5] A. Licsár, L. Czúni, T. Szirányi, Adaptive Stabilization of Vibration on Archive Films, Lecture Notes in Computer Science, CAIP 2003, Vol. LNCS 2756, pp , (IF: 0.515) 8

10 List of Publications Hungarian journal paper [6] L. Czúni, A. Licsár, T. Szirányi, Digitális filmjavító eljárások, Magyar Elektronika, (11), HU ISSN , pp , International conference proceeding [7] A. Licsár, T. Szirányi, L. Czúni, Adaptive Blotch Detection in a Film Restoration Framework, ECCV Workshop on Applications of Computer Vision, Graz, pp , [8] A. Licsár, T. Szirányi; L. Kovács, B. Pataki, Tillarom: an AJAX based folk folk song search and retrieval system with gesture interface based on Kodály hand signs, International multimedia conference. Proc. of the 1st ACM international workhsop on human-centered multimedia, Santa Barbara, USA, pp , [9] A. Licsár, L. Czúni, T. Szirányi, Trainable Post-Processing Method To Reduce False Alarms In The Detection Of Small Blotches Of Archive Films, IEEE International Conference on Image Processing (ICIP), Genoa, Italy, pp , [10] A. Licsár, T. Szirányi, L. Czúni, Blotch Detection in Archive Film Restoration by Adaptive Learning, Workshop on Machine Learning Techniques for Processing Multimedia Content (MLMM), Bonn, pp. 7-11, [11] L. Czúni, G. Császár, A. Hanis, L. Kovács, A. Licsár, T. Szirányi, Semi Automatic Digital Motion Picture Restoration System with Learning Capabilities, Learning for Adaptable Visual Systems (LAVS), Cambridge, UK, [12] A. Licsár, T. Szirányi, Dynamic Training of Hand Gesture Recognition System, ICPR 04, Cambridge, UK, IEEE & IAPR, Vol. 4, pp , [13] A. Licsár, L. Czúni, T. Szirányi, Stabilization Of Vibration On Archive Films By Automatic Multi-scale ROF Selection, Advanced Concepts for Intelligent Vision 9

11 List of Publications Systems (Acivs), Ghent, Belgium, pp , [14] M. Bölecz, L. Czúni, B. Gál, A. Hanis, L. Kovács, B. Kránicz, A. Licsár, T. Szirányi, I. Kas, Gy. Kovács, S. Manno, DIgital MOtion Picture Restoration System for Film Archives (DIMORF), A complex solution for film scanning, processing and recording, Conference of the International Broadcasting Convention, Amsterdam, pp , [15] A. Licsár, T. Szirányi, Supervised training based hand gesture recognition system, 16th ICPR, Vol. 3., IEEE & IAPR, pp , [16] A. Licsár, T. Szirányi, Hand-Gesture Based Film Restoration, 2nd Int. WS on Pattern Recognition in Inf. Systems (PRIS 02), IAPR, Alicante, Spain, pp , Hungarian conference proceeding [17] A. Licsár, L. Czúni, T. Szirányi, Blotch Detection with Trainable Post-processing Method, Joint Hungarian-Austrian Conference on Image Processing and Pattern Recognition (HACIPPR), Veszprém, pp , [18] A. Licsár, L. Czúni, T. Szirányi, Automatic Stabilization of Image Vibration, Képfeldolgozók és Alakfelismerők IV. Konferenciája, Miskolc-Tapolca. pp , [19] L. Czúni, T. Szirányi, A. Licsár, A. Hanis, J. Schanda, B. Kránicz, P. Farkas, DIgitális MOzgóképhelyreállító Rendszer Filmarchívumok számára (DIMORF), Képfeldolgozók és Alakfelismerők III. Konferenciája, Domaszék, [20] A. Licsár, T. Szirányi, Supervised Training Based Hand Gesture Recognition System, Képfeldolgozók és Alakfelismerők III. Konferenciája, Domaszék,

12 List of Publications Publications not Related to the Theses [21] L. Czúni, G. Császár, A. Licsár, Estimating the Optimal Quantization Parameter in H.264, International Conf. on Pattern Recognition (ICPR), pp ,

13 References References [22] A. Ramamoorthy, N. Vaswani, S. Chaudhury, S. Bannerjee, Recognition of dynamic hand gestures, Pattern Recognition, Vol. 36(9), pp , [23] B. Raytchev, O. Hasegawa, N. Otsu, User-independent online gesture recognition by relative motion extraction, Pattern Recognition Letters, Vol. 21(1), pp , [24] P. Read, M.P. Meyer, Restoration of Motion Picture Film, Butterworh-Heinemann, [25] B. Delaney, B. Hoomans, PrestoSpace User Requirements Feedback meeting in London, An integrated solution for Audio-visual preservation and access, [26] Intel Open Source Computer Vision Library: [27] Intel Image Processing Library: [28] C.C. Chang, C.J. Lin, LIBSVM: a library for support vector machines, [29] MUSCLE project: [30] BilVideo: [31] HS-ART Digital: 12