A Survey on Multimedia Information Retrieval System

Transcription

1 A Survey on Multimedia Information Retrieval System Pooja Shinde 1, Prof. J. V. Shinde 2 1 M.E Student, Kalyani Charitable Trust s Late G.N. Sapkal College of Engineering 2 Professor, Kalyani Charitable Trust s Late G.N. Sapkal College of Engineering Savitribai Phule Pune University Abstract This survey paper aims to provide the multimedia researcher with the different fusion approaches. The different fusion approaches are used to have multiple modalities are used to accomplish various multimedia related analysis tasks. Fusion method are describe by using different perspective.the problem of metadata based image retrieval system and a very rapid growth in the quantity and availability of digital image involves the researches into the multimedia information system. Finally in last section we are discussing on the multimedia information retrieval based on late semantic fusion approaches. different dataset, from the image some information was present in the real world is automatically missing. So, the loss of information can be due to the missed details, bad clarification or different viewing angles or any type of imperfectness of an image capturing device like a camera [20][21]. Different techniques of fusion are used in the multimedia information retrieval system [18]. Text based image retrieval system, Content based image retrieval system, content base multimedia information retrieval, and these all techniques are discussed in this paper. Index Terms Late Fusion, Multimedia Information Retrieval, Content based retrieval, Text based retrieval. 1. Introduction The major objectives of multimedia Information Retrieval System is to minimize, overhead of user needed information. The Multimedia Information Retrieval System (MIRS) is that which is able to store, retrieve, maintain, and manage the information. In MIRS, there is difficulty in the communication between an information/image, user and the image retrieval system. The users may have the differing needs and knowledge about the images collection and an image retrieval system must support various forms of query formulation in system. When an image is recorded or observed there can be problem of sensory gap which defines gap between the object/image in the real world and the information in a description resulting from a recording of that different scene for image. That is, when an image is provided to 2. Related Work Multimedia information retrieval of the text, images, audio, and video are may be the best developed technology. As early in 1986, image databases were being developed and deployed the system such as UC Berkeley s Image Query on system, so Developers believe that this software was the first deployed multi-user networked digital image database system in the environment. In the recent times, multimodal fusion techniques have gained much attention of many researchers due to the benefit it provides for various multimedia analysis tasks. One of the earliest considerations is to decide what strategy to follow when fusing multiple modalities. The most widely used strategy is to fuse the information at the feature level, which is also known as early fusion. The other approach is decision level fusion or late fusion which fuses multiple modalities in the semantic space. A combination of these approaches is also practiced as the hybrid fusion approach. 1945

2 1. Level of Fusion The fusion of different modalities is generally performed attwo different levels: feature level or early fusion and decision levelor late fusion [2]. We now discuss the levels of fusion in detail. themonomedia similarity profiles by means of aggregation functions. Several researchers have successfully adopted the decision level fusion strategy. Late fusion is the most used technique in text/image information fusion. 1.1 Feature level multimodal fusion In the feature level or early fusion approach, the features extracted from input data are first combined and then sent as input to a single analysis unit that performs the analysis task at performs the analysis task. The early fusion approach consists in representing the multimedia objects in a multimodal feature space designed via a joint model that attempts to map image based features with text based features. The simplest early fusion method consists in concatenating both image and text feature representations [2][3]. In the feature level fusion approach, the number of features extracted from different modalities may be numerous, which may be visual features, text features, Audio features, motion features and metadata. Image Represe ntation Text Represen tation Fig. 1. Feature level multimodal fusion 1.2 Decision level multimodal fusion Joint features model In the decision level or in late fusion approach, the analysis unit first provide the local decisions D1 to Dn that are obtained based on individual features F1 to Fn [2]. The decision level fusion strategy has many advantages over feature level fusion. For instance, unlike feature level fusion, where the features from different modalities (e.g. audio and video) they may have different representations, the decisions (at the semantic level) usually have the same representation. Concerning late fusion techniques, they mainly consist in merging Fig. 2 Decision level multimodal fusion III. Multimedia Retrieval System: Methodologies Multimedia Information Retrieval is the usually addressed from a textual point of view in most of existing commercial tools, using annotations or metadata information associated with images or videos [4][5]. A.TBIR SYSTEM In TBIR(Text Based Image Retrieval) system the index and search of the images in the collection, based on the textual information in the metadata files associated with each of these images. IDRA tool is used to extract, select and preprocess this information [5]. Finally, both IDRA and Lucene search engines are used for the index and retrieval of the preprocessed textual information, obtaining a ranked result list with the retrieved relevant images for each query. The major component of TBIR system are Textual information extraction, IDRA Processes, Index and Search. B.IDRA Processes IDRA processes the selected text in three steps: 1) Special characters deletion: characters with no statistical meaning, like punctuation marks or 1946

3 accents, are eliminated;2)stopwords detection: exclusion of semantic empty words from specifics lists for each language. 3) Stemming: for reducing inflected or derived words to their stem, base or root form [5]. C. VSM (Vector Space Model) Vector space model is an information retrieval model for representing text documents as vectors of identifiers, such as, for example, index terms [4]. It is used in filtering of information, information retrieval from WWW, indexing and rankings. Ranking which is nothing but adjusting the retrieved documents as per their relevance with the query executed by end user. D.CBIR SYSTEM In CBIR(Content Based Image Retrieval) technique which uses visual contents to search images from large scale image databases according to users' interests, has been an active and fast advancing research area In this case of retrieval system we need to focus on the metadata of the given objects [7][8]. CBIR works on a totally different principle from keyword indexing. Its basic features characterizing such as content of the image, such as colour, texture, and shape of an image.semantic features such as the type of object present in the image are harder to extract, though this remains an active research topic. experiments is not provided to the content based image retrieval module. Then it is observed that results are not upto the mark, hence in the next generation system the better result is obtained by performing merging [12][13]. Using the ImageCLEF 2009 database the system had itself implemented IDRA tool and the CBIR system. In this system a global strategy for all experiments has been conducted and a content based module start working with a filter result list of TBIR system [14]. Finally the merging algorithm is applied on the result of CBIR system and it requires the trec_eval format for the input result list, this is required for the ImageCLEF organization [15]. Afterwards it had included the detailed description of the image i.e. it includes the XML structure for 0.5 million captions of ImageCLEF database. In the multimedia information retrieval system which uses the Wikipedia collection at ImageCLEF 2011 dataset in which it contains almost 240 thousand annotated images for each image the XML file is created that contains the xml version [17], encodings, image id, file and txtxml language. In the MIRS the TBIR subsystem is applied over the whole database acting as a filter to the CBIR systems, selecting the relevant images for certain query. In a second step, the CBIR system works over the set of filtered images reordering this list taking into account the visual information of the image [19]. The CBIR system generates different visual result lists depending on the number of query images. E. CEDD SYSTEM In Color and Edge Directivity Descriptor (CEDD) systems the descriptors consists of different n texture areas [10]. In this system, each textual area is separated into 24 different sub-regions, in which each sub-region describe the color information [11]. CEDD uses the different color information, as it results from the 2 fuzzy systems model that map different colors of the image in a 24-color custom palette. To extract the texture information, CEDD use fuzzy version of the five digital filters which is proposed by the MPEG-7. IV. MERGE Subsystems Descriptions The merging and the text based retrieval method works on the ImageCLEF photographic dataset in which the text based baseline V. CONCLUSION In the MIRSystem detailed description and analysis of textual pre-filtering techniques are used. This textual pre-filtering technique can reduces the size of the multimedia database improving the final fused retrieval results of the system. The combination of textual pre-filtering and image re-ranking lists in a late fusion algorithm outperforms those without prefiltering of the text. Textual information better captures the semantic meaning of a topic and that the image re-ranking (Si) fused with the textual score (St) helps to overcome the semantic gap in between them. In the future, we ll develop methods to speed the re-ranking processes in the different visual related search systems. Beyond using the visual features only, we ll also try to improve the use of a large set of different generic concept detectors in computing similarity in different multimedia document context. 1947

4 Refrences [1] XaroBenavent, Ana Garcia-Serrano, Ruben Granados, Joan Benavent, and Es- ther de Ves Multimedia Information Retrieval Based on Late Semantic Fusion Ap-proaches: Experiments on a Wikipedia Image Collection IEEE transactions on multimedia, vol.15. [2] J. A. AslamandM. Montague, Models formetasearch, in Proc. 24 th Annu. Int. ACM SIGIR Conf. Res. Develop. Inform. Retrieval, New Orleans, LA, USA, 2001, pp [3] S. Clinchant, G. Csurka, and J. Ah-Pine, Semantic combination of textual and visual information in multimedia retrieval, in Proc. 1 st ACM Int. Conf. Multimedia Retrieval, New York, NY, USA, [4] A. Popescu, T. Tsikrika, and J. Kludas, Overview of the wikipedia retrieval task at ImageCLEF 2010, in Proc. CLEF 2010 LabsWorkshop,Notebook Papers, Padua, Italy, 2010 [Online]. Available: clef2010.org/ resources/proceedings/clef2010labs_ submission_124.pdf. [5] S. Clinchant, G. Csurka, and J. Ah-Pine, Semantic combination of textual and visual information in multimedia retrieval, in Proc. 1 st ACM Int. Conf. Multimedia Retrieval, New York, NY, USA, [6] A. García-Serrano, X. Benavent, R. Granados, E. de Ves, and J. Miguel Goñi, Multimedia Retrieval by Means of Merge of Results from Textual and Content Based Retrieval Subsystems, inmultilingual Information Access Evaluation II. Multimedia Experiments: 10th Workshop of the Cross-Language Evaluation Forum, CLEF 2009, Corfu, Greece, September 30 - October 2, 2009, Revised Selected Papers. Berlin, Germany: Springer-Verlag, 2010, pp [7] R. Granados, J. Benavent, X. Benavent, E. deves, and A. Garcia Serrano, Multimodal Information Approaches for the Wikipedia Collection at Image CLEF 2011, in Proc. CLEF 2011 Labs Workshop, Notebook Papers, Amsterdam, The Netherlands, [8] S. A. Chatzichristofis, K. Zagoris, Y. S. Boutalis, and N. Papamarkos, Accurate image retrieval based on compact composite descriptors and relevance feedback information, Int. J. Pattern Recog. Artif.Intell., vol. 24, no. 2, pp , Feb. 2010, World Scientific. [9] A. García-Serrano, X. Benavent, R. Granados, E. de Ves, and J. Miguel Goñi, Multimedia Retrieval by Means of Merge of Results from Textual and Content Based Retrieval Subsystems, inmultilingual Information Access Evaluation II. Multimedia Experiments: 10th Workshop of the Cross-Language Evaluation Forum, CLEF 2009, Corfu, Greece, September 30 - October 2, 2009, Revised Selected Papers. Berlin, Germany: Springer-Verlag, 2010, pp [10] A. García-Serrano, X. Benavent, R.Granados, and J. M.Goñi-Menoyo, Some results using different approaches to merge visual and textbased features in CLEF 08 photo collection, in Evaluating Systems for Multilingual and Multimodal Information Access: 9th Workshop of the Cross-Language Evaluation Forum, CLEF 2008, Aarhus, Denmark, September 17-19, 2008, Revised Selected Papers. Berlin, Germany: Springer-Verlag, 2009, pp [11] M. S. Lew, N. Sebe, C. Djeraba, and R. Jain, Content-based multimedia information retrieval: State of the art and challenges, ACMTrans. Multimedia Comp., Commun., Appl., vol. 2, no. 1,pp. 1 19, Feb [12] P. K. Atrey,M. A. Hossain, A. El Saddik, andm. S. 1948

5 Kankanballi, Multimodal Fusion for Multimedia Analysis: A Survey, Multimedia Syst., vol. 16, pp , [21] DESELAERS, T., KEYSERS, D., NEY, H. Features for image retrieval:an experimental comparison. Information Retrieval, 2008, vol. 11, no. 2, p [13] R. Granados, J. Benavent, X. Benavent, E. de Ves, and A. Garcia-Serrano, Multimodal Information Approaches for the Wikipedia Collection at ImageCLEF 2011, in Proc. CLEF 2011 Labs Workshop, Notebook Papers, Amsterdam, The Netherlands, [14] J. Kludas, E. Bruno, and S. Marchand-Maillet, Information fusion in multimedia information retrieval, in AMR Int. Workshop Retrieval User Semantics, [15] ImageCLEF: Experimental Evaluation in Visual Information Retrieval, in The Information Retrieval Series, H. Müller, P. Clough, T. Deselaers, and B. Caputo, Eds. New York, NY, USA: Springer-Verlag, 2010, vol. 32 [16] T. Tsikrika, A. Popescu, and J. Kludas, Overview of the Wikipedia image retrieval task atimageclef 2011, in Proc. CLEF 2011 LabsWorkshop, Notebook Papers, Amsterdam, The Netherlands, 2011 [17]M. Grubinger, Analysis and Evaluation of Visual Information Systems Performance, Ph.D. thesis, School Comput. Sci. Math., Faculty Health, Engi., Sci., Victoria Univ., Melbourne, Australia, [18] ARAMPATZIS, A., ROBERTSON, S., KAMPS, J. Score distributions in information retrieval. In Proceedings of the 2nd International Conference on Theory of Information Retrieval: Advances in Information Retrieval Theory.Cambridge (UK), 2009, p [19] CHATZICHRISTOFIS, S. A., BOUTALIS, Y. S., LUX,M.Img(rummager): An interactive content based image retrieval system. In Second International Workshop on Similarity Search andapplications SISAP Prague (Czech Republic), 2009, p [20] DATTA, R., JOSHI, D., LI, J., WANG, J. Image retrieval: Ideas, influences, and trends of the new age. ACM Computing Surveys, 2008, vol. 40, no. 2, p