The Future of P2P Audio/Visual Search

Transcription

1 The Future of P2P Audio/Visual Search Fausto Rabitti ISTI-CNR, Pisa, Italy P2PIR Workshop - ACM CIKM 2006

2 Outline of the talk 1. Guessing the future of technologies 2. Today outlook 1. Peer-to-Peer Applications 2. Image and Video search on the Web 3. Improving effectiveness by Similarity Search 4. Scalability issue: P2P solution Search P2PIR Workshop - ACM CIKM

3 Future of Audio/Visual Search Everything we write, see, or hear can now be in a digital form (93% of produced data is digital) In the next three years, we will create more data than has been produced in all of human history, most of it in Audio/Visual form. New trend in MM content production: personal producer VS professional producers Dimensions of the search problem: Effectiveness Efficiency (scalability is the key issue) Search P2PIR Workshop - ACM CIKM

4 Future of Audio/Visual Search Economic dimension of the problem (e.g., personal journalism, cultural tourism, etc.) Social impact of solutions (e.g. community networks) Scientific research activities Results on innovation Is P2P a solution? Search P2PIR Workshop - ACM CIKM

5 What is Peer-to-Peer? P2P is a class of applications that takes advantage of resources storage, cycles, content, human presence available at the edges of the Internet (Shirky) A P2P system is a self-organizing system of equal, autonomous entities (peers) which aims for the shared usage of distributed resources in a networked environment avoiding central services (Steinmetz) P2P is about overcoming the barriers to the formation of ad hoc communities, whether of people, of programs, of devices, or of distributed resources (O Reilly) Search P2PIR Workshop - ACM CIKM

6 Peer-to-Peer systems The traditional client-server approaches require a tremendous amount of effort and resource to meet today challenges Scalability, security, flexibility are the main requirements of future Internet-based applications P2P systems are characterized by decentralized resource usage and decentralized selforganization Search P2PIR Workshop - ACM CIKM

7 Peer-to-Peer systems: Searching One of the major problems of P2P systems is: how to find a data item stored at some dynamic set of nodes in the system Three basic strategies can be used: centralized servers (first generation P2P: Napster) flooding (second generation P2P: Gnutella) distributed indexing (DHTs: Kademilia used in emule): structured P2P system Search P2PIR Workshop - ACM CIKM

8 Structured Peer-to-Peer systems Inspired by the significant possibilities of decentralized self-organizing systems, researches focused on approaches for distributed indexing structures Distributed Hash Tables were developed to provide scalability, reliability and fault tolerance Search P2PIR Workshop - ACM CIKM

9 Searching in the World of Peers Peer-to-peer systems are mostly used for file sharing. This task, which made the fortune of P2P, was not achievable by centralized servers Structured P2P networks such as DHTs, have produced a considerable amount of research but their usage is still limited Today centralized servers are still largely used for searching between (often illegal) file sharing communities data Search P2PIR Workshop - ACM CIKM

10 emule: Servers Search P2PIR Workshop - ACM CIKM

11 emule: Kademilia Search P2PIR Workshop - ACM CIKM

12 emule: Search Search P2PIR Workshop - ACM CIKM

13 emule: Searching from a web server Search P2PIR Workshop - ACM CIKM

14 BitTorrent: web servers Search P2PIR Workshop - ACM CIKM

15 Image and Video search on the Web Today Image and Video Search Engines are trivial applications of Web Search Engines Examples: Google, Yahoo, Ask, etc. Search is performed on the MM Object context (i.e. Web page) or on manually associated text Limits of this approach: who is going to manually tag all A/V material produced by personal devices? Search P2PIR Workshop - ACM CIKM

16 Searching for sea : flickr Search P2PIR Workshop - ACM CIKM

17 Improving effectiveness by Similarity Search on MM Content Exploiting automatically extracted metadata representing MM content: MM features (e.g. MPEG-7) Automatic Context information (e.g., GPS generated info) Solutions based on combination of traditional information (Manual text, Web pages) with automatically generated information (i.e. MM features, context) representing MM Content New searching paradigm based on Similarity Search P2PIR Workshop - ACM CIKM

18 Searching for sea : MILOS PhotoBook Query Search P2PIR Workshop - ACM CIKM

19 Detecting Coat of arms by components Search P2PIR Workshop - ACM CIKM

20 Search P2PIR Workshop - ACM CIKM

21 Face Recognition (TV) Query VideoFrame 2 result Search P2PIR Workshop - ACM CIKM

22 The Importance of Similarity Quotation: An ability to assess similarity lies close to the core of cognition. The sense of sameness is the very keel and backbone of our thinking. An understanding of problem solving, categorization, memory retrieval, inductive reasoning, and other cognitive processes require that we understand how humans assess similarity. MIT Encyclopedia of the Cognitive Sciences, Cambridge, MA, MIT Press 2006, pp Search P2PIR Workshop - ACM CIKM

23 Feature-based Approach similar? image database Search P2PIR Workshop - ACM CIKM

24 Feature-based Approach image layer B feature layer R G Search P2PIR Workshop - ACM CIKM

25 Specific Similarity concepts and definitions Similarity in Chemistry In order to assess the similarity between two molecules A and B we need to: first describe the molecules according to some scheme and, choose an appropriate measure to compare the descriptions of the molecules. Similarity in Social Psychology similarity refers to how closely attitudes, values, interests and personality match between people. similarity leads to interpersonal attraction, i.e. the attraction between people which leads to friendship and relationships. similarity forms social networks of individuals with ties mirrored as friends and acquaintances. Search P2PIR Workshop - ACM CIKM

26 Requirements of New Applications Medicine: Magnetic Resonance Images (MRI) Finance: stocks with similar time behavior Digital library: text retrieval multimedia information retrieval Search P2PIR Workshop - ACM CIKM

27 Similarity Searching Effectiveness the way of formulating the similarity measures - a model of human perception Efficiency the way of achieving the required performance over huge volumes of data index structure Search P2PIR Workshop - ACM CIKM

28 Metric Space an Abstraction of Similarity Metric space: M = (D,d) D domain distance function d(x,y) x,y,z D d(x,y) > 0 -non-negativity d(x,y) = 0 x = y - identity d(x,y) = d(y,x) - symmetry d(x,y) d(x,z) + d(z,y) - triangle inequality Search P2PIR Workshop - ACM CIKM

29 Similarity Search Problem For X D in metric space M, pre-process X so that the similarity queries are executed efficiently. similarity queries range search R(q,r) = { x X d(q,x) r } q D, r 0 q r Search P2PIR Workshop - ACM CIKM

30 Similarity Queries k-nearest neighbours k=5 NN(q,k) = A, q D, k > 0 A X, A = k x A, y X A, d(q,x) < d(q,y) q similarity join X = {x 1, x 2, x N }, Y = {y 1, y 2, y M } μ {(x i,y j ) d(x i,y j ) < μ} similarity self join X = Y Search P2PIR Workshop - ACM CIKM

31 Basic Partitioning Principles ball partitioning { x X d(p,x) r } { x X d(p,x) r } p r generalised hyperplane { x X d(p 1,x) d(p 2,x) } { x X d(p 1,x) >d(p 2,x) } p 2 p 1 Search P2PIR Workshop - ACM CIKM

32 The M-tree (an example) inherently dynamic structure disk-oriented (fixed-size nodes) built in a bottom-up fashion inspired by R-trees and B-trees all data in the leaf nodes internal nodes: pointers to subtrees and additional information Search P2PIR Workshop - ACM CIKM

33 M-tree: Example o 10 o 5 o 3 o 2 o 7 o 11 o 1 o 6 o 4 o 9 o 8 o _ o _ _ _ o _ o _ _ _ o _ o _ o _ o 0.0 o _ _ o 0. 0 o _ _ o 0.0 o _ _ o 0.0 o o 11 o 0.0 o _ _ Search P2PIR Workshop - ACM CIKM

34 Scalability: CPU Costs range query: r = 1,000; 2,000 k-nn query: k = 1; 100 labels: radius or k + D (D-index), M (M-tree), SEQ data: from 100,000 to 600,000 objects M-tree and D-index are faster (D-index slightly better) linear trends Search P2PIR Workshop - ACM CIKM

35 Scalability: I/O Costs the same trends as for CPU costs Search P2PIR Workshop - ACM CIKM

36 Similarity Search Scalability Similarity search is expensive. The scalability of centralized indexes is linear. cannot be applied to huge data archives become inefficient after a certain point Possible solutions: Sacrifice some precision: approximate techniques Use more storage & computational power: distributed data structures Search P2PIR Workshop - ACM CIKM

37 Similarity Search in the World of Peers With P2P systems able to perform similarity search: similarity search becomes scalable P2P communities have new search capabilities While preserving all structured P2P benefits, they will give new search capabilities not available from current centralized servers Search P2PIR Workshop - ACM CIKM

38 Implementation Postulates of Distributed Indexes scalability nodes (computers) can be added (removed) no hot-spots no centralized nodes, no flooding by messages update independence network update at one site does not require an immediate change propagation to all the other sites Search P2PIR Workshop - ACM CIKM

39 Distributed Similarity Search Structures Native metric structures: GHT* (Generalized Hyperplane Tree) VPT* (Vantage Point Tree) Transformation approaches (based on DHTs): M-CAN (Metric Content Addressable Network) M-Chord (Metric Chord) Search P2PIR Workshop - ACM CIKM

40 M-CAN: Range Query Execution Range query R(q,r) map the q on F(q) route the query towards F(q) Reach regions with candidate objects L (F(x),F(q)) r Propagate the query over the candidate regions using a multicast algorithm of CAN Check objects using d Search P2PIR Workshop - ACM CIKM

41 Scalability comparison (INFOSCALE 2006) Compared 4 distributed similarity search structures Query size scalability Dataset size scalability Capability of simultaneous query processing single query multiple queries GHT* excellent poor VPT* good satisfactory MCAN satisfactory good M-Chord satisfactory very good Search P2PIR Workshop - ACM CIKM

42 Further Research Challenges: Complex Similarity Search Problems: different types of queries, involving different features and different similarity measures multiple overlays over the same physical network, distributed incremental similarity search, high communication costs of naïve implementations, collaboration with the load balancing mechanism. Search P2PIR Workshop - ACM CIKM

43 Further Research Challenges: Load balancing Problems: one node contains data of different features, load balancing cost models to measure the load and estimate the reorganization costs, postulates of distributed processing must strictly be respected performance tuning Search P2PIR Workshop - ACM CIKM

44 P2P Solutions for A/V Search P2P-based solution to solve the fundamental Scalability Issue, concerning not only: Distributed Similarity Search structures but also: Cooperative A/V features extraction Support of highly dynamic applications (e.g. videoblogs, photoblogs, etc.) Push-based/cooperative crawling Search P2PIR Workshop - ACM CIKM

45 Technological requirements for MM Search Engines Media specific analysis and feature extraction: e.g. Music Information Retrieval Scalable, dynamic and distributed index structures supporting similarity search Complex/multi-feature query processing: combining evidence from different media indexes, using the similarity paradigm (together with the traditional Web search) Support of distributed push-based crawling, where containers are asked to publish and push information to the search engine (together with the traditional pull-based crawling) Scalable dynamic caching techniques to enhance performance Context based support (based on user location, activity, etc.) and Multi device support (search from PC, mobile phone, PDAs). Search P2PIR Workshop - ACM CIKM

46 P2P and push-based crawling Conventional pull-based crawling techniques face the high refresh rates and huge size of the Web with increasing difficulty and have limitations in dealing with multimedia information In a distributed push-based crawling model, content providers (both professional and personal) are asked to publish and push information to the P2P indexing nodes Collaborative crawling model can effectively deal with important multimedia content that is hidden to traditional crawlers because it is not directly hyperlinked from some HTML page or it is stored in on-line AV specialized repositories that cannot be visited by crawling agents.. Search P2PIR Workshop - ACM CIKM

47 P2P and push-based crawling (2) Multimedia content providers can be helped by the P2P infrastructure in the heavy process of multimedia feature extraction A collaborative and participatory P2P environment can give the providers the possibility of maintaining the control on the contributed material (publish what you want when you want): IPR-protected material is indexed and searched for, but its delivery controlled directly by the owner. New collaborative business models? Search P2PIR Workshop - ACM CIKM

48 Dynamic combination of crawling and feature extraction modes Search P2PIR Workshop - ACM CIKM

49 P2P and Dynamic Caching Caching and replication are routinely used in the Web since they allow bandwidth consumption to be reduced, and user-perceived quality of service to be improved. In decentralized P2P systems, caching and replication permit to achieve a better load distribution, shorter latency, and higher availability. These techniques can be applied to contents, query results, and index entries. Literature proposes several solutions in which caching and replication strategies are managed locally, at the peer or super peer levels, or globally, by deploying a distributed cache over several peers. Search P2PIR Workshop - ACM CIKM

50 P2P and Dynamic Caching (2) New dimension of the problem: due to the peculiarities of multimedia content (size, dynamicity, DRM constraints) It is necessary to enhance search responsiveness, and save computational and communication resources (e.g. by exploiting self-similarities among submitted queries which follow a zipfian distribution) The main problem to deal with is related to dynamicity. In fact, it is not clear how long cached information will remain valid. The variability of data and the dynamicity of the network itself make hard to predict freshness of information for cache entries. To design an on-line caching algorithm, it is necessary to investigate the trade-off between caching-time and validitytime, and explore whether and in which cases there is a correlation between the popularity and the time validity of a cached entry. Search P2PIR Workshop - ACM CIKM

51 Search P2PIR Workshop - ACM CIKM

52 Music Information Retrieval (MIR) Search Mainly based on melody, modeling possible mismatch between the query and the documents Retrieve a song given an excerpt, sung or recorded Retrieve songs being similar to some query excerpts/songs Classification Mainly based on timbre, timing and long-term features Identify author, performers, artist, genre, style, orchestration Recommendation Based on collaborative filtering mixed with content-based Suggest a number of items to purchase or to organize in a playlist; organize programs for Web radios Visualization Represent large personal music collections, for music browsing, audio preview Search P2PIR Workshop - ACM CIKM

53 MIR: Basic Techniques MIR tasks require audio processing for: Tempo identification Transcription of the main melody Recognition of harmonic structures Timbre characterization Similarity is computed using String matching - i.e. Dynamic Time Warping Statistical modeling - i.e. Hidden Markov Models Geometric approaches - i.e. Earth Movers Distance Techniques to visualize, classify, recommend k-nearest Neighbor, Gaussian Mixture Models, Self-Organizing Maps, Markov Models Search P2PIR Workshop - ACM CIKM

54 MIR: Evaluation Problems with copyright issues Researchers have difficulties to obtain large music collections Music Information Retrieval Evaluation exchange (MIREX) Common effort for a TREC-like evaluation framework Participants propose tasks and provide test collections Experiments are carried out by the organizers Main focus on preprocessing techniques Effectiveness of feature extraction, no real need of relevance judgments Initial efforts also for typical retrieval tasks Search P2PIR Workshop - ACM CIKM

55 MIR: Digital Rights Management Audio fingerprinting To recognize copyrighted material Can be exploited for retrieval tasks too Audio watermarking To include copyright ownership and to track users sharing behaviors With watermarks retrieval can be based on metadata Song similarity To identify plagiarism Artist identification Search P2PIR Workshop - ACM CIKM

56 MIR: P2P and Portable Devices The increasing number of large personal collections of digital music, allows for Music retrieval with distributed music indexes, stored in different peers Music recommendation using collaborative filtering based on the analysis of personal collection content Computation of music similarity based on users listening behaviors The audio channel is more suitable for the interaction with portable devices Music retrieval, classification and recommendation through aural interaction Techniques for extracting music snapshots and snippets Search P2PIR Workshop - ACM CIKM

57 Conclusions Starting from today situation Peer-to-Peer Applications Image and Video search on the Web In order to improve effectiveness by adopting the Similarity Search paradigm We need a highly scalable and dynamic solution P2P solution is feasible and promising, also with respect to: Cooperative A/V features extraction Push-based/cooperative crawling Search P2PIR Workshop - ACM CIKM