Model-driven Collaboration and Information Integration for Enhancing Video Semantic Concept Detection

Transcription

1 Mode-driven Coaboration and Information Integration for Enhancing Video Semantic Concept Detection Tao Meng, Mei-Ling Shyu Department of Eectrica and Computer Engineering University of Miami Cora Gabes, FL 33124, USA Abstract With the fast deveopment and popuarity of digita cameras, smart phones, and video surveiance devices, the amount of video data increases dramaticay. Accordingy, automaticay mining and annotating high-eve concepts for video indexing and management become imperative research tass in both mutimedia research and data mining research. The mainstream content-based semantic concept mining approaches suffer from the notorious semantic gap probem, which is the difficuty of associating the ow-eve features to high-eve concepts directy. Recenty, the utiization of concept-concept association has been proven to be effective to address the semantic gap probem. In this paper, a framewor based on muti-mode coaboration and information integration is proposed to integrate the association among concepts to enhance the high-eve concept detection by reusing the information outputs from the primary concept detectors. The experimenta resuts show that the proposed framewor outperforms the other approaches in the comparison and therefore is promising. Keywords: Muti-Mode Coaboration, Video Concept Detection, Re-raning, Information Integration. 1. Introduction In the recent years, we have witnessed the rapid growth of mutimedia data such as videos and images. In order to manage and search video databases efficienty, automatic agorithms for annotating and detecting the high-eve concepts are required. Given the imitations of the traditiona tag-based anaysis for video data, the content-based video anaysis has become increasingy popuar nowadays [5][6][15][22]. However, the semantic gap probem (i.e., the gap between the ow-eve features and high-eve concepts) poses maor chaenges to the video concept mining research community. To address this issue, a ot of research efforts have been put on extracting sophisticated features such as SIFT and HOG, increasing the positive data instances to negative data instances ratio, and improving the cassifier agorithm for concept detection [4][8][16][17]. A these efforts have pushed forward the frontiers of nowedge and contributed to the improvement of semantic concept detection. The concept detection probem coud be viewed as a muti-abe cassification probem in the fied of data mining. Under this scenario, one video shot coud contain mutipe concepts. The traditiona way of soving that probem is the so-caed binary reevance approach [7], which uses the one-vs-a approach to buid one cassifier for each concept and treats a training video shots containing that concept as the positive data instances and a other video shots as the negative exampes. One of the maor disadvantages of this approach is that the information of correations among different concepts, which is hepfu for concept detection, is missing. The concepts are not isoated and usuay co-occur in the video shots, ie the concepts of road and vehice, airpane and airpane fying, etc. Therefore, utiizing the natura correations among different concepts for enhancing the video concept detection has received ots of attention. Some previous studies have been done in this area. In the computer vision domain, some eary approaches expoited the semantic context information to improve the accuracy of obect detection. In [21], the conditiona random fied (CRF) framewor was proposed to maximize the obects abe contextua agreement in an image in order to improve the obect categorization accuracy. Different framewors as the extensions based on CRF were presented in [11] and [25]. In mutimedia research domain, the correations among different concepts were modeed by the semantic mode vectors, the probabiistic graphic modes, the concept ontoogy-based modes, and the concept correation-based modes. In [24], the mode vector approach was proposed

2 to tae the output scores from different cassification modes buit upon different concepts and map those scores to a feature vector. Such feature vectors for a the data instances are used to train a cassification mode, such as the support vector machine (SVM) or K-nearest neighbor (KNN) cassifier. In this way, the correations among different concepts are modeed in the mode vector. In a recent wor [19], a simiar idea was adopted to construct a arge poo of semantic modes for video event detection. The main concern of this approach is the error propagation issue because the output scores might be noisy and the cassifiers buit on such scores are not fuy reiabe. The second approach of modeing the correation is the probabiistic graphic mode. The genera assumption of this type of modes is that the detection of a certain concept coud affect the probabiity of the detection of other reated concepts. Such modes represent each concept as a vertex in the graph and the correation between two concepts are modeed as the edge connecting the two vertices. In [20], a Bayesian networ which is a directed acycic graph (DAG) was buit to mode the semantic contexts. In [2], the detection scores output from each SVM mode were fused using the weights computed based on the conditiona probabiity. These approaches usuay depend on the strong assumption of the independence or the conditiona independence between concepts, which does not necessariy hod in terms of video data sets. Recenty, the ontoogy-based approaches are deveoped to fuse the scores from the concept detectors. Benmohtar [3] combined the neura networ with ontoogy to hep the concept detection in mutimedia data sets. Eeuch [9] integrated the fuzzy ogic with the concept ontoogy and deveoped the deduct engine to infer correations. The correation-based modes are aso proposed. In [13], a domain adaptive semantic diffusion (DASD) framewor was proposed to mode the correations among concepts using the Pearson product and to address the domain change probem using the graph-based semantic diffusion. One probem with the two aforementioned approaches is the strong dependence on the prior nowedge. Even though there are some previous studies in this area, two maor chaenging probems sti ca for better soutions. The first probem is how to seect the significant correations which coud hep the concept detection from a the possibe correations. The second probem is how to mae the best use of both the scores and the abes of the training data to mode the correations to improve the concept detection accuracy in the testing data set. In this paper, we propose soutions for these two probems, respectivey. In order to mine the significant ins, the association rue mining (ARM) technique [26] which is abe to capture the non-trivia associations in the training data sets is utiized. Association rue mining is a data mining technique to find the significant association patterns from a transaction data set. The frequent patterns usuay indicate the significant interna connections among different items. Considering a cassic maret baset anaysis using association rue mining as an exampe, the rue mi bread can capture the purchasing behavior or pattern of the consumers. Such associations or patterns can further be utiized to increase the profits of the companies. Therefore, ARM can provide a promising soution for seecting the significant associations. Coaborative fitering (CF) [10] is an agorithm used in the recommendation systems. One of its appications in CF is to predict the user s ratings for an item, such as a movie, according to (i) the user s previous ratings for the other items and (ii) the ratings for the same item by the other users. The fundamenta assumption of CF is that if two users of the system have simiar behaviors, then most iey they wi act simiary on the other items. Therefore, a the users coaborate to enhance the prediction or recommendation capabiity of the system. This ind of coaborations inspires us to study whether it coud be used as a fusion strategy for integrating information from mutipe reated modes to hep the target concept detection. In addition, it woud be even better if both the training scores and training abes coud be fuy utiized in the framewor. Therefore, we propose a muti-mode coaboration mode to capture the context information contained in the abes and a ogistic regression-based mode to capture the context information contained in the training scores. Afterwards, the resuts from these two modes are combined to provide the fina output scores for each concept. It is important to mae it cear that the proposed framewor is designed to wor under the scenario that both the training and testing data instances are ready, but the concept abes of the testing data instances are unnown. Such a scenario is common in off-ine video annotation appications. Our future wor wi extend the current framewor to address the rea-time video concept detection probem. The paper is organized as foows. In Section 2, the proposed framewor is introduced and a the different components are eaborated. Section 3 presents the experimenta resuts and the insights observed from the resuts are discussed. In Section 4, we give a brief summary and identify the future research directions. 2. The proposed framewor The proposed framewor consists of the training part (shown in Figure 1) and the testing part (shown in Figure 2). In the training part, the concept detection framewor is appied to the video shots and the detection scores for each concept are computed. It is important to point out that the proposed framewor is fexibe and coud be appied to the scores output from any concept detection framewor. The scores are further converted to the posterior probabiities

3 Figure 1. The proposed framewor - training part which are caed posterior probabiistic scores to get a better measurement of the possibiities that the data instances are positive and to convert a the scores into the same range, which is from 0 to 1. The abes are organized into a abe matrix and the ARM technique is appied to find the most significant associations from the abe matrix. Next, a ogistic regressor is trained for each target concept based on the posterior probabiistic scores of both the target concepts and the reated concepts. These modes are then ept for the testing part. At the same time, a coaboration mode is trained and ept for the testing part. In the testing part, after the detection scores are generated from the concept detection framewor, the scores are converted to the posterior probabiistic scores as in the training part. The posterior probabiistic scores for the testing data instances are then input to the ogistic regressor and the coaboration mode. Finay, the scores output from the two modes are fused together to give the fina resuts. The detais of each component are introduced in the foowing subsections. In order to better iustrate the proposed ideas, the foowing definitions are given. Definition 1 (Data Instance and Labe) Since the proposed concept detection is at the shot eve, a data instance refers to a video shot or the features of a video shot, depending on the context. The abe of a video shot for a concept is either 1 or 0, indicating whether the corresponding concept appears in the video shot or not, respectivey. If the abe is 1, the data instance is considered as a positive data instance for that concept; whie if the abe is 0, it is considered as a negative data instance. Definition 2 (Concept-Cass Pair) A concept-cass pair represents a abe for the corresponding concept. In this paper, we denote the concept-cass pair as C ε, where indicates the concept index and ε is the abe. For exampe, C5 1 indicates a positive data instance for concept 5, and C5 0 indicates a negative data instance for concept 5. When ε=1, the concept-cass pair is a positive conceptcass pair; whie when ε=0, the concept-cass pair is a negative concept-cass pair. Figure 2. The proposed framewor - testing part Definition 3 (τ-itemset) A τ -itemset is a set which consists of τ concept-cass pairs. For exampe, {C 1 1, C1 100 } is a 2-itemset. Pease note that a positive concept-cass pair and a negative concept-cass pair for the same concept coud not appear in the same itemset, because one data instance is either be a positive data instance or a negative data instance, but not both. Definition 4 (Support) The support vaue indicates the number of occurrences of the τ-itemset in the training data set. It is denoted as sup(τ-itemset).

4 Definition 5 (Target Concept & Reated Concept) A target concept (TC) is defined as the concept to detect. A reated concept (RC) is a concept that is reated to the TC. There coud be more than one RCs for one TC. Definition 6 (Target Score & Reated Score) A target score is a posterior probabiistic score of a data instance for a TC. A reated score is a posterior probabiistic score of a data instance for a RC with respect to the TC. Definition 7 (Positive Rue & Negative Rue) A positive rue indicates that the occurrence of one concept infers the occurrence of the other concepts. A negative rue indicates that the occurrence of one concept infers nonoccurrence of the other concepts. For exampe, C 1 1 C1 2 is a positive rue. C 1 1 C0 2 is a negative rue. In this paper, ony the rues containing two concept-cass pairs are considered. 2.1 Posterior probabiity cacuation and abe reorganizing Because the proposed framewor is designed to be fexibe, the output scores coud be different depending on the concept detection framewor. In addition, different cassification modes can be buit for different concepts, so even the same concept detection framewor can output the scores with distinct characteristics for different concepts. Given these possibiities, a Bayes theorem-based score conversion agorithm is used here. This agorithm is introduced in detais in our previous wor [18]. Here, a genera equation is given for competeness. Assuming for a data instance i, the output score for concept represented by C is S (i). C = 1 indicates the event that a data instance is positive for concept. C = 0 indicates the event that a data instance is negative for concept. The score after conversion is S (i), which is computed using Equation (1). S (i) = p(s = S (i) C = 1)p(C = 1) 1 d=0 p(s = S(i) C = d)p(c = d). (1) p(c = 1) and p(c = 0) indicate the prior probabiities that the data instance is positive or negative for concept, respectivey. p(s = S (i) C = 1) and p(s = S (i) C = 0) are the vaues of the two conditiona probabiity density functions p(s C = 1) and p(s C = 0) evauated at S (i). The conditiona probabiity density functions coud be estimated from the training data. The output S (i) is defined as the posterior probabiistic score in this study. As is shown in Equation (1), the prior nowedge about the ieihood that the concept appears in the data instance is integrated into the posterior probabiistic score (between 0 and 1). Tabe 1. Labe matrix Instance C 1 C 2... C... C N Instance 1 C1 0 C C 1... CN 0 Instance 2 C1 0 C C 0... CN Instance i C1 1 C C 0... CN Instance m C1 0 C C 1... CN 0 In order to identify the significant associations from the training data set, the abes of a training data instances for a the concepts need to be reorganized into a matrix. Tabe 1 shows an exampe abe matrix after organizing a the abes together. In this matrix, the rows correspond to a the training data instances (1 to m) and the coumns are for a the concepts (1 to N). Each eement is a concept-cass pair indicating whether the data instance is a positive data instance for that concept or not. For exampe, Instance i is positive for C 1, negative for C 2, negative for C, etc. 2.2 Association rue generation In order to discover the significant associations from the training data, the Apriori agorithm [1] is appied to the abe matrix to discover the association rues. The specific agorithm to generate a the 2-item rues is given beow. ASSOCIATION RULE GENERATION 1 List a 1-itemsets which contain positive concept-cass pairs. 2 Combine a 1-itemsets from Step 1 to form candidate 2-itemsets. 3 Seect 2-itemsets from candidate 2-itemsets which have a support vaue greater than zero. 4 For one target concept C t, seect a 2-itemsets which contain C t. 5 Generate the candidate positive rues for C t. 6 Seect the significant rues to identify the significanty reated concepts. Here, Step 1 to Step 4 generate a candidate 2-itemsets which contain the target concept C t. Next, a candidate positive rues are generated for C t with C t in the concusion part. Then a the significanty reated concepts are seected. Two criteria, which are the support ratio R s and the interest ratio R i, are used to prune the rues. Formay, assume for the target concept C t, one candidate positive rue is Cr 1 Ct 1, the support ratio and the interest ratio are defined in Equation (2) and Equation (3), respectivey. R i = R s = sup({c1 t,cr 1 }) sup({ct 1. (2) }) sup({c 1 t,c 1 r }) sup({c 1 t }) sup({c 1 r }). (3)

5 The intuitions of these two criteria are from the TC point of view and the RC point of view. The rationae and ustification of these two vaues for rue seection were presented in our previous wor [18] and therefore is omitted here. Two threshod vaues α% and β% for R s and R i are determined using the cross vaidation set. Since this study focuses on the binary reationship between the RC and TC, ony the 2-item rues are generated. The higher-order reationships coud aso be mined and integrated into the framewor in the future. 2.3 Logistic regression and mode coaboration After the rues are seected for each concept, how to deveop a good re-raning strategy to integrate the posterior probabiistic scores from TC and RCs to improve the detection rate of TC becomes an interesting and important tas. This probem coud be formay formuated as foows. Assume for one target concept and one data instance i, S (i) is the posterior probabiistic score computed using Equation (1), A is the set of IDs for the positive data instances for concept and m indicates the tota number of training instances, {S (e) } (e i) denotes the set of posterior probabiistic scores of the (m 1) data instances in the training data set except data instance i for concept. If a the scores are sorted in the descending order, the raning number for data instance i is given by b(s (i),{s (e) }). Without oss of generaity, if the scores for a RCs of data instance i for concept form a set represented as {S (i) } ( ), then Q (i), (i.e., the score after re-raning for data instance i) coud be expressed using a function f in Equation (4). Q (i), = f (S (i),{s (i) }). (4) If a the scores after re-raning are sorted in the descending order too, the raning number of instance i after re-raning is given by b(q (i),,{q(e), }). Therefore, the re-raning process coud be viewed as soving the optimization probem represented by Equation (4) and Equation (5). f = argmin f b(q (i),,{q(e), i A }) b(s (i),{s (e) }). (5) The minimization is with respect to f. It woud be good if the function b is in the cosed form, which is a specia case. In genera, we want to approximate this function to deveop a re-raning strategy. In this paper, we propose the ogistic regression-based mode and the coaboration mode, both of which aim at approximating this function. In terms of the ogistic regression mode, we want to maximize the ogarithm ieihood function to approximate minimizing the miscassification error, which is another way of viewing Equation (5). For one data instance i and {S (i) the target concept C, et n = {S (i) } be the cardinaity of }, be the concept ID for one RC, and [S (i) ] be the vector of which each eement is a member of {S (i),[s (i) }. The concatenated vector x (i) = [1,S (i) ]] T is the coumn vector of dimension (n+2) by 1, and the corresponding parameter is θ = [θ 0,θ 1,θ 2,...,θ n+1 ] T. The integrated score is given by Equation (6). The cost function which is the ogarithm ieihood times -1 is given by Equation (7). g θ (x (i) ) = h θ (x (i) ) = θ T x (i) exp( h θ (x (i) )) J(θ) = 1 m [ m i=1y (i) og(g θ (x (i) )) (6) + (1 y (i) )og(1 g θ (x (i) ))] (7) θ q θ q δ J(θ). θ q (8) Here, δ is the earning rate, m is the tota number of training data instances, and y (i) is either 1 or 0 indicating the data instance is positive or negative. The estimated vaue for θ is ˆθ which is cacuated using the gradient decent agorithm, and the updating rue is given in Equation (8), where θ q is the q- th eement for θ. The new score of the testing data instance is computed by pugging the posterior probabiistic scores and the ˆθ into Equation (6). The ogistic regressor captures the reationship among different training scores. It woud be even better if the abes coud be taen into consideration as we. Inspired by the coaborate fitering agorithm, a coaboration mode is proposed in this paper. To buid the coaboration mode, a coaboration matrix is first formed. To show the coaboration matrix ceary, a specific numerica exampe is given here. Assume for the target concept C t, without oss of generaity, there are two seected reated concepts, C r1 and C r2, the tota number of training data instances is m and the tota number of testing data instances is z, L t, L r1 and L r2 indicate the coumns of abes for C t, C r1 and C r2 ; S t, S r1 and S r2 indicate the coumn of posterior probabiistic scores for C t, C r1 and C r2, and the coaboration matrix for concept C t is shown in Figure 3. In the matrix, the first m rows are the m training data instances and the rest z rows are the z testing data instances. The testing abes for a the testing data instances are unnown so they are represented as the question mars. The coaboration matrices coud aso be generated in the simiar way for a the other concepts. The coaboration matrix coud be viewed from a different prospective. If we assume the abe coumn is generated by a mode, which is named as the abe mode and aways assigns correct abes for the training data instances. A the entries in one coumn in the coaboration matrix coud be viewed as the ratings given by the mode corresponding to that coumn. The higher the entry is, the

6 indicate the -th eement in vector w (u), 1 o. The cost function to be minimized is defined in Equation (9). To minimize this function, the gradient decent agorithm is utiized, and the updating rue for each parameter is given in Equation (10) and Equation (11). Here, ψ is the earning rate determined by the empirica study. In the testing stage, the prediction coud be made for a testing data instance by mutipying the corresponding earned feature with the parameters corresponding to the target abe mode. For a testing data instance, after the prediction scores from the ogistic regressor and the coaboration mode are computed, the two scores are ineary combined to generate the fina output score. Figure 3. The coaboration matrix higher the mode rates the data instance, which indicates the higher probabiity that the data instance is positive for that concept. In this way, the probem is converted to give the best predictions for the ratings of testing data instances for the target abe mode, which is L t in Figure 3. The vaues which need to be predicted are aso mared using the red rectange in Figure 3. This probem coud be soved using an agorithm simiar to the regression-based coaborative fitering agorithm. The genera idea of the agorithm is to earn a set of new features for a data instances and the parameters for a the modes. The fina predictions are based on these features and parameters. Formay, et G be the coaboration matrix, G(u, v) is the eement corresponding the row u and coumn v in G, where 1 u U, 1 v V, U and V are the tota number of data instances (incuding both the training data instance and testing data instances), and the tota number of modes in the coaboration matrix. r(u,v) = 1 indicates that G(u,v) is nown and r(u,v) = 0 indicates the G(u,v) is unnown. φ (v) is the parameter vector for the mode v and w (u) is the feature vector for data instance u. Both of them are coumn vectors. The dimensions of φ (v) and w (u) are both o, which is determined in the cross vaidation process. G(u, v) coud be factorized as G(u,v) = (φ (v) ) T w (u). Assume λ is the reguarization parameter which is used for adusting the mode compexity to address the overfitting issue. Let φ (v) indicate the -th eement in vector φ (v) and w (u) J (w (1),...,w (U),φ (1),...,φ (V ) ) = 1 2 ((φ (v) ) T w (u) G(u,v)) 2 (u,v) + λ 2 U o (w (u) u=1 =1 ) 2 + λ 2 V o (φ (v) v=1 =1 under the condition: r(u, v) = 1. w (u) J w (u) w (u) ψ J w (u) = v ((φ (v) ) T w (u) G(u,v))φ (v) under the condition: r(u, v) = 1. φ (v) J φ (v) φ (v) = u ψ J φ (v) ((φ (v) ) T w (u) G(u,v))w (u) under the condition: r(u, v) = Experiments and resuts ) 2 (9) + λw (u) + λφ (v) (10) (11) In this paper, the TRECVID 2010 data set [23] for the semantic indexing tas is used. As indicated in Section 2, the proposed re-raning strategy coud be appied to the scores output from any concept detection framewor. Therefore, in this paper, the detection scores generated from CU-VIREO374 [14] framewor downoaded from [12] are used as the inputs. Since ony the TRECVID 2010 testing scores are avaiabe, a the experiments in this paper are done based on the TRECVID2010 testing scores and the abes organized in our group are based on the abes provided from the TRECVID 2011 semantic indexing tas (i.e., the TRECVID 2010 testing data are reused as the deveoping data in TRECVID 2011). After the data preprocessing and ceaning, 144,774 data instances are used to buid and evauate our proposed framewor. The three-fod cross vaidation strategy is adopted to evauate the proposed framewor. In each fod of the cross vaidation, the training data set is further divided into (i) a deveoping data set containing 70% of the data instances in the training data set and (ii) a cross-vaidation data set containing 30% of the data instances in the training data set. The mode is trained using the deveoping data set and the parameters such as λ, α, and β are determined by evauating the framewor based on the cross-vaidation data set. After a the parameters are determined, the whoe training data set is used to train a mode with the seected parameters and the testing data set is used to evauate the fina performance of the framewor for the corresponding fod of cross

7 vaidation. The average performance for a the three fods of cross vaidation is used as the fina criterion to evauate the mode. The Mean Average Precision (MAP) which is a widey used evauation metric in the content-based mutimedia retrieva research society is used in this paper. In order to further evauate the performance of the proposed framewor, the framewor in [2] was impemented as a comparison approach. In our impementation, the parameters w i are earned using the east squares method as described in [2]. Compared with our proposed framewor, one maor difference is that their framewor does not seect the associations among concepts. The contribution of a reated concept to the target concept is modeed using the conditiona probabiity, which is inferred from the training abes. Tabe 3 shows the average of the MAP vaues for the three-fod cross vaidation of a the 130 concepts under different conditions which specify how many data instances are retrieved. For exampe, Top10 indicates the MAP vaue when the top 10 data instances are retrieved and evauated. The Overa indicates the MAP vaue when a the data instances are retrieved. The rows in the tabes indicate the MAP vaues of different framewors. Baseine indicates using the raw scores without appying any re-raning framewor; Aytar denotes the framewor in [2]; Previous is the framewor we proposed previousy in [18]; Proposed is the proposed framewor in this paper. The Impr.R1, Impr.R2 and Impr.R3 rows correspond to the reative improvements of our proposed framewor with respect to the performance of the Baseine, Aytar, and Previous framewors. For exampe, for the MAP vaues in the first coumn Top10, the MAP vaues of Baseine, Aytar, and Previous are , and The MAP vaue of the proposed framewor reaches The reative improvements of the proposed framewor with respect to the other three framewors are 7.94%, 24.19%, and 3.95%. From this tabe, it coud aso be observed that the performance of Aytar s framewor is worse than the baseine, which indicates that the seection of associations is important. Pease note that in their wor, the number of concepts is 39 which is far ess than the 130 concepts in this paper. Therefore, the increase of the number of concepts and associations actuay increases the noise for the re-raning procedure. This aso matches our observation that the seected binary associations are around 3%-4% of a the possibe associations. In other words, the significant associations under the current experiment setting are sparse. On the other hand, the proposed framewor outperforms the baseine, which indicates that the associations among concepts coud serve as an important information source to improve the high eve concept detection. The proposed framewor aso outperforms our previous mode which is based on the concept association networ. The main difference between the current mode and the previous one is the way of modeing the associations between concepts. In our previous wor, the affinities which are computed purey based on the abe matrix are used as the weights to combine scores. In this study, both the abes and the scores are considered in modeing the associations through the muti-mode coaboration. It shows that the second strategy gives a better performance. It is aso weacnowedged that the video data sets are diversified and the data distribution of the training data instances might be different from that of the testing data instances. Therefore, the information based on the training abes coud be biased to a certain degree. This study provides an insight that the integration of the information from the abes and the detection scores coud hep improve semantic concept detection. 4. Concusion and future wor In this paper, a nove mode coaboration strategy to integrate the information from both the detection scores and the abes to improve concept detection in video shots. The scores from any concept detection mode coud be used as the inputs to our proposed re-raning framewor. By using the ogistic regression mode and the coaboration mode, the associations between concepts are captured and the information from the abes and scores are integrated. The experimenta resuts show that our proposed framewor gives promising resuts, which indicates the integration of the information heps video concept detection. To our best nowedge, this is the first wor which expoits the idea of coaborative fitering to sove the re-raning probem in video semantic concept detection. In the future, better strategies to discover the significant associations from the training data set wi be studied. In addition, how to integrate the higher-order associations into the current framewor wi aso be investigated thoroughy. References [1] R. Agrawa and R. Sriant. Fast agorithms for mining association rues in arge databases. In Internationa Conference on Very Large Data Bases, pages , Santiago de Chie, Chie, September [2] Y. Aytar, O. B. Orhan, and M. Shah. Improving semantic concept detection and retrieva using contextua estimates. In IEEE Internationa Conference on Mutimedia and Expo, pages , Beiing, China, Juy [3] R. Benmohtar and B. Huet. An ontoogy-based evidentia framewor for video indexing using high-eve mutimoda fusion. Mutimedia Toos and Appications, pages 1 27, December [4] C. Chen and M.-L. Shyu. Custering-based binary-cass cassification for imbaanced data sets. In The 12th IEEE

8 Tabe 2. The MAP vaues of 130 concepts for different numbers of retrieved data instances Retrieved Instances Top10 Top20 Top40 Top60 Top80 Top100 Top500 Top1000 Overa Baseine Aytar Previous Proposed Impr.R1 7.94% 7.62% 6.85% 8.23% 8.59% 9.06% 8.64% 8.73% 6.39% Impr.R % 22.63% 19.94% 17.94% 13.86% 15.51% 23.02% 25.14% 20.30% Impr.R3 3.95% 4.23% 4.44% 4.35% 4.44% 4.88% 4.56% 5.42% 2.88% Internationa Conference on Information Reuse and Integration, pages , Las Vegas, Nevada, USA, August [5] M. Chen, S.-C. Chen, M.-L. Shyu, and K. Wicramaratna. Semantic event detection via tempora anaysis and mutimoda data mining. IEEE Signa Processing Magazine, 23:38 46, March [6] S.-C. Chen, S. Rubin, M.-L. Shyu, and C. Zhang. A dynamic user concept pattern earning framewor for content-based image retrieva. IEEE Transactions on Systems, Man, and Cybernetics: Part C, 36: , November [7] E. A. Cherman, J. Metz, and M. C. Monard. Incorporating abe dependency into the binary reevance framewor for muti-abe cassification. Expert Systems with Appications, 39(2): , February [8] N. Daa and B. Triggs. Histograms of oriented gradients for human detection. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition, voume 1, pages , San Diego, CA, USA, June [9] N. Eeuch, M. Zara, A. B. Ammar, and A. M. Aimi. A fuzzy ontoogy-based framewor for reasoning in visua video content anaysis and indexing. In The Eeventh Internationa Worshop on Mutimedia Data Mining, San Diego, CA, August [10] D. Godberg, D. Nichos, B. M. Oi, and D. Terry. Using coaborative fitering to weave an information tapestry. Communications of ACM, 35(12):61 70, December [11] X. He, R. S. Zeme, and M. A. Carreira-Perpinan. Mutiscae conditiona random fieds for image abeing. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition, voume 2, pages , Washington, DC, June-Juy [12] Y.-G. Jiang. Prediction scores on TRECVID 2010 data set. ast accessed on September 8, [13] Y.-G. Jiang, J. Wang, S.-F. Chang, and C.-W. Ngo. Domain adaptive semantic diffusion for arge scae context-based video annotation. In Internationa Conference on Computer Vision (ICCV), Kyoto, Janpan, September [14] Y.-G. Jiang, A. Yanagawa, S.-F. Chang, and C.-W. Ngo. CU-VIREO374: Fusing Coumbia374 and VIREO374 for arge scae semantic concept detection. Technica report, Coumbia University, August [15] L. Lin, C. Chen, M.-L. Shyu, and S.-C. Chen. Weighted subspace fitering and raning agorithms for video concept retrieva. IEEE Mutimedia, 18(3):32 43, Juy-September [16] L. Lin, G. Ravitz, M.-L. Shyu, and S.-C. Chen. Correationbased video semantic concept detection using mutipe correspondence anaysis. In IEEE Internationa Symposium on Mutimedia, pages , December [17] D. G. Lowe. Obect recognition from oca scae-invariant features. In IEEE Internationa Conference on Computer Vision, voume 2, pages , Keryra, Greece, August [18] T. Meng and M.-L. Shyu. Leveraging concept association networ for mutimedia rare concept mining and retrieva. In IEEE Internationa Conference on Mutimedia and Expo, Mebourne, Austraia, Juy [19] M. Merer, B. Huang, L. Xie, G. Hua, and A. Natsev. Semantic mode vectors for compex video event recognition. IEEE Transactions on Mutimedia, 14(1):88 101, February [20] M. R. Naphade, T. Kristansson, B. Frey, and T. S. Huang. Probabiistic mutimedia obects (mutiects): A nove approach to video indexing and retrieva in mutimedia systems. In IEEE Internationa Conference on Image Processing, voume 3, pages , Chicago, IL, October [21] A. Rabinovich, A. Vedadi, C. Gaeguios, E. Wiewiora, and S. Beongie. Obects in context. In IEEE Internationa Conference on Computer Vision, pages 1 8, Rio de Janeiro, Brazi, June [22] M.-L. Shyu, Z. Xie, M. Chen, and S.-C. Chen. Video semantic event/concept detection using a subspace-based mutimedia data mining framewor. IEEE Transactions on Mutimedia, 10: , February [23] A. F. Smeaton, P. Over, and W. Kraai. Evauation campaigns and TRECVid. In Proceedings of the 8th ACM Internationa Worshop on Mutimedia Information Retrieva, pages , October [24] J. R. Smith, M. Naphade, and A. Natsev. Mutimedia semantic indexing using mode vectors. In IEEE Internationa Conference on Mutimedia and Expo, voume 2, pages , Batimore, MD, June [25] A. B. Torraba, K. P. Murphy, and W. T. Freeman. Contextua modes for obect detection using boosted random fieds. In Neura Information Processing Systems, pages , Vancouver, British Coumbia, Canada, December [26] I. H. Witten and E. Fran. Data Mining: Practica Machine Learning Toos and Techniques, Second Edition (Morgan Kaufmann Series in Data Management Systems. Morgan Kaufmann Pubishers Inc., San Francisco, CA, USA, 2005.