Fuzzy Knowledge-based Image Annotation Refinement

Transcription

1 284 Int'l Conf. IP, Comp. Vision, and Pattern Recognition IPCV'15 Fuzzy Knowledge-based Image Annotation Refinement M. Ivašić-Kos 1, M. Pobar 1 and S. Ribarić 2 1 Department of Informatics, University of Rijeka, R. Matejčić 2, Rijeka, Croatia 2 Faculty of Electrical Engineering and Computing, University of Zagreb, Unska 3, Zagreb, Croatia Abstract - Automatic image annotation methods automatically assign labels to images in order to facilitate tasks such as image retrieval. Incorrect labels may negatively influence the search results so image annotation should be as accurate as possible. Labels pertaining to objects or to whole scenes are commonly used for image annotation, and precision is especially important in case when scene labels are inferred from objects, as errors in the object labels may propagate to the scene level. To improve the annotation precision, the idea is to infer which labels are incorrect using the context of other labels and the knowledge about objects and their relations. This procedure is here referred to as annotation refinement. The proposed approach used in this paper includes a fuzzy knowledge base and uses the fuzzy inference algorithms to detect and discard automatically obtained object labels that do not fit the context of other detected objects. Keywords: automatic image annotation, annotation refinement, fuzzy knowledge representation scheme, fuzzy inference 1 Introduction Automatic image annotation methods automatically assign labels from a predefined vocabulary to an unlabeled image to facilitate image search and retrieval. The goal is to bridge the so-called semantic gap [1] between the available features that can be extracted from the raw image data, such as color, texture, structure, etc. and the appropriate labels that can be used for retrieval. When searching for images, object or scene labels are typically used, so the vocabularies of automatic image annotation systems contain labels that correspond to objects like skyscraper, fox, flowers, airplane, etc. or to whole scenes like airfield, beach or more general outdoors. In general, a scene may be very complex and be composed of many different objects, so it should also be annotated with many object labels. The automatic recognition of objects depends on the method used for automatic annotation and the features extracted from images. Scene labels can be inferred from object labels if it is assumed that scenes are compositions of objects, as in [2]. Common background objects like sky, grass, etc., may appear in a number of different scenes while some objects are typical for only one scene, e.g. train for scene railway. Correctly detected typical objects greatly help inferring the corresponding scenes, however if an object is annotated with a label of a different typical object, it is very likely that the wrong scene label will be inferred and image will be incorrectly annotated. In scenes without a typical object (e.g. Mountain) even background objects could play an important role for image interpretation and scene inference. Image annotation on object level should therefore be as precise as possible regardless of whether the scene has a typical object or not so that misclassified objects don t negatively influence the image interpretation. The image annotation task is closely related to the problem of classification or the problem of representing the correlations between images and labels, so the most of the automatic image annotation approaches proposed so far belong to the field of machine learning. The methods based on classification, like [3], classify images or image segments into predefined classes based on low-level features extracted from images. Probabilistic methods such as those based on the translation model [4] or on latent semantic analysis [5] learn relevance models to represent the correlations between images and labels. A recent survey of research made in that field can be found in [6, 7]. Graph-based image analysis algorithms have been proposed recently for exploring the correlations between annotated labels [8-14]. To detect and discard the irrelevant labels, a WordNet based semantic similarity is used in [14]. A graphical model is used in [11] for fusing visual content represented by a nearest spanning chain and label correlation by WordNet. A group of authors has examined several different approaches for annotation refinement: Markov chains in [10], conditional random fields (CRF) in [8], normalized Google distance (NGD) in [14] and re-ranking the annotations using the random walk with restarts algorithm in [9]. Incorporating knowledge into automatic image annotation procedure proved as a promising approach for improvement of annotation efficiency. Such an approach was proposed in [15] where a knowledge base and inference algorithms are incorporated into automatic image annotation system for multi-level image annotation. In this paper we propose a pipeline for automatic image annotation with a knowledge based annotation refinement step, presented in Section 2. The refinement step uses a novel procedure for annotation refinement using fuzzy knowledge representation scheme and fuzzy inference algorithms, described in Section 3. The procedure is described with

2 Int'l Conf. IP, Comp. Vision, and Pattern Recognition IPCV' examples from the outdoor image domain. Concluding remarks are given in Section 4. 2 Automatic annotation pipeline The proposed pipeline for automatic image annotation system is shown in Fig. 1. The main stages are image segmentation, feature extraction, segment classification, aggregation of segment labels for image annotation and annotation refinement. It is assumed that an unlabeled image is input to the automatic annotation system. The outputs of the system are labels that are refined using our proposed algorithm. coral, coral, ground,. Note that the labels ground and cloud are a result of misclassification because they are not present in the image. Figure 2. Example of an original unlabeled image and its segmented image Most automatic annotation systems stop at this stage, but because some labels may be the result of misclassification, an additional annotation refinement stage is proposed here. The goal of this stage is to automatically detect and remove the misclassified labels. The procedure for annotation refinement is based on consistency checking and relies on the fuzzy knowledge base. In the fuzzy knowledge base the facts about objects in the domain of interest and their relations are defined. The facts from the knowledge base are used to select a subset of the initial annotation set that satisfies the defined consistency rule. The set is used as the final annotation set and the rest of labels are discarded. Figure 1. Automatic image annotation pipeline with annotation refinement The image is first automatically segmented using the n-cuts algorithm [16] and low-level features such as color, texture, etc. are extracted from each segment. Each segment is represented with a feature vector and then classified into one of the predefined labels It is likely that one label will appear more than once because of actual multiple appearance of an object on the scene or because an object can be split into more than one segment. From the obtained segment labels the annotation set is formed for an image. Next, for each label d in the set A, a value (d) is set depending on the used classifier. If the classifier gives a confidence value for each label, then (d) is set to that value. Otherwise, the value (d) is set to 1. For the unlabeled image in Fig. 2 the annotation set is formed from the segment labels water, cloud, water, fish, coral, fish, 3 Annotation refinement The aim is to annotate an image as precisely as possible, so additional knowledge should be exploited to detect and discard the misclassified labels. Incorrectly detected objects, both typical for a scene or background, can lead to misclassification of the corresponding scenes. Measures of precision and recall are used to evaluate image annotation. Precision of an annotation set A(e) for an image e is here defined as, where r is the number of correct labels and the sum r+w is the number of all labels in the set. Recall is defined as, where n is the number of labels in the reference annotation set R(e) for the image e. Misclassification occurs commonly since different objects can have very similar values of low-level features extracted from images and therefore the classification models cannot be learned well. To reduce the influence of misclassification, the additional knowledge about the domain represented in the knowledge base may be used to check the consistency of labels with respect to the context of the image and the relations among objects. 3.1 A Fuzzy Knowledge-representation Scheme for Annotation Refinement The knowledge about the objects that may appear in images is usually incomplete and uncertain, so a suitable knowledgerepresentation scheme is required. Here, a knowledgerepresentation scheme based on Fuzzy Petri Nets (KRFPN [17]), that supports inference from fuzzy knowledge is

3 286 Int'l Conf. IP, Comp. Vision, and Pattern Recognition IPCV'15 modified and referred to as KRFPNr. The scheme is used to represent the facts in the knowledge base used for annotation refinement. The knowledge base includes pseudo-spatial, spatial and attribute relations between objects in outdoor scenes learned from the data in the training set. The KRFPNr scheme is formally defined as the 12-tuple:, (1) elements of which are described below. is a set of places and is a set of transitions. Each transition is associated with at least one input and output place by the input and output functions, and. A place may be marked with one or more tokens from the set. The tokens are used to define execution of a Fuzzy Petri Net (FPN). Their distribution within places for each execution step w=0,1, is given as, where is a power set of A place p is marked in step w if. In our case, in the initial marking each place can have at most one token. A place that contains one or more tokens is called a marked place and it is important for execution of the transition. To each place from the set, a concept d from the set is assigned by the bijective function. The concepts from the set D may be object labels used for image annotation, e.g. sky, whale, lion or properties of objects like their color (blue, orange, ), position (middle, left, ), etc. To each transition from the set, a relationship from the set is assigned by the function. The set contains all the relationships defined between concepts in the scheme, e.g. occurs_with, is_near, consists_of. The uncertainty and confidence related to the concepts and the relationships between them are expressed by the values of the association functions, and. The degree of truth of the relationship mapped to a transition is given by the transition value. The degree of truth of the concept mapped to the marked place is represented by the token value. The transition values and the initial token values are defined according to the used training dataset Modelling the degree of truth of relations The transition value represents the degree of truth of the relationship associated to the transition. In the KRFPNr scheme, three types of relationships are defined and included in the set : the pseudo-spatial relationship, the spatial and the attribute relationship. The relationship is a pseudo-spatial relationship that describes a mutual occurrence of two objects in the same scene. The reliability of the occurs_with relation between objects and, is denoted as and it is calculated based on the conditional probability of occurrence of object, when object is present on the scene. It represents the reliability that the object appears on the scene along with the object. It is assumed that the appearance of objects are independent events, i.e. that the appearance of one object in the scene does not depend on the occurrence of the other. Using the data in the training set, the reliability of the relationship is computed as: The probabilities and are estimated from the data set using empirical relative frequencies of object occurrence. The joint probability is estimated as the relative frequency of mutual occurrence of both objects and on the scenes in the training set, while the prior probability is estimated as the empirical relative frequency of occurrence of object in the training set. An m estimate is implemented [18] so an estimate of reliability can be obtained in cases when there are no examples of mutual occurrence in the training data set. The size of a virtual set of samples m is s chosen experimentally, and p j is the estimated probability that a sample is the object. The truth value of the relationship is generally not equal to the truth value of relationship For example, the probability of appearance of sky is higher when airplane exists on the scene then the appearance of airplane if sky is on the scene, because airplane in most cases is in the sky but sky can often be without an airplane and occur with a large number of different objects like trees, lion, train,... The relationship is used to check the consistency of segment classification. The truth value of the relation is set to the transition value of the transition between places that correspond to dj and di and to which occurs_with relation is assigned: (3) Spatial relationships such as at_the_top and next_to specify in our scheme the position or relative position of an object in the scene, and are used for consistency checking. Other types of spatial relationships such as topological, distance and internal relations [19], or any new concept or relation can easily be added to the scheme. (2)

4 Int'l Conf. IP, Comp. Vision, and Pattern Recognition IPCV' The truth values of spatial relations are computed using empirical relative frequencies of objects and their spatial positions with m-estimate, in a similar fashion as for computation of reliability of occurs_with relations. The transition value assigned to a spatial relation is set to the truth value of that relation Fuzzy inheritance The fuzzy inheritance algorithm is used in the process of consistency checking for automatic annotation refinement. Input: Concept, depth of inheritance k. Facts in the knowledge base Output: Properties of the concept and the properties of concepts that lie at higher levels of hierarchical structure of concepts. Steps: For the given concept, find the corresponding place using the function :. Define the initial distribution of tokens and set For the distribution construct k levels of the inheritance tree. Collect the leaf nodes. Figure 3. The fuzzy inheritance algorithm All the steps of the inheritance algorithm for the KRFPN are given in detail in [17], and are valid for the KRFPNr as well.the algorithm is based on the inheritance set of the KRFPNr, which is a concept derived from the reachability set of the ordinary Petri nets. The reachability set is defined as the smallest set of all reachable distributions of tokens, starting from an initial distribution and recursively applying the firing of enabled transitions to obtain the immediately reachable distribution of tokens [20]. The enabled transitions are fired in discrete steps in which new token distributions and token values are determined. After the transition has fired at a step w, a new token value is obtained at the output place as where and is the value of the transition. In other words, the token with the maximum value at the input place of the transition together with the transition value determine the value of the token in the output place. A transition is enabled when every input place of the transition is marked: A place is marked if it contains one or more tokens. When a transition fires, tokens simultaneously move from all the transition s input places to the output places. The inheritance set is represented with a fuzzy inheritance tree, while the reachability set is represented by a reachability tree. The main difference between the reachability set and the inheritance set of the KRFPNr are related to the semantic interpretation of places. Namely, to stop further firing of transitions at the places that represent the end of the hierarchical structure among concepts, the corresponding nodes have to be frozen. A node is frozen if it is an output node of a transition associated with attribute and spatial relationships. The generation of the inheritance trees may stop on frozen (F) nodes, on a predefined level k (k-terminal nodes, k-t), in which case a k-level inheritance tree is generated, or on terminal (T) nodes. A terminal node is a node with no enabled transitions. The root nodes of the inheritance trees are formed according to the initially marked places and their corresponding truth values. The nodes of inheritance trees have the form, l, where the first component specifies the place where the token is located and the second one is the token value,, represents the level of the tree, and i is the node index at that level. The steps of the fuzzy inheritance algorithm used for consistency checking are shown in Fig 3.: 3.2 Consistency checking The consistency of labels in set obtained after classification of image segments is checked using the facts from the knowledge base. Among the labels in the annotation set A, there can be correct labels but also some labels that are a result of misclassification. We can assume that correct labels are consistent among themselves with respect to the occurs_with relationship defined in the knowledge base. If there is more than one misclassified label, it is possible that by chance they are also consistent among themselves. Thus, there may be more than one subset of mutually consistent labels in the annotation set A from which only one will be selected for annotation. In order to detect the misclassified labels, the proposed consistency checking algorithm identifies all subsets of mutually consistent labels in the annotation set. Taking into account both the size of such subsets and the confidence values (d) of each object d, the subset A of labels with maximum RV value (Fig. 4) is selected for annotation. The remaining labels in the set A\A are considered misclassified and are discarded. The consistency checking procedure is formally defined in the Fig. 4

5 288 Int'l Conf. IP, Comp. Vision, and Pattern Recognition IPCV'15 Input: Initial annotation set A Steps: for each label in : generate inheritance tree add and leaf nodes to a set define a power set initialization for each : compute if compute if set set Output: The set A of all consistent labels from Figure 4. Consistency checking procedure. First, for each label in the annotation set an inheritance tree is generated using the fuzzy-inheritance algorithm and the facts in the knowledge base. The inheritance trees are used to verify whether an relationship is defined between that object and other obtained object labels. The leaf nodes of the tree correspond to labels that may occur together with the object d. For each object label d in the annotation set A, a set of possible objects that may occur in an image together with that object is defined using the leaf nodes of the inheritance tree. In the next step, all subsets of A, i.e. elements of the power set that are consistent with respect to the occurs_with relationships are identified. A set of labels is consistent if each label in the set has the occurs_with relation defined between it and all other labels in the set. This is equivalent to the condition that a set S is consistent if it is a subset of the intersection of sets of possible labels R(d) for each label d in S,. For each consistent subset of labels, a RV value is computed that considers the size of the set and confidence values (d) of each label in the set S. The set with the highest RV value is selected as the consistent set A and used for annotation. For instance let the image e on the Fig. 2. be considered for consistency checking. Then for the object ground, the appropriate place in the knowledge-representation scheme is determined by the function, and a token is placed in place. According to the initially marked place, the initial token distribution is created. The corresponding root node of the inheritance tree is. The inheritance tree is formed by firing the enabled transitions (whose firing creates new nodes) until the condition for stopping the algorithm is satisfied or the desired depth k of the inheritance tree is reached. Figure 5 shows the 1-level inheritance tree on the KRFPNr scheme for the object ground. Figure 5. 1-level inheritance tree for the object ground detected as possible intruder in automatic annotation of an image e. The truth value of the occurs_with relation between the root node and all other objects is determined by the token value in leaf nodes (the nodes in which the algorithm stops). The arcs of the inheritance tree are labeled with transitions and values. For example, the arc corresponds to the relation occurs_with between the nodes and. The node corresponds to the object at the place and the node to the object water a the place. The truth value of the relation is. The inheritance tree has stopped on output nodes of the transitions, marked as frozen leaf nodes (F). Because the depth of the inheritance tree was 1, the leaf nodes are also marked as 1-terminal (1-T). Leaf nodes of inheritance tree include all objects from the knowledge base that are in relation occurs_with with object ground. The leaf nodes include places that function maps to objects such as, that make the codomain of relation occurs_with for a given object ground. These objects form the set R(ground), R(ground) = {sky, lion, grass,, rock, water}. The inheritance trees are also formed for all other objects in the set For example, in Fig. 6 is the inheritance tree with the object coral as the root node and with all the objects from the knowledge base that are in relation occurs_with with the object coral in leaf nodes. After applying the function on the places at the leaf nodes,, the codomain of relation occurs_with for a given object coral is formed as the set R(coral)={water, fish, sand, rock}. Figure 6. Inheritance tree for the object coral.

6 Int'l Conf. IP, Comp. Vision, and Pattern Recognition IPCV' The intersection of R(coral) and R(ground) contains neither ground or coral so they cannot appear together. By running the rest of the algorithm for each subset of the power set, the sets {}, {water}, {fish},{coral},{cloud},...,{water, fish},, { } are identified as consistent. The subset with the greatest RV value is chosen as the consistent set G={water, fish, coral}. If the chosen set A contains correctly classified labels, the average precision of the image annotation can be increased, Fig. 7a,b. However, if the majority of labels are misclassified, a set of wrong but mutually consistent labels might be chosen. In that case the annotation refinement can discard the correct labels and both recall and precision falls, Fig. 7c. In the Fig. 7, incorrect labels are printed bold. (a) (b) (c) Image example e Reference annotation Automatic annotation Annotation after refinement R(e) sky, wolf, trees, grass airplane, sky, cloud shuttle, astronaut coral, sky, wolf, trees, grass airplane, dolphin, sky, cloud shuttle, train, building sky, wolf, trees, grass airplane, sky, cloud train, building Figure 7. Positive (a), (b) and negative (c) example of annotation refinement. 4 Conclusion In this paper, a knowledge based procedure is proposed for refinement of labels obtained with automatic image annotation. Knowledge about objects and their relationships in the domain of interest is exploited to find consistent sets of labels among the results of automatic annotation and to discard the inconsistent labels. To represent the knowledge about objects and their relationships, a fuzzy knowledge representation formalism, dubbed KRFPNr is used. Vocabularies for automatic image annotation usually contain labels that correspond to objects (building, bear, train, etc.) and scenes (forest, underwater, nature, etc.) because these kinds of terms are typically used when searching for images. If it is assumed that scenes are compositions of objects, then scene labels can be inferred from object labels. In this case, image annotation on object level should be as precise as possible to avoid the case where a misclassified object leads to wrong conclusion about the whole scene. Increased precision can be achieved by including an image annotation refinement step in the annotation process. The approach used in the proposed procedure checks the consistency of each object label obtained with classification with respect to the most likely image context deduced from the rest of the labels. Detected inconsistent labels are assumed to be results of misclassification and are not used for annotation, thereby increasing precision.. The consistency checking procedure relies on the facts about objects and their relations stored in the knowledge base and the fuzzy inheritance algorithm. The functioning of algorithms is demonstrated on examples of images of outdoor scenes. In the future work, instead of discarding the detected inconsistent labels, inference algorithms and facts from the knowledge base willl be adapted for finding suitable replacement labels that are useful for annotation. 5 References [1] Hare JS, Lewis PH, Enser PGB Sandom CJ January Mind the Gap: Another look at the problem of the semantic gap in image retrieval. Multimedia Content Analysis, Management and Retrieval, San Jose, California, USA. [2] Ivašić-Kos, M.; Ribarić, S.; Ipšić, I. Low- and Highlevel Image Annotation Using Fuzzy Petri Net Knowledge Representation Scheme. International Journal of Computer Information Systems and Industrial Management (IJCISIM). 4 (2012) [3] J. Li and J. Z. Wang, ``Real-Time Computerized Annotation of Pictures,'' IEEE Transactions on Pattern

7 290 Int'l Conf. IP, Comp. Vision, and Pattern Recognition IPCV'15 Analysis and Machine Intelligence, vol. 30, 2008, pp [4] Duygulu, P., Barnard, K., Freitas, J.F.G. de, Forsyth, D. A., Object recognition as machine translation: Learning a lexicon for a fixed image vocabulary, ECCV 2002, UK, 2002, pp [5] Monay F. and Gatica-Perez D., On image autoannotation with Latent Space Models, Proc. ACM Multimedia, Berkeley, CA, 2003, pp [6] Datta, R., Joshi, D., Li, J Image Retrieval: Ideas, Influences, and Trends of the New Age, ACM Transactions on Computing Surveys, vol. 20, pp. 1-60, April [7] Zhang, D., Islam, M. M., and Lu, G. (2012). A review on automatic image annotation tecniques. Pattern Recognition, 45(1), [16] Shi, J., & Malik, J. (2000). Normalized cuts and image segmentation. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 22(8), [17] Ribarić, S., Pavešić, N., Inference Procedures for Fuzzy Knowledge Representation Scheme, Applied Artificial Intelligence, vol. 23, January 2009, pp [18] Džeroski, S., Cestnik, B. and Petrovski, I., Using the m-estimate in rule induction. J. Comput. Inf. Technol, 1: [19] Isabelle Bloch, Fuzzy spatial relationships for image processing and interpretation: a review, Image and Vision Computing, vol. 23(2), 2005, pp [20] Chen, S.M., Ke, J.S., Chang, J.F., Knowledge Representation Using Fuzzy Petri Nets, IEEE Transactions on Knowledge and Data Engineering, vol. 2, 1990, pp [8] Wang, Y., Gong, S. (2007). Refining image annotation using contextual relations between words. ACM CIVR 07, July 9 11, Nethelands, [9] Wang, C, Jing, F., Zhang, L., & Zhang, H.-J. (2006). Image annotation refinement using random walk with restarts. ACM MM 06, October 23 27, Santa Barbara, California, USA [10] Wang, C., Jing, F., Zhang, L., Zhang, H.-J. Content based image annotation refinement. CVPR 07. [11] Liu, J., Li, M., Ma, W.-Y., Liu, Q., & Lu, H. (2006). An adaptive graph model for automatic image annotation. ACM MIR 06, Santa Barbara, California, USA, October 26-27, [12] Zhou, X., Wang, M., Zhang, Q., Zhang, J., Shi, B. (2007). Automatic image annotation by an iterative approach: incorporating keyword correlations and region matching. ACM CIVR 07, July 9 11, Amsterdam, Nethelands, [13] Yohan, J., Khan, L., Wang, L., & Awad, M. (2005) Image annotations by combining multiple evidence and WordNet. ACM conference on Multimedia (MM05 ), Singapore, [14] Cilibrasi, R. L., & Vitanyi, P. M. (2007). The google similarity distance.knowledge and Data Engineering, IEEE Transactions on, 19(3), [15] Ivašić-Kos, M.; Ipšić, I.; Ribarić, S. Multi-Level Image Annotation Using Bayes Classifier and Fuzzy Knowledge Representation Scheme, WSEAS transactions on computers. 13 (2014);