Reversible Image Merging for Low-level Machine Vision

Transcription

1 Reversible Imae Merin for Low-level Machine Vision M. Kharinov St. Petersbur Institute for Informatics and Automation of RAS, 14_liniya Vasil evskoo ostrova 39, St. Petersbur, Russia, Abstract: In this paper a hierarchical model for pixel clusterin and imae sementation is developed. In the model an imae is hierarchically structured. The oriinal imae is treated as a set of nested imaes, which are capable to reversibly mere with each other. An object is defined as a structural element of an imae, so that, an imae is rearded as a maximal object. The simulatin of none-hierarchical optimal pixel clusterin by hierarchical clusterin is studied. To enerate a hierarchy of optimized piecewise constant imae approximations, estimated by the standard deviation of approximation from the imae, the conversion of any hierarchy of approximations into the hierarchy described in relation to the number of intensity levels by convex sequence of total squared errors is proposed. Keywords: sementation, pixel clusterin, optimization. 1. INTRODUCTION Nowadays a new problem of automatin the creation of artificial intellience applications arises. This problem is solved, at least in the project PPAML (Probabilistic Prorammin for Advancin machine learnin) of the US aency DARPA (Defense Advanced Research Projects Aency), which is scheduled for In the field of machine vision it seems necessary to create a unified software tool for the detection of an object hierarchy. Such tool should help the inexperienced prorammer to select the «objects of interest» amon many objects found in the imae by computer. Stumblin block for computer object detection is to define the object reardless of the imae content and without preliminary machine learnin. To avoid this difficulty it seems possible to treat the objects as the clusters of pixels of optimal approximations, which minimally differ from the imae in the standard deviation σ or total squared error E = 3Nσ, where the coefficient 3 is the number of color components in the imae. However, althouh such definition is based on the classic cluster analysis [1], the opportunities to minimize of the total squared error E (the approximatin error, for conciseness) in imae processin domain are far from bein exhausted, especially, in the task of multiple optimization for each number of pixel clusters. So, continuin [], we aim to really minimize the approximation errors in the eneralized task of pixel clusterin and imae sementation, as well as to create 1 and implement the unified software for automatic object detection in the primary stae of imae reconition and others processin tasks.. PROBLEM STATEMENT In bein created model for hierarchical pixel clusterin and imae sementation the obtainin of hierarchical sequence of optimized approximations of N pixels partitioned into each number of clusters from 1 to N is treated. The sequence of optimal approximations of clusters is described by the monotonous sequence of approximatin errors E that non-strictly decrease with the rowth of the number of clusters from 1 to N from the maximum value in case of sole cluster to zero when all pixels assined to different clusters. The characteristic property of E sequence is convexity: E 1 + E + 1 E =, =, 3,..., N 1, (1) causin the non-strict rowth of increment Δ E, alon with decrease and increase of approximatin error E. The sequence of optimized approximations should accurately simulate a sequence of optimal approximations. Therefore, for a close similarity the convexity of taret sequence of approximations is also required. Hierarchical approximations that correspond to a convex sequence of E values dependin on the number of clusters are called quasioptimal [3,4]. Since the sequence of optimal approximations in eneral is non-hierarchical, then the sequence of quasioptimal approximations is not uniquely determined as in Fi.1. E Fi.1 Simulatin of optimal approximations Fi. 1 raphically illustrates the approximatin error dependin on the number of clusters in the imae

2 approximations. Lowest bold ray curve corresponds to the optimal imae approximations. Solid curves correspond to the taret quasioptimal approximations that match the optimal at least for three cluster numbers. All the curves divere from a common point for = 1, correspondin to the imae approximation of a sinle cluster, and for = N convere at one point describin imae approximation of N clusters, each containin a sinle pixel. All curves shown in Fi. 1 are convex. Based on experiments with examples of ray imaes [] it may be supposed that taret quasioptimal approximations for color imaes are described by the curves passin throuh a series of optimal values and intricately intertwined dependin on the imae content. Nearly optimal curves within the deviations of quasioptimal curves of Fi. 1 from the optimal values lie in a quite narrow band over the optimal curve. So, in order to reliably appreciate and implement the effect of pixel clusterin optimization, first of all it is hihly desirable to increase the effectiveness of calculatin of optimized approximations for color imaes. To do this would be useful to pose and solve the task of convertin of any hierarchical sequence of approximations into a sequence of quasioptimal approximations, as well as to minimize the approximation error for a fixed number of pixel clusters as in Fi. 1. The latter problem is standard, and it s usually declared in the practical applications of K- means method as well as its numerous modifications [1,5,6]. The first problem looks rather theoretical than practical. But it is the task of the conversion of any pixel clusterin or imae sementation into quasioptimal one turns out most fruitful to unify the software. 3. IMAGE AND OBJECT NOTIONS In the developed model of quasioptimal imae approximations [3,4] a computer processin of the oriinal imae is represented as follows. Let's consider that an imae should be hierarchically structured. Let's aree that the oriinal imae initially is divided into pixels, which are treated as elementary imaes and are capable to reversibly mere with each other. Then durin the processin the oriinal imae is divided into a number of structured sub-imaes that finally mere into а sinle completely structured imae. As a result of discussed processin, a dichotomous set of N 1 pixel clusters, in special case, imae sements, and a sequence of N imae approximations with each number of clusters or sements from 1 to N is enerated and stored in RAM in terms of Sleator- Tarjan dynamic trees without excessive memory usae [3,7,8]. The purpose of the model [3,4] is only producin the resultant hierarchical imae approximations and pixel clusters or imae sements for post-processin. See also the-optimal-sementation-dataset Since any cluster of pixels may be treated as an imae, the model does not involve any restrictions, but simply ascribes to an imae a dichotomous cluster structure. Object 3 in the model is considered to be an element of the imae, i.e. one or another pixel clusters, found in the imae by the computer. The imae is a maximal object that includes other objects detected inside of iven area of the oriinal imae. Completely structured oriinal imae includes all N-1 available objects detectable by computer. Intermediary structured oriinal imae, divided into several nested sub-imaes, contains only these sub-imaes and the objects detected in each of them. Unlike the imae the notion of an object does not imply that any cluster of pixels may be treated as an object, as the dichotomous object hierarchy is calculated for the iven imae by the computer, in compliance with the loic of the problem statement. 4. REVERSIBLE IMAGE MERGING Let I 1 and I be the averae intensities for the sub-imaes S 1 and S, respectively. Let n 1, n be the correspondin numbers of pixels. Then the increment Δ Emere E( S1 S ) E( S1 ) E( S ) of the approximatin error E caused by the merin of specified sub-imaes alon with reduction of their number per unit is iven by the formula: n n 1 Δ E mere = I1 I, () n1 + n where I1 I is the square of the Euclidean distance between the three-component color values of averaed intensity. It is the expression () used to iteratively calculate the hierarchical sequence of quasioptimal imae approximations by Ward's method [9], that are produced accordin to the criterion: Δ mere E = min, (3) whereby at each step of the hierarchy eneration, the merin of clusters causin the minimal increment of the approximation error is performed. The same estimation and criterion (-3) are preferably used to iterative sementin of an imae by pair wise merin of only adjacent sements, as in Mumford Shah sementation model [3,4,10-1]. Since the model of quasioptimal approximations allows any sub-imaes, then it imposes no a priori constraints to merin of the imaes. Therefore, imae merin, in principle, can be carried out in any order, or in accordance with any model of pixel clusterin or imae sementation. Moreover, in the framework of reversible computin technique [13], the reversibility of imae merin operation is supported. This means 3 The term «object» is used as a synonym for the term «sub-imae» and treated mainly in non-modifiable state, in particular to denote the resultant pixel clusters.

3 that if the imaes S 1 and S mere into the imae S 3, then the latter becomes available for reverse dichotomous dividin operation just into the imaes S 1 and S, causin the neative or zero increment of approximatin error: ΔE divide ( S ) ΔE ( S ) = mere 1. (4) 3, S When merin, imaes transform into structural elements referred to as objects. Conversely, the dichotomous division of an imae in twain causes its disinteration into objects. The convexity property of approximatin errors for nested objects is expressed by the non-strictly decreasin of non-neative drop Δ E divide of approximatin error E alon with decreasin of nested objects S ( S ) ΔE ( ) 1 S ΔEdivide 1 divide S, (5) where the notation S should be understood as the imae to be produced. From the standpoint of interpretation, the convexity claims (1), (5) simply means revealin of objects in descendin order of contrast in different or in the same imae locations. Convexity property is obviously not affected by the imae dividin operation, but to preserve the convexity for merin operation the restructurin of composite imae is required. Therefore, the imae merin is carried out as a combined operation consistin of a unitin of the input hierarchies, followed by modification of obtained hierarchical sequence of sub-imaes to match the convexity conditions (1), (5). To mere a pair of structured imaes into a sinle imae, which is to be also structured, most likely, one can simply apply the Ward's method to a joint set of pixels. But to account that each of the two input subimaes is relied preliminary ordered, a more careful alorithm is used. This alorithm aims to keep the established order and does not chane joint hierarchical sequence of sub-imaes, if they are described by the convex sequence of approximatin errors. Otherwise, the computer detects wron embeddins causin convexity violations, eliminates them by dividin a joint imae into sub-imaes and then meres these subimaes into a sinle imae by Ward s method. Since merin of the structured sub-imaes, in eneral, initiates novel convexity disorders, the detection of improper embeddins is performed aain, partitionin into sub-imaes is crushed and the processin is iteratively repeated until obtainin the perfectly structured joint imae. Thus, if the available software supports reversible merin, storin, retrievin, analyzin and transformin of the hierarchy of ordinary clusters [3], then the transition to the advanced reversible merin of imaes, arraned in descendin order of increment of the absolute values of the approximation errors alon increasin of the number of sub-imaes, is provided by simple utilization of the combined operation of imae merin and restructurin instead of cluster merin operation. 5. PIXEL CLUSTERING IMPROVEMENT Our experience of sementation reardless of predetermined imae content, leads to the conclusion that conventional Ward's clusterin [9] is not sufficiently used in the initial processin and streamlinin of video data due to excessive computational complexity. At first lance, mentioned challene is overcome in the task of complete dichotomous structurin of oriinal imae by merin of only adjacent sub-imaes with few pixels in the first stae of imae sementation and subsequent clusterin of the rest several subimaes with many pixels in the second, final stae of eneratin the hierarchically structured imae that simulated by the sequence of hierarchical approximations and described by the convex sequence of approximatin errors. In this case the resultant hierarchy of approximations will be described by piecewise convex curve. But upon closer experimental study it turned out that the correction of approximations to smooth the overall curve still requires heavy computation. To withdraw this problem of data adjustment by refinement of rouh structured imaes obtained in the first stae of processin and at the same time to effectively minimize the approximatin error for iven cluster number the method presented in this section is developed. This method of pixel clusterin, in particular imae sementation for a iven intermediate number of subimaes ensures that the maximal drop of the approximatin error max ΔEdivide, caused by division in twain of certain one from all sub-imaes, would not exceed its minimal increment min ΔEmere, caused by merin of certain sub-imae pair chosen from all pairs of sub-imaes: minδemere max ΔE divide. (6) If the condition (6) is violated, then the division of the sub-imae, which induces maximum drop of approximatin error, is carried out. In the followin step, the calculation of, in eneral case, a new pair of sub-imaes providin while merin the minimal increase of approximatin error, is done. Next, a pair of found sub-imaes meres into one imae by means of the combined operation of merin with the immediate restructurin of the joint imae. Finally, the minimization process is either terminates, if criterion (6) is fulfilled, or resumed while (6) is not fulfilled. Note, that merin of one of two parts of the subimae divided in twain, to the other sub-imae is envisaed in the above method. Since in precedent versions [3,4,14] the discussed method was referred to as SI-method (abbreviated Sementation Improvin), then in current version we call it ASI-method (Advanced Sementation Improvin) to emphasis the utilization of combined merin/restructurin operation that enhances the action of the method in the tasks of approximatin error minimizin. The main advantae of SI and more powerful ASI method that it effectively copes with far from the optimal pixel clusterin or imae sementation,

4 providin more impressive improvement in the visual perception and also by the approximatin error for the rouher initial imae approximations [14]. 6. EXPERIMENTAL RESULTS Fi. illustrates the effect of the combined merin/restructurin operation on the example of the standard color «Lena» imae of 51x51 pixels, shown in the upper left corner. The trivial approximation of oriinal imae, which consists of the same pixels of a sinle color, constitutin a sinle cluster is shown next to the riht. Under the oriinal imae in the left column the approximations that are obtained by ordinary merin of adjacent sements in version [3,4,11] of Mumford-Shah sementation model are placed. These contain from to 5 sements of different colors. In the riht column the appropriate imae approximations with 1-5 colors, which are enerated throuh combined merin/restructurin are placed. They are obtained by the same merin of adjacent sements, followed by the immediate restructurin of the sub-imaes, occupyin these sements, to satisfy the convexity condition (5). Fi. demonstrates an example of conversion of sementation to clusterin. The effect of hierarchical sementation improvement seems obvious. It is expressed both in visual perception, and also in decrease of the standard deviations, written out under the approximations. It is important that, in contrast to the conventional hierarchical sementation (left column in Fi. ), the similar meaninful objects (eyes, pupils, etc.) simultaneously appears in the approximations for hierarchical clusterin (riht column in Fi.). This effect is not accidental and can be used in a processin of stereo-pairs to detect previously unspecified objects for subsequent matchin of their feature points and calculatin the distances [15]. It is also important, that quite similar clusterin results to those in Fi. for color imae "Lena" of 51x51 pixels, are reproduced for the same imae, but in the ray scale and of reduced size of 56x56 pixels. So, we assume that the formal objects i.e. resultant subimaes, stably detected by computer as the parts of visually observable objects, can be treated as workpieces of meaninful objects, at least in a number of practical tasks. ASI-method speeds up the calculations, provides tunin and promotes the object detection. A number of obtained dichotomous sequences of imae approximations is described in Fi. 3 that illustrates the standard deviation σ of approximations from the imae, dependin on the number of clusters shown in the rane from one to thousand. The uppermost curve describes the sequence of approximations obtained by conventional sement merin (left column in Fi.). Lower intertwined curves describe the optimized approximations enerated by combined merin/restructurin, used instead of conventional merin. These describe the time-consumin approximations, obtained by Ward s method and by simple merin followed by immediate restructurin (riht column in Fi.), as well as the approximations, obtained usin ASI-method, which was executed for different sub-imae numbers specified in the rane from 100 to σ=40,9531 σ=37,0860 σ=34,94496 σ=46,04981 σ=7,941 σ=,17105 σ=18,5590 σ=3,68304 σ=15,80636 Fi. Sementation-to-clusterin conversion

5 As can be seen in Fi. 3, the optimized approximations differ from the conventional much more than amon themselves. As for visual perception all look natural and it is difficult to prefer one another. Apparently, it is useful to use several optimized dichotomous sequences of imae approximations. 50 σ Fi.3 Standard deviation σ dependin on cluster number (in loarithmic scale) 7. CONCLUSION Thus, the paper presents the current development of the model of quasioptimal imae approximations, intended for automation of low-level imain usin the classical cluster analysis. The task is to accurately simulate the imae by approximations like that described in Fi.1. To achieve this, the combined merin/restructurin operation for hierarchically structured imae formation and ASI-method for pixel clusterin improvement are proposed in this paper. The fact that improved hierarchical clusterin or sementation is reduced to simply replacin of basic mere operation by advanced combined operation is quite attractive for the unhindered implementation as in ASI-method. But for perfect minimizin of the approximatin error, keepin the cluster numbers, ASImethod itself, especially in the case of a small cluster numbers, must be supplemented by proper version of so called K-meanless method [6,14], which, in fact, is the advanced version of K-means method [1]. The problem of excessive memory usae when operatin with millions of imae approximations is completely overcame, owin to data structure of Sleator-Tarjan dynamic trees [3,7,8]. However, when implementin and even a pilot study it is impossible to inore the routine time-optimizin multi-iterative hierarchy eneration, restructurin and optimization. But it's a one-time job. Therefore, it is likely that the problem of creatin of an auxiliary tool for pixel clusterin and imae sementation will be solved in the comin years. Then the development of the software will be enaed only in specific roups of developers, and the other specialists will be able to use ready-made prorams. 6. REFERENCES [1] S.A. Aivazian, V.M. Bukhshtaber, I.S. Eniukov, L.D. Meshalkin. Applied Statistics: Classification and dimension reduction. M.: Finansy i statistika, p.607. [] M.V. Kharinov. Hierarchical pixel clusterin for imae sementation. Proceedins of the 1th International Conference on Pattern Reconition and Information Processin (PRIP 014), Minsk, 014. pp [3] M.V. Kharinov. Pixel Clusterin for Color Imae Sementation. Prorammin and Computer Software, 015. Vol. 41,. 5, pp [4] M.V. Kharinov. Model of the quasi-optimal hierarchical sementation of a color imae. Journal of Optical Technoloy, 015. Vol. 8, Issue 7, pp [5] A.K. Jain. Data Clusterin: 50 Years Beyond K Means. Pattern Reconition Letters Vol pp [6] S.D. Dvoenko. Meanless k-means as k-meanless clusterin with the bi-partial approach. Proceedins of the 1th International Conference on Pattern Reconition and Information Processin (PRIP 014), Minsk, 014. pp [7] R.E. Tarjan. Efficiency of a Good But Not Linear Set Union Alorithm. Journal of the ACM Vol.,, pp [8] D.D. Sleator, R.E. Tarjan. Self Adjustin Binary Search Trees. Journal of the ACM Vol. 3, 3, pp [9] J.H., Jr. Ward. Hierarchical roupin to optimize an objective function. J. Am. Stat. Assoc Vol. 58, Issue 301, pp [10] D. Mumford, J. Shah. Boundary detection by minimizin functionals, I. Proceedins of IEEE Computer. Vision Pattern. Reconition Conference, San Francisco, pp.-6. [11] A.S. Buaev, A.V. Khelvas Pilot studies and development of methods and tools for analysis and automatic reconition of media flows in lobal information systems. Scientific and Technical Report, Moscow: MIPT Vol. 1, p [1] L. Bar, T.F. Chan, G. Chun G., M. Jun, L.A. Vese, N. Kiryati, N. Sochen. Mumford and Shah Model and Its Applications to Imae Sementation and Imae Restoration. Handbook of Mathematical Methods in Imain p. [13] T. Toffoli Reversible computin. Spriner Berlin Heidelber, pp [14] M.V. Kharinov, I.G. Khanykov. Optimization of piecewise constant approximation for semented imae. SPIIRAS Proceedins Issue 3(40), pp [15] M.V. Kharinov, I.G. Khanykov. A combined method of imae sementation improvement. Bulletin of Buryat State University pp