A system that can learn to recognize two-dimensional

Transcription

1 356 Philips tech. Rev. 38, , 1978/79, No. 11/12 A system that can learn to recognize two-dimensional shapes E. H. J. Persoon In spite of the great progress made in visual information processing during the last twenty years, there are still many 'tasks' in this field that present no difficulties to man but which a machine is as yet unable to carry out satisfactorily. One of these is the recognition of objects by their shape in two dimensions [11. The objects are assumed to be flat or nearly flat, resting on a flat support, arranged at random, and possibly touching or even partly overlapping. A recognition task of this kind is important for the automation of assembly work. After recognizing components scattered indiscriminately on a conveyor belt, the machine would be able to pick them up and position them. In the inspection of assembly-line products, or for identifying and counting objects, it is also important to be able to recognize their shapes. This article describes a recognition system that we have developed, which has two striking features. The first is that the system first 'learns' the shape to be recognized. This feature makes the system very flexible: within certain limits, it can be used for the automatic recognition of any arbitrary shape. Secondly, the system can still recognize an object when other objects touch it or even partly overlap it. To make it clear where the difficulties lie in designing a system capable of recognizing objects that may be partly overlapped, let us briefly consider two methods of recognition with the aid of jig. J. We shall assume that the rectangle in fig. 1a is stored in some way in the memory of the system; the system is asked whether the rectangle is present in the scenes (b) and (c). In the first method the rectangle is translated and rotated over the scenes to see how well it 'fits'. In both (b) and (c) it is found that there is one position in which the rectangle fits very well, which 'implies' that it is present. With this method it is possible, in principle, to recognize partly covered objects; its implementation on a computer, however, takes too much time. Let us assume that the pictures are black and white and consist of 100 x 100 picture elements ('pixels'); each pixel can thus have the value 1 or 0 (i.e. be either black or white). A 'good fit' means that a certain functionf of the position of the rectangle DI' E. H. J. Persoon is with Philips Research Laboratories, Eindhoven. has a sharp minimum; f is, for example, the number of pixels that differ in value in the patterns compared. Now in the first place, calculating a new position, that is to say performing the transformations 'translation' and 'rotation' for every point of the rectangle or of its boundary, is in itself extremely cumbersome. Secondly, the number of possible positions is very large: broadly speaking there are 100 orientations for everyone of 100 x 100 locations. An initial refinement might consist in giving priority to transformations for which f becomes smaller, but even this will often be unsuccessful because the rectangle arrives at a relative minimum offand not at the absolute minimum desired. The second method that should be mentioned here is much more effective, but it can only be used if the objects in the scene do not touch one another. In this method the coordinates of the pixels of each object Q Fig. 1. Example of a recognition problem. The aim is to design systems capable of recognizing the rectangle Ca) in the images Cb) and Cc).

2 Philips tech. Rev. 38, No. 11/12 SYSTEMS THAT LEARN 357 in the scene are determined, for example by scanning the boundary [2J, and from these certain shape features are calculated, such as the greatest distance from the centre of gravity to the boundary, or the 'circularity' (i.e. the ratio of the area to the square of the circumference - a ratio that is a maximum for a circle). The object to be identified is 'recognized' when sufficient agreement is found between its features and those of an object in the scene. If objects touch one another, this method cannot be used: there is no criterion for deciding where one object stops and another begins. In our system an object in scenes like those of fig. 1c is primarily recognized by distinct 'local' features of the uncovered parts, e.g. the right angles I, 2, 3 and 4 in fig. 2. In the following such local features will be called shape elements. It is obvious, however, that even when the system has established that the displayed scene has a number of such elements in common with the given object, this would still be only a weak criterion for recognition if at the same time the relative locations and orientations of the recognized elements were not taken into account. In our system a shape is therefore characterized in two phases: in the first phase a search is made for the shape elements, and in the second phase the relative positions of the elements are characterized. The same two phases recur in the analysis of a scene. First it is established whether the given Fig.2. The right angles I, 2, 3 and 4 are salient 'local' features of the rectangle that can be used for the recognition, even though the rectangle is partly overlapped by a triangle. shape elements are contained in the scene, and next whether they have the appropriate relative positions. This procedure makes it possible to characterize the shape of an object with relatively few elements. Moreover, the same elements often recur in the recognition of more than one shape, so that the memory capacity required does not increase in proportion to the number of shapes to be recognized. Searching for the shape elements and recognizing them The shape elements we use are patterns 'cut out' from the image by the window of fig. 3. This window is a good approximation to a circle with a diameter of eleven pixels on the orthogonal grid. The choice of diameter is a compromise: if the window is made too small, it is no longer possible to discriminate between. shapes, such as straight edges, sharp corners, right angles, and so on; ifitis too large, the number ofpossible shape elements is too large. In conjunction with the measures described below, the window of fig. 3 is found to provide the desired degree of discrimination. It is evident that not every pattern in this window can be used as a shape element: the window has 8 I pixels and there are therefore 2 81 possible patterns. The number of possible shape elements is drastically limited by only accepting a window pattern as a 'shape element' if the centre of gravity of the boundary points of the complete picture as selected by the window is located in the centre of the window. To find such shape elements the screen is scanned from left to right and from top to bottom by the window. As soon as one or more black points appear in the window, the centre of gravity of the selected boundary points is calculated. The window is then displaced in such a way that this point arrives at the centre. Since this has the effect of changing the pattern in the window, the boundary after this step is not usually completely centred, but it is centred after one or two more centre-of-gravity calculations and translations. In all this the boundary points of a picture are taken to be the black and the white points that have a white and a black point respectively as a neighbour in the horizontal or vertical directions (see fig. 4). Fig.3. The window used in searching for shape elements: it is a good approximation to a circle with a diameter of eleven pixels. [1] M. Yachida and S. Tsuji, A versatile machine vision system for complex industrial parts, IEEE Trans. C-26, , W. A. Perkins, A model-based vision system for industrial parts, IEEE Trans. C-27, , B. Neumann, Interpretation of imperfect object contours for identification and tracking, Proc. 4th Int. Joint Conf. on Pattern Recognition, Kyoto 1978, pp [2] E. H. J. Persoon and K. S. Fu, Shape discrimination using Fourier descriptors, IEEE Trans. SMC-7, , P. Saraga and J. A. Weaver, An experiment in flexible automation, this issue, pp

3 358 E. H. J. PERSOON Philips tech. Rev. 38, No. 11/12 The coordinates Xc,Yc of the centre of gravity are calculated as follows from the matrix D(x,y) of the boundary in the window (see fig. 4): LD(x,y)x Xc = L D(x,y), LD(x,y)y Ye = L D(x,y), where the summations (L) extend over all window values of x and y. In fig. 4 we have Xc = 0, Yc = 2t. A boundary section is said to be centred when the distance from the centre of gravity to the centre of the window is smaller than or equal to t J -1 o1 ~ X y o o =D(x,y) Fig. 4. Boundary points (framed) in a simple window pattern and the corresponding boundary matrix D(x,y). Shape elements must be compared with each other to determine whether they are 'identical' or 'nonidentical'. In comparing two pictures Pand Q it is usual to count the pixels that have a different value in P from the value in Q. The number counted, d(p,q), is the conventional 'distance measure' between Pand Q. The two pictures are regarded as non-identical if d(p,q) is larger than a given threshold value. This criterion does not however do justice to the following important boundary effect. In the binary pictures that we process the pixels are white or black depending on whether the video signal exceeds a given threshold or not. At a boundary the signal is close to the threshold. Small variations in the threshold, noise in the signalor variations in the sampling position in relation to the picture can easily cause the value of the boundary points to change. The patterns A and B in fig. 5 may therefore easily originate from the same piece of a given scene. This is very unlikely for patterns A and C. Nevertheless, the conventional distance measure between A and B is equal to that between A and C: d(a,b) = d(a,c) = 5. We therefore use a different distance measure, d'(p,q). In this quantity, asind(p,q), pixels are not counted if they have the same value in Pand Q; however, pixels with different values are also not counted if they are boundary points in both pictures. Our criterion for the identity oftwo pictures Pand Q is now: d'(p,q) = O. This is a more general criterion than the corresponding conventional one (d = 0): it allows differences in the val ues of boundary poi nts. According to the new criterion, A and B in fig. 5 are identical, but A and C are non-identical: d'(a,b) = 0, d'(a,c) = 4. Summarizing, what we understand by two identical shape elements are two window patterns in which the centre of gravity of boundary points is centred in the window and for which d' is equal to zero. During the learning process the shape elements found are stored in the memory if they differ from the elements previously stored. During the recognition process a check is made to see which elements in the memory are also present in the input scene. This, in brief, constitutes the first phase of learning and the first phase of recognition. To make a further saving in memory capacity, a few simple computer operations are used for the identifica- A B c Fig. 5. Three window patterns; the conventional 'distance measure' d (the number of pixels that differ in value) between A and B is equal to that between A and C (d = 5). Measured by this criterion A and C are identical if A and B are taken as identical ('elements are identical if d ~ 5'); conversely, A and B are different if A and C are taken to be different ('elements are different if d ;: 5'). In the system used here, A and B are taken to be identical because the difference only relates to boundary points. This does not apply to A and C, which are thus 'different'. Fig. 6. A shape element (upper left) and the elements derived from it. The window is shown as a circle. Once the element at the upper left is stored in the memory, new elements are compared not only with this one but effectively with the other seven as well. This reduces the memory capacity required by a factor of almost eight. The centring of the boundary has not been taken into account in this figure.

4 Philips tech. Rev. 38, No. 11/!2 SYSTEMS THAT LEARN 359 Fig. 7. Three analyses of a picture into shape elements. In each photograph the elements at the upper right were found in the picture at the upper left. The elements (11 pixels in diameter) are shown greatly magnified with respect to the picture (100 x 100 pixels). The display at the lower left shows the reconstruction of the picture with the elements found. A different background colour makes it easy to find element 3 in the first photograph and element 2 in the two others, seen directly, rotated through 90, 180 or 270, or negated or both. tion of the elements. These are rotation of the elements through 90, 180 and 270 and 'negation' of the pixel values (0 becomes 1 and I becomes 0). In the comparison these operations are applied to the stored element; a new element is therefore in fact not only compared with the stored elements themselves but also with the elements derived from them by these operations. For example, once the element at the upper left oîfig, 6 has been stored, it is also used for identifying the other seven in this figure. The measure reduces the memory capacity required by a factor of about 8. (In fig. 6 and also in figs 9 and 13 the window has been drawn for convenience as a circle.) As can be seen fromfig. 7, a surprisingly small number of elements are sufficient under these conditions, even for fairly complicated shapes. In each of the photographs, the binary picture that was displayed for learning on a screen of 100 x 100 pixels is shown at the upper left. The shape elements that were found are shown, greatly magnified, at the upper right; there are only five in the first and only four in the other two photographs. The picture at the lower left was obtained by displaying each element - at the original scale - at the locations and in the orientations and negation states in which it was found for the first time or recognized later. In the first photograph element 3 has been given a different colour, so that it can easily be identified, in both the vertical and the horizontal strokes. In the other two photographs element 2, picked out in the same way, can be seen to appear directly, rotated or negated or both. The elements found are not in general usable for recognizing the object if it has been rotated through an arbitrary angle in the scene, or 'turned over' to give the mirror image. For this the elements themselves would have to be rotated and mirrored. We decided against such complicated operations. Instead, in the first phase of the learning process we 'show' the system the object, right way up and turned over, in a large number of orientations in one quadrant. Fig. 8 shows the shape elements of an object that were obtained in this way. Despite the large amount of input material, the number of elements remains fairly small, because of the limitation in the discrimination; usually only five orientations per quadrant can be distinguished upon rotation

5 360 E. H. J. PERSOON Philips tech. Rev. 38, No. 11/12 of an element. Our procedure of not counting value differences of boundary points contributes significantly to the reduction in the number of elements. In a recognition attempt the number of elements that can be identified with the elements of fig. 8 is again almost eight times the number of elements in fig. 8. The relative positions of the shape elements Before considering the second phase of the learning and recognition processes, we should note that an object, as displayed in a scene, should be recognized by its shape regardless of size. One reason why this is necessary is to allow for the differences that can easily occur in optical adjustments when viewing the learning scene and the scene to be analysed. This has not been taken into account until now because it is of minor significanee in the recognition of shape elements. For recognizing entire objects however, it has to be considered right from the beginning. The second phase will be discussed by referring to the example given in fig. 9. The shape to be learnt is shown in fig. 9a, and the scene to be analysed in fig. 9b. As a result of the first phase of the learning process the elements I, 2, 3, 4 and 5 in fig. ge have been stored in the memory. As a result of the first phase of the reeognition process, element No. 2 has been identified at the points A, B, C, D, E, F, G, H, K in fig. 9b, either directly or in one of the rotated states 2a, 2b or 2e. We shall first look at the second phase of recognition and come back later to the second phase of learning because this involves recognition as well. Thus, we shall assume for the moment that the second phase of the learning process has been completed. The result of this is a number of 'learning lists' in which the positions of the shape elements of the rectangle have been recorded for each of the five orientations in fig. 9d. As we shall presently see, these do not have to be exactly the orientations at which the elements 1, 2, 3, 4 and 5 have been learnt in the first phase. One such learning list, L 2, contains the shape elements of the rectangle in orientation 2 in fig. 9d and their positions: L2 IShapeelementl_" 1_2_1-..-!.!! 1 l!!_1~1 position... P Q R S --- Besides the shape elements mentioned, L2 contains straight-boundary elements that occur at a large number of positions if the rectangle is large. The memory contains in addition lists Li, L3, L4, L5 for the other orientations. If the object did not possess mirror symmetry, there would be five more lists for the mirrorimage shape. For recognition it must be possible to cover the rectangle in fig. 9b with the one in fig. 9a, and to see Fig. 8. Above: an object that has been presented for learning, both right way up and turned over, in a large number of orientations. Below: the shape elements found as a result. whether this can be done it is a straightforward matter to start from the orientation that has shape elements in common with the scene, i.e. orientation 2 in fig. 9d. However, to obtain coverage the rectangle will not only have to be translated and changed in size, it will usually also have to be rotated through a small angle. For although there is no distinction between the shape elements of the two rectangles, it is still possible, even after the best translation and change in size, for the positions to differ perceptibly because the angular resolution for complete objects is much sharper than for elements (I in 50, i.e. about IO for shapes of 50 pixels in length). However, because the rotation remains restricted to a small angle, the general calculation for rotations, x' = x cos q, - y sin q" y' = x sin q, + y cos q" can be replaced by the simple transformation x' = x - q, y, y' = y + q, x.

6 Philips tech. Rev. 38, No. 11/12 SYSTEMS THAT LEARN 361 The attempt at recognition in the example given now proceeds further as follows. In the analysis of the scene in fig. 9b the recognized shape elements are recorded in a 'scene list' L«at the positions where they were found: Q d o Fig. 9. The two phases of the learning and recognition processes, demonstrated by a simple example. a) The rectangle searched for. b) The scene to be analysed. e) Some of the shape elements of the rectangle that are stored in the memory during the first phase of the learning process. With the aid of No. 2 the elements 2a, 2b and 2e can also be identified. In the scene (b) this has happened at A, B, C, D, E, F, G, Hand K (the first recognition phase). d) Orientations of the rectangle for which, in the second phase of the learning process, the elements found before are recorded with their positions in 'learning lists'. One such list is L2, which contains the items '2 at P', '2a at Q',.. In the second phase of the recognition process a 'scene list' L; is made of the scene in (b):.., '2 at A and P, 'Za at Band G',... The system establishes that Ls and L2 have (at least) three elements in common (2, 2a and 2b), and searches for a translation, a rotation through a small angle and a change in scale that will transform the positions of these elements of L2 into those of L; This does not succeed for the elements at A, Band C or at A, Band H, but it does for those at A, Band D. If the transformation found has the effect of matching at least half of all the elements of L2 (including in this example a large number of straight boundary elements) with elements of Lso the rectangle has been 'recognized'. e) A scene list is effectively also confronted with the learning lists rotated through 90, 180 and In this way the rectangle A' B' D' E is also recognized through the rectangle in orientation 2 of (d). ~ ~ 2e L I position I... I A I BIC I D I~_I FIG I HI KI'" I s element (i2b2b2c122(i2bTc The system looks for elements that are common to L«and the learning lists. In our example Ls and L2 appear to have a number of elements in common. From these elements the system chooses three at particular locations in the scene, e.g. 2 at A, 2a at Band 2b at C. It checks to see whether the corresponding elements of L2 can be made to cover them by a translation, a rotation and a change in size of the rectangle. This does not succeed for the elements of Ls at A, Band C; a trial with A, Band H also fails, but a trial with A, B and D does succeed. Our criterion for 'coverage' oftwo positions with the coordinates x,y and x',y' is that Ix - x' I and Iy - y' I should both be smaller than 2. After this success, the transformation thus found is applied to all the elements of L2. If the percentage of these elements that then covers one of the elements of L«is greater than 50 %, the system gives the signal 'recognized'. The elements of L«that are covered in this way in the example are the right-angle elements at A, B, D, E and a large number of straight-boundary elements. For determining the required transformation one group of three elements from L2 is more suitable than the other. The right-angle elements, for example, are more suitable than straight-boundary elements because they are more characteristic. This is not the place, however, to go further into the choice of a suitable group of three. It remains to describe how the learning lists are obtained. The object to be learnt is at first shown in one orientation. Shape elements are searched for and the elements found are compared with the set found in the first phase; the elements that are recognized are recorded with the appropriate coordinates. This yields the first list (L1). To save memory capacity and computer time the list is restricted to a reasonable number, e.g. 15 elements, evenly distributed around the periphery. Next, the object is rotated through about 2, and a check is made to see whether it can be recognized from list L1. If it can, the object is then rotated further. For the first orientation in which the system fails to recognize the object, a new list is made. Usually this happens after about 20, because the shape elements themselves are no longer recognized. In one quadrant five lists are usually obtained in this way, and another five for the mirror image. As regards orientation the rectangles in fig. 9d lie in the same quadrant as the rectangle of fig. 9b. For the recognition of shapes in the other quadrants a scene is effectively confronted not only with the ten lists mentioned, but also if necessary with the 'lists rotated

7 362 E. H. J. PERSOON Philips tech. Rev. 38, No. 11/12 through 90, 180 and 270 ', which are obtained from the original ones by a short calculation during the confrontation. In this way the rectangle in fig. ge would also be recognized through the rectangle in the orientation 2 in fig. 9d. This measure not only saves memory capacity but also shortens the learning time of the second phase. Fig. 10 gives a more realistic example of a recognition problem. The system has learnt the shape of fig. 8 and is instructed to analyse the picture shown at the upper left. In the first learning phase the shape elements of fig. 8 are stored. In the second phase a number of these elements and a number of the elements derived from them are recorded with the appropriate coordinates in ten learning lists. Fig. 11 shows a representation in each quadrant of one of the lists; the shapes consist of 15 of the 28 elements of fig. 8. The picture at the lower left in fig. 10 is a representation of the scene list Le, where the elements of the scene that were identified with elements in the memory are reproduced at the location of identification. At the lower right are the elements of L; that, after the required transformations of a list based on three elements had been found and applied, were also found to correspond in terms of position to an element of a learning list. Each background colour corresponds to one list. Finally, a visual representation is given at the upper right of the three lists that have been found, after the transformations. It is clear that the system has recognized the three objects in the original scene. For simple shapes much less computer time may be required than for complex ones. A disc, for example, need only be shown in one orientation and thus yields only one learning list. Fig. 12, which is built up in the same way as fig. 10, gives an analysis of a scene containing small plastic balls; the analysis is based on this one learning list. 10 1J Fig. 10. Searching for the object of fig. 8 in a scene. Upper left: the scene to be analysed. Lower left: a visual representation of the scene list Ls: the elements of fig. 8 that have been recognized in the scene, at the location of the recognition. Lower right: the elements of L; that have been covered by the elements of a learning list by means of a transformation. Each background colour corresponds to one learning list. Upper right: visual representation of the recognized lists, after the transformations. Fig. 11. Visual representation in four quadrants of one learning list (15 elements) of the object in fig. 8. Fig. 12. Searching for discs and counting them. The arrangement of the figure is the same as in fig. 10. Because of the simple shape of the object searched for, there is only one learning list. 12

8 Philips tech. Rev. 38, No. 11/12 SYSTEMS THAT LEARN 363 Final comments; further research In the foregoing only a broad outline of the method of operation of the system has been given; many details have been omitted. For example, the elements found in the first phase of learning are tested for the possibility of retrieval before they are stored in the memory. If the element in jig. 13a is found 'by chance' in the search process, then the probability of retrieval is small, because centring leads to a different element when it is approached somewhat differently (fig. 13b and c). With such checks the first phase yields a series of elements that forms suitable starting material for the second phase. Another detail should just be mentioned. To prevent the system from marginally failing to recognize stored elements when it is analysing a scene, Q Fig. 13. Retrievability of shape elements. If the shape element (a) has been found 'by chance' during the learning process, it is hardly ever found again later because the centring of the boundary leads to a different element (c) if the same piece of scene is approached somewhat differently (b). Shape elements that are as difficult to retrieve as the one in this example are not stored in the memory. [3] For further details see E. H. J. Persoon, Principles for selforganisation and their application to picture recognition, Proc. 4th Int. Joint Conf. on Pattern Recognition, Kyoto 1978, pp the window is centred in turn on each ofthe eight pixels surrounding each pixel at which an element was found but not recognized, until recognition takes place or all eight have been examined [3J. Further research is being directed primarily at shortening the processing time. Hardware suitable for specific tasks must be developed which can carry out several processing operations in parallel. This has already met with success for the first phase. We are also investigating the feasibility of applying the method discussed in the foregoing to pictures that are not just black and white but also contain grey levels. Summary. A system has been designed for the automatic reeognition of objects by their shapes in two dimensions. The process of 'learning' an object and the process of recognizing it in a displayed scene are both carried out in two phases. In the first phase a search is made for 'shape elements', i.e. patterns in a small window (with a diameter of 11 pixels) such that the centre of gravity of the selected boundary points is situated in the centre of the window. Two shape elements for which only some of the boundary points differ in value are regarded as 'identical'. In the learning process the (fiat) object is shown to the system both right way up and turned over in a large number of orientations. The small dimensions of the window limit the discrimination between shape elements and so restrict their number, which is not very large even with complex shapes and in spite of the large amount of input material. In the second learning phase the system establishes the relative positions of the elements in a maximum of ten 'learning lists', five of them for five orientations of the object in each quadrant and the same number for the mirror image. In the recognition process a 'scene list' is made of shape elements that have been recognized in the first phase, with their positions. If about half or more of the elements of one learning list can be made to cover the corresponding elements ofthe scene list by transformation, the objecthas beenrecognized. The method can also be used for recognizing objects even if other objects touch or partly overlap them.