MORPHOLOGICAL SEGMENTATION OF IMAGE SEQUENCES

Transcription

1 MORPHOLOGICAL SEGMENTATION OF IMAGE SEQUENCES B. MARCOTEGUI and F. MEYER Ecole des Mines de Paris, Cenre de Morphologie Mahémaique, 35, rue Sain-Honoré, F Fonainebleau Cedex, France Absrac. In image compression, objec-based approaches are adaped o high compression raes, since hey ake ino accoun he geomery of he objecs and he human eye characerisics. Mahemaical Morphology, dealing wih geomerical feaures is a well suied echnique for segmenaion purposes. This paper presens a mehod o segmen image sequences, firs sep of an objec-oriened compression sysem, based on Mahemaical Morphology. Key words: Image Sequences, Segmenaion, Mahemaical Morphology, Image Compression. 1. Inroducion Classical mehods of image compression use specral decomposiion inside spaial uniies called blocks, which are no direcly relaed wih he image conens. These blocks produce arifacs in he coded image ha preven from reaching high compression raes wih an accepable qualiy. As an alernaive, objec-oriened mehods have been developed [1]. In order o avoid block effecs, hese echniques spli he image ino semanic regions and hen encode separaely he shape and he conens of each region. Higher compression raes are expeced, because he geomery of he objecs and he human eye characerisics are aken ino accoun. The firs sep of an objec-oriened approach is segmenaion, on which is based he whole scheme. A good segmenaion, wih he visual conours and only hem, maximizes he compression rae. In image sequences a good ime coninuiy is he basis for a high compression rae. Mahemaical Morphology [2] [3], dealing wih geomerical feaures is a well suied echnique for segmenaion purposes. This paper presens a mehod o segmen image sequences, based on Mahemaical Morphology. The firs image of he sequence is segmened in an inra-frame mode, presened in secion 2. In secion 3 a mehod o reach maximum sabiliy in image sequence segmenaion is presened, as well as he possibiliy o include new regions, in he only cases when i is really necessary. Finally he problem of regulaing he segmenaion is discussed, in order o produce a bisream compaible wih he desired compression. 2. 2D segmenaion Dealing wih image sequences, a 2D segmenaion algorihm is also needed, in order o process inra frames (firs frame of he sequence or iniializing sage). The segmenaion algorihm may be spli up ino he following seps [4]: filering sep: he goal is o remove non percepual feaures in order o simplify

2 B. MARCOTEGUI AND F. MEYER he deecion of homogeneous regions. Morphological filers, (area opening and closing [5], and reconsrucion filers) are used. They remove objecs smaller han a given size preserving he conour of he remaining ones. feaure exracion: he goal is o deec he exisence of homogeneous regions. decision sep: he goal is o place he conour of he objecs ha have been seleced by he feaure exracion. This decision is made by a waershed algorihm o a gradien image [6] [3] [7], which places he conour on he line of highes gradien beween wo objecs. A waershed algorihm applied direcly on a gradien image produces a severe over-segmenaion. Each gradien minimum becomes a region, and mos of hem do no mach wih visual regions. Gradien minima have o be classified according o heir visual significance and he mos significan ones have o be chosen in a number he sysem can afford. This is he aim of he feaure exracion, he main difficuly of he segmenaion. The dynamic [8] applied o he gradien, classifies is minima according o heir relaive conras. Fig. 1 illusraes his ranking for a smooh funcion before and afer adding noise. The wo main srucures have a high dynamics in boh funcions, while he noisy minima have a comparaively lower dynamics. This measure provides for feaure exracion a selecion crierion based on conras. Bu pure conras does no fi exacly wih visual significance. Beween wo regions of same conras, he larger one, would be more visible. For his reason, in order o ge a more visual crierion, we have weighed he dynamics by a funcion of he region size. Grey level Grey level (a) Funcion s dynamics (b) Noisy funcion s dynamics Fig. 1. Dynamics of funcion minima. Some segmenaions a differen resoluions, based on a conras-size crierion are shown in fig D segmenaion In order o reach a ime sabiliy, he successive frames of a sequence are segmened as a 3D volume [9]. In order o avoid unaccepable ime delays, he ime deph of he 3D volume has been reduced o 2 frames(previous and curren frame); his means, a sliding window hrough he sequence [10]. A 3D segmenaion algorihm ransforms a sequence ino a pariion of he 3D space. Conours and exure of each region are subsequenly coded. Coding cos of a region is minimum if i can be prediced (only a moion vecor and predicion

3 MORPHOLOGICAL SEGMENTATION OF IMAGE SEQUENCES (a) Filered image (b) Segmened(75 reg) (c) Segmened(150 reg) (d) Filered image (e) Segmened(150 reg) (f) Segmened(250 reg) Fig. 2. 2D segmenaions. errors have o be coded). This is he case for any region keeping is label from he previous frame. On he oher hand, conours and exure of new regions have o be compleely coded, which is much more expensive. Therefore, o reach a high compression rae, segmenaion algorihms have o favor ime coninuiy acceping new regions only when hey are really imporan Maximum sabiliy We firs rea he ideal case, in which all regions are already presen in he firs frame and have been properly segmened in he inra frame mode. Thus, in iner frame mode, no furher feaure exracion is required. The previous segmenaion works as a se of markers (pu in he firs frame of he sliding window) which invade he curren frame. The decision sep, performed by he waershed algorihm, works on a gradien image in order o locae he conours of he seleced markers. The gradien of hin objecs does no conain conour informaion (see fig. 3). In sill images he filering sep removes he objecs under he gradien resoluion. In video sequences his problem worsens. The uncerainy of spaial conours is proporional o he objec moion (fig. 4). For his reason we perform he waershed on a ime upsampled gradien (fig. 5) which allows us o locae correcly he spaial conour. Taking he previous segmenaion as a se of markers, a waershed on a 3D ime upsampled gradien, produces he conours we are looking for. The image of foreman wih a smooh moion (fig. 6) is segmened in fig. 7. If moion is faser, some objecs may no be conneced wih heir markers. Those objecs disappear in he segmenaion. (The algorihm processes a disconneced objec as wo differen objecs). In fig. 8 we have an example of his siuaion. The ping-pong ball disappears when he moion becomes faser. A high ime dam in

4 B. MARCOTEGUI AND F. MEYER ORIGINAL DILATION EROSION GRADIENT = DILATION - EROSION Fig. 3. Gradien of a hin objec. UNCERTAINTY OF SPATIAL CONTOUR UNCERTAINTY OF SPATIAL CONTOUR ORIGINAL DILATION EROSION GRADIENT = DILATION - EROSION Fig. 4. 3D gradien of a moving objec. he gradien prevens he marker of he disappeared region from reaching he nex frame. This algorihm, inrinsically fails when a new objec appears; he only markers we have are hose of he previous frame. On he oher hand, he algorihm is perfecly able o remove from he segmenaion an objec ha has disappeared. Therefore he nex problem o solve is he inclusion of new regions Inclusion of new regions The algorihm presened in he previous secion offers a maximum sabiliy; i mainains permanen objecs in he scene as long as possible, unil hey evenually vanish. Unable o le new regions appear, i has o be compleed by a procedure for inroducing new regions. Performing a feaure exracion for each frame would inroduce highly unsable segmenaions, since he ranking of regions according o heir visual significance would flucuae one image o anoher. Sabiliy will be inroduced by a hree sep procedure: SPATIAL CONTOUR SPATIAL CONTOUR ORIGINAL DILATION EROSION GRADIENT = DILATION - EROSION Fig. 5. 3D ime upsampled gradien.

5 MORPHOLOGICAL SEGMENTATION OF IMAGE SEQUENCES Fig. 6. Filered image sequence Fig. 7. Segmened image sequence 1. independen feaure exracion. 2. validaion of new markers. 3. merging new and old markers. This procedure is hereafer explained in deail. 1. A feaure exracion (wih a conras-size crierion, as presened in secion 2) selecs he objecs visually imporan. This selecion is performed wihou aking previous segmenaions ino accoun, and is resul is aken as a provisional se of markers. Some of hem already exis in he previous frame and hey have no o be considered as new markers. Ohers are markers of new significan regions in he scene and have o be kep. The res are ranking flucuaions and we wan o rejec hem, in order o keep a good sabiliy. 2. We wan o inroduce new regions only when hey are jusified by an imporan change in he scene. This crierion has been implemened by means of a moving mask, which removes hose markers ha are associaed o regions ha have no changed in ime. The moving mask is generaed as follows: he difference beween wo successive simplified frames is compued. Moving areas appear wih a high grey level. The feaure exracion sep compues a conras hreshold, ha corresponds o he required conras of a region in order o be considered visually imporan. This hreshold is applied o he difference image producing a mask of changing areas. Only markers under his moving mask will be kep as markers of new regions.

6 B. MARCOTEGUI AND F. MEYER (a) Filered image sequence (b) Segmened image sequence Fig. 8. No new regions. 3. Old and new markers are merged ino a 3D image; in he firs frame he previous segmenaion (as markers of permanen objecs) and in he frame corresponding o he curren ime, markers seleced by he feaure exracion ha are under he moving mask (as markers of new regions). Afer he feaure exracion sep, he decision sep is performed in a 3D ime upsampled image as explained in he previous secion. Fig. 9 conains a block diagram of his echnique. Wih his echnique a new region appears only when an imporan change jusifies i. In fig. 10 a segmened image including new regions is presened. ORI SIM GRA PROVISIONAL MARKERS GRA FINAL MARKERS SIM -1 DIF MOVING MASK -1-1 NEW MARKERS SEG SIM conras hreshold SEG -1 Fig. 9. Block diagram of inclusion of new regions Regulaion of he segmenaion For a given compression rae he qualiy should be as high as possible. If moion in he sequence is predicable from he previous frames a fine segmenaion is afforded. On he oher hand if new significan regions appear all he ime he accuracy of

7 MORPHOLOGICAL SEGMENTATION OF IMAGE SEQUENCES Fig. 10. Inclusion of new regions. he segmenaion have o be reduced. Therefore a mechanism should be provided enabling o inroduce more regions upon reques if predicable moion allows i or o merge hem in order o assign bis for new significan regions. Several segmenaions followed by coding sep should be necessary before an opimum is reached. Complexiy and processing ime forbid his opion. Anoher soluion is o produce during he segmenaion a more complee represenaion, such as a ree srucure [11]. Each node of he ree represens a region linked by an edge o an ancesor region conaining i and by wo or several edges o daugher nodes represening all regions conained in i. Going down in he ree means merging regions, going up means spliing regions. Producing he bes segmenaion hen simply resuls in cuing he ree a he level producing he required bisream. For generaing he iniial ree in inra mode, he ree of increasing dynamics is a good candidae. We are currenly developing algorihms for updaing i in ime: his means ranking he fusions according o heir cos/benefi in iner mode and providing a mechanism for inroducing new regions. In order o evaluae he sabiliy of ree represenaions we have made he following experience. We consider he simple siuaion where no new regions appear. We consruc from he firs inra frame a ree of fusions and derive from i wo segmenaions: a fine segmenaion (fig. 11) included in a coarse one(fig. 12). A 3D ime recursive segmenaion as presened in secion 3.1 is applied, one using he fine segmenaion and he oher he coarse one; we obain a fine and a coarse segmenaions of he whole sequence. We hen apply o all frames of he fine segmenaion he same fusions of labels as defined in he iniial inra frame; his gives us again a coarse segmenaion. The coarse segmenaion obained by boh mehods are he same. This shows ha our ree represenaion is indeed sable. Fig. 11. Fine segmenaion.

8 B. MARCOTEGUI AND F. MEYER Fig. 12. Coarse segmenaion. 4. Conclusion A morphological segmenaion algorihm for image sequences has been presened in his paper. Good sabiliy is reached, including new regions only when an imporan change in he scene jusifies i. A way o regulae he segmenaion accuracy is inroduced. In fuure work we inend o generae a ree srucure wih a crierion aking ino accoun no only visual significance bu also coding cos. In ha way we expec o maximize he qualiy/cos rae of he sysem. References 1. M. Kun, A. Ikonomopoulos, and M.Kocher, Second-generaion image coding echniques, Proc. IEEE, vol. 73, April J. Serra, Image analysis and mahemaical morphology. Academic Press, S. Beucher, Segmenaion d images e morphologie mahémaique. PhD hesis, Ecole des Mines de Paris, Juin P. Salembier and J. Serra, Morphological muliscale image segmenaion, in Visual Communicaions and Image Processing, vol. 1818, SPIE, L. Vincen, Grayscale area openings and closings, heir efficien implemenaion and applicaions, in Mahemaical Morphology and is Applicaions o Signal Processing, (Barcelona), Mai S. Beucher and C. Lanuéjoul, Sur l uilisaion de la ligne de parage des eaux en déecion de conours, Rappor Inerne N-598, Ecole des Mines de Paris, Cenre de Morphologie Mahémaique, Mai F. Meyer and S. Beucher, Morphological segmenaion, Journal of Visual Communicaion and Image Represenaion, vol. 1, pp , Sepembre M. Grimaud, La géodésie numérique en morphologie mahémaique. Applicaion à la déecion auomaique de microcalcificaions en mammographies numériques. PhD hesis, Ecole des Mines de Paris, Décembre F. Friedlander, Le raiemen morphologique d images de cardiologie nucléaire. PhD hesis, Ecole des Mines de Paris, Décembre M. Pardas, P. Salembier, and L. Torres, 3d morphological segmenaion for image sequences, in IEEE Winer Workshop on Nonlinear Signal Processing, (Tampere), F. Meyer, Arbre des minima e dynamique, rappor inerne, Ecole des Mines de Paris, Cenre de Morphologie Mahémaique, mars 1990.