Unsupervised color film restoration using adaptive color equalization

Transcription

1 Unsupervised olor film restoration using adaptive olor equalization A. Rizzi 1, C. Gatta 1, C. Slanzi 1, G. Cioa 2, R. Shettini 2 1 Dipartimento di Tenologie dell Informazione Università degli studi di Milano - Polo di Crema via Bramante, 65, Crema (CR), Italy {rizzi,gatta}@dti.unimi.it slanzi@rema.unimi.it 2 DISCo (Dipartimento di Informatia, Sistemistia e Comuniazione), Edifiio 7 Università degli Studi di Milano-Bioa, Via Bioa degli Arimboldi 8, Milano, Italy {ioa, shettini}@diso.unimib.it Abstrat. Chemial proessing of elluloid based inemati film, beomes unstable with time, unless they are stored at low temperatures. Some defets, suh as bleahing on olor movies, are diffiult to solve using photohemial restoration methods. In these ases, a digital restoration tool an be a very onvenient solution. Unfortunately, for old movies olor and dynami range digital restoration is usually dependent on the skill of trained tehniians who are able to ontrol the parameters through olor adjustment, and may be different for a sequene or group of frames. This leads to a long and frustrating restoration proess. As an alternative solution, we present in this paper, an innovative tehnique based on a model of human olor pereption:, to orret olor and dynami range with no need of user supervision and with a very limited number of parameters. The method is ombined with a tehnique that is able to split the movie into different shots and to selet representative frames (key frames) from eah shot. By default, key frames are used to set the olor orretion method parameters that are then applied to the whole shot. Due to the robustness of the olor orretion method the setting used for the key frame is used suessfully for all the frames of the same shot. 1. Introdution Movie hemial materials are the result of a hemially unstable proess, and is subjet to fading with time. This fading is irreversible and in several ases photohemial restoration of faded prints is risky and not always possible. In these ases, digital olor restoration an solve the problem. In this paper, we propose a tehnique for olor digital restoration of faded movies based on a pereptual approah, inspired by some adaptation mehanisms of the human visual system (HVS), in partiular, lightness onstany and olor onstany. The lightness onstany adaptation enables pereption of the sene regardless of hanges in the mean luminane intensity and the olor onstany adaptation enables pereption of a sene regardless of hanges in olor of the illuminant.

2 2 A. Rizzi1, C. Gatta1, C. Slanzi1, G. Cioa 2, R. Shettini2 Restoring film fading and/or olor bleahing an be seen as a problem of hromati noise removal, suh as olor onstany mehanisms [1][2]. Consequently an algorithm is hosen for digital images that performs unsupervised enhanement, alled ACE (Automati Color Equalization) [3][4]. It provides experimental evidene in an automati orretion of the olor balane of an image. Although the number of ACE parameters is very small and their tuning not ritial, their setting an vary widely aording to the image ontent and to the kind of final rendering hosen by the film diretor (e.g. low or high key, artisti olor distortion, et..). 2. Towards unsupervised restoring parameters tuning To implement a standard tuning proedure, we need to extrat a set of still images (key frames) that summarize the video ontent in a rapid and ompat way. Different methods an be used to selet key frames. In general these methods assume that the video has already been segmented into shots by a shot detetion algorithm, and extrat the key-frames from within eah shot deteted. One of the possible approahes to key frame seletion is to hoose the first frame in the shot as the key frame [5]. Ueda et al [6] and Rui et al. [7] use the first and last frames of eah shot. Other approahes inlude time sampling of shots at regular intervals. As an alternative approah [7], the video is time sampled regardless of shot boundaries. In [8][9] the entire shot is ompated into a small number of frames, grouping onseutive frames together, or taking frames from a predefined loation within the shot. Other approahes, suh as [10][11], ompute the differenes between onseutive frames in a shot using olor histograms, or other visual desriptions, to measure the visual omplexity of the shot; the key frames are seleted by analyzing the values obtained. In [12][13] the frames are lassified in lusters, and the key frames are seleted from the larger lusters, or by hierarhial lustering redution. The drawbaks to most of these approahes is the number of representative frames must be fixed in some a priori method, for example, depending on the length of the video shots. This annot guarantee that the seleted frames will not be highly orrelated. It is also diffiult to set a suitable interval of time, or frames: large intervals mean a large number of frames will be hosen, while small intervals may not apture enough representative frames, or those hosen may not be in the right plaes. We apply here a new algorithm that dynamially selets a variable number of keyframes depending on the shot s visual ontent and omplexity. After the extration of key frames, these images are used as a set for the parameter tuning of ACE, the hosen algorithm for olor orretion. By default the key frames are used to set the olor orretion method parameters, whih are then applied to the whole shot. Due to the robustness of the olor orretion method the setting used for the key frames is used suessfully for all the frames of the same shot.

3 Unsupervised olor film restoration using adaptive olor equalization 3 3. ACE: Automati Color Equalization ACE is an algorithm for unsupervised enhanement of digital images. It is based on a omputational approah that merges the "Gray World" and "White Path" equalization mehanisms, while taking into aount the spatial distribution of olor information. Inspired by some adaptation mehanisms of the human visual system, ACE is able to adapt to widely varying lighting onditions, and to extrat visual information from the environment effiiently. The implementation of ACE follows the sheme as shown in Fig. 1: first stage: hromati spatial adaptation (responsible for olor orretion); and seond stage: dynami tone reprodution saling, to onfigure the output range, and implement aurate tone mapping. No user supervision, no statistis and no data preparation are required to run the algorithm. Fig. 1. ACE basi sheme. In Fig. 1 I is the input image, R is an intermediate result and O is the output image; subsript denotes the hromati hannel. The first stage, the Chromati/Spatial adaptation, produes an output image R in whih every pixel is reomputed aording to the image ontent, approximating the visual appearane of the image. Eah pixel p of the output image R is omputed separately for eah hromati hannel as shown in equation (1). R( p) = j Im, j p r j Im, j p ( I( p) I( j) ) d( p, j) Ymax d( p, j) (1) Fig. 2 displays the used r( ) funtion.

4 4 A. Rizzi1, C. Gatta1, C. Slanzi1, G. Cioa 2, R. Shettini2 Fig. 2. r( ) funtion. The seond stage maps the intermediate pixels array R into the final output image O. In this stage, a balane between gray world and white path is added, saling linearly the values in R with the following formula O ( p) = round[127.5 s R ( p)] (2) + where s is the slope of the segment [(m,0),(m,255)], with M m = = max[ R ( p)] p min[ R ( p)] p using M as white referene and the zero value in R as an estimate for the medium gray referene point to ompute the slope s. A more detailed desription of the algorithm an be found in [3][4]. The appliation of ACE for movie restoration, is not a straight forward proess; several aspets have been modified or introdued in order to fulfill the tehnial needs of the film restoration field. (3) 4. Color frame restoration The prinipal harateristi of ACE is its loal data driven olor orretion; ACE is able to adapt to unknown hromati dominants, to solve the olor onstany problem and to perform an image dynami data driven strething. Moreover, ACE algorithm is unsupervised and needs little involvement of the user. These properties make it suitable for film restoration, a problem in whih usually there is no referene olor to ompare the results of the filtering, subjetivity is used to determine the pleasantness and naturalness of the final image. Faded movie images are dull, have poor saturation and an overall olor ast. This is due to the bleahing of one or two hromati layers of the film. Sine it is neessary to deal with lost hromati information, restoring the olor of faded movies is more omplex than a simple olor balane. The tehnique, presented here, is not just an

5 Unsupervised olor film restoration using adaptive olor equalization 5 appliation of ACE on movie images, but an enhanement of ACE priniples to meet the requirements of digital film restoration pratie. ACE is used to remove possible olor asts, to balane olors and to orret ontrast of every single frame of the movie. This preliminary tool does not use any inter-frame orrelation to improve its performane. This will be a subjet for future researh. In this instane, ACE parameters have to be properly tuned and new funtions have been added to ahieve image naturalness, to preserve the natural histogram shape and to add new funtions for the restoration proess. These new funtions an obtain satisfatory results even though the input frame is exessively orrupted. The new funtions are: Keep Original Gray (KOG): This funtion is devised to relax the GW mehanism in the seond stage. Instead of entering the hromati hannels around the medium gray, keep original gray funtion preserves the original mean values (independently in R, G and B hannels). This results in histograms more similar in shape with the original. Relaxing GW mehanism in the seond stage does not affet the ACE olor onstany property, ahieved in the first stage. This prevents to muh modifiation of a low or high key images. This funtion is also important for the fade-in and fade-out sequenes. Keep Original olor Cast (KOC):In some instanes, to reate a more artisti quality, film diretors use unnatural olor in a sequene, (e.g. Nosferatu, 1922 direted by F.W. Murnau 1 ). Even though the first stage of ACE removes the olor ast, an estimation an first be undertaken of the olor ast and then replae bak the olor ast at the seond stage. Keep Original Dynami Range (KODR): Sometimes film diretors use a limited dynami range of the film to obtain speifi visual effets and low or high key pitures. In these ases, the use of KODR respets the original intention of the diretor. This funtion an also be used to manage frames that are exessively orrupted. In Summary, ACE parameters an be tuned to ahieve : The SLOPE of the r( ) funtion: greater the slope the more the final ontrast. KOG: retains the original mean lightness of the frame. KOC: retains the original olor ast. KODR: retains the original dynami range. All these parameters need to be set aording to the harateristis of eah single shot. To omplete a restoration of the entire movie, a set of representative frames need to be extrated. 1 [ ] Many senes featuring Graf Orlok were filmed during the day, and when viewed in blak and white, this beomes extremely obvious. This potential blooper??is orreted when the "offiial" versions of the movie are tinted blue to represent night. [ ] from Internet Film Database (

6 6 A. Rizzi1, C. Gatta1, C. Slanzi1, G. Cioa 2, R. Shettini2 5. Choosing the representative frames The video is segmented into shots (a ontinuous sequene of frames taken over a short period of time) by deteting abrupt hanges and fades between them, sine these are more ommon than other editing effets. For abrupt hanges, a threshold-based algorithm is implemented, oupled with a frame differene measurement omputed from histograms and textures desriptors. To detet fades a modified version of the algorithm proposed by Fernando et al. [14] is implemented. The results obtained by these algorithms are submitted for evaluation to a deision module, whih gives the final response. This opes with onfliting results, or with groups of frames that are not meaningful, suh as those between the end of a fade-out and the start of a fade-in, by inreasing the robustness of the detetion phase. A gradual transition detetion algorithm is urrently being developed; it will be integrated in a similar manner. The key-frames seletion algorithm proposed, dynamially selets the representative frames by analyzing the omplexity of the events depited in the shot. The frame differene values initially obtained are used to onstrut a umulative graph that desribes how the frames visual ontent hanges over the entire shot, an indiation of the shot s omplexity: sharp slopes indiate signifiant hanges in the visual ontent due to a moving objet, amera motion or the registration of highly dynami event. These ases are onsidered interesting event points that must be taken into aount in seleting the key frames to inlude in the final shot summary. Event points are identified in the umulative graph of ontiguous frame differenes by seleting those points at the sharpest urve of the graph (urvature points). The representative frames are those orresponding to the mid points between eah pair of onseutive urvature points. In more detail: Three different desriptors are omputed: a olor histogram, an edge diretion histogram, and wavelet statistis. The use of various visual desriptors provides a more preise representation of the frame and aptures small variations between the frames in a shot. The olor histogram used is omposed of 64 bins determined by sampling groups of meaningful olors in the HSV olor spae [15]. The edge diretion histogram is omposed of 72 bins orresponding to intervals of 2.5 degrees. Two Sobel filters are applied to obtain the gradient of the horizontal and the vertial edges of the luminane frame image. These values are used to ompute the gradient of eah pixel and those pixels that exhibit a gradient over a predefined threshold are taken to ompute the gradient angle and then the histogram. Multiresolution wavelet analysis an provide information about the overall texture of the image at different levels of detail. At eah step in the multiresolution wavelet analysis four sub-images (or sub-bands) are obtained with the appliation of a low-pass filter (L) and high-pass filter (H) in the four possible ombinations of LL, LH, HL and HH. These bands orrespond to a smoothed version of the original image (the LL band) and the three oeffiient matries of details (the LH, HL and HH bands). We apply the multiresolution wavelet analysis on the luminane frame image, using three-step Daubehies multiresolution wavelet expansion to produe ten sub-bands. Two energy features, the mean and the variane, are omputed on eah sub-band, resulting in a 20- valued desriptor.

7 Unsupervised olor film restoration using adaptive olor equalization 7 To ompare two frame desriptors a differene measure is used to evaluate the olor histograms, wavelet statistis and edge histograms. The differene between two olor histograms (d H ) is based on the intersetion measure. The differene between two edge diretion histograms (d D ) is omputed using the Eulidean distane as suh in the ase of two wavelet statistis (d W ). The three resulting values are then ombined to form the final frame differene measure (d HWD ) as follow: d HWD ( d d ) + ( d d ) + ( d d ) = (4) H W W Signifiant differenes are obtained only if at least two of the single differenes exhibit high values. By weighing eah differene against the other, the measure detets signifiant hanges, ignoring the smaller differenes due to the moving amera, or aquisition and ompression noise. If we were to use, for example, only the olor histogram, a highly dynami frame sequene (i.e. one ontaining fast moving or panning effets) but with the same olor ontents, would result in a sequene of small frame differene values. On the ontrary, if the frame sequene ontained a flash, or some olor effets, the orresponding frame differene value would be greater than the real ontents of the sequenes all for. With the use of three desriptors, only if the hanges in the sequene are signifiant in terms of olor, texture and overall edges, will d hwd result in high values. The algorithm proposed by Chetverikov et al. [16] is used to detet the high urvature points. The algorithm defines as a orner a loation where a triangle of speified size and opening angle an be insribed in a urve. Using eah urve point P as a fixed vertex point, the algorithm tries to insribe a triangle in the urve, and then determines the opening angle in orrespondene of P. Different triangles are onsidered using points that fall within a window of a given size entered in P, and the sharpest angle, under a predefined threshold θ max, is retained as a possible high urvature point. Finally, those points in the set of andidates high urvature points that are sharper than their neighbors (within a ertain distane) are lassified as high urvature points. The algorithm does not require proessing the whole video, unlike some earlier methods that extrat, for example, key frames based on the length of the shots. Another advantage of our algorithm is that it an be easily adapted in order to extrat the key frames on-the-fly: to detet a high urvature point we an limit our analysis to a fixed number of frame differenes within a predefined window. Thus the urvature points an be determined while omputing the frame differenes, and the key frames an be extrated as soon as a seond high urvature point has been deteted. It must be noted that the first and last frames of the shot are impliitly assumed to orrespond to high urvature points. If a shot does not present a dynami behavior, i.e. the frames within the shot are highly orrelated, the graph does not show evident urvature points, signify that the shot an be summarized by a single representative frame: The middle frame in the sequene is hosen as the key frame. D D H

8 8 A. Rizzi1, C. Gatta1, C. Slanzi1, G. Cioa 2, R. Shettini2 6. Experimental results In this setion results are presented with omments regarding the ACE parameter tuning. All the pitures are key frames seleted with the previously desribed method. We also present results on different key frames of the movie La iudad en la playa (direted by Ferruio Musitelli) an Uruguayan short movie of the 60 s. Color ast removal: Fig. 3a shows a frame from La iudad en la playa with a bluish olor ast. Fig. 3b shows the ability of ACE filtering to remove the olor ast without a priori information. ACE eliminates the bluish olor ast restoring the olor of the wall inside the window. Fig. 3. ACE removal of the olor ast. a) Original frame from La iudad en la playa. b) Frame filtered with SLOPE=2.5 Controlling the ontrast tuning the slope: Fig. 4 shows the effet of ACE filtering with different values of SLOPE. Fig. 4a shows a frame without a strong olor asts. Fig. from 4b to 4d show ACE filtering results of Fig. 4a, with inreasing SLOPE value (1, 2, 2.5). As it an be notie the ontrast inrease as the SLOPE inrease. Fig. 4. Effet of ACE filtering. a) Original frame from La iudad en la playa. b) Frame filtered with SLOPE=2.

9 Unsupervised olor film restoration using adaptive olor equalization 9 ) d) Fig. 4. (ont.) Effet of ACE filtering. ) Frame filtered with SLOPE=2.5. d) Frame filtered with SLOPE=5. Keeping seleted properties of the input image: Fig. 5 shows the behavior of the KOC funtion. The algorithm hanges ontrast and mean lightness keeping its olor ast. Fig. 5. Behavior of the KOC funtion. a) Original frame from Nosferatu. b) Frame filtered with SLOPE=10 and KOC. Fig. 6 shows the behavior of the KOG and the KODR funtions. Fig. 6b shows the filtering without any funtions. While KOG (Fig. 6) keeps the same mean lightness, KODR (Fig. 6d) keeps the dynami range and thus the reddish olor ast.

10 10 A. Rizzi1, C. Gatta1, C. Slanzi1, G. Cioa 2, R. Shettini2 Fig. 6. Behavior of the KOG and the KODR funtions. a) Original frame from La iudad en la playa. b) Frame filtered with SLOPE=3.3. ) Frame filtered with SLOPE=3.3 and KOG. d) Frame filtered with SLOPE=3.3 and KODR. Finally, Fig. 7 shows the appliation of ACE on a blak and white film in order to restore its original dynami range. The example is from TOM TIGHT ET DUM DUM by Georges Méliès (1903). Fig. 7. Appliation of ACE on a blak and white film. a) Original frame from Tom tight et dum dum. b) : Frame filtered with SLOPE=20.

11 Unsupervised olor film restoration using adaptive olor equalization 11 More Examples Fig. 8. Examples of key frames filtering. On the left (a,, e) the original key frames from La iudad en la playa. On the right (b, d, f) the filtered key frames.

12 12 A. Rizzi1, C. Gatta1, C. Slanzi1, G. Cioa 2, R. Shettini2 7. Conlusions This paper has presented a tehnique for digital olor and dynami range restoration of faded movies ombining a method that identifies different shots and automatially selets key frames from a movie. The olor and dynami range restoration uses an unsupervised olor equalization algorithm, based on a pereptual approah. To meet the requirements in the digital restoration field new funtions have been added and results are satisfatory and suggest potential researh. The objetive for future researh will onsider the problem of ACE parameter fine tuning, by investigating improvements to the speed of film proessing and optimizing and aelerating the algorithm. Aknowledgments The authors want to thank Ferruio Musitelli for the permission to use the film and his preious and enouraging friendship. Referenes [1] M. Chambah, B. Besserer, P. Courtellemont, Reent Progress in Automati Digital Restoration of Color Motion Pitures, SPIE Eletroni Imaging 2002, San Jose, CA, USA, January 2002, vol. 4663, pp [2] M. Chambah, B. Besserer, P. Courtellemont, Latest Results in Digital Color Film Restoration, Mahine Graphis and Vision (MG&V) Journal, Vol. 11 no. 2/3, [3] A. Rizzi, C. Gatta, D. Marini, A New Algorithm for Unsupervised Global and Loal Color Corretion, Pattern Reognition Letters, Vol 24 (11), pp , July [4] A. Rizzi, C. Gatta, D. Marini, From Retinex to Automati Color Equalization: issues in developing a new algorithm for unsupervised olor equalization, Journal of Eletroni Imaging, Vol 13 (1), pp , January [5] Y. Tonomura, A. Akutsu, K. Otsugi, and T. Sadakata, VideoMAP and VideoSpaeIon: Tools for automatizing video ontent, Pro. ACM INTERCHI 93 Conferene, pp , [6] H. Ueda, T. Miyatake, and S. Yoshizawa, An interative natural-motion-piture dediated multimedia authoring system, Pro. ACM CHI 91 Conferene, pp , [7] Y. Rui, T. S. Huang and S. Mehrotra, Exploring Video Struture Beyond the Shots, appeared in Pro. of IEEE Int. Conf. on Multimedia Computing and Systems (ICMCS), Texas USA, [8] F. Arman, A. Hsu and M.Y. Chiu, Image Proessing on Compressed Data for Large Video Databases, Pro. ACM Multimedia '93, Annaheim, CA, pp , 1993.

13 Unsupervised olor film restoration using adaptive olor equalization 13 [9] A. Pentland, R. Piard, G. Davenport and K. Haase, Video and Image Semantis: Advaned Tools for Teleommuniations, IEEE MultiMedia 1(2), pp , [10] S. Han, K. Yoon, and I. Kweon, A New Tehnique for Shot Detetion and Key Frames Seletion in Histogram Spae. Pro. 12 th Workshop on Image Proessing and Image Understanding, [11] A. Hanjali, R. Lagendijk, and J. Biemond, A New Method for Key Frame based Video Content Representation. Pro. First International Workshop on Image Databases and Multimedia Searh, [12] Zhuang Y., Rui Y., Huang T.S., Mehrotra S.: Key Frame Extration Using Unsupervised Clustering, in Pro. of ICIP 98, Chiago, USA, [13] A. Girgensohn, J. Borezky, Time-Constrained Keyframe Seletion Tehnique, Multimedia Tools and Appliation, vol. 11, pp , [14] A. C. Fernando, C. N. Canaharajah, D. R. Bull, Fade-In and Fade-Out Detetion in Video Sequenes Using Histograms, Pro. ISCAS 2000 IEEE International Symposium on Ciruits and System, IV , May 28-31, Geneva, Switzerland, [15] G. Cioa, I. Gagliardi, R. Shettini, Quiklook 2 : An Integrated Multimedia System, International Journal of Visual Languages and Computing, Speial issue on Querying Multiple Data Soures, Vol. 12, pp , [16] D. Chetverikov and Zs. Szabo, A Simple and Effiient Algorithm for Detetion of High Curvature Points in Planar Curves, Pro. 23rd Workshop of the Austrian Pattern Reognition Group, pp , 1999.