How to Detect Moving Shadows: Theory and Practice

Transcription

1 How to Detect Moving Shadows: Theory and Practice Andrea Prati ImageLab * D.I.I. Università di Modena e Reggio Emilia * Staff: Prof. Rita Cucchiara (head), Grana Costantino, Vezzani Roberto

2 Introduction and motivations Theory of the shadows Proposed taxonomy Relevant approaches Performance evaluation metric Summary How to select? Conclusion and open discussion

3 Introduction (1) Moving object detection is a key task in many computer vision related topics Video-surveillance Traffic analysis Data compression (MPEG) Indoor scene analysis and understanding Intelligent vehicles

4 Introduction (2) What s wrong here?

5 Introduction (3) Motivations of shadow detection Segmentation without shadow suppression Shadow detection (white=shadow) Segmentation with shadow suppression

6 Introduction (4) Motivations of shadow detection Object points and shadow points share two important visual features: motion detectability

7 Introduction (5) Motivations of shadow detection Shadow suppression is essential if further steps are required, as: tracking classification modeling object counting E.g., in traffic-related applications, the systems typically aim at computing:» queues length» turning rates» lane occupancy» vehicles speed» traffic flow

8 Theory of the shadows (1) Physical explanation Shadows are due to the occlusion of the source light by an object in the scene still shadows shadows shadows moving shadows umbra self-shadow penumbra

9 Theory of the shadows (2) Physical explanation The photometric reflectance model resulting from the Kubelka theory 1 is: s x E x x x E x x 2 k( λ, ) = ( λ, )(1 ρf( )) ρλ (, ) + ( λ, ) ρf ( ) where: x is the position in the image plane E(?,x) is the irradiance? f (x) is the Fresnel reflectance?(?,x) is the material reflectance (or reflectivity) 1 P. Kubelka. New contribution to the optics of intensely lightscattering materials. Part I. J. Opt. Soc. Am., 38(5): , 1948

10 Theory of the shadows (3) Physical explanation For a white illumination, the irradiance is constant over the wavelengths: s x Ex x x x 2 k( λ, ) = ( ){ ρf( ) + (1 ρf ( )) ρλ (, )} For dull surfaces the Fresnel reflectance can be approximated as zero 2 : s ( λ, x ) = Ex ( ) ρλ (, x ) k We will consider a single wavelength, i.e.: s ( x ) = Ex ( ) ρ( x ) k 2 J. Geusebroek, A.W.M. Smeulders, R. van den Boomgaard. Measrument of color invariants. in Proc. of CVPR 2000, vol. 1, pp

11 Theory of the shadows (4) Physical explanation We assume that the light source is far from the object, the distance between light source and surface S is constant, the light source emits parallel light rays and the observation point is fixed. Using the Phong s model we can write the irradiance as: A + c cos ( N ( x, E ( x, y) = c + k( x, y) c cos ( N( x, y), L) k c c A A P P y), L) if if if illuminated penumbra umbra where c A and c P are the intensity of the ambient light and of the source light, and k(x,y) is the softening introduced by penumbra

12 Theory of the shadows (5) Shadow can be detected by means of background differencing D ( x, y) = s ( x, y) s0( x, y) k k - the reflectance is supposed constant D D ( x, y) = ρ( x, y)( E ( x, y) E0( x, y)) k k - in the case of umbra ( x, y) = ρ( x, y) c cos ( N( x, y), L) k P if the light source is strong General framework Shadow can be also detected by means of frame ratio computation FRk ( x, y) = the reflectance is supposed constant FRk ( x, y) = The surface normal N is supposed fixed in a neighborhood s k ( x, y) s0( x, y) E ( x, y) E0( x, y) FR k ( x, y) = f ( ca, c P ) k Difference in presence of umbra will be high! The frame ratio is supposed to be locally constant in presence of shadows

13 Proposed taxonomy (1)

14 Proposed taxonomy (2)

15 Proposed taxonomy (3) Selection of the features Features to be selected in shadow detection approaches are numerous To have a valuable taxonomy we selected the more relevant ones: Spectral: using or not color can dramatically change the performance Spatial: trade-off between accuracy and time performance Temporal: as above, and additional delay in the response. Typically not applicable in on-line applications

16 Statistical nonparametric (Snp) Proposed taxonomy (4) Selection of the approaches Statistical parametric (Sp) Deterministic non-model (Dnm1) Deterministic non-model (Dnm2) A. Elgammal,, D. Harwood,, and L.S. Davis, Non-parametric model for R. Cucchiara,, C. Grana, G. Neri, M. Piccardi, background subtraction, in A. Prati, The Sakbot system for moving Proceedings of IEEE ICCV 99 object detection and tracking,, in Video Video- FRAME-RATE RATE Workshop, based Surveillance Systems - Computer Vision and Distributed Processing, Chapter 12, pp , 158, eds. Kluwer Academic Publishers,. I. Mikic,, P. Cosman,, G. Kogut,, and M.M. Trivedi, Moving shadow and object detection in traffic scenes, in Proceedings J. of Int l Conference on Pattern Recognition, Stauder,, R. Mech,, and J. Ostermann, Detection Sept of moving cast shadows for object segmentation, IEEE Transactions on Multimedia, vol. 1, no. 1, pp , Mar

17 Proposed taxonomy (5) Selection of the approaches Why these approaches? Snp: selected cause one of the more cited (W 4 ) system among the statistical non-parametric approaches to shadow detection Sp: selected because it utilizes features from all three domains (spectral, spatial and temporal) Dnm1: because is the only one that uses HSV color space for shadow detection Dnm2: is one of the few approaches that deals with penumbra Not easy to obtain code from someone else!!

18 Relevant approaches (1) Statistical non-parametric (Snp) approach E s [ µ R ( i), µ G ( i), B ( i) ] [ σ ( i), σ ( i), σ ( i) ] i = µ i = R G B

19 Relevant approaches (2) Statistical non-parametric (Snp) approach Key features of this algorithm: color exploitation automated parameter selection very fast algorithm due to code optimization

20 Relevant approaches (3) Statistical parametric (Sp) approach Pixel (x,y) features: v = R G B [ ] T A priori p(c 1 ),p(c 2 ),p(c 3 ) BG SH FG p ( C v) i yes = j p( v Ci) p p( v C ) p i j p( C i v) Iterations completed? no classification C = arg max p ( C ) ( C v) i i ( C ) Key features of this algorithm: color exploitation spatial propagation fast implementation The pixel (x,y) is assigned the membership probabilities based on the probabilities of its neighbors p(c 1 ),p(c 2 ),p(c 3 ) good discrimination between shadow and objects j

21 Relevant approaches (4) Deterministic nonmodel (Dnm1) based approach - SAKBOT

22 Relevant approaches (5) Deterministic non-model (Dnm1) based approach - SAKBOT Shadow is a semi-transparent region in the image, which retains a representation of the underlying surface pattern, texture or color value and decreases its brightness

23 Relevant approaches (6)

24 Relevant approaches (7)

25 Relevant approaches (8)

26 Relevant approaches (9) Deterministic non-model (Dnm1) based approach - SAKBOT Changing parameters can result in different performance

27 Relevant approaches (10) Deterministic non-model (Dnm1) based approach - SAKBOT

28 Relevant approaches (11) Deterministic non-model (Dnm1) based approach - SAKBOT

29 Relevant approaches (12) Deterministic nonmodel (Dnm1) based approach - SAKBOT Key features of this algorithm: HSV color space exploitation ghost and ghost shadow management good detection performance

30 Relevant approaches (13) Deterministic non-model (Dnm2) based approach Key features of this algorithm: penumbra detection spatial reasoning completeness

31 From 3 : Evaluation metric (1) How to evaluate performance? 1. Visually judge the automatic segmentations directly or comparing them with manual segmentations of experts 2. Show images and their segmentation accompanied by validating comments like good results, quite acceptable, 3. Only express the difficulty of evaluation, without an explicit evaluation itself A straightforward evaluation results is possible only if the purpose of the segmentation process is well defined. 3 K.L. Wincken et al., Model-based evaluation of image segmentation methods, Performance Characterization in Computer Vision, eds. Kluwer Academic Publ.

32 Evaluation metric (2) How to evaluate performance? An evaluation method should be: independent of the purpose of segmentation objective, i.e. independent of human factors not restricted to images of a particular dimension or type They 3 proposed to evaluate performance in terms of post-processing editing effort needed to obtain a satisfactory segmentation 3 K.L. Wincken et al., Model-based evaluation of image segmentation methods, Performance Characterization in Computer Vision, eds. Kluwer Academic Publ.

33 Evaluation metric (3) How to evaluate performance? Typical approach for evaluating performance of segmentation is the use of ground-truths. They can be generate mainly with three approaches 4 : by using synthetic data: the problem is that it will probably not faithfully represent the full range of real data by manually generating ground-truths: since manual mark-up is tedious and time consuming large volumes of ground truthed data are likely to have errors. by directly evaluating the segmentation by a human panel (actually no ground-truths is provided): very time consuming and difficulty in incorporating new algorithms 4 P.L. Rosin, E. Ioannidis, Evaluation of Global Image Thresholding for Change Detection, unpublished

34 Evaluation metric (4) To evaluate the four algorithms we use two classes of metrics Shadow detection accuracy Shadow discrimination accuracy Quantitative metrics TP F is the number of ground truth points minus the number of points detected as shadow but belonging to foreground Qualitative metrics robustness to noise flexibility to shadow object independence scene independence computational load detection of special shadow (ghost shadows, indirect cast shadows, penumbra)

35 Evaluation metric (5) Benchmark suite Object speed has been measure in average pixels per frame

36 Evaluation metric (6) Quantitative comparison Experimental results are obtained in two ways Manual ground truth segmentation of almost 20 frames for each sequence Extensive manual ground truth segmentation for one sequence (Intelligent Room - indoor)

37 Evaluation metric (7)

38 Evaluation metric (8)

39 Evaluation metric (9) Qualitative comparison Highway I outdoor sequence Visual result examples Intelligent room indoor sequence

40 How to select? (1) The best shadow detection algorithm depends on the application you have to deal with General-purpose application minimal assumptions more assumptions Dnm1 Specialized application Deterministic model-based too many object classes Dnm2 Indoor application Noisy sequence Statistical approaches Statistical approaches Dnm1

41 How to select? (2) Does this really work? NO! What are the key, physically important, independent variables upon which the performance of such algorithms should be evaluated? How might the experimental analysis be performed so that some future researchers, who wants to incorporate one or more such shadow detection algorithms into a larger system, could measure these variables in the environment in which the application is to be conducted and then choose the correct algorithm?

42 How to select? (2) Examples Restricting the analysis only to indoor environments under solely artificial illumination and identifying a taxonomy of illumination conditions (e.g., overhead fluorescent, overhead incandescent, single desk lamp, etc.), the approaches can be tested and evaluated in such conditions. Consequently, the selection of the best algorithm is more reliable In outdoor environments one might have stratified by surface type upon which the shadow is cast (grass, concrete, asphalt), as well as the illumination condition (diffuse, midday sun, early or late sun, etc.) Very onerous

43 Conclusion Metric refinement Even more extensive comparative evaluation Additional algorithms for comparative evaluation Multi-camera shadows related problem investigation Proposal of novel shadow detection techniques

44 Open discussion THANK YOU!! Visit us at: and Computer Vision and Robotics Research Lab University of California, San Diego (UCSD)