Rain Removal in a Video Sequence

Transcription

1 Li Hao Qi Yingyi Zhang Xiaopeng Presentation of CS4243 Project

2 Outline Objective 1 Objective 2 3

3 Objective of the project Dimming or removing the rain streaks in the video.

4 Our observations Objective It rains heavily or slightly. Rain streaks are in focus or out of focus. Scene object moves or not. Camera moves or not.

5 Videos for experiment Rain Camera Scene object motion Large Fixed Stationary Moving Moving Stationary Small Fixed Stationary Moving Moving Stationary

6

7 Why K-means works?

8 Workflow of the method The method is mainly divided into 4 steps:

9 Result Objective Result of a certain frame (a) Before removal (b) After removal

10 Method for dynamic video sequence

11 Overview Objective in Nayar s paper Detection Applying color constraint Removing detected rain streaks

12 Physical property of a raindrop Spherical shape Uniform istribution Speed: v = 200 a, where a is radius of raindrop.

13 Photometric model of rain Brightness of a stationary raindrop (c) The field of view of a raindrop (d) Experiment on the brightness of raindrops

14 Photometric model of rain Cont. Photometry of rain streaks Figure: The intensity change of a pixel due to a falling raindrop. I = βi bg + α, β = τ T, α = τēd

15 Dynamic model of rain Binary representation of pixels { 1, if drop projects to location r at time t b( r, t) = 0, otherwise Spatio-temproal correlation in a discrete space-time volume

16 Assumptions Objective Raindrops are assumed to be spherical and evenly distributed over space and time. Straight line motion of raindrops leads to high correlation of pixels along the rain streak over a period of time. The motion of the background is slow, so the irradiance of background is constant over the exposure time. The average irradiance of raindrop can be assumed to be constant for pixels that lie on the same streak, because the brightness of the drop is weakly affected by the background intensity.

17 Detection Nayar s method Method used in Nayar s paper I = I n I n 1 = I n I n+1 c (1) Figure: Nayar s detection method

18 Detection Nayar s method Cont. Detection results (a) Light rain (b) Detection (c) Heavy rain (d) Detection Limitation on large rain When it is a large rain, one pixel may be affected by different drops in two successive frames.

19 Detection Improved method Check the intensity changes of five successive frames. There would be three cases. Figure: Improved method

20 Detection Improved method Detection Result (a) Light rain (b) Detection (c) Heavy rain (d) Our Detection (e) Nayar s Detection

21 Apply Nayar s photometric constraint Reject those rain streaks which don t satisfy the linearity constraints of the photometric model: How to do it? Use Matrix! I = βi bg + α, where β [0, 0.039]

22 Apply Nayar s photometric constraint Each pixel p i in the component has I i and I bgi. Let I bg1 I 1 I bg2 I bg = I bg3 C A, I = I 2 I 3 C A.. Figure: One connected component So we can deduce a linear matrix equation as follows: «β ( I bg 1) = I α Then solve for the linear least squares solution «β = (A T A) 1 A T I where A = ( I α bg 1)

23 Apply Nayar s photometric constraint Cont. Rejection Result Problems: Most of the rain streaks do not satisfy the linear constraint, and β [0, 1]. Rain and non-rain areas have similar β and are very hard to be seperated. Figure: Result of photometric model

24 Problems with photometric constraint Example: I = I d I bg According to photometric constraint, β = I bg1 I bg2 I 1 I Therefore, β / [0, 0.039]. i.e. Photometric constraint does not hold.

25 Why? The model may be too general and ideal. The variation of size and velocity of raindrops will violate the general assumption. This model cannot distinguish rain and non-rain components if they are connected with each other.

26 Our method - Using color constraint Assume camera is static. In the R, G, B channels of a pixel, the values contributed by rain are more or less the same.

27 Our method - Using color constraint Cont. Reject the candidates which dissatisfy this constraint. For a detected candidate: 1 Let 2 Compute R = R R, G = G G, B = B B e = ( R G) 2 + ( R B) 2 + ( B G) 2 (2) 3 If e > ɛ, this candidate will be rejected.

28 Our method - Using color constraint Cont. Rejection result: Figure: Result after applying color constraint

29 Removal method in Nayar s paper For each pixel affected by rain in n th frame, Removal results: I n = I n 1 + I n+1 2 (3) Figure: Result of Nayar s removal method Artifacts: The edges of rain streaks are very sharp.

30 Our method of rain removal Cont. The flow chart of our removal method: Li Hao Qi Yingyi Figure: Zhangflow Xiaopeng chart Rain of our Removal method in a Video Sequence

31 Our method of rain removal Cont. Comparison with Nayar s: Figure: Comparison between ours and Nayar s

32 Appendix Reference I K. Garg and S.K. Nayar Detection and Removal of Rain from Videos Computer Vision and Pattern Recognition,2004. K. Garg and S.K. Nayar When Does a Camera See Rain? International Conference on Computer Vision, 2005