Main Subject Detection via Adaptive Feature Selection

Transcription

1 Main Subject Detection via Adaptive Feature Selection Cuong Vu and Damon Chandler Image Coding and Analysis Lab Oklahoma State University

2 Main Subject Detection is easy for human 2

3 Outline Introduction Algorithm Results and Analysis Conclusion and Future work 3

4 Outline Introduction Algorithm Results and Analysis Conclusion and Future work 4

5 Outline Introduction Algorithm Results and Analysis Conclusion and Future work 5

6 Outline Introduction Algorithm Results and Analysis Conclusion and Future work 6

7 Outline Introduction Algorithm Results and Analysis Conclusion and Future work 7

8 Introduction Photographers convey their ideas in photos via one or more main subjects. Human visual system pays more attention to some parts of images. Applications of Main Subject Detection (MSD) Auto cropping a photo Image compression Unequal error protection Object recognition Etc. 8

9 Previous work Algorithms to detect the main subject in an image o Y. Ma et al., Contrast-based image attention analysis by using fuzzy growing, in MULTIMEDIA 03, New York, USA, pp , ACM, o J. Luo et al., A computational approach to determination of main subject regions in photographic images, Image and Vision Computing, vol. 22, no. 3, pp , o T. Liu et al., Learning to detect a salient object, IEEE Conference on Computer Vision and Pattern Recognition, pp. 1-8, o R. Achanta et al., Frequency-turned Salient Region Detection, IEEE Conference on Computer Vision and Pattern Recognition,

10 Previous work Algorithms predict visual fixation o L. Itti et al., A model of saliency-based visual attention for rapid scene analysis, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 20, no. 11, pp , o O. L. Meur et al., A coherent computational approach to model bottom-up visual attention, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 28, no. 5, pp ,

11 Outline Introduction Algorithm Results and Analysis Conclusion and Future work 11

12 Algorithm Adaptive Feature Selection & Weighted Sum Feature Measurement Multi stage Refinement Saliency Map & Bounding Box 12

13 Outline Introduction Algorithm o Feature Measurement Results and Analysis Conclusion and Future work 13

14 Features Sharpness Color Distance Lightness Distance Contrast Edge Strength 14

15 Original image block 24 pixels overlap Sharpness [1] Spectral sharpness map S 1 Final sharpness map S 3 Geometric mean Convert to gray scale image 8 8 block No overlap Spatial sharpness map S 2 [1] C. T. Vu, and D. M. Chandler, S3: A Spectral and Spatial Sharpness Measure, IEEE Intl. Conference on Advances in Multimedia,

16 Color Distance (1) Original image (2) Convert to a*, b* Measure dist. from avg. block a*, b* to avg. bkgnd a*, b* (3) 8x8 blocks w/ 50% overlap Local color dist. map 16

17 Lightness Distance (1) Original image (2) Convert to L* Measure dist. from avg. block L*, to avg. bkgnd L* (3) 8x8 blocks w/ 50% overlap Local lightness dist. map 17

18 RMS Contrast (1) Original image (2) Convert to luminance Measure avg. block RMS contrast (3) 8x8 blocks w/ 50% overlap Local contrast map 18

19 Edge Strength (1) Original image (2) Run Robert s edge detector Measure avg. block edge pixels (3) 8x8 blocks w/ 50% overlap Local edge strength map 19

20 Outline Introduction Algorithm o Feature Measurement o Adaptive Feature Selection Results and Analysis Conclusion and Future work 20

21 21

22 22

23 23

24 24

25 25

26 26

27 27

28 Adaptive feature selection Sharpness Contrast Lightness Color Edge 28

29 Adaptive feature selection Sharpness Contrast Lightness Color Edge 29

30 Adaptive feature selection How to measure the usefulness of a feature? Sharpness Contrast Lightness Color Edge 30

31 Cluster distance Clustered Non-clustered The more clustered, the better!!! 31

32 Saliency region, includes P pixels Distance d j A saliency pixel Centroid Cluster Distance β = d j α P 32

33 Cluster distance Low cluster distance High cluster distance Cluster Distance = Cluster Distance =

34 Outline Introduction Algorithm o Feature o Adaptive Feature Selection o Multi-stage Refinement Results and Analysis Conclusion and Future work 34

35 35

36 Sharpness Stage 1: Measure Features Lightness Distance Contrast Color Distance Spatial weighting Edge Strength 36

37 Sharpness Stage 1: Measure Features Lightness Contrast Color Spatial weighting High cluster distance Low cluster distance Edge Strength 37

38 Stage 1 map Stage 1 rectangle Saliency pixels: Greater than T = 1.5 mean(map) 38

39 Stage 2: Refine Features Sharpness Lightness Distance To the new background Contrast Background Color Distance To the new background Spatial weighting Edge Strength 39

40 Sharpness Stage 2: Refine Features Lightness Contrast Color Spatial weighting Edge Strength 40

41 Stage 2 map Stage 2 rectangle Saliency pixels: Greater than T = 2 mean(map) Take only 75% of saliency pixels 41

42 Sharpness Stage 3: Refine Features Again Lightness Distance To the new foreground Contrast Foreground Color Distance To the new foreground Spatial weighting Edge Strength 42

43 Stage 3: Refine Features Again (Distance to the foreground) Sharpness Lightness Contrast Color Spatial weighting Edge Strength 43

44 Final map Final rectangle Saliency pixels: Greater than T = 1.5 mean(map) 44

45 Outline Introduction Algorithm Results and Analysis Conclusion and Future work 45

46 Original image Ground truth First stage map Final map MSD Result 46

47 MSD Results on Objects 47

48 MSD Results on Plants 48

49 MSD Results on People 49

50 MSD Results on Animals 50

51 MSD Failure Cases 51

52 Image testing sets MSRA database [2] : 5000 images (set B) with bounding box groundtruth EPFL database [3] : 1000 images from MSRA database with object-contour based groundtruth [3] [2] T. Liu et al., Learning to detect a salient object, IEEE Conference on Computer Vision and Pattern Recognition, [3] R. Achanta et al., Frequency-turned Salient Region Detection, IEEE Conference on Computer Vision and Pattern Recognition,

53 Evaluation Criteria Boundary-based Criterion: BDE: Boundary Displacement Error [4] between the boundaries of Detected and Groundtruth rectangles Region-based Criteria Precision = Recall = F-measure = A( D G) A( D) A( D G) A( G) (1 + α) P α P + R R 53 [4] J. Freixenet et al., Yet another survey on image segmentation: Region and boundary information intergration, In ECCV, pages , 2002.

54 Overall Results MSRA database BDE Better Y. Ma et al.[5] L. Itti et al.[6] T. Liu et al.[2] Our algorithm Fixed weights First stage map [5] Y. Ma et al., Contrast-based image attention analysis by using fuzzy growing, in MULTIMEDIA 03, New York, USA, pp , ACM, [6] L. Itti et al., A model of saliency-based visual attention for rapid scene analysis, IEEE Transactions on Pattern Analysis and Machine 54 Intelligence, vol. 20, no. 11, pp , 1998.

55 Overall Results MSRA database Better Precision Recall F-Measure 0 Y. Ma et al.[5] L. Itti et al.[6] T. Liu et al.[2] Our algorithm Fixed weights First stage map 55

56 Overall Results EPFL database 1 Better Precision Recall F-measure 0 Y. Ma et al.[5] L. Itti et al.[6] J. Harel et al.[7] X. Hou et al.[8] R. Achanta et al.[3] AFS AFS_MS [7] J. Harel et al., Graph-based visual saliency, Advances in Neural Information Processing Systems, 19: , [8] X. Hou et al., Saliency detection: A spectral residual approach, IEEE Conference on Computer Vision and Pattern Recognition,

57 Average weighs of features Sharpness Lightness Distance Contrast Color Distance Edge Strength Stage Stage Stage Overall

58 Conclusion and Future work Relatively simple low-level features can be effective for MSD if they are combined in an adaptive and multi-stage fashion. Our results are very promising. This adaptive feature selection can be a useful strategy for other applications. Future works: Improve the algorithm: new feature, using segmentation, etc Multiple MSD 58

59 Thank you! {cuong.vu,

60 Stage 1: Measure Features Sharpness ω = 1 Lightness Distance ω = 1/3 Contrast Color Distance Spatial weighting Edge Strength ω = 2/3 60

61 Stage 2: Refine Features Sharpness ω = Lightness Distance Contrast Color Distance Background Spatial weighting Edge Strength ω = 1 61

62 Stage 3: Refine Features Again Sharpness ω = 0.95 Lightness Distance Contrast ω = 1 Foreground Color Distance Spatial weighting Edge Strength 62