A490 Machine Vision and Computer Graphics

Transcription

1 A490 Machine Vision and Computer Graphics Lecture Week 11 (More on Convolutions and Introduction to Boundary, Shape and Skeletal Models) November 6, 2012 Sam Siewert

2 More on Convolution FCG Chapter 9 - Final Words Sam Siewert 2

3 Audio and PCM 1D compared to 2D Image Processing in terms of Convolutions Applied Useful for Digital Media Systems VoIP Digital Cable Infotainment and Mobile Media One of the PIDs in an MPEG Transport Stream (Audio and Video PIDs) Sam Siewert 3

4 Digital Audio Analog to Digital Encoding PCM Pulse Code Modulation ADC Sample Rate (e.g. 6 to 48KHz per Channel) 8-bit, 16-bit and 20-bit Monotone 8/16/20-bit Stereo (Right and Left Channel) Dolby Surround Sound Channel Mappings.1 Refers to Low Frequency Speaker 2.x, 3.x, 5.x, 7.x Speaker Placement Compression and Encoding Standards MP3 MPEG-1, Audio Layer 3 (Lossy PCM Encoded Data Compression) AC-3 - Audio Codec #3 (Dolby Digital), Multi-Channel Compression Transport Audio Elementary Stream in MPEG Packets (188 Byte) Multiplexed with Video in UDP or RTP Transport Stream Sam Siewert 4

5 Basic Voice/IP Setup Four Tasks (Services) On Each Phone Host Record, Play-back Streaming, Transport Two Phone Hosts for Point-to-Point Phone Call Record Buffer Audio ADC Codec Data ( 8 - bit Mono, 16 - bit Stereo, etc.) Record Half Buffer Audio Source ( Microphone, MP 3 ) waveform Sound Card ADC / Codec / DMA Codec Sample IRQ Record Sound Bite Message Data Audio Streaming Transport Data Sound Bite Messages tnettask UDP or TCP Speaker s waveform Sound Card DMA / Codec / DAC Playback Control Playback Sound Bite Remote Message Playback Half Buffer Audio Codec DAC Data Playback Buffer Sam Siewert 5

6 Point-to-Point VOIP Host - 1 Replicate Services on Each Host Audio Source ( Microphone, MP 3 ) waveform Record Buffer Audio ADC Codec Data ( 8 - bit Mono, 16 - bit Stereo, etc.) Record Half Buffer Sound Card ADC / Codec / DMA Codec Sample IRQ Record Sound Bite Message Data Audio Streaming Transport Data Sound Bite Messages tnettask ( UDP or TCP ) Establish Control and Data Transport Speaker ( s ) waveform Sound Card DMA / Codec / DAC Playback Control Audio Codec DAC Data Playback Host - 2 Sound Bite Remote Message Playback Half Buffer Playback Buffer Ethernet LAN Record Buffer Either Side Should Be Able to Initiate or Answer a Call Audio Source ( Microphone, waveform MP 3 ) Speaker ( s ) waveform Audio ADC Codec Data ( 8 - bit Mono, 16 - bit Stereo, etc.) Sound Card ADC / Codec / DMA Sound Card DMA / Codec / DAC Codec Sample IRQ Playback Control Record Playback Record Half Buffer Sound Bite Message Data Sound Bite Remote Message Audio Streaming Transport Data Sound Bite Messages tnettask ( UDP or TCP ) Playback Half Buffer Audio Codec DAC Data Playback Buffer Sam Siewert 6

7 Notes on Discrete Convolution a[1,-1] a[0,-1] a[-1,-1] a[1,0] a[0,0] a[-1,0] a[1,1] a[0,1] a[-1,1] Sam Siewert 7

8 Anti-aliasing Removal of stair-step look to diagonal lines in graphics Blurring and filling in Looks Better Interesting Discussion of Sampling Fourier Transform Sequence of Sine waves to produce square wave f*g (convolution of f and g) transformed to frequency space is same as transform of f and g followed by convolution Useful to Understand the Frequency Components of Sampling for 1D time series and 2D images Ties back to Fundamentals of Application of the DCT to a Macro-block Aliasing Can Come from Sampling and Filtering and smoothing functions can anti-alias Sam Siewert 8

9 Boundary, Shape, Skeletal Models CMV Chapters 9 & 10: Steps Toward Object Recognition Sam Siewert 9

10 Start with Threshold Assumption Assume that Threshold transforms image to 1 s on background of 0 s, or PBM from PGM Define What Connected Means Assertion that Foreground is 8-connected (all nearest neighbors) and Background is 4- connected (only orthogonal neighbors) x 0 x x 0 x Sam Siewert 10

11 Label Connected Regions (Skeletal Model Concepts) Raster Image with Definition of Connectedness for FG and BG Uniquely Assign ID to Each Connected Region Filter Regions Below a Size Threshold Skeleton is the Medial Line Along Limbs or Regions E.g. A GENERALIZED MORPHOLOGICAL SKELETON TRANSFORM AND EXTRACTION OF STRUCTURAL SHAPE COMPONENTS, by Jianning Xu, Computer Science Department, Rowan University Glassboro, NJ, IEEE ICIP Sam Siewert 11

12 Skeletal Models for Motion Tracking Like Fiducial Markers on Robotic System, used to Track Limbs (Kinematics, inverse Kinmatics for rigid bodies) E.g. for human figures - Sam Siewert 12

13 Skeletal Models to Robotics Useful for Basic Gesture Recognition Requires Clear Separation of FG/BG Works Better with Stable Camera (Stereo) Motion in Near Foreground E.g. Kinect Light Coding for Image-Based 3D Reconstruction 20 Joints per Player, 6 people in theory 640x480 RGB, 8-bit SDK Popularized by Willow Garage and ROS Sam Siewert 13

14 Shape Analysis Use of Distance Functions and Filling/Thinning Sam Siewert 14

15 Distance Metric for 8-Connected Foreground d 8 = max( x i x j, y i y j ) Computes Distance From Edge Such that Center of Object Has Highest Value From Computer and Machine Vision, E.R. Davies, page 241. Sam Siewert 15

16 Method to Compress Objects Object Shapes Described by Max Distance Values At Medial Locations Compressed Form, Can be Expanded by Propagation to Min Distance Edge When Rendered See Page 243, Figure 9.7 Sam Siewert 16

17 Flood Fill Fills in Shape Between Defined Edges 4-Connected vs. 8-Connected Target Color (Shade) and Replacement Color (Shade) are used with Starting Pixel and Algorithm Runs Recursive E.g. 4-Connected Flood Fill From Wikipedia Flood-fill (node, target-color, replacement-color): 1. If the color of node is not equal to target-color, return. 2. Set the color of node to replacement-color. 3. Perform Flood-fill (one step to the west of node, target-color, replacement-color). Perform Flood-fill (one step to the east of node, target-color, replacement-color). Perform Flood-fill (one step to the north of node, target-color, replacement-color). Perform Flood-fill (one step to the south of node, target-color, replacement-color). 4. Return. Sam Siewert 17

18 Thinning (Skeletons) Useful for Character Recognition and Gesture Recognition Eliminates Redundant Shape Information and Preserves Topological Information Only (a Form of Compression) Connected Set of Medial Lines Along Limbs Fig 9.8 Mathematical Idealized Medians Fig 9.9 From Thinning Algorithm From Computer and Machine Vision, E.R. Davies, page 246. Sam Siewert 18

19 Chi Function Used To Determine Which Pixels Can Be Removed and Which Must Be Retained If Pixel is on Boundary, it Can Be Removed and Chi=2, Sigma!= 1 Sigma is Sum of Neighbors (Where FG=1, BG=0) Chi is a Complex Conditional Expression to Encode When Pixel of Interest is On An Edge and Can Be Removed Note Edge Conditions to Understand Chi Line End N edge E edge S edge W edge NW edge SE edge Sam Siewert 19

20 Thinning Algorithm Chi Consider All Cases of Proper Edges Where Pixel Can Be Thinned (Chi=2) and Sigma!= 1 Chi = (A1!= A3) + (A3!= A5) + (A5!= A7) + (A7!= A1) + 2*((A2 > A1) && (A2 > A3)) + ((A4 > A3) && (A4 > A5)) + ((A6 > A5) && (A6 > A7)) + ((A8 > A7) && (A8 > A1)) E.g. Here are Some Non-Edges to Retain Isolated Chi=4 Chi=8 Sam Siewert 20

21 Combined Distance and Thinning for Guided Compute Maximum Distance and Thin Better for Parallel Speed-up (P. 252) From Computer and Machine Vision, E.R. Davies, page 252. Sam Siewert 21

22 Boundary Patterns Toward Feature Vectors and Invariants Sam Siewert 22

23 Centroidal Profile Find Centroid Based on Threshold and Pixel COM (As Discussed Before) Define Shape In Terms of Radius as a Function of Angle Around Centroid (Shape Signature) From Computer and Machine Vision, E.R. Davies, page 271. Sam Siewert 23

24 Centroidal Profile Problems Notched or Partially Occluded Objects Computationally Complex Must Have Accurate Centroid First Suggested for Inspection Applications (Within Shape Tolerances?) From Computer and Machine Vision, E.R. Davies, page 272. Sam Siewert 24

25 Alternative Strategies Determine Invariant Signature Feature R, theta Plot is a Signature (Centroidal Profile), Consider Alternative Shape Invariants E.g. Boundary / Perimeter Length Symmetry SIFT Scale Invariant Feature Transform (David Lowe, 1999) Keypoint Analysis (Extracted from Reference Images and Stored in Database) Compare Each Feature from Test Image to References Candidates that Match Based on Euclidian Distance Measure of Feature Vector Becomes a Search Problem Sam Siewert 25

26 Keypoint Concepts Inspired Based on Observations of Human Recognition Processes E.g. Saccades quick eye movement tracked when humans are presented images to recognize (e.g. a face) Centroid of pupil Key Features are Clearly Traced Sam Siewert 26

27 Feature Vectors Any N dimensional vector that represents and object A form of compression for the reference database and faster compare Vector and Distance Metric Definitions are Key, But Once Defined, Simple Distance Measures Can Be Applied Corners of Objects Radial Distance Profile (Centroidal) Various Histograms Derived from Scans with Binning (Blobs) Sam Siewert 27

28 Feature Vectors Unique Patterns That Define Objects of Interest How to Generalize Keypoints of Feature Vectors is the Challenge Bio-inspired (Human Observation) is A Promising Approach, but Search is Costly! Operation in Real-Time? PCA Principal Component Analysis Sam Siewert 28

29 N Dimensional Distance Harder to Visualize than 3D Vector Differences, but Same Concept as Euclidian Quick Review of Euclidian One Dimension: abs(p 1 p 2 ), or Sqrt((p 1 p 2 ) 2 ) Two Dimension: Sqrt((p 1 -q 1 ) 2 + (p 2 q 2 ) 2 ) Three Dimension: Sqrt((p 1 -q 1 ) 2 + (p 2 q 2 ) 2 + (p 3 - q 3 ) 2 ) N Dimension: Sqrt((p 1 -q 1 ) 2 + (p 2 q 2 ) (p n - q n ) 2 ) Sam Siewert 29

30 Other Distance Measures Defines the Difference Between {A} and {B} in General with a Metric (Number) Euclidian Distance is One Such Metric Manhattan Distance is Another (Sum of Orthogonal Line Segments on a Grid Between 2 Points) Hamming Distance is Another - # of Positions at Which 2 Equal Length Strings Have Different Symbols E.g. 1111_0000 and 0000_1111 have maximum distance or toned and roses has distance of 3 in ASCII Sam Siewert 30