Machine Vision for Railroad Track Inspection

Transcription

1 Slide 1 Machine Vision for Railroad Track Inspection Esther Resendiz* Esther Resendiz Luis Fernando Molina, John M. Hart* J. Riley Edwards, Christopher P. L. Barkan, Narendra Ahuja* Railroad Engineering Program and *Beckman Institute University of Illinois at Urbana-Champaign

2 Slide 2 Overview Overview of applications for machine vision in railroad inspection Railroad track inspection background Machine vision for defect analysis and turnout detection in railroad track inspection Signal processing motivation One-dimensional periodic object detection Two-dimensional periodic object detection Future direction Conclusion

3 Slide 3 Overview Overview of applications for machine vision in railroad inspection Railroad track inspection background Machine vision for defect analysis and turnout detection in railroad track inspection Signal processing motivation One-dimensional periodic object detection Two-dimensional periodic object detection Future direction Conclusion

4 Slide 4 Overview of Applications for Machine Vision in Railroad Inspection Undercarriage Inspection of Passenger Railcars Structural Underframe Inspection of Railcars

5 Slide 5 Overview of Applications for Machine Vision in Railroad Inspection Intermodal Freight Train Monitoring Safety Appliance Inspection Hemispherical Video Camera

6 Slide 6 Overview Overview of applications for machine vision in railroad inspection Railroad track inspection background Machine vision for defect analysis and turnout detection in railroad track inspection Signal processing motivation One-dimensional periodic object detection Two-dimensional periodic object detection Future direction Conclusion

7 Slide 67 Initial Inspection Focus Cut spikes Missing Raised Inappropriate patterns Rail anchors Missingi Shifting Inappropriate p patterns

8 Slide 78 Current Inspection Focus - Turnout General Switch Frog Missing bolts Missing cotter pins Distance of switch points from stock rail Worn or broken switch points Worn or broken frog points

9 Slide 89 Experimental Data Acquisition System Power source Lateral Cameras Track Cart Over-the-rail Computer

10 Slide 910 System Overview - Over-the-rail View Used for measuring the distance between the spike head and the base- of-rail and verifying spiking patterns Measure the distance between switch points and stock rail, and identify worn or broken frog points

11 Slide System Overview - Lateral View Optimal view for measuring the distance between the rail anchors and the edge of the ties and verifying i anchor patterns Also used to Identify missing bolts/cutter pins and worn or broken switch points in turnouts

12 Slide 12 Railroad Track Viewpoints Lateral Over-the-rail

13 Slide 13 Overview Overview of applications for machine vision in railroad inspection Railroad track inspection background Machine vision for defect analysis and turnout detection in railroad track inspection Signal processing motivation One-dimensional periodic object detection Two-dimensional periodic object detection Future direction Conclusion

14 Slide 14 Track Components Ballast, rail, tie plate, and tie Anchor, spike, and tie plate holes

15 Slide 15 Track Component Isolation Delineate ties, ballast, and rail using edge detection Original image Edge image (base-of-rail highlighted) Incorporate texture classification in areas surrounding edge detection Makes component detection robust Tie patches Ballast patches Steel patches

16 Slide 16 Base-of-Rail Delineation The original image, where we will isolate the base of the rail

17 Slide 17 Base-of-Rail Delineation Strong horizontal edges are detected (candidate edge is shown)

18 Slide 18 Base-of-Rail Delineation Surrounding pixels above and below candidate edge are examined

19 Slide 19 Base-of-Rail Delineation Ballast below edge Ballast below edge Base of the rail is confirmed

20 Slide 20 Defect Detection Results Detect tie plates using similar edge/texture method Tie with shifted anchor

21 Slide 21 Defect Detection Results Individual components are located and measured with spatial template and prior knowledge Tie with shifted anchor

22 Slide 22 Defect Detection - Shifted Anchor Anchor displacement detected Tie with shifted anchor

23 Slide 23 Track Panorama Visualization of cumulative track components using panoramic stitching Tie / Tie plate delineation on test panorama Component detection on test panorama

24 Slide 24 Over-the-Rail Video Over-the-rail video

25 Slide 25 Over-the-Rail Component Detection

26 Slide 26 Overview Overview of machine vision applications for railroad inspection Railroad track inspection background Machine vision for defect analysis and turnout detection in railroad track inspection Signal processing motivation One-dimensional periodic object detection Two-dimensional periodic object detection Future direction Conclusion

27 Slide Lateral Tie Detection GOAL: isolate frames that contain a tie in the center of the frame Lateral video Frame 86 Frame 113 Frame 140

28 Slide Lateral Tie Detection Convert the image into a sinusoidal signal that represents a tie centered in the current frame Convert each frame into a binary mask that classifies ballast and non-ballast image patches White: Ballast Black: Non-ballast Oi Original i limage Binary mask

29 Slide Lateral Tie Detection Compare each frame s binary mask to a template mask abs Masked lateral video Plot the resulting signal and detect maximum points in the signal Tie Detection at x(10 4 ) 10 9 Frame Frame 55 Template mask Frame Frame Frame Frame Number

30 Slide 30 Over-the-Rail Tie Detection GOAL: Isolate each video frame that contains a tie centered at location w w w Over-the-rail video

31 Slide Over-the-Rail Tie Detection GOAL: Isolate each frame that contains a tie centered at location w. w w Frame 18 Frame 46 Frame 74 Frame 105 Convert the video into a sinusoidal id signal that t represents tie detection at w. abs White: Ballast, Black: Non-ballast

32 Slide 32 MUSIC Overview MUltiple SIgnal Classification (MUSIC) is a spectral estimation technique From a received signal, detect the composite frequencies 1 T = 8 f = 1/T = y Unit of Space or Time

33 Slide 33 MUSIC Overview Detect frequencies from signal y 2 T Power (d db) Frequency in radians ( )

34 Slide 34 Switch Area Detection GOAL: Detect a switch area in a video T =? T =? Lateral video Inspect for periodicity (T) in the components along the rail

35 Slide 35 Switch Area Detection (contd.) Convert video of the mid-rail area to a panoramic mosaic Panoramic mosaic Convert this mosaic to the Gabor frequency domain Gabor frequency of panoramic mosaic Perform spectral analysis on Gabor-transformed image to find T Gab bor Respo onse Gabor Block Number Spectral Estimation T

36 Slide 36 Legend Color corresponds to numerical value

37 Slide 37 Switch Area Detection (contd.) Classify frame as either closure area or switch area based on periodicity (T) Use edge-detection and template-based methods to find heel block joint bar and switch rod bolts Turnout detection video

38 Slide 38 Overview Overview of machine vision applications for railroad inspection Railroad track inspection background Machine vision for defect analysis and turnout detection in railroad track inspection Signal processing motivation One-dimensional periodic object detection Two-dimensional periodic object detection Future direction Conclusion

39 Slide 39 Detect and Segment Periodic Objects GOAL: Detect periodic components along a panoramic inspection image Panorama formed from acquired video Assume objects only occur along one dimension (horizontal) Inspection panorama from lateral view

40 Slide 40 Image Model Represent each panoramic image by its rows, where each row consists of N pixel-rows I T T T x ) [ I ( x ) I ( x ) I ( x ) ] ( 1 2 R I 1( x ) I ( x 2 ) I R (x) Row-wise inspection panorama Each row analyzed independently

41 Slide 41 Image Model Represent each panoramic image by its rows, where each row consists of N pixel-rows I 1( x ) I ( x 2 ) I T T T x ) [ I ( x ) I ( x ) I ( x ) ] ( 1 2 R I R (x) Row-wise inspection panorama Each row is further decomposed and processed as blocks N N I r ( 1 x) [ b0 ( x) b ( x) bm ( x)]

42 Slide 42 Block-wise Gabor Each block undergoes G-dimensional blockwise Gabor transform Let G j ( I r ( x)) represent the blockwise Gabor transform of row I r (x) using the jth Gabor filter Color/illumination proportional to value in feature space (x) I r G ( I ( x )) 1 r x 12 3 M Block-wise Gabor decomposition of row (one of G dimensions)

43 Slide 43 Block-wise Gabor Each block undergoes G-dimensional Gabor transform Color/illumination proportional to value in feature space I (x) I r 12 3 M 1 4 G 2( I x r ( )) 3 r x Block-wise Gabor decomposition of row (one of G dimensions)

44 Slide 44 Row-wise Signal Each block-wise Gabor row is a received signal Each block b m contains y(m) M Alternating 0/1 (with T=8) blocks

45 Slide 45 MUSIC Overview MUSIC = MUltiple SIgnal Classification From a received signal, detect the composite frequencies 1 T = 8 f = 1/T = y Unit of space (m)

46 Slide 46 MUSIC Overview Frequencies are detected from signal y 2 T Power (d db) Frequency in radians ( )

47 Slide 47 Results - Track bolts Original panorama Labeled periodicities

48 Slide 48 Results - Track bolts Labeled periodicities Detected bolts (Object 7A)

49 Slide 49 Results Primary Periodicity Original panorama Labeled periodicities in rows

50 Slide 50 Results Primary Component Labeled periodicities in rows Tie Plate (Object 8A) Anchors (Object 6B) Ties (Object 10B)

51 Slide 51 Results Primary Component Labeled periodicities in rows Left-side Tie Plate (7A) Lower Tie Plate (9B)

52 Slide 52 Secondary Periodicities Labeled periodicities in rows Object 7B Object 7A

53 Slide 53 Secondary Periodicities Labeled periodicities in rows Object 6A Object 10B

54 Slide 54 Overview Overview of applications for machine vision in railroad inspection Railroad track inspection background Machine vision for defect analysis and turnout detection in railroad track inspection Signal processing motivation One-dimensional periodic object detection Two-dimensional periodic object detection Future direction Conclusion

55 Slide 55 Repeating Objects in Images 2D GOAL: Detect and segment objects that repeat along g j p g arbitrary directions, and are of an arbitrary size

56 Slide 56 Method for Detecting and Segmenting Repeating Objects in Images Filter images with different Gabor filters Project these filtered images along various orientations Detect one-dimensional periodicity for each orientation Extract the spatial frequencies containing object and create a new image Segment the object from new image

57 Slide 57 Gabor Filters Gabor filters are defined by and B : Direction of oriented Gabor filter =0 = /4 = /2 = 3 /4 B : size of Gabor filter Height=B B pixels, Width=B B pixels

58 Slide 58 Legend Color corresponds to numerical value

59 Slide 59 Gabor-transformed Images Original Image G G B (x,y) Image Filtered with = 0, B=64 G B (x,y) Image Filtered with = /2, B=64, B ( x, y) I( x, y ) g, B ( x x, y y ), x y

60 Slide 60 Periodicity Orientation direction of periodic repetition =0 = /4 = /2 = 3 /4

61 Slide 61 Detect and Segment Repeating Objects in Example Image Apply our method (Filter, Project, Detect, Extract, and Segment) on an example image Example image

62 Slide 62 Filter by, Project by Apply Gabor filters at various orientations ( ) and sizes (B) Original Image Rotate each by all orientations G B, ( x, y) G B (x,y) ( = 3 /4, B=64) G B (x,y) ( = 0, B=64) = 3 /4, B=64, =0 = 3 /4, B=64, = /4 Sum across rows to form projection signals = 3 /4, B=64, = /2 = 3 /4, B=64, = 3 /4

63 Slide 63 Project by Signal formed from projection Signal analyzed with MUSIC to estimate T s, B, ( t) G, B ( x, y ) ( t x cos y sin ) dx dy Amplitude measured at T intervals to form a quality metric Q,B B Accumulate measures for all (, ) L(, ) Q, B, B,

64 Slide 64 Detect with Mapping L(, ) ) Orienta ations of Gabor (0 to ) = = = = 3 = =0 = /4 = /2 =3 /4 Direction of Periodicity (0 to )

65 Slide 65 Extract Spatial Frequency Spatial Spatial frequencies frequencies similar to dissimilar to Image with repeating component Image without repeating component

66 Slide 66 Segment Repeating Object Gradient detection Works well for narrow objects (e.g. line-like) Works well for objects where and are separated by /2 Block-wise detection (along ) Works well for larger objects Works well for arbitrary directions Gradient detection Block-wise detection

67 Slide 67 Iterative Demonstration Original image L( )

68 Slide 68 Extract Spatial Frequency Spatial frequencies similar to Spatial frequencies dissimilar to Image with repeating component Image without repeating component

69 Slide 69 Segment Object Gradient detection Image with repeating component Block-wise detection

70 Slide 70 Iterate on Image without Repeating Component Image without repeating component L( )

71 Slide 71 Extract Spatial Frequency Spatial frequencies similar to Spatial frequencies dissimilar to Image with repeating component Image without repeating component

72 Slide 72 Segment Object Gradient detection Image with repeating component Block-wise detection

73 Slide 73 Iterate on Image without Repeating Component Image without repeating component L( )

74 Slide 74 Extract Spatial Frequency Spatial frequencies similar to Spatial frequencies dissimilar to Image with repeating component Image without repeating component

75 Slide 75 Segment Object Gradient detection Image with repeating component Block-wise detection

76 Slide 76 Overview Overview of applications for machine vision in railroad inspection Railroad track inspection background Machine vision for defect analysis and turnout detection in railroad track inspection Signal processing motivation One-dimensional periodic object detection Two-dimensional periodic object detection Future direction Conclusion

77 Slide 77 Future Direction With more development, algorithm may be used to Adjust to new track conditions in an unsupervised manner Incorporate domain knowledge and operate in a semi- supervised manner Algorithm could be modified to Detect symmetry Determine vanishing point along horizon Other infrastructure inspection may benefit from the current work

78 Slide 78 Overview Overview of applications for machine vision in railroad inspection Railroad track inspection background Machine vision for defect analysis and turnout detection in railroad track inspection Signal processing motivation One-dimensional periodic object detection Two-dimensional periodic object detection Future direction Conclusion

79 Slide 79 Conclusion Machine vision is useful for defect detection in railroad track inspection The regular structure of the railroad track makes it ideal for signal processing-based methods Signal processing algorithms provide an efficient method for processing images and video Promising results provide inspiration for future work

80 Slide 80 Thank you!