Automated Detection for Baseball Batter Analysis Using Image Processing and Segmentation Methods

Transcription

1 Automated Detection for Baseball Batter Analysis Using Image Processing and Segmentation Methods Jeremy Storer Bowling Green State University Department of Computer Science Bowling Green, Ohio Abstract This paper presents a method to segment a baseball, baseball bat, and waistline from an image of a baseball player swinging a bat using OpenGL. Several fundamental image processing techniques are demonstrated and used such as, thresholding, region growing, convolution matrices, and hough transforms. The three components were successfully found and displayed in several different images reliably. This is a task that in the future could be handled well by GPU processing. I. INTRODUCTION Image segmentation is the process of intelligently partitioning an image into a set of desired regions. Typically, these regions are used to represent meaningful areas of an image such as objects. Image segmentation has two primary objectives. The first objective is to decompose an image into parts for further analysis. In ideal cases, parameters are able to be found that can completely and reliably separate the desired objects out of their environment. However, the ideal case rarely occurs, which results in segmentation being a complex and often times difficult task. If, for example, the desired object is of similar shade, color, or pattern to its environment, it becomes difficult to extract it and a variety of filters and other image manipulation methods must be used. The second objective of image segmentation is to perform a change of representation. This means that the object of importance is found and highlighted so that it is easy to comprehend and identify for the user. This paper demonstrates the use of both of these objectives in the automated detection of a baseball, bat, and baseball player s waist orientation. A player was videotaped taking batting practice and then the video was made into a series of images that were used as the basis of creating a method that segments the image appropriately using OpenGL. A series of convolution matrices and methods such as thresholding, region growing, and hough transformations were used to assist in achieving the goal of detection. The reason for detecting the ball, waist orientation, and bat is to be able to assist players with their swing mechanics. If we can find these items we will be able to detect flaws in the batters swing by looking at the position of each item relative to one another. This will be able to show the player if the position of the bat or hips is at the wrong or right angle in relation to the ball. II. PREVIOUS WORK Ideas in how to get started and efficiently detect the required objects came from observing previous efforts in the field of image segmentation. Several papers were studied and common methods of image segmentation were extracted from them and studied to figure out how to apply them to the detection of the baseball items. In this section, fundamental image processing and image segmentation methods that were considered for this study are discussed. A. Thresholding The first item investigated was thresholding. Thresholding is the simplest of commonly used image segmentation techniques. The pixels of an image are partitioned depending on their intensity value and a new image is generated. A survey conducted by Mehmet Sezgin and Bulent Sankur shows a large amount of thresholding methods [7]. The most important method, in regards to this project, presented in the survey was a very simple and general global threshold algorithm. The selected threshold level, T, will create a binary image, g, such that: { 1, f(x, y) > T g(x, y) = (1) 0, f(x, y) T where x and y represent a pixel in x,y space. The idea is to find a T value that eliminates as much background data as possible and then assigning it a 0 value and the object a 1 value. This will result in an image of only the highlighted object. An example of how powerful thresholding can be is displayed in figure 1[4]. The left side of the figure shows a normal picture of an apple against a solid background. For this figure a T value was selected that had a lesser intensity than that of the background. This results in the binary image on the right side of the figure. Although this is an example of an ideal case, the power of thresholding can be clearly seen and is of importance to consider when attempting to detect a baseball, bat, and waist orientation of a baseball player during a swing.

2 Figure 1. The left image is unaltered. The right is the binary image after a threshold is applied. B. Convolution Matrices Convolution matrices allow various filters to be applied to an image resulting in various outputs depending on the content of the matrix applied. The filters are used in the process of identifying the image by locating sharp edges which are discontinuous. These discontinuities bring changes in pixels intensities which define the boundaries of the object. Shrivakshan and Chandrasekar used a convolution matrix to get an outline of a shark [6]. They used several different edge detection convolution matrices, one of which is the Sobel operator. They showed that the Sobel operator is the magnitude of the gradient computed by: where pixel[i, j] = S x = S y = S 2 xs 2 y (2) This operator emphasizes pixels that are closer to the center of the matrix and is one of the most commonly used edge detection methods. This work gave insight into how the outline of images can be created so that they can be easier to find after thresholding. C. Region Growing Region growing is a region-based image segmentation method that involves the selection of initial seed points. In region growing, a seed pixel is selected. Then, the neighboring pixels are checked and added to the region if they are declared similar enough to the seed. This is repeated until there are no neighboring pixels that fit the criteria of being similar enough to the seed. The idea behind this concept is that an object will have unique defining characteristics so when a seed is planted it will ideally be able to separate itself from its environment [1]. Region growing adds another method to our toolbox in the goal of segmenting the baseball related objects. (3) D. Template Matching Template Matching is a technique that identifies the parts of an image that match a predefined template. This technique is flexible and relatively straightforward to use, which makes it one of the most popular methods of object detection. Template Matching works by creating a reference image of what is being looked for and a source image that can be inspected. The template is slid around the inspected image and analyzed at each stop to decide where it best matches. The position with the best match probability is then assumed to be the object the template is representing. Fonseca and Manjunath [3] discovered an Area-based method which is a combination of feature detection and feature matching. It was found that this method is best suited for the templates which have no strong features with the image, since they operate directly on the bulk of values. Matches are estimated based on the intensity values of both image and template. This is the method used in this project. E. Hough Transform The Hough Transform [2] is a more complex technique which can be used to isolate features of a particular shape within an image. It requires that the desired features be specified in some parametric form, the classical Hough transform is most commonly used for the detection of regular curves such as lines, circles, and ellipses. A generalized Hough transform can be employed in applications where a simple analytic description of a feature is not possible. Due to the computational complexity of the generalized Hough algorithm it is not often used as it is time consuming to run. However, when this project is ported to GPU processing an adaptive Hough transform could be used, which is a more powerful variant than the standard version but still too slow to use without the assistance of the GPU. The Hough transform technique is tolerant of gaps in feature boundary descriptions and is relatively unaffected by image noise. To describe how the Hough transform technique works, consider the common problem of fitting a set of line segments to a set of discrete image points. We can describe a line segment in a number of forms. However, a convenient equation for describing a set of lines uses the Hesse normal form. x cosθ + y sinθ = r (4) Here, r, is the length of a normal from the origin to this line and θ is the orientation of r with respect to the x-axis. For any point on this line, r andθ are constant. In an image analysis context, the coordinates of the points of edge segments in the image are known and therefore serve as constants in the parametric line equation, while r and θ are the unknown variables we seek. We plot the possible (r,θ) values defined by each (x,y) point in cartesian image space to curves in the polar Hough parameter space. This point-to-curve transformation is the Hough transformation for straight lines. When viewed in Hough parameter space, points which are collinear in the

3 cartesian image space become readily apparent as they yield curves which intersect at a common (r,θ) point. The transform is implemented by quantizing the Hough parameter space into finite intervals or accumulator cells. As the algorithm runs, each (x,y) is transformed into a discretized (r,θ) curve and the accumulator cells which lie along this curve are incremented. Resulting peaks in the accumulator array represent strong evidence that a corresponding straight line exists in the image. This same concept can be applied to any shape that can be created with a normal form. III. M ETHODOLOGY Each of the techniques presented in the previous section provide a tool to use to complete the task of segmenting the baseball objects. The challenge comes in how to use these tools together to accomplish the intended goal. Figure 2 is one of Figure 3. Sobel operator applied to the unaltered image template to represent the ball as it has roughly the same shape in all of the images it is captured in due to the speed being roughly the same throughout the images. This then allows us to find the location of the baseball where we can then plant a seed for region growing. The seed is planted onto the Sobel image and goes outward until it hits the white border of the ball resulting in figure 4. We can now clearly see the location Figure 2. Unaltered image of baseball player the images captured from the recorded video stream. These images presented a few unique challenges. There is a very busy and noisy background due to the netting, the baseball is blurry from moving quickly and the bat is a color that is difficult to distinguish form the sky and rest of the background. There is also something advantageous in the image, which is the fact that the player has a red shirt on, which will help to differentiate him from the rest of the image. There needed to be a way to manipulate the image on a per pixel and RGB component level basis so it was converted to ppm format. In ppm format each pixel is represented by the three RGB component values. This means we now have access to each pixel and its RGB components in the image allowing us to freely manipulate it. A. Baseball Segmentation The first object chosen to segment was the baseball. The baseball is somewhat blurry due to the speed at which it is moving so using a classical Hough transform would not work very well and other Hough variants proved to be too slow for the time being. Instead, a series of filters and methods will be used to automatically find the ball. First, a Sobel operator was applied to the entire image resulting in Figure 3. A lot of the useless background information has been cast aside and a clear shape is left that represents the ball even if it is not exactly round. This shape is clear and distinct enough to build a template with. The next step we perform is constructing a Figure 4. Region growing applied to the center of the baseball causes the baseball to be shaded. of the baseball and store its location for future use. B. Waist The next object to detect was the waist location and orientation of the player. Looking back at figure 2 we can see there is a clear divide of the red of the players jersey and white of the players pants to use as unique features. The first step to finding the waist is to create a threshold that uses the redness of the player s jersey. After creating a binary image that highlights the extreme redness of the jersey we can start seeing the outline of the waist area as can be seen in Figure 5. From Figure 5 it is clear that we are getting close to finding the waistline of the player. Eliminating a bit more of the left over noise in the background will prove beneficial to applying a round of the sobel operator so a threshold to eliminate the non-red elements is used resulting in Figure 6. It is now a sobel operator combined with one more round of thresholding to find the waist as is shown in Figure 7. Any line generated from the sobel filter that does not have a white pixel a certain specified distance below it will be discarded, as well as any data that falls on the border of the image since it is assumed that the waist of the player will be relatively near the center of the image. The left and right most points of this line are

4 passing throughout the image as the bat can be in multiple orientations throughout the swing. D. All Together We can now use the locations that were found from detecting the ball,the bat, and the waist to overlay them onto the image so they can easily be seen as in Figure 8. The results were consistent across multiple images proving that detection of the ball and player s waist line and orientation are possible. Figure 5. Thresholding applied to the unaltered image to separate out elements containing a certain level of red. Figure 6. The thresholding values are further refined from figure 5 to eliminate background noise Figure 8. Result of running image processing. The ball has a blue box around it, the waist is represented by a yellow line and the bat a pink line. IV. C ONCLUSION taken and used for a very close approximation of the location of the player s waist. C. Bat The bat is segmented in much similar ways to the baseball and the waist. A threshold was found to eliminate much of the background image other than the handle of the baseball bat. It was difficult to separate the entire barrel as its of similar shade to the sky in the background. The handle, however, provides all the information we need to extrapolate the orientation and angle of the bat. After the thresholding was applied a sobel filter was run and then the handle area of the bat had region growing applied to it. Finally, a template was matched to the bat to find it in the remaining image. This template matching was more involved as the template had to be rotated while The ability to reliably detect these three key items in relation to a baseball players swing can provide valuable insight into how a player can correct swing mechanics. The angular relationship between the three items allows us to calculate appropriate swing timings and waist and bat location optimums. Currently, we are able to do this on an image by image basis. Each image takes approximately 3 minutes to process. A faster CPU will not greatly decrease processing time as all the tasks are done in a serial manner. In the future GPUs with the CUDA language can be used to do these calculations as it is a highly parallelizeable task. The only limiting factor in how much speed up could occur is how many GPUs are able to be used. With enough GPU power this task will be able to be done in real time allowing real time analysis of a players swing mechanics. This will result in immediate feedback for players and coachese to use to allow them to more quickly and more accurately correct swing mechanic mistakes. R EFERENCES Figure 7. Waist line of the player after image processing is applied to the unaltered image [1] Rolf Adams and Leanna Bischof. Seeded region growing. IEEE Transactions on Pattern Analysis and Machine Intelligence, 16(6), [2] Richard Duda and Peter Hart. Use of the hough transformation to detect lines and curves in pictures. Comm. ACM, 15(1):11 15, January [3] L.M.G. Fonseca and B.S. Manjunath. Registration techniques for multisensor remotely sensed imagery. Phtogram. Eng. Rem., pages , [4] Ms.K.Deepika J. Prakash, P.Harish. Android based object recognition into voice input to aid visually impaired. In International Conference On Recent Trends In Engineering Science And Management, pages , March [5] Bulent Sezgin, Mehmet Sankur. Survey over image thresholding techniques and quantitative performance evaluation. Journal of Electronic Imaging, 2004.

5 [6] G.T. Shrivakshan and Dr.C. Chandrasekar. A comparison of various edge detection techniques used in image processing. International Journal of Computer Science, 9(1), [7] S. Venkatesh and P. L. Rosin. Dynamic threshold determination by local and global edge evaluation. CVGIP: Graph, Models Image Process, 1995.