Advanced Image Processing, TNM034 Optical Music Recognition

Transcription

1 Advanced Image Processing, TNM034 Optical Music Recognition Linköping University By: Jimmy Liikala, jimli570 Emanuel Winblad, emawi895 Toms Vulfs, tomvu491 Jenny Yu, jenyu080 1

2 Table of Contents Optical Music Recognition Introduction Purpose Problem description Method Research and theories Pre-processing StaffLineHeight and StaffSpaceHeight Finding the Clefs Detecting the lines Compressing and Uncompressing the image Vandermonde Perspective control Finding the position of the note heads Determine the duration of notes Working Process Pre-process StaffLineHeight and StaffSpaceHeight Finding the Clefs Detecting the lines Compressing and Uncompressing the image Vandermonde Perspective control Finding the position of the note heads Determine the duration of notes Determine the pitch of the notes Summary Results and analysis Discussion Errors Other possibilities References Attachments

3 1 Introduction 1.1 Purpose The purpose of this paper is to describe and evaluate the progress of the development of a program for optical music recognition. Working Process mainly covers the final solution that have been selected and implemented within the final program, while Discussion somewhat compares with other possible solutions and improvements. 1.2 Problem description A program shall be developed for Optical Music Recognition (OMR). The resulting program shall be able to read a music score and return a character string that represents the detected notes. The detected notes shall only include crotchets and quavers. Minims and semiquavers shall be excluded from the result string, as well as any characters marking pauses and so on. 1.3 Method MATLAB is the framework used during the whole project. The method is based on creating a OMR system step-by-step by both using methods developed by others and solutions crafted by ourselves. The results of each part are evaluated along the way to give feedback of which solution that is best suited for each part of the OMR process. 2 Research and theories There is a lot of research available on optical music recognition. Some theories are developed through reading different research papers [1-9]. There are training images available to test the implemented program. Since the training images of the music sheets are both scanned and photographed, a pre-processing method should be implemented to be able to handle perspective, noise, cutting of the image and geometric transformations. Since most of the papers have staff line detection, a technique to detect these lines is researched. Control-points for a picture are always useful for recognizing various things on the image, therefore research is done on recognition of clefs. Note head detection and note duration are the most important parts in OMR. 3

4 2.1 Pre-processing A pre-processing function is needed to compute an image with higher quality than the input image. Improving the quality of an image is done by eliminating skew and other geometry distortions, reducing noise and, finally, converting it into black and white image as Mariusz Szwoch writes in his paper Guido: a Musical Score Recognition System [1]. In this paper the rotation of the image is done later because the staff lines need to be detected for the specific algorithm used. During the pre-process, the image is turned into a cut version of the binary- and grey-scale image. 2.2 StaffLineHeight and StaffSpaceHeight StaffLineHeight is the thickness of the staff lines and staffspaceheight is the distance between the staff lines. These distances are calculated based on the reasoning Mariusz Szwoch use in his paper on A robust detection for distorted musical staves [2]. The distances are needed to determine the positions of the staffs and to decide the size of different objects in the music sheet. With the correct space between the lines, dynamic variables can be created to compute the approximate size of for example the note height. 2.3 Finding the Clefs Filtering, template matching and correlation is the easiest and most accurate method for finding distinct symbols, as mentioned in Lecture 3 [3]. The clefs are to be concerned as very distinct symbols and therefore this procedure would be good to use for detecting the position of these. The obtained positions can later be used as information of distinct positions in the music sheet. 2.4 Detecting the lines As mentioned in Mariusz Szwoch s paper on A robust detection for distorted musical staves [2], many staffs detection algorithms analyze original image searching for five equally spaced and sized run-lengths of black pixels, or straight lines by using the Hough transform or mathematical morphology. To compute a result with a lower complexity, it was decided that the location of the staffs in a music score should be found by matching local histograms. We follow three of the steps presented in Szwoch s report: 1. Clearing histogram entries with small values 2. Successive break down histogram columns that are wider than 1*(the distance between lines on the same stave). 3. Locating a distinct local maximum value in histogram and clear the neighbouring columns 4

5 The detected lines are removed from the image, though the positions (equations) of them are stored for later usage during the process. 2.5 Compressing and Uncompressing the image The compressing and uncompressing of the image has the purpose of eliminating noise from the image and return a cleaner version that will be easier to work with because the result image will have less data. 2.6 Vandermonde The Vandermonde method can be seen as a second pre-process where the image is rotated and corrected according to how the staff lines are aligned. It uses a Vandermonde matrix that stores information about the positions of the lines [3]. Those positions can then be used for image correction by telling where we actually want the points to be. Figure 1. Illustration of staff lines before and after the geometric correction using Vandermonde. 5

6 2.7 Perspective control The perspective of an image can be fixed by finding four certain points in the image and calculate the distance between them. With theses lines it is possible to calculate how the images differ from the alignment of an image without perspective. It can then make all lines that are vertical in reality, vertical in the image. Figure 2. Illustration of the perspective before and after perspective control. 2.8 Finding the position of the note heads The most straight forward approach to detecting the notes and their positions, as mentioned in the articles [6] and [8], are template matching. By first filtering the preprocessed image with a primitive structural element, preferably a disk, candidates can be selected and used to create a mean image of the note heads. This creates the possibility to make a second filtering, which detects the actual position of the note heads. 2.9 Determine the duration of notes All notes/objects should at first be separated, therefore stafflines need to be removed at first. After the separation all objects are labeled, and those objects not containing note head coordinates are removed. Information is also stored to know which noteheads belongs to which objects. All notes are sorted according to there position relative to the stafflines. Finally the duration of the notes are determined by looking at the object area, euler number of objects, position relative to other notes and size of objects. 6

7 3 Working Process 3.1 Pre-process By using morphological operations, noise could be removed from the image. To get a cut version of the image it was needed to find the edges of the area of the image that contains the details and objects of interest. The area can be determined and used for edge detection with the Prewitt method which is a built in function. When this is done it is possible to find where the first and last white pixel lies in each row. The pre-process also creates a grayscale and binary version of the image. 3.2 StaffLineHeight and StaffSpaceHeight Both stafflineheight and staffspaceheight were found by collecting information from each column in the binary image. When running through the columns, the continuity of the black (when searching for stafflineheight) or white (when searching for staffspaceheight) pixels was counted and added to an vector which counts how many times each height occur is the column. The height that occurred most times is the desired height that is used to find objects in the image. 3.3 Finding the Clefs First, the space between the staff lines was estimated. This is just a mean value of the space between spots/lines with high value. The clefs was found by using template matching, since the clefs have a distinguishable shape and do not vary as much as notes. A clean image of a clef was imported and the size was decided with respect to the size of the estimated space between the staff lines. The image of the clef was run throughout the whole music score image, placing values in a matrix depending on how much the image matches point the music score. By examining projections onto both the horizontal and vertical plane of the 7

8 filtered image a dynamic threshold can be obtained. The points with values over the specified threshold are the centre position of an existing clef in the music score. 3.4 Detecting the lines In [2] after the three steps of line detection the fourth was implemented with a method that was created to use the built in function polyfit to obtain equations for all the lines. When the positions of the clefs had been found, a more absolute position of the staff lines could be estimated, since it was known how the lines lie in comparison to the clef. From the position of each clef, an approximate height of the staff could be calculated. The area after the clef was cut vertically into small areas that each were checked if they contained five staff lines. Figure 3. The first run of the staff line reconstruction Figure 4. After the first run of staff line reconstruction (during second run) 8

9 By using the locations of the five line segments in each piece, the equations for each staff could be obtained and described as second degree equations and stored in a matrix on the form Ax 2 +Bx+C. These equations were later used to perform geometric correction on the current image. 3.5 Compressing and Uncompressing the image An empty matrix is created to store the information about the compressed image. An binary version of the image is used when compressing. Each column in the binary image is compressed so that the matrix stores the length of whites and blacks in that column. When uncompressing the image, the matrix that stores the length of whites and blacks is checked. If the length of the whites is larger than a given value then the whites will be placed to their original positions in the image. If the length of whites were smaller than the given value the white-runs were set to zero, this yielded a cleaner image with mostly the staff lines left therefore the image could be sent to the next function that determined the lines. 3.6 Vandermonde After the equations of the staff lines have been obtained and stored in the Vandermonde matrix, the basic geometry of the note sheet (lines) is known and this can be used to correct the image in a way that makes all staff lines run horizontally from left to right. 3.7 Perspective control Deciding if there is a perspective in the image is done by first measuring the length of the staff lines of the top staff and comparing it with the length of the bottom staff. This is done by finding certain control points that are located in the left and right end of the top- and bottom staff. If there in fact is a perspective, it was needed to calculate how much the image needed to be corrected to fix the perspective. That can be done by first checking the distance between the control points and then with that decide what kind of perspective that the image is in. The information about the original and desired points are provided to the built in functions cp2tform and imtransform, which applies the transformation to the image. 9

10 3.8 Finding the position of the note heads After all the pre-processing had finished, the process of detecting the notes started. By using some morphological operations with structure elements that suited our needs, the image was enhanced even further, with respect to the possibility of detecting notes. On this image, a filtering was made using a primitive structural element with the shape of a disk. By using the peaks in the resulting matrix that was larger than 75% of the maximum peak, a mean image of a note head was created. This mean image of a note head was then used as filter when filtering the enhanced image once more. The major peaks of the matrix obtained in the second filtering were used to enhance the result of a more general filtering. From the peaks in the matrix obtained after the final filtering the note heads positions was given. Figure 5. The first filtering using the mean note image. 10

11 Figure 6. Enhanced result of mean note filtering. To enhance the result of the note head detection, some local horizontal histogram projections around each detected head was made. Using a couple of rules to separate actual note heads from faulty detected symbols, such as stronger lines or other symbols, the result was improved. 3.9 Determine the duration of notes Staff lines were removed using a vertical median filter on the grayscale image with the height of staffspaceheight and increasing the contrast with the built in function imadjust. The grayscale image is converted to binary by using the built in functions graythresh and im2bw, where graythresh calculates a good threshold to use when converting the image to binary. At this points all objects hopefully were separated, so now the image was labeled using the built in function bwlabel, and all objects not containing a note head were removed. Note heads that were situated close to each other were separated. The separation is done because groups of notes often intersect with each others. The information of which note head that belongs to which label is also stored. The notes were then sorted in the correct order (chronologically). The built in function regionprops was used to easily extract a single object from the image. regionprops was also used to calculate the area and Euler number of each object. By using the 11

12 extracted objects, width and height of each object were obtained. For each label it was investigated whether or not the label contained one or many notes. All this information was stored for later use. At first all labels containing more then one note were marked as crotchets and labels containing only one note as quavers. At this stage labels containing multiple notes were evaluated by estimating the area (with a bit of overestimation) if the object would be a crotchet. Vertical projection of each of the objects were made and the height of x largest peaks were found (where x is the amount of notes contained in the label). From the height of the peaks and stafflineheight, the estimated area of the stems were calculated. The area of the note head and the area of the beams were both estimated using staffspaceheight. If the actual area of the object was larger than the estimated area, then the object should be something else than a quaver or crotchet and is of no interest. If the actual area of the object were smaller than the estimated area then the object was marked as a quaver. All the labels containing multiple notes were then evaluated and marked as done. Now the labels containing only one note were evaluated. Labels (having only one note) that had a label with multiple notes above or below itself (close above or below) inherited the duration from the label with multiple notes and were marked as done. Labels still not marked as done (at this point there were only a label containing one note left to evaluate) were evaluated by looking at the width and the Euler number of the objects. If the Euler number were 0 or the width were to large then the object were marked as quaver, else a crotchet, and as done Determine the pitch of the notes The pitch of each note was determined by using the note heads distance to the center line in the current stave. 4 Summary 4.1 Results and analysis An analysis was made over the results of both note detection in attachment (2) and the resulting string output from the program in attachment (1). When analyzing the string output, it is quite clear that the part causing the most errors is the determination of the notes respective duration, especially the differencing between quavers and semiquavers. If errors of this kind are ignored, the accuracy is noticeably higher. 12

13 4.2 Discussion Errors Many of the errors in our implemented OMR system occurs as a result of faulty detection of the note heads positions. Since this process is based on matching with a mean note image, it is inevitable that the position of some note heads are detected one or two pixels in the wrong height. This leads to a small error, but in low resolution images one or two pixels can be the difference between two notes. A solution to this might be to sample up the image before doing the actual note detection. Another solution, which we have partly implemented, is to use additional checking of the notes properties and by this deciding the position. The part implemented in our system just tries to verify that the note actually is a note and not some other marking. The other major source of errors is misclassification between crotchets, quavers and semiquavers. Our approach here was to estimate the areas, Euler numbers, distance between note heads. Estimating the area wrong can lead the program to think the duration is something that it isn t. Euler number can also be different from what we expect if we get notes containing holes that should not be there (or containing holes that we do not expect), that could also lead the program to think the note has a different duration than it is. Inherited durations can also be wrong, but by looking at the training images this help more than it hurts. Late in the project we realized that one of many alternative solutions would have been to detect the vertical lines separating the beats, count the number of notes, comparing them and determine their respective duration. One more approach would have been to skeletonize the notes in some way and count the intersections in/of the horizontal bars connecting the note heads. These two types of errors yield a faulty string where some of the values obtain correct answers for semiquavers and incorrect answers where extra notes have been detected. Also the function can return too little note results due to a few misses in the correlation of the mean note. In the attachment (1) all of the training images have been run with the program and the results are registered with r for right and f for false underneath every value of the string. It can be clearly seen, comparing with the results of note detection, that for most of the correct notes detected a correct string is generated even for the semiquavers that should have been discarded Other possibilities When detecting the notes, one possible approach is template matching using a database of all possible notes (along with statistics of which template has the best match). The biggest drawback 13

14 with this template matching, with one template for each possible type of note, on all objects in the image is that there are notes that are too similar to each other, especially when the image is a photograph of the music sheet. One of the, in the long term, most stable and accurate approaches would have been to have a quite robust, but not perfect, detection algorithm implemented in a web service. By letting users submit their musical sheets, analyzing them and correcting the parsing manually, a huge neural network would have been constructed that takes both mistakes in previous detections and similarities between different sheets into concern. The main benefit of this approach would be the huge amount of data both from actual sheets and corrections made by users. If we were given more time to work with the project, our first priority would have been to correct the detection of the notes durations. This is the part generating the most errors and could have been solved with another approach, as mentioned above, hopefully with a better result. 5 References [1] M.Szwoch, Guido: a Musical Score Recognition System [2] M.Szwoch, A robust detection for distorted musical staves [3] B.Kruse, Lecture Slides TNM034 [4] R.Fergus and B.Singh and A.Hertzmann and S.T.Roweis and W.T.Freeman, Removing Camera Shake from a Single Photograph [5] D.Nehab, Staff Line Detection by Skewed Projection [6] X.F.Hermida and C.SB.Vargas, Development of an Optical Music Recognizer (O.M.R.) [7] J.S.Cardoso and A.Capela and A.Rebelo and C.Guedes, A CONNECTED PATH APPROACH FOR STAFF DETECTION ON A MUSIC SCORE [8] A.Rebelo and G.Capela and J.S.Cardoso, Optical recognition of music symbols [9] F.Rossant and I.Bloch, Robust and Adaptive OMR System Including FuzzyModeling, Fusion of Musical Rules, and Possible Error Detection 6 Attachments (1) Results of running the program on test images (2) Statistics of detected notes 14