Vision System-based, Grape Leaves Processing, in Real Time

Transcription

1 Vision System-based, Grape Leaves Processing, in Real Time Theodore P. Pachidis Dept. of Industrial Informatics Kavala Institute of Technology Kavala, Greece Ilias T. Sarafis Dept. of Mechanical Engineering Kavala Institute of Technology Kavala, Greece John N. Lygouras Dept. of Electrical and Computer Engineering Democritus University of Thrace Xanthi, Greece Abstract In this paper, automatic grape leaves processing and preparation of them (fitting of the proper cutting shapes on leaves area) to cover food industry needs, are presented. In the experimental setup, images are captured by means of a single firewire camera and a LED-based lighting system, while grape leaves are moving on a conveyor belt. For image processing, a novel software application (called NtolcutPT) has been developed in visual C++. It is based on known and novel proposed in this paper image processing algorithms and methodologies that ensure automatic leaves processing in real time, with less wastage and more reliability. Adapting the proposed system in a food industry production line the whole process can be optimized. Experimental results prove the effectiveness of this system. Keywords-grape leaves feature extraction; thinning algorithm; NtolcutPT software application; standard cutting models I. INTRODUCTION In food industry, a number of processes related to the production of specific products has not yet been automated. On the other side, automatic image processing suffers from problems concerning: a) image conversion from color or gray scale images to binary, b) object segmentation, c) reliable extraction of features. The aforementioned problems introduce much more difficulties in cases where real time processing is necessary. An important number of algorithms and methodologies has been proposed in the literature in terms of thresholding [1], [11], image segmentation [2], feature extraction [3], thinning and skeletonization [4]. Papers referring to leaves processing have also been presented. In [5], Zhang et al. proposed image processing techniques for the automatic inspection of tobacco leaves. Kondou et al. [6] used grape leaves as an experimental material to examine a trees growing technique by means of a digital camera. Srivastava et al. [7] developed a color based detector to detect Downey Mildew (DM) disease that appears in grape leaves. In [8], Tzionas et al. presented a leaf classification method based on their morphological features. Neto et al. [9] developed a feature extraction application from leaves. Finally in [10], Li et al. developed an interactive tool for vein extraction of leaves using independent component analysis. A basic problem regarding grape leaves before preparing some kind of food is to subtract the petiole. Another one is to cut each leaf properly following specific shapes to reduce leaf wastage. In details the objectives of the work presented in this paper are: a) To combine well-known algorithms and methodologies for image processing with novel ones (proposed in this paper) and finally to achieve grape leaves basic features extraction with the help of a novel software application as these leaves are moved on a conveyor belt with a constant speed. The application is called NtolcutPT and is running in a personal computer Pentium IV. b) To calculate the proper location and orientation of two different in size shapes called Standard Cutting Models (SCMs) on each leaf surface. A correct combination of these two shapes has to properly fit in each leaf surface reducing at the same time leaf wastage. These shapes are designed according to the requirements of a local food industry. Leaf pieces as a result of a cutting procedure can be utilized in preparing a Greek food made from boiled grape leaves and rice (stuffed grape leaves). c) To drive a robotic head providing the correct cutting instructions. Thus, it will be possible for each leaf to cut the leaf petiole from its base and to extract one or two pieces of the previously referred shapes. d) Although NtolcutPT application is running in a personal computer with Windows XP as operating system (which is not a real time one), the whole process has to be executed in real time (the desirable cycle time is 1 sec). By means of a system that satisfies the aforementioned objectives, the whole process in an industry production line for specific food preparation can be optimized while the installation cost remains low. Experimental results prove the effectiveness of the proposed system. In section II, theory and algorithms are presented. In section III the proposed system is briefly described. Experimental results are illustrated in section IV. Finally, in section /10/$ IEEE

2 V, the conclusions of this work, as well as, suggestions for future work are given. II. THEORY AND ALGORITHMS A. First Stage of the Processing Algorithm As it is previously mentioned many different solutions concerning image processing, process or product quality and reliability, have been presented by the scientific community. However, in most cases, image processing in real time is a challenge. The challenge of this work was to reduce the computational cost to ensure processing in real time. This goal was achieved by using known or novel algorithms and methodologies with low computational cost. Two different requirements are taken into consideration. According to the first one, each image contains only one object (leaf). This is possible because a worker cannot place a leaf on the conveyor belt so fast. Even more, in a fully automated system, capturing frame rate (up to 30 frames / sec but it can be increased) and system settings can ensure that an image contains only one object. According to the second requirement, the object must be found near the center of the image. Previous requirements permit the easy separation of the object from its background, without the need of a segmentation algorithm, where generally, the computational cost is high. The processing algorithm adopting the above requirements is separated in two stages. According to the first stage, calculations are focus on capturing an image for processing in which the object will be found near the center of the image. In this stage the processing cost has to be very low so that the system captures the correct frame. In this first stage, smoothing is achieved with the help of a median filter [11]. This filter reduces salt and pepper noise. Gray scale images are converted to binary by use of the Ridler and Calvard algorithm [1] for auto thresholding. This method calculates a proper threshold for each image with very low computational cost. Alternatively an adaptive thresholding method [11] can be selected by means of NtolcutPT application. If N is the number of object (leaf) pixels in the binary image with function I(i,j), leaf center coordinates are calculated from: The leaf area from: = 1 1 i i and j = N N j (1) A = I( i, j) (2) Then in binary images, moments [8], [11] are calculated using the relation: p q μ = ( i i ) ( j j) (3) p, q and angle θ expressing leaf orientation is given as: 1 2 μ 1 1,1 θ = tan (4) 2 μ μ 2,0 0,2 Steps of the first stage of the processing algorithm are the following: 1. Capture a color image by means of the vision system (image resolution 640x480 pixels). 2. Convert this color image to gray scale one. 3. Smooth the resulting image using a proper filter. 4. Calculate the best threshold and convert image to binary. 5. Calculate grape leaf center, area and angle the main leaf axis forms with the horizontal axis. 6. Check by means of leaf center coordinates if the leaf is properly located in the image frame. 7. If the leaf is properly located, store the initial color image in a matrix with resolution 512 x 480 pixels. 8. Repeat the loop to capture a new image. B. Second Stage of the Processing Algorithm In the second stage, the processing algorithm processes the correct image and finds the desirable features of each leaf. These are: the leaf center, the leaf area including leaf cavities and holes, the petiole s base point, maximum and minimum distance from the leaf center and the angle that the leaf axis forms with the horizontal axis. It also fits the proper shapes (SCMs) on the leaf area. In this part of algorithm, Roberts operator [11] is used to find image edges. Comparing Roberts operator to Canny algorithm for edge detection as the 2 nd stage of the processing algorithm is executed (in a single core, Pentium IV personal computer with Windows XP), it was confirmed that Roberts operator is two times faster than Canny algorithm (16 ms instead of 32 ms respectively). Moreover, Roberts operator provides a continuous line (it is desirable to extract the leaf contour) while Canny algorithm does not provide a continuous line (Fig. 1). The last result is undesirable regarding the next steps of the processing algorithm. The proposed Color Edge Thinning Algorithm (CETA) thins edges so that successive pixels have a common bound (4-neighbours) and it is based on an important number of different templates (Fig. 1). (a) (b) (c) Figure 1. Part of leaf contour after the implementation of a) Roberts operator. b) Roberts operator +CETA. c) Canny edge detection algorithm. While CETA is based on templates, it is fast enough (it is executed in 16 ms) and consequently it can be used in real time applications. This algorithm was successfully used to generate robotic trajectories [12] by means of Pseudo Stereovision System (PSVS) and it is implemented in lines. In HumanPT robotic application [13], it was utilized to thin color edges (up to 11 pre-specified colors). The final result of the implementtation of CETA is a line with 4-neighbor pixels. Based on this feature of lines (leaf contour), it is then possible to calculate the remaining features. In this work, CETA is implemented in one color image (binary images). During image scanning a counter watches the number of pixels that are removed. The algorithm stops when the counter remains zero after an execution. In Fig. 2 the selected templates and the characterization for each

3 template are presented. From 220 totally templates these 33 templates are carefully selected and contribute to edge thinning. The basic steps of the algorithm as it is modified for the needs of this work are the following: 1. In the binary edge image read a 3 X 3 window starting from the upper left corner. 2. If the central pixel of the window is white then: a) set in a variable the value that represents the white color (default color) and b) compare it with a template type. Else go to step If template type isn t one of -3-, -4-, -5-, -6-, -8-, Æ Go to Step If the template type is one of -3-, -4-, -5-, -6-, then: a) If the template is one of I, II, III, IV, of type 3_1, 4_1, 5_1, 6_1, remove the template central pixel respectively. b) Increase pixels counter. c) Repeat steps (a) and (b) for one of the templates I, II, III, IV, of type 3_2, 4_2, 5_2, 6_2 respectively. d) Go to Step If the template type is -8- then: a) If the template type is 8_1_I remove the template central pixel. b) Increase pixel counter. c) Go to Step If image scanning has finished, check if pixel counter is equal to zero. a) In any case set pixel counter equal to zero and repeat line thinning procedure. b) If during last cycle no pixels have been subtracted then stop the loop. To find the feature point that corresponds to the petiole s base, as well as, to the maximum distance from the leaf center, two different sets of points are created. Points that searching in narrow ranges of angles are found in the minimum distance from the leaf center of gravity and points that searching in narrow ranges of angles are found in the maximum distance from its center. Studying and comparing a sufficient number of grape leaves throughout the research it was realized that petiole s base is related with the leaf main axis and the minimum distance from the leaf center. Exploiting these two findings, the 2nd stage algorithm was designed to search for a point found in a locally minimum distance from the leaf center. Proposition: The point corresponding to the petiole s base is the point found in the minimum distance from the leaf center searching in a narrow range of angles around the main axis. The angle θ1 the leaf main axis forms with the horizontal axis is calculated from (4). In any angle θ1, this narrow range of angles is shifted respectively by θ1, so that petiole s base point is always correctly calculated. The area calculated for the leaf surface by means of (2) does not include cavities or holes that could exist on the leaf surface. This fact could dramatically increase leaf wastage. The problem has been solved as follows. From edge image, edge points are classified to be successive. This was another reason for adopting Roberts operator and CETA. From these successive points, a small number of them is selected. These points, just like before, are located, searching in different narrow ranges of angles, in the maximum distance from the leaf center. They are stored in the so-called peak matrix and if these successive points are connected each other then a surface surrounding the leaf is created while at the same time it covers leaf cavities and holes. If xi, yi are the coordinates of the successive points the closed line surrounding the leaf forms, the area surface is calculated by (5): Figure 2. The selected templates of the proposed edge thinning algorithm. AL = x 1 i 2 N yi xi+1 where i=1 N yi +1 (5) Cutting shapes L (L stands for Large) and S (S stands for Small) have dimensions in mm. Using a simple calibration method, dimensions of image pixels are accurately determined in mm. For the calibration, a sheet of paper with a printed chessboard on it (successive black and white squares) is taken as a measuring pattern. Then using the resulting scale factors, horizontally and vertically calculated, the cutting shapes (SCMs) are properly scaled and rotated so, when they are displayed, present the real cutting state. The algorithm compares leaf area ΑL to areas of cutting shapes and displays the proper combination of these shapes with the correct orientation while simultaneously stores as data, center coordinates and the orientation of these shapes. These data may drive a robotic head with two cutting tools and when the leaf is under the

4 robotic head a cutting instruction is given. If a properly captured color image exists, the 2 nd stage of the processing algorithm is executed. The basic steps of this algorithm are the following: 1. Convert the color image to gray scale one. 2. Apply the Median filter. 3. Calculate the leaf center. 4. Apply the edge detection algorithm to create a binary edge image. 5. Apply the proposed thinning algorithm (CETA). 6. Calculate the maximum and the minimum distance on the leaf area from its center. 7. Calculate the angle formed by the line connecting the leaf center to each pixel of edge image with the horizontal line. 8. Find a desirable number of points having as unique feature to be found in the maximum distance from the center of gravity, on the leaf surface. 9. Classify these points as successive points and store them in a matrix (peak matrix). 10. Add and connect these points with a pre-selected color in the same frame with the color image. 11. Add to the previous frame, as a circle with a pre-selected color, the point found in the maximum distance from the center. Store its coordinates. 12. Add to the same frame, as a circle with a pre-selected color, the point a) which is found in the minimum distance from the leaf center searching in a narrow range of angles around the main axis and b) simultaneously leaf main axis forms angle θ 1 with the horizontal axis. Store its coordinates. 13. Find the first pixel of peak matrix scanning matrix. 14. Calculate the leaf area A L from the line segments surrounding the leaf by means of (5). 15. Scale and rotate the leaf properly according to the angle the main axis of the leaf forms and calculate SCM area for the two different types of them (L, Large and S, Small). 16. Calculate SCMs center coordinates and their orientation. 17. Add in proper locations on the image together with their center coordinates the correct combination of L and S shapes according to the leaf area Α L. 18. Send cutting data that is petiole s base point, SCMs center coordinates, leaf orientation angle and type of shapes to the robotic cutting head if this is desirable. Thus it can be properly prepared, moved and rotated. 19. For each leaf, calculate its exact arrival time in the area under the robotic head according to the conveyor belt velocity and cut the leaf if this is desirable (if the robotic head has been installed). 20. Repeat previous steps if a new image containing a leaf in the proper location exists. Figure 3. Basic steps of the 2 nd stage of the processing algorithm with images: a) initial b) gray-scale c) binary d) after erosion e) implementation of Roberts operator f) CETA implementation g) center, petiole s base and point in the maximum distance from the center h) leaf area including caves and holes i) SCMs fitting. In Fig. 3 the basic steps of the 2 nd stage of the processing algorithm are presented with images. The final result (Fig. 3 (i)) is designed on the current color image and appears on the screen. Algorithm stages as they are previously described are continuously executed. The procedure is stopped by the system operator. III. SYSTEM DESCRIPTION The grape leaves processing system is composed of a) a conveyor belt that can be moved with constant speed using a controller attached on its base. b) The imaging system that consists of a color firewire camera (IEEE-1394) with resolution 640 x 480 pixels and a LED-based lighting system in a dark room above the conveyor belt creating this way steady lighting conditions (Fig. 4). c) A personal computer Pentium IV running at 3.4 GHz with Windows XP as operating system. d) A software application (called NtolcutPT) developed in visual C++ which is executed in the personal computer and permits the processing of grape leaves and the control of the procedure. In Fig. 5 the main window of NtolcutPT application appears. From this window a user can select the type of the camera (usb or firewire) or capture video for statistical reasons. The operator can also start the procedure, process data relating to the correct location of the robotic head or be aware of the petiole s coordinates, the leaf rotation angle, the moving speed, the real camera frame rate and even more statistical data that are also stored in files. In Fig. 6, a window regarding application parameters is illustrated. From this, a user can select, regulate or activate different parameters that ensure the improved system operation. In Fig. 7 (a) and (b) two different windows are shown. These windows, if the procedure is active, present in Fig. 7 (a) the current state on the conveyor belt as a

5 result of the execution of the first stage of the processing algorithm, while in Fig. 7 (b) the current state as a result of the execution of the second stage of the algorithm. NtolcutPT software application has even more features of minor importance. Future work will include: a) the completion of a robotic head construction with two cutters (2 different shapes) to accurately cut each leaf. b) a feeding system that will be able to place every time a single leaf on the conveyor. SCMs fitting as a leaf is rotated, leaf wastage calculation and the process cycle execution time. In Fig. 8, the leaf center (red cross), the petiole s base point (orange circle), the fitted SCMs (blue and cyan colors), their centers (yellow and blue crosses) and the point in the maximum distance from the leaf center (red circle) are presented. In Fig. 9 results from different leaves are illustrated. From the leaves that we have processed, we found that the mean value of the uncovered leaf area from SCMs is 6,45 cm 2 while the mean value of the wastage is 7,94%. Taking into consideration the experimental results concerning 90 images, it is found that the petiole s base point is correctly located in angles of about ο, while SCMs are adapted to the leaf on any orientation of leaf (0 360 o ). A sample of these images is shown in Fig. 10. Finally the whole process cycle found is less than 1 sec. This cycle can be more improved. Particularly, the mean value measured for each cycle for the 1 st stage of the processing algorithm is 99 ms while the mean value measured for each cycle of the 2 nd stage is 801 ms. Figure 4. System s view. Figure 6. Program parameters regulation window. Figure 5. The main window of NtolcutPT application. IV. EXPERIMENTAL RESULTS Experimental results concern examples of features extraction from different leaves, the accuracy in finding petiole s base and (a) Figure 7. a) The initial image window. b) The image processing window. (b)

6 Figure 8. A leaf, the basic features and the fitted SCMs. Figure 10. Samples of the same leaf as it is rotated 360 ο. Figure 9. Results from different leaves. V. CONCLUSIONS In this paper several methodologies and algorithms concerning grape leaves processing were presented. The whole procedure permits grape leaves basic features extraction, SCMs fitting and control of the whole system. A novel software application developed in visual C++ includes and implements the aforementioned methodologies and algorithms and it has the features to be applied in a food industry procedure. It can automatically process grape leaves images as leaves are moved on a conveyor belt, extract useful features of them and finally control a robotic head with the objective to cut the leaves properly. Leaf pieces can be used then to prepare a specific type of food made from boiled grape leaves and rice. Our future work includes steps to improve even more NtolcutPT software application on issues regarding the quality of the results, the reliability of the system as well as the processing speed. It also includes the construction of a system that in each cycle automatically places one leaf on the conveyor belt and the completion of the robotic head construction. REFERENCES [1] T. Ridler and S. Calvard, Picture Thresholding Using an Iterative Selection Method, IEEE Transactions on System, Man and Cybernetics, SMC-8, 1978, pp [2] R. Unnikrishnan, C. Pantofaru, M. Hebert, Toward Objective Evaluation of Image Segmentation Algorithms, Pattern Analysis and Machine Intelligence, IEEE Transactions on, vol. 29, no. 6, 2007, pp [3] W. Xin, Laplacian Operator-Based Edge Detectors, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 29, no. 5, May 2007, pp [4] X. Bai, L. J. Latecki, W. Y. Liu, Skeleton Pruning by Contour Partitioning with Discrete Curve Evolution, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 29, no. 3, 2007, pp [5] J. Zhang, S. Sokhansanj, S. Wu, R. Fang, W.Yang, A trainable grading system for tobacco leaves, Computers and Electronics in Agriculture, vol. 16, no. 3 (1997), pp [6] H. Kondou,, Y. Motonaga,, Kitamura, Y. Nishikawa,, A., H. Hashimoto,, K. Nakanishi, and T. Kameoka,, Shape Evaluation by Digital Camera for Grape Leaf, Proc. of the Third Asian Conference for Information Technology in agriculture, pp , [7] Srivastava, M. Shugen and I. Kousuke, Development of a Sensor for Automatic Detection of Downey Mildew Disease, in Proc International Conference on Intelligent Mechatronics and Automation (ICIMA 2004), pp , Chengdu, Sichuan, China, [8] P. Tzionas, S. Papadakis and E. D.Manolakis, Plant Leaves Classification Based on Morphological Features and a Fuzzy Surface Selection Technique, 5 th International Conference on Technology and Automation ICTA'05, Thessaloniki, Greece, pp , Oct [9] J.C. Neto, G.E. Meyer, D.D. Jones Individual leaf extractions from young canopy images using Gustafson Kessel clustering and a genetic algorithm, Computers and Electronics in Agriculture, vol. 51, 2006, pp [10] Y. Li, Z. Chi, and D. Feng, Leaf vein extraction using independent component analysis, in 2006 IEEE International Conference on Systems, Man and Cybernetics, 8-11 Oct 2006, Taiwan. [11] R. Jain, R. Kasturi and B. Schunck, Machine Vision, McGRAW- HILL, International Editions Artificial Intelligence Series, [12] T. Pachidis and J. Lygouras, Vision-based Path Generation Method for a Robot-based Arc-Welding System, Journal of Intelligent and Robotic Systems, Vol. 48, No. 3, 2007, pp [13] T. Pachidis, J. Lygouras and K. Tarchanidis, HumanPT: An Open- Source, HumanPT Architecture-based, Robotic Application for Low Cost Robotic Tasks, Journal of Intelligent and Robotic Systems, Vol. 55, No. 4, 2008, pp