Pool Player Bot. Jiaying Zhang Apr 20, 2013

Transcription

1 Pool Player Bot Jiaying Zhang Apr 20, 2013 University of Florida Department of Electrical and Computer Engineering EEL5666 Intelligent Machines Design Laboratory Instructors: Dr. A. Antonio Arroyo, Dr. Eric M. Schwartz TAs: Ryan Chilton, Josh Weaver

2 Table of Contents Abstract Introduction Integrated System Mobile Platform Actuation Sensors Behaviors Conclusion Appendices

3 Abstract The Pool Player Bot is an antonymous robot that can play the table top pool. The Pool Player Bot can move along one side of the pool table and stop before the white ball. It can also rotate the cue by a specific angle to align with the white ball and an objective ball. After a nice shot, the robot goes back to starting point and waiting for another task. All the performance is under the observation from a camera mounted over the pool table, telling the robot when to stop can how much angle should it rotate by. This article introduces how the Pool Player move and finish its shot task step by step. The detail working process and control system are also included in the content. Introduction This antonymous robot is designed to play a 20x12x3 inch table top pool (see Pic1). It allows the customer to select one of the three color balls as the objective ball and the robot will shot the white ball to strike the selected objective color ball into pocket. The platform is based on a 3 wheel vehicle and the robot is able to move forward or backward with the control of either IP camera or IR sensor on the side. Two servos are used to control the position of cue, one enables cue rotate in horizontal plane and the other is to extend the cue length. Solenoid is the key part because it performs the striking process. The robot return to its initial position after each shot round, and the customer can select another ball for the robot to shot in the new round. Integrated System Pic1: Table-Top Pool Table The system is mainly consisted of a camera, a computer and a robot (See Pic2). The camera is used as eye of god, taking instant video of the position of white ball, color balls and cue tip. The video reaches the computer by DroidCam Application and the computer process the images by OpenCV color detect program. The output of OpenCV program on computer are the angle of balls, the line of balls, the position of white ball and the position of cue tip which is highlighted in blue. These outputs are represented as different characters sending to the computer serial port that is connected with Xbee terminal. Through the Xbee communication, the board receives 3

4 Pic2 System Flow specific signal telling what to do next. For example, if the board is received a character S from on board Xbee terminal, she will stop the motor and start approaching the cue to the white ball. The system feedback is LCD screen. It reflects the current behavior of the robot. Another input for the board is IR range sensor. When the robot moves apart from the camera range, then it can detect the relative position between the robot and the pool-table side (See Pic3). Mobile Platform Pic3 Camera s Shooting Range The robot is basically consisted of two layer platforms shown in Pic4. The bottom platform is designed to move the robot and the top one is execute the motion of cue. On the bottom, there is a base platform with two wheels and a caster, carrying the board, batteries, motors, IR sensor, LCD and servo1. The wheels are driven by DC motors. The board I use is EpiphanyDIY. In the middle, there is a servo which drives the turning table as well as the cue system. LCD is supported by a frame on the corner in order to provide convenience for customers to read. There is wood piece fixed on the side of robot which faces the pool table. It works as the support for turning table and IR sensors are mounted on it. 4

5 Pic4 Mechanical Structure On the top, there is a turning table holding servo2, a cue and a solenoid structure. This platform is driven by a steering servo fixed underneath it. The cue is screwed on the solenoid pivot, and a wedge is mounted underneath the solenoid to supply a downward orientation for the pool cue. The solenoid support is pined with three link rods and one of them is connected to servo. When the servo works, these link rods rotates and the angle between them changes, as a result the solenoid moves on the turning table and the cue is kinds of extended. This servo is fixed at the edge of the turning plate and the solenoid is guided with a special rail to provide the on degree of freedom for the cue motion. Because of the link rods and the servo, the pool cue is allows to reach the ball in a wide range. Actuation DC Geared Motor (x2) Purpose: drive the robot Application: wheels Servo 1 Purpose: push the solenoid link Size: 35x16.9x32.0 mm Weight:24.3h/0.86oz Speed: 0.19sec/60 Torque: 3.4kg-cm/47 oz-in Servo 2 Purpose: hold and turn the turning table Size: 35x16.9x32.0 mm Weight:24.3h/0.86oz Speed: 0.19sec/60 Torque: 3.4kg-cm/47 oz-in Solenoid Purpose: perform the shooting of the cue Type: Push type C-Frame 5

6 Series S-17-85QH 0.84 x 0.92 x1.75 Duty cycle: 1/4 Voltage: 12 V Stroke: 1 in Sensors Solenoid-Relay-NPN schematic Infrared Proximity Sensor (x2) Purpose: Detect the distance between robot and pool table Solenoid Push force versus VS stroke IP Camera (Android Phone) Purpose: Take instant video of balls and cue, and output the angle of balls, the line of balls, the position of white ball and the position of cue tip. Behaviors The Pool Player Bot allows the customer to decide which ball it will shoot. The Human-Robot interface in Pic6 is an execution program created by Visual Studio. When people click the target ball for the robot, which can be purple, red or yellow ball, the corresponding program will run to calculate whether the balls position are within the robot s playing scope. If not, the customer should either select another color ball or move the balls on the pool table. If yes, then the robot will start its shoot progress. Pic6 Human-Robot interface 6

7 To finish a nice shot, the Pool Player Bot works follow several steps. Step 1: System initializing. This step is to initialize the robot position at the lower-left corner to the pool table. The distance between the inner wheel and table side wall is around 2 inch. Secondly, the servo that controls the turning table is set as its 180 degree. The purpose of this step is to make angle calculation more convenient. Finally, the servo linked with solenoid is set as its 100 degree to pose the cue in the shortest length. When the LCD shows Please select an objective ball, this means the robot is ready to shot. Step 2: Cue Paralleling. If the selected objective ball is within the shot scope, the robot will receive the angle of the two balls. Then the cue turns by this received angle value so that the cue will parallel to the line connecting the center of two balls. Step 3: Robot Moving. Although the cue is paralleled to ball centers line, it still has distance. So the robot will start to move until the cue is aligned with two balls. The camera will computer the position of cue tip blue and, if the center of blue is on the left hand side of balls line, it will send forward signal to robot, while if it is on the right hand side, backward signal will be sent. This step will process for several times to make sure the robot is arrived at the right position. Step 4: Cue Approaching. After the cue is aligned and stopped, the cue starts approaching the ball in the way of changing servo s angle. The two link rods allow the cue to extend. When the cue tip blue is very close to the white ball, OpenCV will send out a reach signal and then the cue stops extend and strike the white ball. If the robot is happy, the objective ball will fall into the pocket. Step 5: Strike the solenoid and initialize. This allows the robot goes to the initial position and prepare for another shot. The communication between the computer and robot is very frequent and Pic7&8 are the flow chart of OpenCV program and robot program with Xbee terminals. 7

8 Pic6 OpenCV program flow chart 8

9 Pic7 Robot program flow chart 9

10 Conclusion Through this project I have learnt a lot knowledge regarding autonomous robot. My robot Pool Player Bot is quite smart when it can communicate with camera. Thanks to OpenCV, the robot can see the ball and know where it is. Although very often, the robot does a fantastic job, it still has a lot of room to improve. The robot has limited striking range. Cue tip is easy to move out of the camera range when the striking angle is very large. While on the other side, if the white ball is very close to the edge, the cue is not able to reach it. Maybe in the future I can design a smart robot with flexible arms. Another main problem is an obstacle that I haven t conquered. I was thinking to use line following sensor to keep the robot move around the pool table but the robot keep shaking all the time even on a straight line. I have to get rid of this idea and switch to wall following sensor. However, there still exist problems when the robot needs to make a 90 degree turn. As a result, the Pool Player now is not able to move around the pool table which causes people to put the balls at reachable position to serve the robot otherwise it will not finish the task. All in all, the small robot has low precision. The robot moves after the cue is turned an angle. If the wheels do not go a straight line it will cause some problem. I think it might need some function in OpenCV program such as feedback check. Further, the camera does not detect holes, and more work needs to do if the color balls are far from the holes or at special angles. 10

11 Appendices 1. Atmel program for EpiphanyDIY board /* * Epiphany_v2.c * * Created: 2/14/ :50:18 PM * Author: Jiaying Zhang */ #include <avr/io.h> #include <util/delay.h> #include <math.h> #include <stdio.h> #include "clock.h" #include "ATtinyServo.h" #include "uart.h" #include "adc.h" #include "motor.h" #include "lcd.h" int main(void) clockinit(); adcinit(&adcb); ATtinyServoInit(); motorinit(); LCDInit(); usartinit(&usartc0,115200); usartinit(&usarte1,57600); PORTE_DIRSET=0x80;//Xbee control PORTC.DIRSET = 0x08; sei(); fprintf(&lcd_str,"\r System \ninitializing"); setservoangle(20,180);//servo for turn table initialized _delay_ms(1000); setservoangle(21,105);//servo for cue initialized _delay_ms(1000); int i=0; int j=110; double step=0; char msg[10]; 11

12 int dir=0; int speed=650; int speedlow=speed-30; void movedir() if (dir==0) setmotoreffort(1,650,motor_dir_forward);//outer/right wheel setmotoreffort(2,650,motor_dir_forward);//inner/left wheel if (dir==1) setmotoreffort(1,650,motor_dir_backward);//outer/right wheel setmotoreffort(2,650,motor_dir_backward);//inner/left wheel void move() adcchannelmux(&adcb,3,3);//front IR sensor adcchannelmux(&adcb,0,0);//rear IR sensor int IRfront= analogread(&adcb, 3); int IRrear= analogread(&adcb, 0); //fprintf(&xbee_str," F %d R %d \r\n", IRfront,IRrear); if(irfront>1000 &&IRrear<1000)//at corner dir=0;movedir (); elsedir=dir; movedir(); if(irrear>1000 &&IRfront<1000)//at corner dir=1; movedir(); elsedir=dir; movedir(); fprintf(&lcd_str,"\rplease Select \nobjective Ball"); while(1) if(step==0)//check play scope fprintf(&lcd_str,"\rplease Select \nobjective Ball"); if(datainbufe1()) fscanf(&xbee_str,"%c",msg); if(strcmp(msg,"n")==0)fprintf(&lcd_str,"\rsorry!! Out \nof my scope");_delay_ms(3000); if(strcmp(msg,"y")==0) 12

13 if(step==1) fprintf(&lcd_str,"\rcue Paralleling"); if(datainbufe1()) fscanf(&xbee_str,"%c",msg); if(strcmp(msg,"a")==0) for(i=180;i>29;i--) setservoangle(20,i);_delay_ms(100); if(strcmp(msg,"b")==0) for(i=180;i>34;i--) setservoangle(20,i);_delay_ms(100); if(strcmp(msg,"c")==0) for(i=180;i>39;i--) setservoangle(20,i);_delay_ms(100); if(strcmp(msg,"d")==0) for(i=180;i>43;i--) setservoangle(20,i);_delay_ms(100); if(strcmp(msg,"e")==0) for(i=180;i>47;i--) setservoangle(20,i);_delay_ms(100); if(strcmp(msg,"f")==0) for(i=180;i>51;i--) setservoangle(20,i);_delay_ms(100); if(strcmp(msg,"g")==0) for(i=180;i>55;i--) setservoangle(20,i);_delay_ms(100); if(strcmp(msg,"h")==0) for(i=180;i>59;i--) setservoangle(20,i);_delay_ms(100); if(strcmp(msg,"i")==0) for(i=180;i>63;i--) setservoangle(20,i);_delay_ms(100); if(strcmp(msg,"j")==0) 13

14 for(i=180;i>67;i--) setservoangle(20,i);_delay_ms(100); if(strcmp(msg,"k")==0) for(i=180;i>71;i--) setservoangle(20,i);_delay_ms(100); if(strcmp(msg,"l")==0) for(i=180;i>75;i--) setservoangle(20,i);_delay_ms(100); if(strcmp(msg,"m")==0) for(i=180;i>79;i--) setservoangle(20,i);_delay_ms(100); if(strcmp(msg,"n")==0) for(i=180;i>83;i--) setservoangle(20,i);_delay_ms(100); if(strcmp(msg,"o")==0) for(i=180;i>87;i--) setservoangle(20,i);_delay_ms(100); if(strcmp(msg,"p")==0) for(i=180;i>91;i--) setservoangle(20,i);_delay_ms(100); if(strcmp(msg,"q")==0) for(i=180;i>95;i--) setservoangle(20,i);_delay_ms(100); if(strcmp(msg,"r")==0) for(i=180;i>99;i--) setservoangle(20,i);_delay_ms(100); if(strcmp(msg,"s")==0) for(i=180;i>103;i--) setservoangle(20,i);_delay_ms(100); if(strcmp(msg,"t")==0) for(i=180;i>107;i--) setservoangle(20,i);_delay_ms(100); if(strcmp(msg,"u")==0) 14

15 for(i=180;i>111;i--) setservoangle(20,i);_delay_ms(100); if(strcmp(msg,"v")==0) for(i=180;i>115;i--) setservoangle(20,i);_delay_ms(100); if(strcmp(msg,"w")==0) for(i=180;i>119;i--) setservoangle(20,i);_delay_ms(100); if(strcmp(msg,"x")==0) for(i=180;i>124;i--) setservoangle(20,i);_delay_ms(100); if(strcmp(msg,"y")==0) for(i=180;i>129;i--) setservoangle(20,i);_delay_ms(100); if(strcmp(msg,"z")==0) for(i=180;i>135;i--) setservoangle(20,i);_delay_ms(100); if (step==2) //robot start to move until the cue is align with the balls fprintf(&lcd_str,"\rcue Moving"); //move(); //setmotoreffort(1,speed,motor_dir_forward);//outer/right wheel //setmotoreffort(2,speed,motor_dir_forward);//inner/left wheel if (datainbufe1()) fscanf(&xbee_str,"%c",msg); if (strcmp(msg,"s")==0) fprintf(&lcd_str,"\rcue Aligned"); setmotoreffort(1,0,motor_dir_forward);//outer wheel setmotoreffort(2,0,motor_dir_forward);//inner wheel _delay_ms(1000); step=3; else if(strcmp(msg,"f")==0) 15

16 setmotoreffort(1,speedlow,motor_dir_forward);//outer wheel setmotoreffort(2,speedlow,motor_dir_forward);//inner wheel else if(strcmp(msg,"b")==0) setmotoreffort(1,speedlow,motor_dir_backward);//outer wheel setmotoreffort(2,speedlow,motor_dir_backward);//inner wheel if(step==3)//double check position fprintf(&lcd_str,"\rcheck Cue Position"); if (datainbufe1()) fscanf(&xbee_str,"%c",msg); if (strcmp(msg,"d")==0) _delay_ms(1000); step=4; j=110; if (strcmp(msg,"k")==0) step=step-1; if(step==4)//approach the cue to the white ball fprintf(&lcd_str,"\rapproaching \nthe ball"); j=j+1; if(j<=180) setservoangle(21,j); _delay_ms(200); if(j>180) fprintf(&lcd_str,"\rsorry!\nout of Scope"); _delay_ms(1000); PORTC.DIRSET=0x0F;//strike the solenoid once PORTC_OUTSET=0x0F; _delay_ms(500); PORTC_OUTCLR=0x0F; PORTC.DIRSET=0x80; _delay_ms(2000); fprintf(&lcd_str,"\rsystem \ninitializing"); 16

17 setservoangle(21,150);//initialize servo position for(i=150;i>100;i--) setservoangle(21,i); _delay_ms(100); _delay_ms(1000); setservoangle(20,90); //initialize servo position for(i=90;i<180;i++) setservoangle(20,i); _delay_ms(100); _delay_ms(1000); step=5; _delay_ms(1000); if (datainbufe1()) fscanf(&xbee_str,"%c",msg); if(strcmp(msg,"a")==0) fprintf(&lcd_str,"\rstricking!!"); _delay_ms(1000); PORTC.DIRSET=0x0F;//strike the solenoid once PORTC_OUTSET=0x0F; _delay_ms(500); PORTC_OUTCLR=0x0F; PORTC.DIRSET=0x80; _delay_ms(2000); PORTC.DIRSET=0x0F;//strike the solenoid twice PORTC_OUTSET=0x0F; _delay_ms(500); PORTC_OUTCLR=0x0F; PORTC.DIRSET=0x80; _delay_ms(2000); fprintf(&lcd_str,"\rsystem \ninitializing"); setservoangle(21,150);//initialize servo position for(i=150;i>100;i--) setservoangle(21,i); _delay_ms(100); _delay_ms(1000); setservoangle(20,90); for(i=90;i<180;i++) setservoangle(20,i); 17

18 _delay_ms(100); _delay_ms(1000); step=5; if(step==5)//move back adcchannelmux(&adcb,3,3);//front IR sensor adcchannelmux(&adcb,0,0);//rear IR sensor int IRfront= analogread(&adcb, 3); int IRrear= analogread(&adcb, 0); if( IRfront<1000 &&IRrear<1000)//starting point setmotoreffort(1,0,motor_dir_backward);//outer wheel setmotoreffort(2,0,motor_dir_backward);//inner wheel step=0; else setmotoreffort(1,speed,motor_dir_backward);//outer wheel setmotoreffort(2,speed,motor_dir_backward);//inner wheel 2. OpenCV / Visual Studio 2012 program for shot the purple ball as a sample // camera1.cpp : Defines the entry point for the console application. #include "stdafx.h" #include <cv.h> #include <cxcore.h> #include <highgui.h> #include "SerialPort.h" SerialPort* serialport; IplImage* GetThresholdedImageBlue(IplImage* img) // Convert the image into an HSV image IplImage* imghsv = cvcreateimage(cvgetsize(img), 8, 3); cvcvtcolor(img, imghsv, CV_BGR2HSV); IplImage* imgthreshed = cvcreateimage(cvgetsize(img), 8, 1); cvinranges(imghsv, cvscalar(90, 50, 150), cvscalar(105, 200, 255), 18

19 imgthreshed); // threshold to get the blue color block cvreleaseimage(&imghsv); return imgthreshed; IplImage* GetThresholdedImageWhite(IplImage* img) // Convert the image into an HSV image IplImage* imghsv = cvcreateimage(cvgetsize(img), 8, 3); cvcvtcolor(img, imghsv, CV_BGR2HSV); IplImage* imgthreshed = cvcreateimage(cvgetsize(img), 8, 1); cvinranges(imghsv, cvscalar(18, 1, 200), cvscalar(150, 50, 255), imgthreshed);//white ball //cvinranges(imghsv, cvscalar(22, 130, 200), cvscalar(32, 255, 255), imgthreshed);//yellow ball cvreleaseimage(&imghsv); return imgthreshed; IplImage* GetThresholdedImageBall(IplImage* img) // Convert the image into an HSV image IplImage* imghsv = cvcreateimage(cvgetsize(img), 8, 3); cvcvtcolor(img, imghsv, CV_BGR2HSV); IplImage* imgthreshed = cvcreateimage(cvgetsize(img), 8, 1); //cvinranges(imghsv, cvscalar(22, 130, 200), cvscalar(32, 255, 255), imgthreshed);//yellow ball //cvinranges(imghsv, cvscalar(0, 150, 150), cvscalar(8, 250, 255), imgthreshed); //red ball cvinranges(imghsv, cvscalar(110, 50, 40), cvscalar(170, 150, 200), imgthreshed);//purple ball cvreleaseimage(&imghsv); return imgthreshed; double step=0; int last5x1=0; int last4x1=0; int last3x1=0; int last2x1=0; int last1x1=0; int main() 19

20 // key variable for exiting the viewing program. Given garbage //default. int key = 'c'; // allocate memory for an image to hold snapshots from the camera. IplImage *img; // capture from video device #0 or IPWebcam. Only one should be //uncommented at a time. CvCapture* capture = cvcapturefromcam(4); cvnamedwindow("ip Webcam", CV_WINDOW_AUTOSIZE); cvnamedwindow("cue Blue"); cvnamedwindow("white Ball"); cvnamedwindow("purple Ball"); IplImage* imgscribble = NULL; // position the window cvmovewindow("ip Webcam", 600, 5); cvmovewindow("cue Blue", 1000, 480); cvmovewindow("white Ball", 1000, 230); cvmovewindow("purple Ball", 1000, 5); // While loop that constantly updates the image in the display //window from the camera. This can be exited when the 'q' key is //pressed. while(key!= 'Q' && key!= 'q') // retrieve the captured frame img=cvqueryframe(capture); IplImage* frame = img; //frame = cvqueryframe(capture); if(imgscribble == NULL) imgscribble = cvcreateimage(cvgetsize(img), 8, 3); IplImage* imgbluethresh = GetThresholdedImageBlue(img); CvMoments *momentsblue = (CvMoments*)malloc(sizeof(CvMoments)); cvmoments(imgbluethresh, momentsblue, 1); double moment10blue = cvgetspatialmoment(momentsblue, 1, 0); double moment01blue = cvgetspatialmoment(momentsblue, 0, 1); double areablue = cvgetcentralmoment(momentsblue, 0, 0); static int posx1 = 0; static int posy1 = 0; last5x1=last4x1;//store the previous cue tip blue position in order to make last4x1=last3x1;//sure the robot is already stoped last3x1=last2x1; 20

21 last2x1=last1x1; last1x1 = posx1; posx1 = moment10blue/areablue; posy1 = moment01blue/areablue; IplImage* imgwhitethresh = GetThresholdedImageWhite(img); CvMoments *momentswhite = (CvMoments*)malloc(sizeof(CvMoments)); cvmoments(imgwhitethresh, momentswhite, 1); double moment10white = cvgetspatialmoment(momentswhite, 1, 0); double moment01white = cvgetspatialmoment(momentswhite, 0, 1); double areawhite = cvgetcentralmoment(momentswhite, 0, 0); int posx2 = moment10white/areawhite; int posy2 = moment01white/areawhite; IplImage* imgobjectivethresh = GetThresholdedImageBall(img); CvMoments *momentsball = (CvMoments*)malloc(sizeof(CvMoments)); cvmoments(imgobjectivethresh, momentsball, 1); double moment10ball = cvgetspatialmoment(momentsball, 1, 0); double moment01ball = cvgetspatialmoment(momentsball, 0, 1); double areaball = cvgetcentralmoment(momentsball, 0, 0); int posx3 = moment10ball/areaball; int posy3 = moment01ball/areaball; //compute the angle between white ball and objective ball double a=posy3 posy2; double b=posx3 posx2; double tan=a/b; double angle=atan(tan)*180/ ; angle=(int) angle; double c=(posx2 posx1)*(posx2 posx1); double d=(posy2 posy1)*(posy2 posy1); double dis=sqrt(c+d); dis=(int) dis; printf("blue (%d,%d) \t WHITE(%d,%d) \t PURPLE(%d,%d) \t Angle=%f \t dis=%f \r\n", posx1, posy1, posx2, posy2, posx3, posy3,angle,dis);//320x180 if(step==0)//check playing scope printf("blue (%d,%d) \t WHITE(%d,%d) \t PURPLE(%d,%d) \t Angle=%f \t dis=%f \r\n", posx1, posy1, posx2, posy2, posx3, posy3,angle,dis);//320x180 int posy2min= 1.2*abs(angle)+155; int posy2max= 0.8*abs(angle)+190; printf("scope=[%d,%d]",posy2min,posy2max); if (posy2>=posy2min && posy2<=posy2max && abs(angle)>45 && 21

22 abs(angle)<=90)//in the playing scope serialport = new SerialPort(); serialport >send("y", 1); printf("send to serial port 'Y' \r \n"); step=0.5; else//out of scope serialport = new SerialPort(); serialport >send("n", 1); printf("send to serial port 'N' \r \n"); //break; if (step==0.5) printf("angle=%f \r\n",angle); if (angle> 49 && angle<= 45) serialport = new SerialPort(); serialport >send("c", 1); printf("send to serial port 'c' \r \n"); if (angle> 53 && angle<= 49) serialport = new SerialPort(); serialport >send("d", 1); printf("send to serial port 'd' \r \n"); if (angle> 57 && angle<= 53) serialport = new SerialPort(); serialport >send("e", 1); 22

23 printf("send to serial port 'e' \r \n"); if (angle> 61 && angle<= 57) serialport = new SerialPort(); serialport >send("f", 1); printf("send to serial port 'f' \r \n"); if (angle> 65 && angle<= 61) serialport = new SerialPort(); serialport >send("g", 1); printf("send to serial port 'g' \r \n"); if (angle> 69 && angle<= 65) serialport = new SerialPort(); serialport >send("h", 1); printf("send to serial port 'h' \r \n"); if (angle> 73 && angle<= 69) serialport = new SerialPort(); serialport >send("i", 1); printf("send to serial port 'i' \r \n"); if (angle> 77 && angle<= 73) serialport = new SerialPort(); serialport >send("j", 1); printf("send to serial port 'j' \r \n"); if (angle> 81 && angle<= 77) 23

24 serialport = new SerialPort(); serialport >send("k", 1); printf("send to serial port 'k' \r \n"); if (angle> 85 && angle<= 81) serialport = new SerialPort(); serialport >send("l", 1); printf("send to serial port 'l' \r \n"); if (angle>= 90 && angle<= 85) serialport = new SerialPort(); serialport >send("m", 1); printf("send to serial port 'm' \r \n"); if (angle>86 && angle<=90) serialport = new SerialPort(); serialport >send("n", 1); printf("send to serial port 'l' \r \n"); if (angle>82 && angle<=86) serialport = new SerialPort(); serialport >send("o", 1); printf("send to serial port 'o' \r \n"); if (angle>78 && angle<=82) serialport = new SerialPort(); serialport >send("p", 1); 24

25 printf("send to serial port 'p' \r \n"); if (angle>74 && angle<=78) serialport = new SerialPort(); serialport >send("q", 1); printf("send to serial port 'q' \r \n"); if (angle>70 && angle<=74) serialport = new SerialPort(); serialport >send("r", 1); printf("send to serial port 'r' \r \n"); if (angle>66 && angle<=70) serialport = new SerialPort(); serialport >send("s", 1); printf("send to serial port 's' \r \n"); if (angle>62 && angle<=66) serialport = new SerialPort(); serialport >send("t", 1); printf("send to serial port 't' \r \n"); if (angle>58 && angle<=62) serialport = new SerialPort(); serialport >send("u", 1); printf("send to serial port 'u' \r \n"); if (angle>54 && angle<=58) 25

26 serialport = new SerialPort(); serialport >send("v", 1); printf("send to serial port 'v' \r \n"); if (angle>50 && angle<=54) serialport = new SerialPort(); serialport >send("w", 1); printf("send to serial port 'w' \r \n"); if (angle>45 && angle<=50) serialport = new SerialPort(); serialport >send("x", 1); printf("send to serial port 'x' \r \n"); if (step==1)//compute the line of two balls and detect the postion of cue correspongding with the line //if (posy2 posy3==0&&posy1==posy2) if(posy2 posy3==0) if( posx1>=posx2 10 && posx1<=posx2+10 ) serialport = new SerialPort(); serialport >send("s", 1); printf("serial port S \r \n"); step=step+1; else if (posx1<posx2 10) serialport = new SerialPort(); serialport >send("f", 1); printf("serial port F \r \n"); 26

27 else if (posx1>posx2+10) serialport = new SerialPort(); serialport >send("b", 1); printf("serial port B \r \n"); if(posy2 posy3!=0) int posx1align=(posy2 posy1)*posx3/(posy2 posy3)+(posy1 posy3)*posx2/(posy2 posy3); if( posx1>=posx1align 10 && posx1<=posx1align+10 ) serialport = new SerialPort(); serialport >send("s", 1); printf("serial port S \r \n"); step=step+1; else if(posx1<posx1align 10) serialport = new SerialPort(); serialport >send("f", 1); printf("serial port F \r \n"); else if(posx1>=posx1align+10) serialport = new SerialPort(); serialport >send("b", 1); printf("serial port B \r \n"); //end of step 1 if(step==2) printf("cue lastpos=%d\t currentpos=%d \r\n",last5x1,posx1); if (last5x1==posx1)//double check after stop if (posy2 posy3==0&&posy1==posy2) serialport = new SerialPort(); serialport >send("d", 1); 27

28 printf("serial port D \r \n"); step=step+1; if(posy2 posy3!=0) int posx1align=(posy2 posy1)*posx3/(posy2 posy3)+(posy1 posy3)*posx2/(posy2 posy3); if( posx1>=posx1align 10 && posx1<=posx1align+10 ) serialport = new SerialPort(); serialport >send("d", 1); printf("serial port D \r \n"); step=step+1; else serialport = new SerialPort(); serialport >send("k", 1); printf("serial port K \r \n"); step=step 1;//go back to previous step //compute the distance between the cue tip and white ball if (step==3) //printf("dis=%f \r\n",dis); if (dis<20 && dis>0) serialport = new SerialPort(); serialport >send("a", 2); printf("send to serial port 'A' \r \n"); step=step+1; break; cvshowimage("cue Blue", imgbluethresh); cvshowimage("white Ball", imgwhitethresh); cvshowimage("purple Ball", imgobjectivethresh); 28

29 //cvshowimage("video", frame); // show the image in the window cvshowimage("ip Webcam", img ); cvreleaseimage(&imgbluethresh); cvreleaseimage(&imgwhitethresh); cvreleaseimage(&imgobjectivethresh); delete momentsblue; delete momentswhite; delete momentsball; // wait 10 ms for a key press key=cvwaitkey(10); // Close down the img capturing and windows. cvdestroywindow("ip Webcam"); cvreleasecapture(&capture); return 0; 3. Human-Robot interface program by Visual Studio 2010 <Global.Microsoft.VisualBasic.CompilerServices.DesignerGenerated()> _ Partial Class Form1 Inherits System.Windows.Forms.Form 'Form overrides dispose to clean up the component list. <System.Diagnostics.DebuggerNonUserCode()> _ Protected Overrides Sub Dispose(ByVal disposing As Boolean) Try If disposing AndAlso components IsNot Nothing Then components.dispose() End If Finally MyBase.Dispose(disposing) End Try End Sub 'Required by the Windows Form Designer Private components As System.ComponentModel.IContainer 'NOTE: The following procedure is required by the Windows Form Designer 29

30 'It can be modified using the Windows Form Designer. 'Do not modify it using the code editor. <System.Diagnostics.DebuggerStepThrough()> _ Private Sub InitializeComponent() Dim resources As System.ComponentModel.ComponentResourceManager = New System.ComponentModel.ComponentResourceManager(GetType(Form1)) Me.Button1 = New System.Windows.Forms.Button() Me.Label1 = New System.Windows.Forms.Label() Me.Button2 = New System.Windows.Forms.Button() Me.Button3 = New System.Windows.Forms.Button() Me.Label2 = New System.Windows.Forms.Label() Me.Label4 = New System.Windows.Forms.Label() Me.PictureBox1 = New System.Windows.Forms.PictureBox() CType(Me.PictureBox1, System.ComponentModel.ISupportInitialize).BeginInit() Me.SuspendLayout() ' 'Button1 ' Me.Button1.Font = New System.Drawing.Font("Microsoft Sans Serif", 12.0!, System.Drawing.FontStyle.Bold, System.Drawing.GraphicsUnit.Point, CType(0, Byte)) Me.Button1.ForeColor = System.Drawing.Color.FromArgb(CType(CType(64, Byte), Integer), CType(CType(0, Byte), Integer), CType(CType(64, Byte), Integer)) Me.Button1.Location = New System.Drawing.Point(137, 281) Me.Button1.Name = "Button1" Me.Button1.Size = New System.Drawing.Size(80, 54) Me.Button1.TabIndex = 0 Me.Button1.Text = "Purple Ball" Me.Button1.UseVisualStyleBackColor = True ' 'Label1 ' Me.Label1.AutoSize = True Me.Label1.Font = New System.Drawing.Font("Broadway", 18.0!, System.Drawing.FontStyle.Bold, System.Drawing.GraphicsUnit.Point, CType(0, Byte)) Me.Label1.ForeColor = System.Drawing.Color.Yellow Me.Label1.Location = New System.Drawing.Point(157, 201) Me.Label1.Name = "Label1" Me.Label1.Size = New System.Drawing.Size(435, 27) Me.Label1.TabIndex = 1 Me.Label1.Text = "Please Select an Objective Ball" ' 'Button2 ' Me.Button2.Font = New System.Drawing.Font("Microsoft Sans Serif", 12.0!, 30

31 System.Drawing.FontStyle.Bold, System.Drawing.GraphicsUnit.Point, CType(0, Byte)) Me.Button2.ForeColor = System.Drawing.Color.Red Me.Button2.Location = New System.Drawing.Point(541, 281) Me.Button2.Name = "Button2" Me.Button2.Size = New System.Drawing.Size(84, 53) Me.Button2.TabIndex = 2 Me.Button2.Text = "Red Ball" Me.Button2.UseVisualStyleBackColor = True ' 'Button3 ' Me.Button3.Font = New System.Drawing.Font("Microsoft Sans Serif", 12.0!, System.Drawing.FontStyle.Bold, System.Drawing.GraphicsUnit.Point, CType(0, Byte)) Me.Button3.ForeColor = System.Drawing.Color.Orange Me.Button3.Location = New System.Drawing.Point(345, 280) Me.Button3.Name = "Button3" Me.Button3.Size = New System.Drawing.Size(79, 54) Me.Button3.TabIndex = 3 Me.Button3.Text = "Yellow Ball" Me.Button3.UseVisualStyleBackColor = True ' 'Label2 ' Me.Label2.AutoSize = True Me.Label2.Font = New System.Drawing.Font("Jokerman", 26.25!, CType((System.Drawing.FontStyle.Bold Or System.Drawing.FontStyle.Italic), System.Drawing.FontStyle), System.Drawing.GraphicsUnit.Point, CType(0, Byte)) Me.Label2.ForeColor = System.Drawing.Color.Violet Me.Label2.Location = New System.Drawing.Point(57, 99) Me.Label2.Name = "Label2" Me.Label2.Size = New System.Drawing.Size(644, 51) Me.Label2.TabIndex = 4 Me.Label2.Text = "Welcome to Pool Player Bot Home" ' 'Label4 ' Me.Label4.AutoSize = True Me.Label4.ForeColor = System.Drawing.SystemColors.ControlLightLight Me.Label4.Location = New System.Drawing.Point(109, 416) Me.Label4.Name = "Label4" Me.Label4.Size = New System.Drawing.Size(507, 13) Me.Label4.TabIndex = 6 Me.Label4.Text = "Designed for Course IMDL University of Florida Apr 10, 2013 " & _ 31

32 " Designer: Jiaying Zhang " ' 'PictureBox1 ' Me.PictureBox1.Image = CType(resources.GetObject("PictureBox1.Image"), System.Drawing.Image) Me.PictureBox1.Location = New System.Drawing.Point(-4, 0) Me.PictureBox1.Name = "PictureBox1" Me.PictureBox1.Size = New System.Drawing.Size(749, 115) Me.PictureBox1.TabIndex = 7 Me.PictureBox1.TabStop = False ' 'Form1 ' Me.AutoScaleDimensions = New System.Drawing.SizeF(6.0!, 13.0!) Me.AutoScaleMode = System.Windows.Forms.AutoScaleMode.Font Me.BackColor = System.Drawing.Color.ForestGreen Me.ClientSize = New System.Drawing.Size(744, 438) Me.Controls.Add(Me.Label2) Me.Controls.Add(Me.PictureBox1) Me.Controls.Add(Me.Label4) Me.Controls.Add(Me.Button3) Me.Controls.Add(Me.Button2) Me.Controls.Add(Me.Label1) Me.Controls.Add(Me.Button1) Me.ForeColor = System.Drawing.SystemColors.ActiveCaptionText Me.Name = "Form1" Me.RightToLeft = System.Windows.Forms.RightToLeft.No Me.Text = "Pool Player Control Panel" CType(Me.PictureBox1, System.ComponentModel.ISupportInitialize).EndInit() Me.ResumeLayout(False) Me.PerformLayout() End Sub Friend WithEvents Button1 As System.Windows.Forms.Button Friend WithEvents Label1 As System.Windows.Forms.Label Friend WithEvents Button2 As System.Windows.Forms.Button Friend WithEvents Button3 As System.Windows.Forms.Button Private Sub Form1_Load(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles MyBase.Load End Sub Private Sub Label1_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) 32

33 Handles Label1.Click End Sub Private Sub Button1_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Button1.Click Process.Start("C:\Users\Erica\Desktop\EEL 5666 Intelligent Machines Design Lab\OpenCV\Purple Ball\Debug\camera.exe") End Sub Private Sub Button3_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Button3.Click Process.Start("C:\Users\Erica\Desktop\EEL 5666 Intelligent Machines Design Lab\OpenCV\Yellow Ball\Debug\camera.exe") End Sub Private Sub Button2_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Button2.Click Process.Start("C:\Users\Erica\Desktop\EEL 5666 Intelligent Machines Design Lab\OpenCV\Red Ball\Debug\camera.exe") End Sub Friend WithEvents Label2 As System.Windows.Forms.Label Private Sub Label2_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Label2.Click End Sub Friend WithEvents Label4 As System.Windows.Forms.Label Private Sub Label4_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Label4.Click End Sub Private Sub PictureBox1_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) End Sub Friend WithEvents PictureBox1 As System.Windows.Forms.PictureBox End Class 33