i Real Time Wireless Robotic Arm System by Using Leap Motion Controller ONG TING YONG This Report Is Submitted In Partial Fulfillment Of Requirements For The Bachelor Degree of Electronic Engineering (Computer Engineering) Faculty of Electronic and Computer Engineering University Technical Malaysia Malacca
JUNE 2016 ii
iii
iv
Dedicated to my beloved father and mother v
vi ACKNOWLEDGEMENT First of all, I would like to express my greatest gratitude to Dr. Nurulfajar Bin Abd Manap, my thesis advisor and also my project supervisor for his guidance and advice throughout this project. He helped me out with his greatest knowledge in order to solve some of the problem I faced for my final year project and encourage me to participate some of the competition in order to gain more valuable experience. Next, I would like to thanks to my panel examiners Dr. Lim Kim Chuan and Dr. Syafeeza Binti Ahmad Radzi for spending their valuable time in order to listen my project presentation. Besides, I would like to thanks to all the staff of the Department of Electronic and Computer Engineering (FKEKK), UTeM for their guidance and valuable comments. Nonetheless, I would like to thanks to my teammate supervised under Dr. Nurulfajar Bin Abd Manap as well for sharing their knowledge and experiences throughout this project. Last but not least, I would like to thanks to my family and my friends for giving me support throughout the implementation of this project and I would like to apology for all the inconvenience that I might had made during this project implementation. Thanks.
vii ABSTRACT This report describe the design and the integration of a 5 Degree Of Freedom (DOF) robotic arm into a wireless robotic arm system. Nowadays, the evolution of the technology from time to time bring lots of changes to the human life. One of the evolution until nowadays is the wireless system. A wireless system is a system that can convey the information between two or more points and it is not physically connected by any electrical conductors. The advantages to go wireless can increase the mobility of the technology, increase the scalability, save cost and also save spaces. In this project, a 5 DOF robotic arm controlled by the Leap Motion controller in wireless system are proposed. Generally, a robotic arm have the same function as a normal human arm. It can be used to perform some high-precision task and also handle some hazardous situation for us. However, most of the robotic arm are wire connected to the system which limit the mobility of the robotic system and took up more spaces compared with the wireless system. Hence, the approach to integrate the robotic arm into wireless system are proposed. All the process to integrate the robotic arm into a wireless system will be presented in this thesis. Besides, the method to integrate the robotic arm into a wireless system will be discussed and demonstrated. Furthermore at the end of this project, a prototype robotic arm in wireless system will be presented and the results for this project will be recorded into this thesis.
viii ABSTRAK Laporan ini menerangkan reka bentuk dan integrasi ke atas robotik lengan yang mempunyai 5 darjah kebebasan kepada sistem wayarles. Pada zaman sekarang, evolusi teknologi dari semasa ke semasa telah membawa banyak perubahan kepada kehidupan manusia. Salah satu evolusi adalah sistem wayarles. Sistem wayarles ini merupakan salah satu sistem yang menyampaikan maklumat diantara dua atau lebih sistem dan ia tidak disambungkan secara fizikal oleh konduktor elektrik. Evolusi teknologi dalam sistem wayarles ini membawa manfaat seperti meningkatkan mobiliti, meningkatkan produktiviti, menjimatkan kos dan juga menjimatkan ruang. Dalam projek ini, robotic lengan yang mempunyai 5 darjah kebebasan, dikawal melalui Leap Motion controller secara sistem wayarless akan dibentangkan. Secara umumnya, robotic lengan ini mempunyai fungsi seperti lengan manusia biasa. Ia boleh digunakan untuk melaksanakan tugas yang memerlukan ketepatan tinggi dan juga digunkan untuk menangani keadaan yang berbahaya untuk kita. Walau bagaimanapun, kebanyakan robotic lengan yang digunakan adalah dalam sistem berwayar. Ia menghadkan mobiliti sistem robotik yang digunakan dan ia mengambil ruang yang lebih berbanding dengan sistem wayarles. Oleh itu, cara untuk mengintegrasikan robotic lengan kepada sistem wayarles dicadangkan. Proses mengintegrasikan robotic lengan kepada sistem wayarles akan dibentangkan dalam tesis ini. Selain itu, kaedah- kaedah yang digunakan untuk mengintegrasikan robotic lengan ini akan dibincangkan. Pada keakhiran projek ini, satu prototaip robotic lengan dalam sistem wayarles akan dibentangkan dan hasilnya akan direkodkan ke dalam tesis ini.
ix TABLE OF CONTENT CHAPTER CONTENT PAGE TITLE VERIFICATION FORM AUTHOR RECOGNITION SUPERVISOR RECOGNITION DEDICATION ACKNOWLEDGEMEN ABSTRACT ABSTRAK TABLE OF CONTENT LIST OF TABLE LIST OF FIGURE LIST OF ABBREVIATIONS i ii iii iv v vi vii viii ix xii xiii xvii 1 INTRODUCTION 1.1 Background 1 1.2 Problem Statement 3 1.3 Objectives 4 1.4 Scope of Project 4 1.5 Report Structure 5
x 2 LITERATURE REVIEW 2.1 Introduction 7 2.2 Robotic Arm 7 2.3 Leap Motion Controller 9 2.4 NRF24L01 Transceiver Module 11 2.5 Wireless System 12 2.6 Reviews on Related Works 14 2.6.1 Wireless Robotic Arm via Bluetooth Connection 14 2.6.2 Wireless Robotic Arm via Wi-Fi Connection 16 2.6.3 Wireless Robotic Arm via ZigBee Connection 16 2.6.4 Wireless Robotic Arm via Radio Frequency Connection 17 2.6.5 Comparison between Related Works 19 2.7 Summary 19 3 METHODOLOGY 3.1 Introduction 20 3.2 Project Overview 20 3.2.1 Description 20 3.2.2 Block Diagram 22 3.2.3 Project Implementation 23 3.3 Hardware Part 24 3.3.1 Components and Hardware 24 3.3.2 Integration Process 26 3.4 Software Part 29 3.4.1 Leap Motion 29 3.4.2 Node.js 31 3.4.3 Arduino IDE 32 3.4.4 Notepad++ 34 3.5 Programming Part 35 3.6 Summary 36
xi 4 RESULTS AND DISCUSSION 4.1 Introduction 37 4.2 Input Data from Leap Motion Controller 37 4.3 From JavaScript to Arduino 51 4.4 From Arduino Transmitter to Arduino Receiver 60 4.5 Arduino and Circuitry for Both Transmitter and Receiver Module 62 4.6 From Arduino to Robotic Arm 65 4.7 Visualizer 73 4.8 Summary 74 5 CONCLUSION AND RECOMMENDATION 5.1 Conclusion 75 5.2 Recommendation 76 REFRENCES 78
xii LIST OF TABLES NO. TITLE PAGE 2.1 NRF24L01 transceiver pin and pin description 12 2.2 Comparison of related work 19 3.1 Connection for NRF24L01 RF module and Arduino Mega 2560 board 27 3.2 Pin connection between Arduino Uno and Arduino Mega 2560 28 3.3 Connection between Arduino Mega 2560 (receiver) and servo robotic arm 28 3.4 Details for EmBoss EB-500W power supply 29
xiii LIST OF FIGURES NO. TITLE PAGE 2.1 Robotic Arm with 6 Degree of Freedom (DOF) 9 2.2 Schematic view of the Leap Motion controller 10 2.3 Leap Motion coordinate system 10 2.4 NRF24L01 transceiver module 11 2.5 Robotic system to handle dangerous task 14 2.6 HC-05 Bluetooth module 15 2.7 ZigBee module 17 2.8 NRF24L01 module 18 3.1 Block diagram of the project 22 3.2 PCB board, female header pin and jumper wires 24 3.3 Leap Motion controller 24 3.4 Arduino Mega 2560 and Arduino Uno board 25 3.5 NRF24L01 transceiver module 25 3.6 The hardware for the robotic arm 26 3.7 Leap Motion Control Panel 30 3.8 Leap Motion Visualizer 30 3.9 Node.js command prompt 31 3.10 Standard Firmata code from Firmata library 33 3.11 Receiver code (left) and transmitter code (right) 33 3.12 Notepad++ 34 3.13 Executing the JavaScript coding 34
xiv 4.1 Sample data obtained from the Leap Motion controller when the hand is in center position 38 4.2 Sample data obtained when the hand is positioned towards the left end 39 4.3 Sample data acquired when the hand moved towards the right end 39 4.4 Sample data acquired when the hand is positioned towards the upper end 40 4.5 Sample data obtained when the hand moved towards the lower end 40 4.6 Sample data acquired when the hand is positioned towards the front end 41 4.7 Sample data obtained when the hand moved towards the back end 41 4.8 Leap Motion controller tracking data in x, y and z axis 42 4.9 X-axis rotational data obtained from Leap Motion Controller 43 4.10 Sample data obtained when the hand's position is in the center of Z rotation 43 4.11 Sample data acquired when the hand rotated upward according X-axis 44 4.12 Sample data obtained when the hand spin downward according X-axis 44 4.13 Z-rotational data obtained when the hand rotated rightward 45 4.14 Z-rotational data acquired when the hand spin leftward 45 4.15 Leap Motion controller tracking data in x rotation and z rotation 46 4.16 Sample data obtained for the middle finger and ring finger 47 4.17 Sample data obtained when the fingers in x-axis opened 48 4.18 Sample data obtained when the fingers opened according y-axis 48 4.19 Sample data obtained when the fingers in z-axis opened 49 4.20 Sample data for finger distance when the fingers are closed 50 4.21 Sample data obtained when the fingers are opened 50 4.22 Sample data obtained when the hand is in center position 52 4.23 Sample data obtained when the hand moved to the left end 53 4.24 Sample data acquired when the hand is in the right end 53 4.25 Sample data acquired when the hand moved to the upper end 54 4.26 Sample data obtained when the hand is in the lower end 54 4.27 Sample data obtained when the hand rotated to the right end 55 4.28 Sample data acquired when the hand rotated to the left end 55 4.29 Sample data obtained when the hand rotated to the lower end 56 4.30 Sample data acquired when the hand rotated to the upper end 56
xv 4.31 Sample data for fingers in normal position 57 4.32 Sample data obtained when the fingers are in closed position 57 4.33 Sample data acquired when the fingers are in opened position 58 4.34 Sample of data when the data are grouped as struct data 58 4.35 Sample data obtained when the hand moved to the left end and the palm of the hand rotated toward the right end with fingers closed in Leap Motion controller 59 4.36 Sample data acquired when the hand stand still in middle position and the palm of the hand rotated toward the left end with fingers opened in Leap Motion controller 59 4.37 Schematic diagram designed for both transmitter and receiver module in Arduino board 62 4.38 PCB layout routing for both transmitter and receiver module in Arduino board 62 4.39 Printed PCB layout for both transmitter and receiver module in Arduino board 63 4.40 Transmitter board (Arduino Mega 2560+ Arduino Uno + NRF24L01 transceiver module) connected with host PC 64 4.41 Receiver board (Arduino Mega 2560 + NRF24L01 transceiver module) connected with robotic arm 64 4.42 Orange (Signal pin), Red (VCC pin), Black (ground pin) 65 4.43 Overall connection for the wireless robotic arm system 67 4.44 The robotic arm moved rightward according to the hand's position detected in Leap Motion controller 68 4.45 The robotic arm moved leftward according to the hand's position detected in Leap Motion controller 68 4.46 The robotic arm moved upward according to the hand's position detected in Leap Motion controller 69 4.47 The robotic arm moved downward according to the hand's position detected in Leap Motion controller 69
xvi 4.48 The robotic palm rotated leftward according to the palm's position detected in Leap Motion controller 70 4.49 The robotic palm rotated rightward according to the palm's position detected in Leap Motion controller 70 4.50 The robotic palm moved upward according to the palm's position detected in Leap Motion controller 71 4.51 The robotic palm moved downward according to the hand's position detected in Leap Motion controller 71 4.52 The robotic gripper opened according to the fingers detected in Leap Motion controller 72 4.53 The robotic gripper closed according to the fingers detected in Leap Motion controller 72 4.54 Visualizer by Leap Motion App Home software 73 4.55 Visualizer using HTML and JavaScript coding 73
xvii LIST OF ABBREVIATIONS COM - Communication port DOF - Degree of Freedom FIFO - First In First Out GFSK - Gaussian Frequency Shift Keying IDE - Integrated Development Environment ISM - Industrial, Scientific and Medical PC - Personal Computer PCB - Printed Circuit Board PWM - Pulse Width Modulation RF - Radio Frequency RX - Receiver SPI - Serial Peripheral Interface TX - Transmitter USB - Universal Serial Bus Wi-Fi - Wireless Fidelity
1 CHAPTER 1 INTRODUCTION In this chapter, all the project introduction, background, problem statement, objectives, scope of project and report structure will be discussed. This project is all about controlling a real time robotic arm in wireless system by using the NRF24L01 transceiver module and Leap Motion controller. The robotic arm moved according to the hand detected by the Leap Motion controller while NRF24L01 transceiver module act as the transmitter and receiver module for the robotic arm system. The background of this project will be discussed next, continue with problem statement, objectives, scope of project and the structure of this report. 1.1 Background Nowadays, the invention of the robotic system grows rapidly from time to time to ease human in daily activities and serve the world for a better future. In order to build out a robotic system, the three main branch to develop a good robotic system are mechanical engineering, electronic engineering and computer science engineering. The mechanical engineering will handle the hardware part, the computer science engineering will handle the software part while electronic engineering will handle both hardware and
2 software part. The hardware developer will cover up the design and build up the robotic system while the software developer will focus on how to function the robotic system. Different robotic system are designed according to the requirement of the user to perform specific tasks. The evolution of the robotic system serve human in wide range of areas such as performing hazardous task for human, environmental interaction and navigation, resemble human in appearance, education, trainings and much more. Generally, most of the robotic system are used as industrial robots to perform human task. It can be used to aid human in performing specific task which could bring long term or short term harm for humans, aid in hazardous task, dirty task or even boring task for humans. Besides, robots are widely used in many fields such as underwater exploration, space exploration, military services, fight against crime, car production, duct cleaning, fix oil spills, investigate hazardous environment and so on. To design and construct a robotic system, the most important element to power up a robotic system is the power source. The power source can be batteries, solar power, pneumatic, nuclear energy, hydroelectricity and so on. Without the power sources, it is impossible to function the robotic system. Next, the second element to build up a robotic system is the actuator. Actuator is the control system for the robotic system and it act as a motor to rotate and move the robotic system accordingly. Normally, the actuators are powered up by using the power sources and rotate according to the input signal from the system. Next, the third element to function the robotic system is the computer programming code. Programming code are the most important element for the robotic system to move. Depend on what robot designed, programming code are the essential element to program a robot on how the robot move according to the input signal. It can be C programming, C# programming, VB programming, HTML programming, JavaScript programming and so on. In the past few years, the most common controller for a robotic system are keypad, mouse, keyboard or joystick. However, with the growth of the technologies, numerous of the controller has been invented especially in human-computer interaction technologies. Besides, most of the robotic system are interconnected via wired
3 connection. With the latest technologies nowadays, the interaction between the human and the robotic system in wired system can be done as well by using wireless system. With the integration of wireless communication over the wired communication, the advantages are greatly increase in the mobility for the robotic system, no cables to mess up the system, save cost and controllable from far to handle tasks. The main purpose of this project is to build a wireless robotic arm system controlled by using a hand gestures controller to control the robotic arm simultaneously. This robotic arm system will perform according to our hand gesture movement and the connection between the controller and the robotic arm will be connected via wireless connection. 1.2 Problem Statement Although there are various methods to control a robotic arm system, not all controller are as simplest as possible to control a robotic arm system. Some common controller to control a robotic arm system include mouse controller, keyboard controller, joystick controller, keypad controller and so on. Although all these controller are the most common controller to control a robotic arm system, there are still some difficulties to use these controller. The difficulties to use these controller include lack of the understanding about functionality of each button, time consuming to think and move the robotic arm according ideal movement and hard to control the robotic arm precisely and accurately. In order to overcome these difficulties, a better solution to control the robotic arm can be implemented by using the Leap Motion controller. Leap Motion controller, a hand gestures sensor controller is a new technology that tracks the user's hand simultaneously with accuracy approximately 0.01mm and a frame rate up to 300 frame per second. It can detect the user's hand within its surface area of 24 cm² and field of view up to 150. It is the best solution to control the robotic arm since it is user friendly, easy to understand, easy to use and no trainings are required.
4 Besides, the robotic arm nowadays are fixed in a workplace and connected with the controller via wired connection. If the robotic arm are fixed in a place and connected to the controller via cable connection, it limit the mobility of the robotic arm to move from place to place and control it from far distance. Furthermore, exposed cable without cares could easily damage by the rats, cleaners or even the workers. It may cause power failure, system failure or even worst is sending someone to the emergency room. With the integration of the robotic arm into a wireless system, it allow the users to control the robotic arm from far or handle some dangerous activities such as handling toxic substances, bomb disposal or exploring hazardous environments. Besides, it can be used for asepsis preservation in operating room, military service, deep sea discovery or space discovery to discover the existing and educate the future about the discovery. 1.3 Objectives The purpose of this project is to build up a robotic arm in wireless system by using the Leap Motion controller to detect our hand gesture movement and move the robotic arm simultaneously. In short, the objectives are: 1.3.1 To assemble the robotic arm with Leap Motion controller in wired system. 1.3.2 To integrate the robotic arm system into a wireless robotic arm system. 1.3.3 To develop a real time wireless robotic arm system. 1.4 Scope of Project The scope of work in this project are divided into three main parts where the first part of this project will focus on how to get the Leap Motion data from the Leap Motion controller while the second part of this project will focus on how to utilize the Leap Motion data to control a 5 DOF robotic arm in wired connection. If the robotic arm in the second part moved simultaneously according to the hand detected by the Leap Motion controller, the third part of this project will be the integration of this robotic arm system into a wireless robotic arm system.
5 For Leap Motion controller, the Leap Motion controller is a hand gesture sensor in built with two monochromatic IR cameras and three infrared LEDs to interpret the user's hand accurately and precisely. The Leap Motion data can be obtained via some programming language and perform some formula calculation to convert it into PWM signal from 0 to 255 corresponds to the angular range of 0º to 180º [16] for the servo motors. Next, the PWM signal are feed into a microcontroller to control each servo motor in the robotic arm simultaneously. When the robotic arm move according to the hand detected by the Leap Motion controller, NRF24L01 transceiver module can be added to integrate the robotic arm into a wireless robotic arm system. One will be added to the microcontroller attached with the robotic arm while another one will be added to another microcontroller connected with the PC. NRF24L01 transceiver module were chosen for this project because this module are the cheapest module with high data rate, low power consumption and support up to 100 meter for wireless communication. 1.5 Report Structure For the structure of this report, this report consists of five different chapters. The first chapter of this report will be the introduction of this project and the second chapter will be the literature review of this project. The methodology used for this project falls under Chapter 3 while result and discussion of this project will be included into Chapter 4. Lastly, the last part which is the conclusion and the recommendation of this project will be described in Chapter 5. In Chapter 1, this chapter will introduce about the introduction of this project. The background of this project and the problem statement to be solved in this project were stated. Besides, the objectives of this project and the scope of this project were mentioned in this chapter. For Chapter 2, this chapter will fully discuss about the literature reviews related to this project. The literature review gathered from various sources are divided into two main parts, the first part will be the basic theory and the principles for this project while the second part will be the overall review from some journals related to this project.
6 For Chapter 3, this chapter discuss about the methodology used for this project. In order to achieve the objectives of this project, some problem solving methods and work are stated here. Moreover, the process to construct and the implementation for the wireless robotic arm system will be further discussed in this chapter. For Chapter 4, this chapter identifies the results obtained from this project. Besides, this chapter will present and explain about the project setup and some sort of problem encountered throughout the project. Lastly for Chapter 5, this chapter will be the conclusion for this project. Some future work and recommendations are stated here in order to improve this project.
7 CHAPTER 2 LITERATURE REVIEW 2.1 Introduction This chapter discusses about some published information regarding this project and it will cover with some basic theories together with its principles for a better understanding about this project. Besides, there will be a section discusses about some published project related to this project and a short comparison between each project. Furthermore, this part will conclude the entire summary for the literature review and the alternative to be used throughout this project. 2.2 Robotic Arm Generally, robotic arm are similar as the human arm and it is programmable according to the requirement of the users. It is not easy to build up a robotic arm system because it required the branch of electronic engineering to identify the chips or component used in the robotic arm, mechanical engineering to design and to construct the robotic arm, electrical engineering to control the power input and the power output for each servo motor and computer science to program the operation of the robotic arm system. Generally, a robotic arm system are classified into two main categories. The first