Department of Electrical Power Engineering and Mechatronics HAPTIC GLOVE FOR AUGMENTED AND VIRTUAL REALITY

Transcription

1 Department of Electrical Power Engineering and Mechatronics HAPTIC GLOVE FOR AUGMENTED AND VIRTUAL REALITY TAKTIILNE KINNAS TÄIUSTATUD JA VIRTUAALNE REAALSUSE JAOKS MASTER THESIS Student: Nikita Katushin Student code: Supervisor: Professor, Vu Trieu Minh Tallinn, 2018

2 (On the reverse side of title page) AUTHOR S DECLARATION Hereby I declare, that I have written this thesis independently. No academic degree has been applied for based on this material. All works, major viewpoints and data of the other authors used in this thesis have been referenced Author:... /signature / Thesis is in accordance with terms and requirements Supervisor:... /signature/ Accepted for defence Chairman of theses defence commission:... /name and signature/ 1

3 TTÜ Department's title THESIS TASK Student: Nikita Katushin, MAHM Study programme, main speciality: MAHM02, Mechatronics Supervisor(s): Professor, Vu Trieu Minh Consultants: Thesis topic: (in English) Haptic glove for augmented and virtual reality (in Estonian) Taktiilne kinnas rikastatud ja virtuaalse reaalsuse jaoks Thesis main objectives: 1. Programming STM32 microcontroller 2. MATLAB/Simscape Simulation 3. Making prototype Thesis tasks and time schedule: No Task description Deadline 1. Analysis of possible sensors and microcontrollers Software programming Hand prototyping Simulink model Haptic glove assembly Finalizing Master thesis Language:... Deadline for submission of thesis: a Student: a /signature/ Supervisor: a /signature/ Consultant: a /signature/ Terms of thesis closed defence and/or restricted access conditions to be formulated on the reverse side 2

4 CONTENTS PREFACE... 5 List of abbreviations and symbols... 6 INTRODUCTION HUMAN WRIST ANALYSIS DESIGN Block diagram Hardware Microcontroller PWM Driver Vibration Motors Accelerometer/Gyroscope Multiplexer Bluetooth USB Bluetooth module Battery Voltage converter Charger Inertial measurement units Accelerometers Gyroscopes Battery calculation GETTING AND PROCESSING RAW ANGLE Microcontroller adjustment USART communication protocol I2C protocol Reading from the slave Raw data processing Angle calculating Sensor fusion

5 3.7 Character encoding Character decoding SIMULATION SolidWorks Model MATLAB software Serial port inputs Joints D Model PROTOTYPING Experemental results Summary and future recommendations COST ANALYSIS SUMMARY APPENDICES... Ошибка! Закладка не определена. GRAPHICAL MATERIAL

6 PREFACE Motivation for writing the diploma on this topic was found and offered by the student. The final thesis topic was found and proposed by the author of the thesis. First of all, the topic was interesting because it combines several branches of engineering. The thesis is focused on obtaining a working prototype as a proof of concept that it is possible to make a low-cost device using only IMU sensors. Last semester of the study was dedicated to it. I would like to gratitude Programme Director of Mechatronics Mart Tamre for his advices at the initial stages. Also I wold like to gratitude Dhanushka Chamara Liyanage and my supervisor Trieu Minh Vu for the support and their advices, which they were giving me during this semester. 5

7 List of abbreviations and symbols VR AG DoF IMU MEMS MCP TP DMA ADC PWM IDK GPIO SDA SCL MSB LSB ASCII PCB Virtual Reality Augmented reality Degree of Freedom Inertial Measurement Unit Micro-electro-mechanical Metha-carpophalangial Trapeziometacarpal Direct memory access Analog to Digital Converter Pulse-Width modulation Integrated Development Environments General-Purpose input/output Serial Data Line Serial Clock Line Most Significant Bit Least Significant Bit American Standard Code for Information Interchange Printed Circuit Board 6

8 INTRODUCTION Smart gloves have the potential as an interface for human-machine interaction. In the future, they can replace the usual mouse, keyboard and joysticks. Such kind of devices offers a more intuitive interface. Therefore, engineers are finding ways to improve their efficiency. In recent years, different augmented and virtual reality gadgets have been developed. It is due to the new virtual reality wave. In the year 2012 new start-up called Oculus was launched on the Kickstarter campaign to raise funds for the production of new virtual reality headset. The developers promised users a "full immersion effect" due to the use of displays with a resolution of 640 by 800 pixels for each eye. The result of this event was the beginning of new VR technologies development and the rapid growth of investments to the industry. Since 2015, virtual reality technologies have become new technological revolution. From year to year, the possibilities of virtual reality are becoming more available to the mass consumer. Well-known companies are engaged in the development of these technologies with as much effort as possible and make it more widely spread. Virtual reality is the industry in which technologies evolve at the same time with the development of content. After all, if there is virtual reality helmet or glasses, there must be something to interact and to operate. That is why there are various kinds of applications of virtual reality gadgets such as: Movies Games Online streams Social networks Medicine Education Trading Industry Virtual reality software development kit gives an opportunity to create and implement different applications and devices. There are various types of gadgets made by commercial and non-commercial developers and companies. Virtual reality glasses immerse their users into the virtual reality, but different types of interaction with virtual reality, such as gamepads and joysticks, interfere with the effect of presence. The founders of some VR devices often complained that the reason why virtual reality gadgets cannot become wide spread is that there are not well-developed gadgets and tools for interaction with virtual reality. Haptic 7

9 gloves have become an excellent solution. However, one of the disadvantages of such kind of devices is a high price, which sometimes exceeds even the cost of original glasses for virtual reality. Although, virtual reality gloves can offer the user to feel the objects in the virtual environment using haptic feedback, which gives an in-depth effect of presence. In addition, haptic gloves can have various applications in remote, virtual and human-robot interaction such as manipulation tasks. It is also possible to interact not only with virtual reality but with augmented reality as well. As an example, these gloves can be used together with VR headset and stereo cameras remotely controlling robot arm. There are many different devices for virtual and augmented reality that were made by commercial and non-commercial companies and developers. Most of the devices have their own advantages and disadvantages in terms of their abilities and use restrictions. The most part of the devices have very similar implementation, but such devices as FESTO ExoHand [1] and DextaRobotics Dexmo Haptic Feedback Exoskeleton Gloves [2] have external mechanical exoskeleton, which makes them bulky and expensive. At the same time, FESTO ExoHand has good accuracy and can be used in different areas such as the robot and virtual reality manipulation and rehabilitation. From the other hand there are various amount of similar and popular solutions as Manus VR [3], Senso gloves [4], Goldfinger Smart Glove. These devices are based on flex sensors [5], which change their resistance by bending. These sensors are expensive and cannot register resistance change in several bending points. That is why in contrast to high-priced haptic gloves, many low-cost gloves do not provide more than one degree of freedom per finger [6], [7]. In order to solve this problem, several sensors are quite often used to increase possible DoFs. In such kind of implementations several resistive sensors per finger [5], which increases DoF. While these gloves provide additional DoFs, they lacks other important features and increase their price due to resistive (flex) sensors cost. Currently, different types of sensors know as integrated measurement units are presented on the market. Such kind of IMUs has an embedded 3-axis MEMS gyroscope, a 3-axis MEMS accelerometer and even magnetometers, which gives more features and increases flexibility of device based on these sensors. One of the possible disadvantages is need to be recalibrated after some time. The second issue is that existing solutions do not provide high accuracy in hand gesture tracking. All these problems gives an opportunity for further developments in this area. The objective of the thesis is to develop a device that will enable interaction of real arm movements with 3D computer models. For this purpose, three different engineering fields have to be integrated in one device: mechanical, computer and electrical engineering. From mechanical part, it is needed to 8

10 create 3D model that will represent the movements of the human arm in SolidWorks environment. Basic knowledge of electronics and programming have to be used to program microcontroller [8]. STM32 microcontroller will be used for connecting the real world with the computer environment and transferring data. These microcontrollers are well supported by manufacturer. A lot of information is available on the STMicroelectronics website. [8] CubeMX will be used for adjusting microcontroller pinouts and Keil studio for programming a software part. These gloves will be developed using IMU sensors. The sensors will be used in order to gather data movement of human hand. This approach should increase accuracy of device and give additional flexibility of human hand increasing DoF. For this purpose, IMU should be tested and calibrated using complementary filters. The design and implementation of hardware and software as well as proof-ofconcept experiments are presented. Software such as MATLAB, which is able to simulate virtual reality, will be used to program the control of the 3D model. 9

11 1 HUMAN WRIST ANALYSIS Based on anatomical and medical hand analysis of previous studies and research [9], the hand skeleton model has 23 internal DoFs (Figure 1.1). Fig 1.1. Human wrist model Each of the four fingers has four DoFs. The DIP and PIP joints both have one DoF, and the remaining two DoFs are located at the MCP joint. Different from the four fingers, the thumb has five DoFs. Two DoFs are at the trapeziometacarpal joint (also referred to as the carpometacarpal joint), and two are at the MCP joint. The remaining one DoF of the thumb is at the IP joint. The basic flexion/extention and abduction/adduction of the thumb and fingers are performed by the articulation of the 21 DoFs just described. As shown in Figure 1.2, the motions are used to describe rotations toward and away from the palm, which occur at every joint within the hand. The abduction is the movement of separation (e.g., spreading fingers apart), and the adduction motion is the movement of approximation (e.g., folding fingers together). The abduction/adduction only occurs at each finger s MCP joint and at the thumb s MCP and TM joints. Another two internal DoFs are at the base of the fourth and fifth (ring and little finger s) metacarpals, which perform the curve or fold actions of the palm. Fig 1.2. Denotation of flexion/extension and abduction/adduction motions of thumb and fingers. The main goal of the thesis is develope a device using the IMU sensors and knowledge about human hand physiology, which will provide realistic hand animation. For this purpose MPU6050 was chosen. 10

12 [10] MPU6050 has accelerometer and gyroscope on his board and its work will be considered in the following sections. Raw data from these two sensors (gyroscope and accelerometer) can be combined and filtered using complementary filter, which gives an opportunity to calculate accurate angle and position of each finger phalange. 11

13 2 DESIGN The main core of the glove is STM32 F413ZH [8]. The microcontroller implements system control. For hand position control MPU6050 IMU was chosen. MPU6050 has 3-axis accelerometer and 3-axis gyroscope. The sensor provides raw data of gravity acceleration and angular velocity, which can be converted into angles using trigonometric equations. The MPU-6050 features three 16-bit ADCs for digitizing the gyroscope outputs and three 16-bit ADCs for digitizing the accelerometer outputs. For precision tracking of both fast and slow motions, the parts feature a user-programmable gyroscope fullscale range of ±250, ±500, ±1000, and ±2000 /sec (dps) and a user-programmable accelerometer fullscale range of ±2g, ±4g, ±8g, and ±16g [10]. Communication with all registers of the device is performed using I2C communication protocol at 400kHz. In current work, 16 IMU sensors will be used and multiplexer is needed to gather data from all sensors. STM32 sends registers to 74HC4067 and reads data sequentially. Then, the microcontroller gathers data and sends package to computer. In the prototype the communication between STM32 and the computer is established by DMA UART. The vibration motors are used for simulation of tactile feedback and their work are performed by PWM driver. Figure 2.1. System block diagram 2.1 Block diagram The MATLAB software will use data from on-glove accelerometer/gyroscope sensor to move a hand model into virtual space. It will detect hand movement in the virtual environment with one degree precision. In addition to a hand model, which will be the primary collider object for the hand, there will 12

14 be nodes at points corresponding to motor placement map on the hand (see motor placement map), which can act as reference points for force calculation. Fig Software Flowchart 2.2 Hardware In the following section different components will be considered for making working and wearable prototype Microcontroller The main core of the glove is STM32 F413ZH [8]. The microcontroller implements system control and manage to process incoming data passed to it from the accelerometer/gyro and from USART. It will send signals to the PWM Driver TLC5947 to perform the motors control within the glove. It also converts the data from the IMUs to USART format to be sent to the PC. 13

15 The STM32F413 devices are based on the high-performance Arm Cortex-M4 32-bit core operating at a frequency of up to 100 MHz. The STM32F413 devices belong to the STM32F4 access product lines (with products combining power efficiency, performance and integration) while adding a new innovative feature called Batch Acquisition Mode (BAM) allowing to save even more power consumption during data batching. The STM32F413 devices incorporate high-speed embedded memories (up to 1.5 Mbytes of Flash memory, 320 Kbytes of SRAM), and an extensive range of enhanced I/Os and peripherals connected to two APB buses, three AHB buses and a 32-bit multi-ahb bus matrix. All devices offer a 12- bit ADC, two 12-bit DACs, a low-power RTC, twelve general-purpose 16-bit timers including two PWM timer for motor control, two general-purpose 32-bit timers and a low power timer. They also feature standard and advanced communication interfaces. Up to four I2Cs, including one I2C supporting Fast-Mode Plus Five SPIs Five I2Ss out of which two are full duplex. Four USARTs and six UARTs An SDIO/MMC interface An USB 2.0 OTG full-speed interface Three CANs [8] All listed features provide an affordable and flexible way for user to try out concepts and build prototypes, by choosing from the various combinations of performance and power consumption features provided by the STM32 microcontroller. In addition, STM32 microcontrollers are supported by wide choice of Integrated Development Environments. During development the project were used Keil and CubeMX. Keil was chosen as IDK to program using C-code generation. Its features allows debugging code in online and it is supported by STMicroelectronics company, which means that it can use all libraries provided by STM. In CubeMX it is provides wide variety of features. Figure STM32 F413ZH [8] 14

16 2.2.2 PWM Driver Additional hardware is needed to be used in order to control motors. For this purpose, 24-channel PWM Driver will be used. The TLC5947 is a 24-channel, constant-current sink PWM driver. Each channel is individually adjustable with 4096 pulse-width modulated steps.. The software will generate the code to send the data, which will correspond to the correct DC motors. [11] Figure TLC5947 PWM Driver [11] Vibration Motors Variable speed motors will be placed within the glove at contact points that provide optimal coverage of the hand. The variable speed aspect allows for the controlling of the intensity at which vibration is felt, allowing the device to simulate any nature of contact. These motors with diameter 10mm are small, but can reach high motor speeds Vibration motors Accelerometer/Gyroscope The MPU-6050 is integrated 6-axis MotionTracking device that combines a 3-axis gyroscope, 3-axis accelerometer in a small 4x4x0.9mm package. With its dedicated I2C sensor bus. The MPU-6050 MotionTracking device, with its 6-axis integration, on-board MotionFusion, and run-time calibration 15

17 firmware, enables manufacturers to eliminate the costly and complex selection, qualification, and system level integration of discrete devices, guaranteeing optimal motion performance for consumers. The MPU-6050 is also designed to interface with multiple non-inertial digital sensors, such as pressure sensors, on its auxiliary I2C port [10]. The MPU-6050 features three 16-bit analog-to-digital converters for digitizing the gyroscope outputs and three 16-bit ADCs for digitizing the accelerometer outputs. For precision tracking of both fast and slow motions, the parts feature a user-programmable gyroscope full-scale range of ±250, ±500, ±1000, and ±2000 /sec (dps) and a user-programmable accelerometer full-scale range of ±2g, ±4g, ±8g, and ±16g [10]. An on-chip 1024 Byte FIFO buffer helps lower system power consumption by allowing the system processor to read the sensor data in bursts and then enter a low-power mode as the MPU collects more data. With all the necessary on-chip processing and sensor components required to support many motion-based use cases, the MPU-6050 uniquely enables low-power MotionInterface applications in portable applications with reduced processing requirements for the system processor [10]. Communication with all registers of the device is performed using I2C at 100kHz or 400kHz. Additionally, MPU-6050 has temperature sensor on its board. For the power supply the MPU-6050 operates from VDD power supply voltage range of 2.375V-3.46V. Additionally, the MPU-6050 provides a VLOGIC reference pin (in addition to its analog supply pin: VDD), which sets the logic levels of its I2C interface. The VLOGIC voltage may be 1.8V±5% or VDD [10]. In the following work GY-521 module will be used. The choice of the module is related to the simplicity of prototyping, i.e. it has convenient output pins for connecting wires GY-521 [10] 16

18 2.2.5 Multiplexer 74HCT4067 is a single-pole 16-throw analog switch suitable for use in analog or digital 16:1 multiplexer/demultiplexer applications. The switch features four digital select inputs (S0, S1, S2 and S3), sixteen independent inputs/outputs (Yn), a common input/output (Z) and a digital enable input (E). When E is HIGH, the switches are turned off. Inputs include clamp diodes. This enables the use of current limiting resistors to interface inputs to voltages in excess of VCC [12]. Fig CD74HC4067 [12] Bluetooth USB To ensure wireless data transmission, the Bluetooth receiver must be used. The purpose of the USB Bluetooth receiver/transmitter is to receive and transfer data to or from the computer using the serial port as bits. Calculated angles will be encoded and sent to a computer. After that, the data has to be decoded into understandable form for the 3D simulation. In addition, this device should be able to transfer data from the computer generated by the simulator when the virtual hand touches objects and causes activation of the vibration motors. As in the case of receiving data, the data for feedback must be encoded and later decoded for a PWM driver. The use of wireless communication is important in order to preserve freedom of movement and reduce disbelief of environment. Fig USB Bluetooth Receiver/Transmitter 17

19 2.2.7 Bluetooth module The blue tooth receiver should be embedded in the glove and will collect the data and transmit it as it does the USB receiver. The microcontroller will collect the data received by the sensors and then it will send the data packet over the Bluetooth. The data will be transferred serially, once received by the Bluetooth chip connected to the microcontroller on the glove. Once the receiver receives the signals (in bit/character) encoded form, it will have to be decoded through the microcontroller. Once the data is decrypted, it can activate the current to the correct vibration motors through the mapping logic. Fig AT-09 Bluetooth module Battery A 3.6V Lithium Ion battery will be chosen as the power supply for all electronic components on the glove. It will supply power to the vibration motors, sensors and microcontroller. The battery was chosen in that way to be rechargeable and have sufficient lifetime to support any use case of the system. The part that will be purchased is needed to be covered by protection, in order to be safety enough. Fig Lithium Ion Battery [13] 18

20 2.2.9 Voltage converter The boost voltage regulators generate higher output voltages from input voltages as low as 2.5 V. They are switching regulators and have a typical efficiency between 80% to 90%. The available output current is a function of the input voltage, output voltage, and efficiency, but the input current can typically be as high as 1.4 A. This regulator is available with a fixed 5 V, 9 V, or 12 V output. The input voltage, VIN, must be at least 2.5 V and should not exceed the output voltage. Based on this, U3V12F5 model will be chosen. The presented model has output voltage 5V. According to the battery, the input voltage will be 3.6V and it does not exceed the output voltage [14]. Fig Voltage converter [14] Charger A charger is needed to supply battery on the board. Using such kind of devices allows to connect micro USB cable to supply the device. The following charger is a complete constant-current/constant-voltage linear charger for single cell lithium-ion batteries, which is very suitable for portable applications. The charge voltage is fixed at 4.2V, and the charge current can be programmed externally with a single resistor. The TP4056 automatically terminates the charge cycle when the charge current drops to 1/10th the programmed value after the final float voltage is reached. Other features include current monitor, under voltage lockout, automatic recharge and two status pin to indicate charge termination and the presence of an input voltage. The maximum programmable charge current is up to 1000mA and input supply voltage up to 8V [15]. Fig TP4056 [15] 19

21 2.3 Inertial measurement units Inertial measurement unit is an electronic device, which indicates changing orientation in smart phones, video games remote control joysticks, quadcopters, etc. These devices contains several sensors such as accelerometers, gyroscopes or magnetometers. The number of sensor inputs in an IMU are referred to as DoF. Therefore, a chip with a 3-axis gyroscope and 3-axes accelerometer will be a 6-DoF IMU Accelerometers Accelerometers are sensitive to both linear acceleration and the local gravitational field. There are different types of accelerometers. During this study, digital accelerometers will be used, which communicate over I2C communication protocols. The accelerometers will be mounted on glove according to each phalange. Accelerometers sensors are not sensitive to rotation about the earth gravitational field. The most common use us in consumer electronics, such as smartphones, portrait frames, etc. Figure Aceelerometer inside The understanding of MEMS accelerometers can be understood by looking at a microscope image of it. On the figure is seen that the upper proof mass is suspended by the restoring springs. Both a gravitational field directed to the left and a linear acceleration of the package to the right will deflect the proof mass to the left. The deflection of the proof mass is measured from the change in capacitance between the fingers of the proof mass and the sensing plates. A simplified transducer is shown on the figure [16]. 20

22 Figure Simplified model and the equivalent circuit Internal circuit converts the tiny capacitance to a voltage signal, which is digitized and output. Each accelerometer has a zero-g voltage level, which can be found in Appendix 3. Accelerometers also have a sensitivity, usually expressed in mv/g. To produce the final reading it is needed to divide the zero-g level by the sensitivity, which will be shown in the following sections. Computing orientation from an accelerometer relies on a constant gravitational pull of 1g (9.8 m/s^2) downwards. If no additional forces act on the accelerometer (a risky assumption), the magnitude of the acceleration is 1g, and the sensor s rotation can be computed from the position of the acceleration vector. If the Z-axis is aligned along the gravitational acceleration vector, it is impossible to compute rotation around the Z-axis from the accelerometer. Digital accelerometers give information using a serial protocol like I2C or SPI [16] Gyroscopes In contrast to accelerometer, a gyroscope measures angular velocity, not angular orientation. It is needed to initialize sensor and then measure the angular velocity around the x, y and z axes at measured time intervals [16]. ω dt = α, ( ) where ω angular velocity, deg/min, dt measured time intervals, min, α change in angle, deg. Therefore, to get the new orientation angle, it is needed to sum the original and changed angle. Basically, it is integrating, adding many smalls calculated intervals to find current orientation. At the same time, it means that if there is some error, it will become magnified over the time [16]. For example gyroscopic drift will become increasingly inaccurate, which will be considered in the 3.3 section. 21

23 Figure Gyroscope inside Gyroscopes use Coriolis Effect to transform an angular velocity into a displacement. The Coriolis Effect forces act perpendicular to the rotation axis and to the velocity of the body in the rotating frame. The displacement induces a change in capacitance between the mass and the housing. Hence, it transforms the angular rate input to the gyroscope into an electrical output. 2.4 Battery calculation All STM32 boards operate with 3.3V logic. However, the Nucleo-144 board and its shield boards can be powered in three different ways from an external power supply, depending on the voltage used. The three power sources are summarized in the Table When STM32 Nucleo-144 board is power supplied by VIN or E5V, the jumper configuration must be the following: Jumper JP3 on pin 1 and pin 2 for E5V or jumper JP3 on pin 5 and pin 6 for VIN Jumper JP1 OFF (shown on the Fig ) [8]. Figure JP1 is off 22

24 Table External power sources Input power name Connector pins Voltage range Max current Limitation Vin E5V 3.3V CN8 pin 15 CN11 pin 6 CN8 pin 7 CN11 pin 16 7 V to 12 V 4.75 V to 5.25 V 3 V to 3.6 V 800 ma From 7 V to 12 V only and input current capability is linked to input voltage: 800 ma input current when Vin=7V 450mA input current when 7V<Vin<12V 500 ma - - Two possibilities: ST-Link PCB is cut SB3 and SB111 OFF (ST-Link not powered) According to the table 2.4.1, the second way to supply will be chosen and the battery will be connected to the pin 6. To use following settings, the jumper JP3 should be relocated to E5V pins as it is shown on the figure Figure JP3 is set When choosing a battery, the following rules have to be followed: 1. The total current of the model, which is required for normal operation of all prototype. 2. Working time, i.e. how long the model will last on one charge. 3. Operating voltage. The first two requirements directly determine the parameters of the required battery. The third one sometimes varies within small limits. The total current of the model can be calculated adding operating current of all nodes, including microcontroller. The current of the microcontroller can be calculated using CubeMX power consumption calculator. In this software, it is possible to select settings for particular project and put all needed data including battery. 23

25 Fig Setting battery The next step is choosing power mode for MPU. In a step window, it is needed to setup power mode, power range, memory type, voltage source, operating frequency, etc. After that, the user has to choose which peripherals it will use. Either it is possible to choose it manually or using import button from the selected pinout. Figure Edit step window According to this data, the software will calculate total MCU power consumption, which is equal to ma. Figure Power consumption After that, it is necessary to define operating current remaining nodes, which can be found in datasheets. The following table shows how much current will be consumed. 24

26 Table Total current Node name Operating current Number of elements Summary Units GY-521 3, ,4 ma CD74HC ma STM32 F413ZH ma DC motor ma TLC ma Total current: 368,4 ma As it was mentioned above, Li-Ion battery with 3.6V and 2600 mah was chosen for this project. However, according to the table , to use second type of external power sources it is needed to boost voltage in range 4.75V to 5.25V. For this purposes a voltage regulator can be chosen. Voltage regulators take a lower voltage level and bring it to 5 V. In this case, it is necessary to use boost converter, which will bring up the voltage to 5 V. The following equation is used to find power: P = V I, (2.4.1) where P power, W, V voltage, V, I current, A. For example, if Li-Ion battery is used with 3.6V output voltage and 2600 mah, the power can be considered in the following way: V battery I battery = V boosted I boosted (2.4.2) Where, V battery battery voltage, V, I battery battery output current, A, V boosted needed voltage, V, I boosted boosted current, A. Moreover, the result will be: I boosted = = (2.4.3) 25

27 Calculated current is less than max possible current for E5V, which is shown in the table and which means that it is enough for supplying. I max > I boosted > I total (2.4.4) The final step is estimation of working time of the battery. The discharge time directly depends on the total current of the circuit. Therefore, it is possible to calculate lifetime by dividing battery capacity and load current. T = C I (2.4.5) where T working time, hours, C battery capacity, mah, I load current, A. And estimated working time of the battery equals to: T = C I = 2600 = 7.05 (2.4.6) The battery lifetime will be 7.05 h or approximately 7 h and 3 minutes and the last thing what has to be calculated is maximum discharge speed. In the datasheet can be found, that the permissible discharge rate of this battery is 0.2C [13]. This means that the battery can give all stored energy for the time that is determined by dividing one hour by the amount of "C". Take the presented battery and divide 1 hour by = 5 (2.4.7) This means that it can give all stored energy in 5 hours. Therefore, the total current calculated above will not damage the battery, because its discharge time is approximately 7 hours. So the time, which it is possible to discharge the battery without causing damage, is calculated for. And multiplying 0.2 figure by battery capacity it is also possible to calculate the maximum discharge current that it can output = 0.52 (2.4.8) This corresponds to the value presented in the datasheet for the battery. 26

28 3 GETTING AND PROCESSING RAW ANGLE In order to act the model to move, it is necessary to prepare data processing from the sensors. The next section will focus on the adjusting the microcontroller, explaining the algorithm of encoding and decoding data. Methods of obtaining data will be considered using C libraries. Since, the data is raw, it is necessary to convert them into a form that is understandable for 3D model. In addition, the sensors should be calibrated according to the algorithm proposed below. As it was mentioned above, the chosen sensors only can give information about angular displacement or gravity force vectors. Therefore, it means that these data has to be converted into angles. Getting angles is possible by using Euler angles and integrating in case of gyroscopes. There are different ways how to calculate angles. For example, instead of Euler angles it is possible to calculate them using quaternions. Comparing to quaternions, Euler angles are simple and intuitive and they can be used to perform a simple analysis and control. On the other hand, Euler Angles are limited by a phenomenon called "Gimbal Lock". In applications where the sensor will never operate near pitch angles of +/- 90 degrees, Euler Angles are a good choice. Euler angles provide a way to represent the 3D orientation of an object using a combination of three rotations about different axes. 3.1 Microcontroller adjustment CubeMX is a graphical tool for STM32 microcontrollers, which provide an opportunity of easy microcontroller adjustment. CubeMX has the following key features [8]: Easy microcontroller selection covering the whole STM32 portfolio Board selection from a list of STMicroelectronics boards Easy microcontroller configuration (pins, clock tree, peripherals, middleware) and generation of the corresponding initialization C code Easy switching to another microcontroller by importing a previously-saved configuration to a new MCU project Easy exporting of current configuration to a compatible MCU Generation of configuration reports Generation of embedded C projects for a selection of integrated development environment tool chains. STM32CubeMX projects include the generated initialization C code, MISRA 2004 compliant STM32 HAL drivers, the middleware stacks required for the user configuration, and all the relevant files for opening and building the project in the selected IDE. Power consumption calculation for a user-defined application sequence 27

29 Self-updates allowing the user to keep STM32CubeMX up-to-date CubeMX is free of charge application, which can be downloaded from the official STMicroelectronics website. The program itself looks as it is presented on the following figure. Figure CubeMX main window Each of the GPIO pins can be configured by CubeMX software as output, as input or as peripheral alternate function. Most of the GPIO pins are shared with digital or analog alternate functions. All GPIOs are high-current-capable and have speed selection to better manage internal noise, power consumption and electromagnetic emission. If it is needed the input configuration can be locked by following a specific sequence in order to avoid writing to the input/output registers. Fast input/output handling allowing maximum input/otput toggling up to 100 MHz [8]. The following figure is a graphical representation of microprocessor, which the board has. As it can be seen, it has up to 140 adjustable pins. Figure Microprotsessor 28

30 In the following prototype only 10 pins has been set, which means that microcontroller can decrease needed supply current. Therefore, it allows us to decrease power consumption and increase lifetime of the chosen battery. According to the following pin settings, the battery estimation lifetime was considered in the section 2.4. First of all it is needed to adjust microcontroller clock frequency using build in crystal oscillator and clock configuration parameters in CubeMX. Three different clock sources can be used to drive the system clock (SYSCLK): HSI oscillator clock HSE oscillator clock Main PLL (PLL) clock The clock controller provides a high degree of flexibility to the application in the choice of the external crystal or the oscillator to run the core and peripherals at the highest frequency and guarantee the appropriate frequency for peripherals that need a specific clock such as USB OTG, SPI and etc. To provide the highest clock speed HSE clock configuration will be chosen for generation clock speed. STM32 board provides an opportunity to use several clock sources for generation the high-speed external clock signal. For STM32 F413ZH two possible clock sources are granted: HSE external crystal/ceramic resonator HSE external user clock These settings can be chosen in the pinout configuration window under RCC pop-up window. The external crystal/ceramic resonator or HSE has the advantage of producing a very accurate rate on the main clock. Figure RCC window After that, in the clock configuration window the output frequency can be chosen. There are two possible way how to do that. First, user can adjust system clock frequency manually. The second way is that the software can choose appropriate settings automatically. The following settings can be changed in the clock configuration window. 29

31 Figure Clock configuration On the figure it is seen that input frequency for HSE is 8 MHz. After that, PLL control system has to be generated in such way that generates an output signal whose phase is related to the phase of an input signal. By applying empirical method, the PLL has been adjusted at the maximum permissible frequency for the particular board, which equals to 72 MHz. The output frequencies are presented on the left side on the figure Figure Ceramic resonator pins As the result, on the main window two pins has been set to green, which indicates that oscillation has been chosen and crystal ceramic resonator has been switched on. Once the clock speed has been found, the next step will be configure GPIO pins for multiplexer, accelerometer/gyroscope sensors and USART communication. As in the previous section, USART settings can be found in the left configuration window (figure ). STM32 F413ZH has several pins for USART communication. In this case it is needed only one USART, which is seen on the following picture. Figure USART configuration A universal synchronous asynchronous receiver-transmitter is a type of serial interface that can be used to communicate asynchronously or synchronously. Asynchronously means that this protocol does not 30

32 use synchro signal as I2C or SPI protocol. In order of synchronously it is opposite. Receiver-transmitter means that data can be transmitted or/and received. It this project asynchronous signal will be used, which means that there has to be used special conditions: a strict requirement for observing clear time intervals on both sides. UART has two lines RxD and TxD, receive and transmit respectively. It means that it is needed 2 pins. These pins are presented on the figure Figure Rx and Tx pins Next step is adjusting pins for demultiplexer. A demultiplexer has 4 digital inputs, which allows to control analog input signal. A multiplexer has 2 N inputs and N selected lines, which are used to select which single input line to send to the output signal. The described demultiplexer has 4 selected lines, which means 2 4 input signals. Figure I/O pins In addition, for each pin, its speed, mode and state will be adjusted. Pins can work in both analog and digital modes, which is suitable because multiplexer is controlled by changing, which is suitable because multiplexer is controlled by changing these four pins. All pins has to be set to the low state. Figure GPIO configuration The last thing that is neede to do is adjusting I2C pins. For these pins 400 khz speed has been chosen. Figure SDA and SCL pins 31

33 3.2 USART communication protocol The interaction of the microcontroller with personal computer is established through the built-in UART and USB to Serial converter, which allows to connect RS232 serial device to USB port. The data can be also transferred using Bluetooth modules, which also have Rx and Tx. The following figure shows a general representation of the protocol. It is seen that Tx line of the device 1 is connected to Rx line of the device 2 and vice versa. Figure Serial communication between two devices When a device is transmitting a byte, the transmitter initially sets the logical 0 to the Tx line. This is start bit, which defines beginning of the transfer. After that, the transmitter sets the bits of the transmitted byte at certain time intervals, i.e. frequency. A parity bit can be transmitted, which is needed to check the transmission quality. When all bits are transmitted, a stop bit will be set, i.e. logic high level on the transmission line. The number of stop bits can be different. In this case, one stop bit has been chosen. The time intervals are set by baud rate. Both devices must know about this speed. For example, if it is assumed that one baud equals to one bit transferred over the bus in one second, then 9600 baud means that 9600 byte has been transferred during one second. Nevertheless, as it was described before there can be start and stop bits. If these bits will be taken into consideration, then 9600 divided by 10 bits is 960 bytes per second. However, all listed calculations are performed by software. User have to define only baud rate, numbers of stop bits and how big will be data package. Figure Byte transfer In addition, a big advatage of this protocol is possibility to use it for transferring data between device and computer. For this it is needed to make or use UART-RS232-USB adapters. For this project the following configurations were chosen. 32

34 3.3 I2C protocol Getting raw values from gyroscope is performed by I2C library. Library is necessary because of complicity of getting data from IMUs and I2C protocol itself. The physical representation of the protocol is just two wires, called SCL and SDA. SCL is a clock line, it is used to synchronize all data receiving from the bus. However, SDA is the data line, where data transfers over the bus. The SDA and SCL lines have to be connected to the all devices on the I2C bus. Both SDA and SCL lines are open drain, which means that they can drive its output low, but never high. Only if both lines have pull-up resistors to the five volts supply. The whole bus itself represents as it illustrated below: Figure I2C bus The value of resistor is not very important and can vary from 1.8 kohms up to 47 kohms. In the chosen module it is used 4.7 kohms resistors and they are connected to 3.3 V power supply. It means that only 3.3 volts can supply MPU A built-in voltage regulator is used in order to use five volts power supply. This voltage regulator is also shown in Appendix 2. The SCL and SDA lines will always be low, if resistors are missing (nearly 0 volts) and the I2C bus will not work. The devices, which are connected over the I2C bus are either master or slave. The master is always the devices that drives the SCL line and the slave is devices that respond to the master. The difference between slave and master is that only master can initiate a transfer over the I2C bus, slave cannot do that. There can be several slaves on the I2C bus and normally only one master. However it is possible to use more than one master, but it is unusual and do not covered in this work. In this project, the STM32 controller will be the master and the slaves will be GY-521 modules, which consist MPU6050 chips, and PWM drivers. Both master and slave can transfer data over the bus, but this transfer is always controlled by the master [17]. When the master wants to get data from a slave, it begins sending start sequence on the I2C bus. A start and stop sequenes are two special sequences that defined for I2C bus. They are special in that sense that they can be placed only where the SDA is allowed to change while the SCL is high level. If the data 33

35 has being transferred, SDA must remain stable and not change while SCL is high. As the name implies, the start and stop mark the beginning and end of a transaction with the slave device [17]. Figure Start and stop sequence The data is transferred in sequence of 8 bits. The bits are placed on the SDA line starting with the MSB. The next sequence will be transferred to Least Significant Bit. The SCL line always oscillates between high and low statement (figure 3.3.2). In the real life the resistor pulls it high. The device receiving the data sends back an acknowledge bit. So there are actually more than 8 bits. When acknowledge bit is set to low position, then it means that it has received the data and is ready to get another byte. The master should terminate the transfer by sending a stop bit if it sends a high level. The high acknowledge level indicates that cannot accept any further data. Figure SCL and SDA lines The standard I2C speed is from 100 khz up to 400 khz, which is called fast mode. The addresses of I2C devices can be either 7 bits or 10 bits long. The 10 bits addresses are rarely used and will not be considered here. The MPU6050 has two 7 bits addresses depending on which power source has been chosen. If five volts are used to supply it, the address will be 0x68 and for 3.3 volts the address will be 0x69. The 7 bits address means that it is possible to use up to 128 devices on the I2C bus, i.e. 7 bit number can be from 0 to 127 (zero is also counts as an address). However, it is still sent by 8 bits. The extra bit is indicating if the master writes to slave or reads from it. If the bit is zero, then the master will write to the slave and if it is one, then the master will read. The 7 bit address is placed in the upper 7 bits of the byte and the read/write bit in the least significant bit. Figure SCL and SDA lines 34

36 It can be confusing, because all 7 bits will be shifted over by one. In this case, if it is sent 0x68 in hex, 104 in decimal or in binary, the actual result will be in case of writing. From another point of view, instead of 104 the master sends 208 in decimal. And to read it is 209. Therefore the read/write bit just makes the number an odd or even. To start sequence, the master should send a start sequence. It will alert all slave devices that sending is strating and they should be ready for the next commands. Then the master sends the slave device address. The slaves that matches this address will start sending data and any others, which have another addresses will be ignored. Next, the master will send register number inside the slave that it wishes to write to or read from. The number addresses depends on how many registers the slave actually has. It can be found in datasheet of the particular sensor. The MPU6050 module has up to 33 different registers, which can be found using datasheet [10] Reading from the slave Having sent the I2C addresses and the internal register address the master can receive the data byte or bytes (it can be more than one) from the slave. Before reading data from the slave, the master should send internal address that is needed to be read. Hence, a read from the slave starts by writing to it some initial address. So it is needed to send the start sequence, the I2C address of the slave with the read/write bit low and the internal register number. Then, another start sequence will be sent and the I2C address again with the read bit set. Now it is possible to read as many data as user wishes and terminate the transaction with a stop bit sequence. So to read data from the gyroscope/accelerometer: 1. Send a start sequence 2. Send 0xD0 (I2C address of the MPU6050 if 5V supply is used and the R/W bit is low) 3. Send power management bit 0x6B 4. Send a start sequence again 5. Send 0xD1 (I2C address with the R/W bit high) 6. Initializes MPU6050 (Choosing gyroscope and accelerometer sensitivity) 7. Read data from MPU Send the stop sequence [17] 35

37 Generally, the bit sequence looks like this: Figure Complete I2C data transfer. [10] The raw data angles contain in 16 bit format. That is why the MSB and LSB has to be merged after reading. After that it gets 16 bit register, which consists one angle raw data. 3.4 Raw data processing As soon as I2C protocol is needed, a lot of work performed with sequences and registers. That is why it is more convenient and wisely to use already existed libraries. For MPU6050 a library TM_STM32_MPU6050 was found. The use of libraries makes program more readable and makes work with sensors more suitable for controlling data. For example, only one line is responsible for initializing all 16 sensors. Sensitivity is easily changeable by changing arguments. All possible sensitivity settings are listed in Appendix 3. In this example accelerometer with +/- 2 g and gyroscope with +/- 250 /sec sensitivity were chosen. Figure Sensors initializing Once all sensors are initialized in main function, there is no need to return to this function. After that infinite while loop is performed. In this while loop all program works and reads data from sensor. All raw data writes to the following variables: 36

38 Figure Write raw data to variables The received values contain information about the gyroscope rotation and forces acting on the accelerometer axes. These data has to be converted to understandable view according to the sensitivity, which has been chosen during initialization. In the other words, the received data has to be divided by the corresponding sensor sensitivity value, which is shown in Appendix 2. Figure Raw data processing In addition, the clamping function has been written in order to cut excessive information from accelerometers to prevent and clamp instabilities when rotation axes happen to become aligned with gravity. The clamp function restricts values in +/- 1g range. 3.5 Angle calculating The approach how to find angle of various objects with respect to gravity of the earth is executed by an instrument called an inclinometer. It is also known as a tilt indicator, tilt sensor, tilt meter and etc. There are several physical principles, on the basis of which an inclinometer can be created. Most often, the slope is determined by the gravity force, geomagnetic field, gyroscopic effect or indirect measurements. Any of the listed principles has its pros and cons. 37

39 If the gravity force is the only force acting on the object, then in this case the MEMS-accelerometer, an instrument that measures the projection of acceleration on its axis, can be used to determine the angle of inclination of the object. The magnitude of the measured projection determines the angle of inclination. In the real life, besides the force of gravity the other forces can act on the object, such forces caused by rotation, jolting, shaking, etc. Since the force of gravity has a constant value, any additional forces acting on the object will change the output value of the accelerometer, and consequently an error will appear during the calculation of the inclination angle. By applying preliminary processing of the output signal of the accelerometer, it is possible to reduce the influence of the other forces. In the ideal case, where the force of gravity is always in the same plane of X axis of the object, it is possible to calculate and obtain an expression of the projection of the gravity force on the axis using simple trigonometry equations. In a single-axis case, achievement of high resolution on a wide range of measurements is possible only using a high-resolution accelerometer. In addition, such circuit cannot operate in a full range of angles (0º - 360º), because the sine values are the same for angles from Nº to 180º-Nº. To get rid of these disadvantages, the additional axis y, orthogonal to x, should be included. Two axes using gives an opportunity to find all possible angles. In the starting position the x and y axis are in the horizontal plane and the z axis is orthogonal to the x and y axes (Figure 3.5.1). Figure Initial position of accelerometer In the initial position, when the gravity force acts only in the opposite direction along the z axis, it is seen that all angles are equal to 0. From the properties of the sine and cosine functions it follows that while the sensitivity along one axis will decrease, it will also increase on the other. Moreover, the angles can be calculated by the following formulas: 38

40 α = arctan ( A x A z ) (3.5.1) β = arctan ( A y A z ) (3.5.2) γ = arctan ( A y A x ) (3.5.3) where A x gravity force vector along x axis, g, A y gravity force vector along y axis, g, A z gravity force vector along z axis, g. Figure Rotational position of three axes The second step is getting angles using gyroscope. The gyroscope measures not the acceleration and not the angle, but the angular velocity. Therefore, it is much less sensitive to the noise vibrations. To obtain the angle from the angular velocity, the gyroscope readings must be integrated and an initial angle (i.e., zero angle of the gyroscope) added to them. Integration is performed according to the algorithm ( ): α(t) = α(t 1) + rawdata dt (3.5.4) Where, α(t) - current angle, deg, α(t 1) - angle at the previous time period, deg, rawdata - raw data from the accelerometer, deg/ms, dt step time, ms. The program is performed in infinite loop and the time step for this code is defined to be sec. The proposed algorithm is performed on STM32 board by: 39

41 Figure Converting data The program performs in infinite loop and the time step for this code is defined by HAL_GetTick() function. The get tick function provides a tick value in millisecond since the STM32 board has begun running the current program. In contrast to the build-in delay function, using the get tick function and if statement allows the microcontroller to continue operating and executing the other part of the code without any delays and the microprocessor pauses. Therefore, the get tick function performs the recalculations of the complementary filter every 20 milliseconds, because T_OUT constant has been set on Sensor fusion An accelerometer measures inertial force, such as gravity (and ideally only by gravity), but it might also be caused by acceleration (movement) of the device. Even if the accelerometer is relatively stable, it is very sensitive to vibration and mechanical noise. A gyroscope is less sensitive to a linear mechanical movement, the type of noise that the accelerometer suffers from. The gyroscopes have the other types of problems like drift (not coming back to zero-rate value when rotation stops). It is seen that in stable position on the figure 3.6.1, the angle from accelerometer is stable, but received angle from gyroscope has static error called drift. 40

42 Angle (deg) Angles Readings θ_accel θ_gyro Figure Zero angle Averaging the data that comes from the accelerometers and the gyroscopes can produce a better estimate of orientation than obtained using accelerometer data only. For this purpose, several approaches can be chosen such as a complementary filter. At the same time, when the gyroscope rotates around the z axis, no drift is observed. To prove this, the data was read and plotted. This allows to use the rotation around the z axis without using the filter. The Complementary filter is a combination of two or more filters that combines the information from different sources and gets the best value. In this case, the filter combines and estimates readings from accelerometer and gyroscope. The following algorithm performs an integrating and adding part: θ filtered = FK θ accel (t) + (FK 1) θ gyro (t 1) (3.6.1) Where, θ accel angle from the accelerometer, deg, θ gyro angle from the gyroscope, deg, θ filtered filtered angle, deg, FK complementary filter coefficient can be adjusted between 0 and 1. The complementary filter implementation on STM32 board is: Figure Implementation of complementary filter 41

43 Angle (deg) The received data from the gyroscope are presented on the figure 3.6.3, where the blue line represents the raw data from the accelerometer, the green line is the raw data from the gyroscope and the red line represents the result of the complementary filter Angles Readings θ_accel θ_gyro θ_filtered Figure Complementary filter It is seen when the coefficient FK is set to the 0.1, the filtered angle is stable and almost does not have disadvantages of the accelerometer and gyroscope. The result is more stable and almost corresponding to the blue line comparing to the gyroscope. At the same time, the noise from the accelerometer has been reduced due to the stability of gyroscope data in short time terms. However, all the described calculations for all three axes of the accelerometer are based on the assumption that an ideal accelerometer is used. Therefore, it has an ideal sensitivity and does not have any displacement. In the real life, the MEMS accelerometer is a mechanical device and, despite the fact that it is adjusted, it still will have a static "load". At the same time, this will lead to a change in sensitivity and a shift in the zero level of the accelerometer. As a result, the accelerometer will give the tilt angles with an accuracy much worse than the specified. Sensor fusion can reduce the tilt error and its sensitivity. There is no need to use expensive equipment to calibrate the accelerometer. It is enough to remove several readings of the accelerometer, if only gravity acts on it. Taking into account the initial displacement and sensitivity of the sensor, all the obtained values from the accelerometer can be represented using the following equation: 42

44 A 1 = A 0 A g sin α (3.6.2) Where, A 0 initial shift, K sensitivity coefficient, A g the actual force acting on the sensor, which equals 1g, α angel between acting force and sensor axis. The problem of initial calibration is focused to find the quantities A0 and K. In order to find the indicated values, the readings from the accelerometer were taken at 0, 90, 180 and 270 position. And the mathematical result will be: A 2 = A 0 A g sin (α + π 2 ) (3.6.3) A 3 = A 0 A g sin(α + π) (3.6.4) A 4 = A 0 A g sin (α + π 2 ) (3.6.5) Taking into account that sin(α) = sin(α + π) and sin(α + π ) = sin(α π ), after addition of 2 2 expressions (3.4.2), (3.4.3), (3.4.4) the following result will be obtained: A 0 = 1 4 (A 1 + A 2 + A 3 + A 4 ) (3.6.6) To find the sensitivity coefficient, the following trigonometric identities will be used: sin (α + π ) = cos(α) (3.6.7) 2 sin (α + π ) = cos(α) (3.6.8) 2 sin 2 (α) + cos 2 (α) = 1 (3.6.9) Finding sum of squared difference A 1 A 3 and A 2 A 4 will be obtained: (A 1 A 3 ) 2 + (A 2 A 4 ) 2 = 4 K 2 A g 2 (sin 2 (α) + cos 2 (α) ) (3.6.10) Therefore: K A g = 1 2 (A 1 A 3 ) 2 + (A 2 A 4 ) 2 (3.6.11) 43

45 The considered method of calibration does not depend on the initial orientation of the axis, which simplifies its implementation. The described sequence must be performed for each axis of the accelerometer. 3.7 Character encoding In different operating systems, special character sets are used to represent text information. Typically, such set is represented as a table, where each symbol corresponds to a binary sequence of one or more bytes. It is often called character encoding. The most common used table is the American Standard Code for Information Interchange (ASCII) code, which is used for the representation of character information in the MS DOS operating system, in the Windows Notepad and other devices and software, as well as in MATLAB. It is seen that the table below contains 128 characters, where each character is encoded by one byte (8- bits). Each number has decimal, hexadecimal and octal representation. In addition to the characters, this table contains special characters or functions coming first in the list. Some of them can be used as terminator in order to sign end of the line. Figure ASCII table According to the ASCII table, the numbers which correspond to the symbols 0, 1, 2, etc. has different meaning in decimal. For example, character 1 corresponds to 49 in decimal. Therefore, all sent data should be converted from integer to character. Since the function HAL_UART_Transmit_DMA(UART_HandleTypeDef *huart, uint8_t *pdata, uint16_t Size) performs data 44

46 transmit, it also encodes numbers into characters. From the library description, it is seen that this function transfers the data, which were written to the pdata buffer. Data are transferred via USART with 8 bits long; it means that each element of this buffer must be unsigned integer. In other words, each number will be considered as ASCII symbol. However, before sending data each number should be encoded into an understandable form, because ASCII table does not contain negative numbers and numbers greater than 9. In this situation, this is not convenient and imposes restrictions on the data transferring. Gyroscopes can find angles in the range +/-180 degrees, which are floating numbers. Since the ASCII table contains characters from 0 to 9, this means that it is possible to divide the number into separate numbers. For example, if there is a number 359.5, then using the coding algorithm, it is possible to get separate characters "3", "5", "9",,, 5. As soon as character, does not contain any useful information during transferring, It is needed to get rid of it. Practically it means that each number has to be multiplied by 10 and casted to get four-digit integer number. After that number will be turned. It is needed to know which number corresponds to what digit. Coded number writes to the buffer as it is shown below. Figure Keeping an angle in buffer In addition, the number will be checked if its remainder after division contains zero. For example, a number 132,0. In this particular case 0,1 has to be added to the number. It is needed to keep data package equal to quadruple. Having fixed data length is important in terms of decoding them. Decoding algorithm is strictly depends on data package length. Adding 0,1 will not prevent to 3D model position inaccuracy. First, because number will be casted into integer values without floating point. 3.8 Character decoding After sending the data package, it must be processed and converted into an understandable format for the simulator and the 3D model. In general, the data package is a set of digits corresponding to the numbers that were specified during the transfer over the HAL_UART_Transmit_DMA function. The data 45

47 package ends with a terminator NULL or \ 0, i.e. when the buffer is no longer contains the data, the simulator understands that the data package was sent and waits for the next one. To process the data package, a special function for decoding the data was created in MATLAB. Knowing that the package consists of an even numbers of data, this allows to create an algorithm that will decode the data. Using presented algorithm, which is shown below, first it determines the buffer size. This was done to ensure the flexibility and ability to make changes of the glove. For example, to change the number of required axes, the number of used sensors or to change the order of transferred data. However, after determining the buffer size, an intermediate unsigned integer variable x is created. Unsigned integer 16 bit type means that it can contain numbers from 0 to It is important, because data are sent in the range from 180 to 540 degree. The variable fc calculates how many numbers are in one data package, in order to use this variable in the decoding cycle later. In addition, an intarr array is created to contain the decoded numbers. Figure Decoding algorithm The decoding itself occurs in the double for loop, where the data package is divided into 4 cells and then 48 is subtracted from each number to get correct number. Then the obtained number is turned back by multiplying the tenths and hundredths according to the order of the sent number. Next, the number is cast into a signed integer type and the number 360 is subtracted. The entire data package is divided and placed into intarr array. This array will be sent as the function output. 46

48 4 SIMULATION For the visualization of the prototype, it is needed to make simulation in appropriate software. Such kind of programs as MATLAB/Simulink have simulation features for such kind of issues. Using Simulink it is possible to visualize and implement received data in Simscape add-on. Simscape enables to rapidly create models of physical systems within the Simulink environment. With Simscape, it is possible to build physical component models based on physical connections that directly integrate with block diagrams and other modelling paradigms. It is modelled by assembling fundamental components into a schematic. Simscape add-on products provide more complex components and analysis capabilities [18]. Simscape helps to develop control systems and test system-level performance. It is possible to create custom component models using the MATLAB based Simscape language, which enables text-based authoring of physical modeling components, domains, and libraries. The functionality of Simscape enables to parameterize models using MATLAB variables and expressions, and design control systems for the physical system in Simulink. In addition, Simscape supports C-code generation [18]. First, it is needed to install Simscape/Simmechanics add-on using get hardware supported packages features of MATLAB. Second, it is needed to create 3D model, which will be a representation of a human arm. For this purpose, Solidworks software will be used and then the model will be transferred in into MATLAB/Simulink using Simscape. 4.1 SolidWorks Model The 3D arm is representation of human arm should be make in order to transfer it into MATLAB. The design of the following prototype is made corresponding to the human hand analysis. The model must correspond and have sufficient DoFs according to the analysis of the human hand, which was considered in previous paragraphs. The following arm consists of 8 main elements: five fingers, elbow, wrist, biceps. Figure Hand initial position 47

49 4.2 MATLAB software Simulink model consists of three main parts: serial port inputs, join actuators, 3d model. Figure Simulink model Serial port inputs Communication between microcontroller and PC performs due to UART DMA. For this purpose, Serial port configuration function is used. The Serial Configuration block configures parameters for a serial port that user can use to send and receive data. It is needed to set the parameters of used serial port before setting up the Serial Receive and the Serial Send block. It is shown on the following picture. Figure Serial configuration 48

50 For this particular case, communication port 10 was chosen with 9600-baud rate and 8 data bits. After that, the Receive and Send blocks will prompt to add a Configuration block to configure serial port properties. The end of the transferred data ends with the terminator \0 shown in the picture. The terminator is needed to show that data package has ended and not to mix or shift data packages. Data size is corresponds to the value of outputs of sensors. In this particular case, it equals 72 corresponding to the amount of transferred bytes. Figure Serial receive configuration All received data has to be converted from string to float. This problem can be solved using the prevoisly considered algorithm in section 3.8, where each received character will be converted into a number. After that, each number should be multiplied by the number of tens and hundreds according to calculated and transferred number on the microcontroller. All listed calculations are performed in MATLAB function block. As a result function sends array of calculated figures. In addition, the Real-Time Pacer block function is used. The RealTime Pacer block slows down ("paces") simulation time to track real elapsed time. The degree of slowdown is controllable via the Speedup parameter. To use this block, it is needed to copy it anywhere in the model. Figure Serial port inputs block 49

51 4.2.2 Joints Joint actuators were generated automatically during transferring data from Solidworks to Simulink. They are used to receive upcoming data. The joint actuator block actuates hand joints. Joint actuator actuates joints with genetalized torque or angular position, velocity, acceleration motion signals, etc. Basefollower sequence and joint axis determines sign of forward motion. Inputs are Simulink signals. The following parameters can be chosen: Figure Joint actuator menu D Model The final part of Simulink model is implementation of 3D model in Simulink. This block can be created by transferring Solidworks model to the Simulink using Simscape add-on. It allows to create models of physical systems within the Simulink environment. Using this software, it is possible to build models based on connections. Simscape helps to develop control systems and allows to simulate different types of joints. It is possible to parameterize models using MATLAB function blocks. Each block is representation of assembly parts and their connection. 50

52 Figure Representation of the 3D model As it was mentioned in the section 1, each finger has 4 degrees of freedom. One degree of freedom can be simulated by revolute joint. As it was mentioned above, the Metacarpophalangeal joint has two degrees of freedom. Therefore, to simulate this joint several ways can be found. It is possible to use spherical joint, which has 3 degree of freedom or this joint is also can be simulated by two revolute joints. They have to be perpendicular to each other, so first will rotate about x axis and second about z axis. This solution has been chosen because it provides 2 degrees of freedom. Figure Metacarpophalangeal joint 51

53 5 PROTOTYPING In order to justify all presented theoretical algorithms and math, a primary prototype has been made. The prototype has been created using the same parts, which were considered in the section 2. In addition, the glove was bought and the sensors were placed on it. A double-sided tape was used to fix the sensors. Wiring has been performed according to circuit schematics in appendix 5. Figure 5.1 Glove prototype All sensors are placed according to each phalanges and placement map in appendix 5. Each sensor has his own LED, which indicates that it is powered and can be used. A breadboard is used to wire sensors and connect multiplexer to the microcontroller. As it can be seen, each sensor is connected in parallel to prevent voltage drop and supply them correctly. SCL line is also connected in parallel. SDA line is connected to the multiplexer. Four lines S0, S1, S2 and S3 are connected to the microcontroller, which performs the multiplexer control. 52

54 Figure 5.2 Multiplexer and microcontroller In order to send data, a TTL converter MAX3232 [19] was connected to the Tx and Rx pins. This circuit is very suitable for STM32 controllers because it allows to operate a single 3 V to 5.5 V supply. As soon it uses RS232 connector, it is needed to use USB to Serial Converter [20]. Such kind of converters are needed to convent RS232 serial devices to a USB port. Figure 5.3 RS232 and MAX3232 converters 5.1 Experemental results In this section, experimental results and tests will be shown. As soon as the thesis is focused on virtual reality, the result should be presented graphically, which means that different angles of hand and finger phalanges will be plotted. All tests were performed using MATLAB. The visualization follows each graph in order to be more informative as a proof of concept. Since the entering data, which goes through the simulator is quite large and the principle of each finger is similar, several tests with two fingers has been done as a proof of prototype. On the picture below the graphs of the middle finger and the index finger are presented. The figure shows the opened hand. 53

55 Figure Fingers initial position After that, the fingers were squeezed, whis can be seen as angle change. Figure Squeezed fingers Several tests of hand were made to show its position along x, y, z axes to see wrist movements. Figure Hand initial position 54

56 Figure Arm turning In addition, testing videos were recorded, which show that prototype works correctly and corresponds to hand movement in real life. It also shows that hand movement are performed in real time. These materials can be found on YouTube by the following links: Summary and future recommendations One of the advantages of the system is that it can provide 23 degrees of freedom, which equals to the real human arm DoFs and it was the main goal. The glove can fit the different people with different hands. The ability to change software for the personal needs depending on the desired degrees of freedom of model. The same applies to the flexibility of measuring angles. The main disadvantages and future recommendations are mainly connected to the prototyping part. First, the wiring has to be changed in that way that will not to constrain human hand movements. Hence, the way how to connect sensors and microcontroller must be changed. The wires have to be wired in parallel without breadboard. The solution can be found in flexible PCBs, which can be embedded to the glove. The MEMS sensors itself, which are embedded to the used GY-521 modules, are several times smaller. The MEMS sensors can be embedded into flexible PCB without need to use 55

57 modules GY-521, which occupy a lot of space. As an alternative, thin wires can be soldered in parallel and connected to the MEMS. These solutions have to provide more flexibility of human hand movement and avoid its constraints. In addition, the 3D hand can be changed in order to correspond to the real hand, i.e. arrange joints precisely. 56

58 6 COST ANALYSIS In the following section, the entire project cost is considered. Cost estimating is the practice of forecasting the cost of completing a project with a defined scope. It is the primary element of project cost management, a knowledge area that involves planning, monitoring, and controlling a project s monetary costs. The approximate total project cost, called the cost estimate, is used to authorize a project s budget and manage its costs. This is important for understanding of how much effort and money was spent during the prototyping. This is also important for determining a project s eventual scope and for ensuring that projects remain financially feasible, i.e. as one of the most important things of the success of the project is its price in comparison to analogs. The following aspects are considered as labor, equipment, services, software, hardware and facilities costs. During this study different types of software were used. The software was received from official websites for free, that is why its cost will mot be considered. Facilities and labor cost are not considered as well. Therefore, the total cost is the cost of hardware, which were used. The following table shows the total cost of this project. Table 6.1 Cost analysis Item 3-Axis Gyroscope/Accelerometer Vibration Motors Details Tracking hand movement Variable speed, multiple operating voltages Unit Cost Quantity Cost 0, ,12 0, ,20 Battery Lithium Polymer 3,6V 8,50 1 8,50 Bluetooth USB USB receiver 3,03 1 3,03 Bluetooth Module Communicates with dongle and microcontroller 1,74 1 1,74 STM32 F413ZH Data proceeding 16, ,00 Glove One size glove 3,00 1 3,00 PWM Module Drives motors 2,75 1 2,75 DC-DC converter Converts current 3,50 1 3,50 Charger Allows to charge battery using usb port 1,30 1 1,30 Multiplexer Multiplex signals 4,00 1 4,00 Software All used software in the project and its cost Shipping Cost to ship all parts ,82 57 Total: 75,96

59 SUMMARY This thesis was aimed at creating glove prototype, which will be able to read data from human hand. This work is also aimed at immersed studying of STM32 microcontoller. Compering to existing analogs of programmable microcontrollers, the STM32 controller provides more flexibility and at the same time more complex in use, because of features. During the study, two different communication protocols were considered and communication was established between different devices such as computer, microcontroller and sensors. The use of the six-axis sensor makes it possible to achieve sufficient accuracy of the angle. Also, it makes possible to track hand movements, which was impossible for the different versions of the virtual reality glove due to another working principle. The most interesting part is using multiplexer for reading 16 different sensors. As soon as these sensors are using rather difficult communication protocol and given number of sensors together with unknown microcontroller, it makes the performing task more difficult. However, the use of analog multiplexer allows solving this problem. For the future recommendations it is needed to note that there is a big opportunity for modifying the prototype. First, the use of Euler s angles algorithm allows getting correct angles without using difficult math. But at the same time these angles are constrained by 90 degrees. These angles have big disadvantage called gimbal lock, which constrains the use of created prototype. A possible solution is the use of quaternions, which allows to rotate sensors in any position. In addition, the first prototype itself is rather bulky and it is required in modifications. A good idea would be to use flexible PCB, which would remove all unnecessary wires and allow the hand to move more naturally and make own circuits instead of modules. Third, change the engine to a more suitable one. Instead of MATLAB it is possible to use game engines such as Unity, which can be more appropriate for that purposes. There can be created conditions for creating interaction with 3D objects. This would allow realizing of the possibility of feeling objects, which were not considered in this work. In general, the result can be considered as positive, since the task of interacting with virtual reality and transferring hand movements into virtual reality were executed. 58

60 LIST OF REFERENCES [1] "Festo. ExoHand," [Online]. Available: [Accessed ]. [2] "Dexta Robotics," [Online]. Available: [Accessed ]. [3] "Manus VR," [Online]. Available: [Accessed ]. [4] "Senso. Senso Me," [Online]. Available: [Accessed ]. [5] P. Weber, E. Rueckert, R. Calandara, J. Peters and P. Beckerle, "A Low-cost Sensor Glove with Vibrotactile Feedback and Multiple.". [6] E. Rueckert, R. Lioutikov, R. Calandra, M. Schmidt, P. Beckerle and J. Peters, "Low-cost Sensor Glove with Force Feedback for Learning from Demonstrations using Probabilistic Trajectory Representations," [7] D. Nedelkovski, "How to mechatronics," [Online]. Available: [Accessed 11 January 2018]. [8] "STM32 32-bit ARM Cortex MCUs," [Online]. Available: [Accessed ]. [9] L. Kang, C. I-Ming, H. Y. Song and K. L. Chee, "Development of finger-motion capturing device based on optical linear," [10] "Invensense. MPU-6050," [Online]. Available: [Accessed ]. [11] "Texas Instruments. TLC Channel, 12-Bit PWM Driver," [Online]. Available: [Accessed ]. [12] "Texas Instruments. CD74HC4067," [Online]. Available: Everything. [Accessed ]. [13] Lithium-ion Battery (Datasheet), [Võrgumaterjal]. Available: [Kasutatud ]. [14] Robotshop. U3V12F5 (Datasheet), [Võrgumaterjal]. Available: [Kasutatud ]. [15] "NanJing Top Power Corp. TP4056," [Online]. Available: [Accessed ]. 59

61 [16] Using the MPU-6050, [Võrgumaterjal]. Available: [Kasutatud ]. [17] "Robot electronics. I2C protocol," [Online]. Available: [Accessed ]. [18] "MathWorks. Simscape," [Online]. Available: [Accessed ]. [19] Texas Instruments. MAX3232 Multichannel RS-232 Line, [Võrgumaterjal]. Available: [Kasutatud ]. [20] Trendnet. USB to Serial Converter, [Võrgumaterjal]. Available: [Kasutatud ]. [21] G. Saggio, A. Lagati ja G. Orengo, Wireless Sensory Glove System developed for advanced, International Journal of Information Science, pp , [22] G. Mithul, M. Vince ja Q. Ellie, Haptic Glove for VR, October 4, [23] S. Kapoor, P. Arora, V. Kapoor, M. Jayachandran, M. Tiwari and S. Kapoor, "Haptics Touchfeedback Technology Widening the Horizon of Medicine," PubMed Central, 15 March [24] S. Cobos, M. Ferre, M. A. Sanchez Uran, C. Pena ja J. Ortego, Efficient Human Hand Kinematics for Manipulation Tasks, France, [25] "Walyou. GoldFinger," [Online]. Available: [Accessed ]. [26] "Solvelight Robotics. Dexmo," [Online]. Available: Dexmo Haptic Feedback Exoskeleton Gloves for VR. [Accessed ]. [27] "Cyber Glove Systems. CyberGlove," [Online]. Available: [Accessed ]. 60

62 GRAPHICAL MATERIAL APPENDICES Appendix 1 Voltage Regulator Efficiency 61

63 Appendix 2 GY 521 schematic 62

64 Appendix 3 MPU6050 product details 63

65 Appendix 4 Physical pinout MPU-60X0 series 64