Internship Report - Trondheim, April 2016 3D Model Programming Students : Tutors : Raphaël FITOUSSI Matthias POTTIEZ M. Tomas HOLT M. Jan H. NILSEN
1
Acknowledgements We would like to sincerely thank Mr. Tomas Holt and Mr. Jan H. Nilsen for helping and guiding us for the project. Thanks to them, we learned a new language and we were able to fully understand what we were doing. We would like to thank the NTNU university for welcoming us into their building. Finally, we would also like to thank Mr. Patrick Lebègue and the "Relations Internationales" from our IUT departement to enable us to do the internship in Norway. 2
Abstract This project is an application that allows the user to graphically visualize a set of markers in a 3D environnement. The data used to display the markers are coordinates that are recorded through motion capture. The application has several features and provides a way to modify the data. The software's main purpose is to enable the user to check if some errors have occured during the recording through motion-capture and to correct these errors. Indeed, recording motion with motion -capture is not accurate yet, and markers are likely to swap position or move in an unexpected way. Therefore, this application must provide some tools which bring relevant information to the user regarding the motion of the markers (animation, marker selection and higlighting, display of the previous steps through time ) and allow data modification. Moreover, this software should provide a way to «communicate» with a parent program, and therefore be a module to another program. 3
Summary Introduction...5 I - User guide...6 At first sight...7 1. First part : The screen...8 2. Second part : Time steps manager...9 3. The third part : the functionalities...10 3.1 Camera...10 3.2 File...11 3.3 Selection...11 3.4 Link...12 3.5 Former/Further Steps...13 3.6 Swapping...14 II - Technical documentation...15 1 Languages and libraries used...16 1.1 C++...16 1.2 OpenGL...16 1.2.1 Version used...16 1.2.2 OpenGL optionnal libraries...16 1.3 Qt...16 1.3.1 Version used...17 2 Set up the environment...17 3. Description of the classes...18 3.1 Marker class...18 3.2 Data class...18 3.3 DisplayWindow class...19 3.3.1 Camera movements...19 3.3.2 Picking and picking mode...19 Picking handling...20 3.4 ProgramWindow class...20 3.5 FileWindow class...20 3.6 CoordinatesWindow class...20 3.7 MarkerCoordinatesWidget class...21 3.8 SwapWindow class...21 3.9 SwappingCorrectorProgram class...21 4. Discussion...22 4.1 Problems encountered...22 4.2 Possible improvements...22 Conclusion...23 Figures...24 References...25 4
Introduction Being second-year students at the Institut Universaire de Technologie A of Lille 1 in computer sciences, we had to make a choice between doing an internship in France with a company, or doing an internship in a foreign country. We have chosen to do it outside France for several reasons : Firstly we were interested in getting in touch with the Norwegian culture. Secondly we wanted to improve our english and speak it more fluently. We had heard that Norwegians were speaking english almost as well as their mother tongue so we have decided to go there. Lastly, the NTNU university has a strong reputation and the projects which the former students had to work on looked very interesting and challenging regarding their novelty. Our internship took place in Trondheim. Our tutors were working in VizLab, which is a visualization laboratory. 5
I - User guide 6
At first sight Parts of the program The program is divided into 3 main parts : 1) The screen that is the view of a model in 3D. Each dot in blank represents one point that is represented with X,Y and Z coordinates. The blue bar is the z axis, the green one the y axis and the red one the x axis. 2) The part where the user manages the time steps of the view. This is related to the demo of the view. 3) Here is the main functionalities that the user can use to interact with the screen and modify the model view. 7
1. First part : The screen Screen with multiple cases After the basics (axis shown in red/blue/green and the points in blank), the screen can have differents displays possibilities, depending on what the user has done. Here is a list of what display you could have on the screen and their signification : A blank cross : represents the selection for linking A red cross : represents the selection for swapping Lines between points : the links Unique colored points : the markers Colored lines or same colored points : represents the further/former steps 8
2. Second part : Time steps manager Part of the time steps management In this part, the user can manage the time steps as he wants, to get a view on the screen at any time step he would like. The "Start demo" button can be used to perform a demo and plays without stop the time steps of his model. The "Pause" button is useful to pause at any time step the user would like while he is playing the demo with the "Start demo" button. The "Stop" button is useful to stop at any time step while the user is playing the demo with the "Start demo" button and get back to the first time step. The backward button is used to move backward from one time step each time the user hit it. The hit it. forward button is used to move forward from one time step each time the user These 2 last buttons are useful if the user would like to be really precise and just move slightly before or after. The last tool is the slider : with it the user can drag to any time step he would like to set. The number of the current time step is known with "Step number : ". 9
3. The third part : the functionalities The six tabs of the program The functionalities are separated in 6 tabs that allow the user to modify the view and see some information about it. 3.1 Camera Tab "Camera" "Reset camera" button : reset the view of the screen at the beginning (for example if the user moves the camera, it will be back to where it was at first). "Back side" button : move the camera to make the view set to the back aligned with axis. "Front side" button : move the camera to make the view set to the front aligned with axis. "Left side" button : move the camera to make the view set to the left aligned with axis. "Right side" button : move the camera to make the view set to the right aligned with axis. 10
3.2 File Tab "File" Here there is a choice between two displays of the file coordinates that is used to make the view on the screen : have a view with timesteps in rows and markers in columns or the contrary. After the user has selected one of the two, the "Display file coordinates" have to be pressed to show the file coordinates. 3.3 Selection Tab "Selection" Here is the selection part where the user can select points to put them as "markers". The markers are colored in different color from each other and he can follow them through timesteps. Once the user has selected "select markers" and make sure it is checked, he can directly select the points he wants to mark on the screen. Each time he clicks on a point, the user will add a marker on the tab "Selection" and will see some information : the color of the marker, the x,y and z coordinates of the point for the current step. To remove one marker the user just have to press the button "remove". 11
3.4 Link Tab "Link" The "Link markers" button is used to draw a link between two points that the user wants to connect. Once he has selected "Link markers" and make sure it is checked, he can pick two points on the screen to draw a link between them. The checkbox "display links" is used to let the user the possibility to display links or not that he has drawn before. The "erase one link" button is used to erase one link that the user selects on the screen. The "erase all links" button is used as the "erase one link", but instead, it erases every link the user has drawn. The "save skeleton" button is used to save all the links that the user has drawn into a file. The user can chose where to store the file. 12
3.5 Former/Further Steps Tab "Former/Further Steps" The "display former steps" combined with the number of steps given next to the button displays the former steps on the screen. The "display further steps" do the opposite of the "display former steps", it displays the further steps instead. The user has differents possibilities : he can chose to display the former steps only for the markers that he has selected in the tab "Selection" if he checks "display for selected markers only" and he can also display the former steps and the further steps with lines or with points, it depends on which one is checked in the radiobuttons. 13
3.6 Swapping Tab "Swapping" The swapping tab allows the user to swap two points coordinates with how many steps he wants. To start the swapping mode, the user has to click on the button "swap mode". To swap two points, the user has to select two points like he would do in the "Selection" tab. Every time he selects a point, there is a cross to warn him that he has selected that point. In the tab "Swapping", there are coordinates that are displayed depending of time steps selected with "Numbers of furthers steps to be swapped" that are the coordinates after the current time step. In the display of the points, the user can identify the points he has selected with the color under the number of marker. After the user has selected his points and his number of further steps to be swapped, the user can click on the button "swap markers" to swap the coordinates between these two points. If the user wants to deselect one marker, he can simply click on the point he does not want or he can also click on the black part of the screen to deselect his last point selected. 14
II - Technical documentation 15
1 Languages and libraries used This program has been developed with the C++ programming language as well as the Qt library and the OpenGL API. 1.1 C++ C++ is an Object-oriented programming language. C++ has been chosen to program this application because this language is often used in combination with OpenGL and 3Drendering due to its efficiency. When we could, we tried to use some features of the C++11 standard. 1.2 OpenGL OpenGL is an API which is used for rendering 2D and 3D graphics. OpenGL has been used for this project because it is an open source API and it is more or less cross platform. 1.2.1 Version used We used version 2.0 of OpenGL to program our software. The main reason for this choice is because further versions of OpenGL use shaders and therefore involves shaderprogramming. As our program only needs basic 3D-rendering, version 2.0 of OpenGL suited our needs well and was faster to learn than more recent versions of OpenGL. 1.2.2 OpenGL optionnal libraries OpenGL offers a limited amount of functions for 3D-rendering. Several external libraries such as GLU, GLUT, or GLEW have been created to add new commands to the set of original OpenGL functions. We chose not to use those as most of them are now deprecated. Furthermore, the QtGui module that we used provides functions that can replace some of the commands of those external libraries. 1.3 Qt Qt is a multi-platform User Interface framework. We had to use Qt because the parent program of our module also uses this framework. Moreover, Qt also provides tools to enable the integration of OpenGL in the graphical interface. Lastly, the Qt company allows programmers to use an open source distribution of Qt. 16
1.3.1 Version used We used the last version of Qt included in Qt Creator. To integrate OpenGL in the program we used the QtGui module rather than the deprecated QtOpenGL module. 2 Set up the environment Follow this link to download the open-source version of Qt Creator : https://www.qt.io/download-open-source To create a new project, click on " New project " and select " Empty qmake project " from the " other project " tab. Give a name to the project and go through the following steps. Once the project is created, simply add these two lines at the top of the.pro file created : QT += core gui greaterthan(qt_major_version, 4): QT += widgets Then compile the project. Once the compilation is done, create a "files" and an "icons" directory in the build directory created in the workspace. You can put the files you want to use for the program in the "files" directory and icons in the "icons" directory. 17
3. Description of the classes This section provides a brief description of the classes used in the program. Please refer to the documentation included in the source code if you want further details regarding the classes. Simplified class diagram 3.1 Marker class This class is a representation of the markers displayed on the screen. It holds an array of 3 integers which correspond to the x, y and z coordinates of the marker in space. It also has three different attributes which are RGB color-identifiers. these color variables are used when picking is involved. This topic will be covered in the " DisplayWindow class " section. 3.2 Data class the Data class contains the data used in the program. It has one attribute which is a twodimensionnal QVector of Markers. This class provides methods to load and save the data (loaddata(), savedata()) as well as a member-function to modify the data (swapmarkersdata()). A pointer to a constant instance of the Data class created in the ProgramWindow class is sent to three different classes so that they can access the data : DisplayWindow, CoordinatesWindow and SwapWindow. 18
3.3 DisplayWindow class This class handles the 3D rendering and all the events happenning on the screen where the graphics are displayed. As a result, it inherits from QOpenGLWidget (which enables the rendering of OpenGL graphics) and QOpenGLFunctions_2_0 (which allows the use of OpenGL functions from version 2.0). It contains member variables and methods which set what is to be displayed on the screen (displayfurthersteps, displaylinks) and the way the mouse events are processed (camera movements and picking mode). It also manages QVectors that store the markers picked with the different picking modes (selectedmarkerindexes, linkedmarkerindexes, markerstobeswappedindexes). 3.3.1 Camera movements The user can move the camera around the scene by using the mouse. By pressing the left button and moving the mouse, the camera is turned according to the distance between the mouse start position and its current position. The glrotatef() OpenGL function is used to rotate the camera. Four methods were also implemented to set the camera to a specific angle (movecameratoleftside, movecameratorightside, movecameratobackside, movecameratofrontside). 3.3.2 Picking and picking mode Picking refers here to the ability of the user to select or choose a specific 3D element displayed on the screen. Depending on the current picking mode, clicking on an element allows the user to perform certain actions on it. Only one mode can be used at a time.there are four different picking mode : - Selection : when this mode is turned on, the marker picked is highlighted with a specific color and is added to the coordinatewindow where its coordinates are shown. - Swap : this mode enables the user to choose the two markers which he wants the position to be swapped. - Link : the index of the element picked is stored in a QVector. Once two indexes are stored, a line is drawn between the two markers - Remove Picked Link : this specific mode only works on lines linking two markers. When this mode is turned on and a line is picked, the two marker-indexes are removed from the QVector and the line is not displayed any longer. 19
Picking handling The glupickmatrix() GLU function enables the handling of picking. However, as already stated, we decided not to use the GLU library. Therefore, we had to implement picking in a different way. We decided to a use specific process which is sometimes refered to as object selection using the back buffer and which works as follows : when a click is performed on the screen, the entire scene is redrawn with each element painted with a specific color. The color of the pixel on which the click as been performed is then read and this color is used to identify the element picked. In our program, we decided to use the index of the marker in the QVector member of the Data class, and turn it into a RGB color-identifier. That way, when the glreadpixel() command is used and the color obtained is transformed back, we get an index that allows us to know which marker has been picked. A similar procedure has been used for link/line picking (color-identifier which corresponds to the index of the link in the QVector). 3.4 ProgramWindow class This class represents the main window of the program. It inherits from QWidget and it contains all the different widgets, tabs, buttons and instances of the other classes (Data, DisplayWindow, FileWindow, CoordinatesWindow, SwapWindow). Therefore, in addition to containing the different widgets of the program, it also connects all the different parts of the program together (the signals from the widgets which activate the slots in the different parts of the application). 3.5 FileWindow class This class allows for the graphical representation of the coordinates in a table. 3.6 CoordinatesWindow class The CoordinatesWindow class is a class that controls markers that have been picked with the "selection" mode. It inherits from QWidget. When a marker is picked on the DisplayWindow, a signal containing the index of the marker in the two-dimensional QVector from Data and its color is sent to the CoordinatesWindow. this window shows the color of the selected markers and their x, y and z coordinates for the current step. Each marker is shown on a line. A selected marker can be deselected thanks to the "remove" button on the left of each line. If this button is clicked, a signal is sent to the DisplayWindow which removes the index of the marker from its selectedmarkerindexes QVector member. 20
3.7 MarkerCoordinatesWidget class This class is used to display the markers picked with the "swap" mode. It inherits from QFrame. Each instance of this class shows the color of the marker picked and the x, y and z coordinates for the current step. It also displays the x, y and z coordinates for a number of following steps depending on the numberoffurthersteps member variable of the SwapWindow class (see "SwapWindow class" section below for further details). 3.8 SwapWindow class The SwapWindow class is the class which contains and handles the MarkerCoordinatesWidgets. Each time a marker is picked in "swap" mode, a MarkerCoordinatesWidget is created and added to the SwapWindow. Only 2 markers at the same time can be added to the SwapWindow. Depending on the currentstep and on the numberoffurthersteps attributes, the SwapWindow, updates and modifies the MarkerCoordinatesWidgets. The numberoffurtherstepstoupdate attribute is used when the numberoffurthersteps member variable is set too high considering the currentstep and the total number of steps. For instance, if the total number of steps is 300, the current step is 295 and the number of further steps is 10, the number of further steps to update will be 5 (300-295) as 10 goes beyond the limit of the total number of steps. This member variable is also used when the swapping process is done. 3.9 SwappingCorrectorProgram class This class is a wrapper for the whole program. It is used by the parent program to create an instance of the application and communicate with it by sending a file to this instance so that it can be processed. 21
4. Discussion 4.1 Problems encountered We encountered several issues during the project that we have solved. Firstly, we had trouble setting up the application properly. We had difficulties integrating version 2.0 of OpenGL with a modern version of Qt and the QtGui module. Thanks to our tutors, we managed to fix these problems. The books they advised us to read were also helpful to understand how OpenGL works. Secondly, our program was designed to enable the user to detect the swapping and to correct it when it occured. Nevertheless, it was difficult to implement a feature which would indicate to the user when the swapping happened and which markers were swapping. 4.2 Possible improvements The first thing that could be improved is the camera. We managed to move the camera according to the motion of the mouse. Yet, the movement of the camera is not accurate and consistant depending on the starting angle. We also did not add the possibility to zoom in and zoom out. The most important improvement and one of the hardest would be to implement an automatic correction of swapping. At this stage of the development, the user has to handle the swapping process himself : detect when it happens, select the right markers to swap back and select the number of steps he wants to correct. 22
Conclusion This internship has been a very rewarding experience. This project enabled us to learn new languages. It was also a great introduction to 3D programming. Considering that it is uncommon to do an internship involving 3D programming and that this field is becoming more and more important, we gained an invaluable knowledge by working on this project. We acquired skills in C++, Qt, and OpenGL, understanding how to combine them to achieve our project. We were able to get in touch with advance notions such as picking, swapping and 3D rendering. Being abroad also gave us the opportunity to improve our english speaking skills Discovering a different culture was really rewarding and we look forward to coming back to Norway in the future. 23
Figures Parts of the program...7 Screen with multiple cases...8 Parts of the time steps management...9 The six tabs of the program...10 Tab "Camera"...10 Tab "File"...11 Tab "Selection"...11 Tab "Link"...12 Tab "Former/Further Steps"...13 Tab "Swapping"...14 Simplified class diagram...18 24
References Books Dave Shreiner, Mason Woo, Jackie Neider, & Tom Davis, "The Official Guide to Learning OpenGL, Version 2", OpenGL Programming Guide Fifth Edition, Addison Wesley Edward Angel, "A primer", 3 rd Edition OpenGL, Addison Wesley Websites Opengl-tutorial, Home (www.opengl-tutorial.org) [April 2016] OpenClassrooms, OpenClassrooms (https://openclassrooms.com/dashboard) [April 2016] Qt Centre, Qt Centre Forum Qt Centre Community Portal (www.qtcentre.org/content/) [May 2016] Qt Home, Qt official website (www.qt.io/) [May 2016] 25