Aircraft Smooth Motion Controls with Intel Perceptual Computing SDK. Cédric Andreolli - Intel

Transcription

1 Aircraft Smooth Motion Controls with Intel Perceptual Computing SDK Cédric Andreolli - Intel 1

2 Contents 1 Introduction Playing with the aircraft orientation The forces in our game Plane orientation and direction vector Working with the SDK data Retrieving the data Position world Motion controls implementation The gameplay Changing the pitch Changing the roll Changing the yaw Camera calibration How to define the model s rotation Implementation of the pitch Implementation of the roll Implementation of the yaw Section Conclusion Smoothing the motion controls Analyzing the data A first solution Our solution Section Conclusion Integration in Unity Introduction Project settings PlaneMotion.cs script Conclusion

3 1 Introduction This document explains how to use the data provided by the Intel Perceptual Computing SDK to implement smooth motion controls in a 3D game. The algorithm explained in this paper was used to create a demo with Unity* and the Intel Perceptual Computing SDK where the player controls a plane in a 3D world. To understand how to achieve smooth motion controls, we will first explain how the aircraft moves in a 3D world. Then we will take a look at the data produced by the SDK and show our first version of the motion controls and an algorithm for it. In the last part, we will explain how to implement this algorithm in Unity. This paper focuses on the methodology we used to get the proposed solution. 3

4 2 Playing with the aircraft orientation 2.1 The forces in our game Our goal is not to create a realistic flight simulator, we just want the plane to move in a way that the player thinks is normal. We also want to minimize the impact of the motion controls on the CPU workload because in a game, the CPU usually has a lot of other jobs to process. A good solution to simplify this problem is to use the direction vector instead of the forces vectors. Figure 1 shows what happens when the plane is flying straight forward, up, and down. Figure 1: Flying in three different directions As we can see from the figure, the plane orientation is exactly the same as the direction vector. Taking only this vector in to consideration simplifies the problem. This representation is very close to what happens in the real world as long as the speed of the plane is fast enough. If the speed is not fast enough, the plane will fall, which is called a stall. Figure 2 shows an example of stall. Implementing this behavior will not be explained in this document and to simplify the problem, our plane will always have a speed sufficient to avoid stalls. 4

5 Figure 2: Representation of a stall 2.2 Plane orientation and direction vector In a game, your plane s position must be refreshed at each frame. Fortunately, as we ve noticed in the previous section, the plane orientation and the direction vectors are the same. In most of the 3D engines, you can easily retrieve the orientation of your objects and their positions as well. With those two values, and with the speed of your plane, you can compute its position at the next frame. To simplify the explanations, the unit of the speed will be a distance by frame. The formula used to compute the plane s position at the current frame is: position = lastposition + planeorientation speed position: A 3D vector that holds the position of the plane at the current frame. lastposition: A 3D vector that holds the position at the previous frame. planeorientation: A 3D vector that holds the orientation of the plane at the current frame. speed: A float that holds the speed (distance / frame). The equation would look similar to: This calculus will have to be done at each frame. We will soon see the only parameter we are going to play with is the plane orientation. This will allow the plane to move in an almost realistic way in the 3D world. 5

6 3 Working with the SDK data 3.1 Retrieving the data You can use C++ and C# to access the Intel Perceptual Computing SDK data. Retrieving the data can be done in different ways. You can inherit the UtilPipeline class and read the data from the method: 1 bool OnNewFrame( void ) Or you can request a frame with the method: 1 bool AcquireFrame( bool wait ) The wait parameter can be set to true to wait until a new frame is ready. This also creates a lock on the frame so the function 1 bool ReleaseFrame( void ) will have to be called to release the lock. In a game, we want to evaluate the controls at each frame. For this reason, we will retrieve the SDK data with the second method. Here is a snippet of code in C# (Unity): 1 i f (! pp. AcquireFrame( true ) ) return ; 2 / 3 Put some code here to access SDK data 4 / 5 pp. ReleaseFrame () ; 6 / 7 Process the data here to create smooth controls. 8 / Data processing will be done outside of the critical section to avoid any unnecessary lock. Once the frame is locked, we can request an access to the data by calling the function: 1 bool QueryGeoNode(PXCMGesture.GeoNode. Label body, 2 out PXCMGesture.GeoNode data This function takes two parameters: body: data: Describes the node(s) you are interested in. Contains the position of the selected nodes. 6

7 This function also returns false if the requested nodes are not available, which occurs if the player s hands are not recognizable by the SDK. If the function returns true, the node positions can be read in the data parameter. In our case, we are requesting information between the two hands. When specifying the body parameter, you must specify the hand and the node(s) of interest. To read data from the left hand, you need to send: 1 PXCMGesture.GeoNode. Label.LABEL BODY HAND LEFT 2 PXCMGesture.GeoNode. Label.LABEL HAND MIDDLE 3.2 Position world In the previous part, we explained how to retrieve hand positions with a call to: 1 bool QueryGeoNode(PXCMGesture.GeoNode. Label body, 2 out PXCMGesture.GeoNode data Now that the variable data is set, we want to retrieve a 3D vector with the position of the hand in 3D space. This can be achieved with the positionworld attribute. You must keep in mind that this attribute represents the position of player s hands from the camera point of view (not the screen!). positionworld has three attributes (x, y, and z) that you can use to get the location of player s hands. Figure 3 shows the axis representation used in the SDK. Figure 3: Axis representation 7

8 4 Motion controls implementation 4.1 The gameplay Now that the data can be accessed, it s time to define the gameplay, or the movements players make to go up, down, turn, etc. We explained previously that the plane will have a fixed speed so there is no need of any control on this side. The following section describes the plane motion controls associated with the player movements Changing the pitch The pitch can be described as the orientation of the plane s nose regarding the horizon. For this part, we want to use a movement natural to someone who has piloted a real plane. When flying a plane, to orient the nose in the direction of the sky (go up), you pull back on the joystick. If you want to orient the nose in the direction of the ground (go down), you push the joystick forward. In the game, we will implement the following behavior (see Figure 4): To orient the nose in the direction of the ground, players will bring their hands close to the camera. If they want to orient the nose in the direction of the sky, they will move their hands away from the camera. Figure 4: Controlling the pitch Changing the roll Rolling is the ability to change the angle of the wings regarding the horizon. When a plane rolls left (resp. right), its left (resp. right) wing is under the horizon and the right (resp. left) one is on the horizon. To map the rolling, we want to use a natural movement. Our choice is as follows: 8

9 To roll left, players must put their left hands down and right hands up. To roll right, players must put their right hands down and left hands up. That way, the plane will follow the hand movements as shown in Figure 5. Figure 5: Controlling the roll Changing the yaw The yaw can be described as the possibility to turn without rolling. It is a rotation around the vertical axis. Concerning the game play (Figure 6), we choose to use the following movements: To yaw left, pull your left hand. To yaw right, pull your right hand. Figure 6: Controlling the yaw 4.2 Camera calibration When implementing the motion controls, you will need to define a reference point. This point will be used to determine the amplitude of the movements. For example, if your two hands are close to the reference point, your plane should go straight forward. If you pull your hands just a little bit, your plane should rotate in direction of the sky very slowly. On the other hand, if you pull hard, your plane should rotate really fast. 9

10 When players start to play, they must be able to define a reference point in 3D space from which the commands will be computed. Developers should not choose the reference point because their apps could be used with a lot of different cameras and an absolute value would not work in all cases. The best solution is to let the players choose (when they want) this point. Figure 7 illustrates the reference point. Figure 7: Reference point and pitch In our game, players can define the reference point by putting their thumbs up. As long as their thumbs are up, motion controls are not activated. This way, players can redefine the reference point whenever they want. 4.3 How to define the model s rotation Now that the game play is defined, we must link it with how rotations will be applied to the plane Implementation of the pitch Implementing the pitch is not very complicated once you have your reference point set. To change the pitch, we just need to rotate the model around its x axis. For now, the only missing value is the rotation angle along this axis. This angle is a function that depends on the distance between the hands and the reference point. The problem is that the player s two hands will never be at the exact same Y distance from the reference point. Figure 8 illustrates this problem. 10

11 Figure 8: Y distance from the reference point In our implementation, the solution is quite simple. To compute the pitch angle, we always use the y value of the closest hand to the camera. In Figure 8, we would have chosen the left hand (because y1 < y2). Right now, let s say our game has 30 frames per seconds. The attribute positionworld (that we are using) returns the hand position (from the camera s point of view) in meters. So when users are playing, the values will probably be between -0.4 and 0.4 meters if they want to rotate very fast. With a game around 30 fps we get : = 12 degrees per second It s definitely not a big rotation speed value, but if we just add a coefficient in the multiplication, we can rotate very quickly. Here is how to implement this function in generic language: /* * mainhandy: The Y distance between the main hand * and the camera * secondaryhandy: The Y distance between the secondary * hand and the camera */ void controlpitch(float mainhandy, float secondaryhandy){ //check which hand is the closest to the camera float positiony = (mainhandy<secondaryhandy)? mainhandy : secondaryhandy; //calibrationy is the Y position of the reference point float pitch = calibrationy - positiony; //sensibilityfactor is a coefficient used to increase //the rotation speed 11

12 pitch *= sensibilityfactor; transform.rotatearoundlocal(transform.right, pitch); Implementation of the roll The roll is a rotation along the Y axis. Roll can be done by moving one hand up to the same height as the other (also see Section 4.1.2). This time, the reference point will not be used because the only thing that matters is the height difference between the left and the right hands (Figure 9). The rotation is simple to compute: Figure 9: Z distance between the hands roll = mainhandz secondaryhandz For the orientation of your plane, world coordinates, and implementation, you may have to reverse the rotation. To do that, just apply a -1 factor to the roll or reverse the subtraction parameters: roll = secondaryhandz mainhandz Once again, you can multiply your roll by a coefficient to increase the rotation speed. See the implementation in generic language: /* * mainhandz: The Z position of the main hand in * world coordinates * secondaryhandz: The Z position of the secondary * hand in world coordinates */ void controlroll(float mainhandz, float secondaryhandz){ float roll = mainhandz - secondaryhandz; 12

13 //sensibilityfactor is a coefficient that //we use to increase the rotation speed roll *= sensibilityfactor; RotateAroundYAxis(myPlane, roll); As you can see, the two code snippets are symmetric Implementation of the yaw The yaw works in the same way as the roll but with a rotation around the Z axis. This time, we need to measure the distance between the hands along the Y axis. Figure 10 illustrates this point. Figure 10: Y distance between the hands The rotation is computed as follows: yaw = mainhandy secondaryhandy If the motion controls are reversed, you just need to reverse the subtraction: yaw = secondaryhandy mainhandy Here is the code in a generic language: /* * mainhandy: The Y distance between the main hand * and the camera * secondaryhandy: The Y distance between the * secondary hand and the camera */ void controlyaw(float mainhandy, float secondaryhandy){ 13

14 float yaw = mainhandy - secondaryhandy; //sensibilityfactor is a coefficient that //we use to increase the rotation speed yaw *= sensibilityfactor; RotateAroundZAxis(myPlane, yaw); Section Conclusion Implementing the motion control this way was not too hard, but the result is pretty bad. Whether you are close or far from the reference point, rotation acceleration regarding your hand positions is linear. What we really want is to have very small moves when the hands are close to the reference point, and large moves when the hands are far from this point. We re going to show you how to do this in the next section. 14

15 5 Smoothing the motion controls Analyzing the data If you take a look at the data we are manipulating, the pitch, roll, and yaw computed at each frame are between and Figure 11 shows the distribution of the motions realized by the player around the reference point. As we can see, most of the moves of the player are very small. This means that most of the time, the player is looking for accuracy. What we want is to decrease the rotation factor of small moves and to increase the rotation factor of larger moves. This way, it would be possible to have very accurate controls when you are close to the reference point and very fast rotations when you are far from it A first solution Figure 11: Pitch distribution on a flight In an actual game, motion controls should not be a CPU consuming task. Indeed, the CPU should be used to compute physics, animate characters, etc. If smooth motion controls means a lot of CPU work, the rest of the game s performance will suffer. The well-known square function, which can decrease small values and increase bigger ones, seems to be well adapted to our problem. Let s try it with our data (we will only use absolute values here). Figure 12 shows the square function applied to the pitch values. 15

16 Figure 12: The square function We are not too far from what we would like to get. We can see in Figure 12 that small values are even smaller. This would allow players to have really accurate motion controls when they are close to the reference point. But the problem here is that we still can t get a high rotation factor Our solution This problem can be solved easily. The square function has a very interesting property. x R,x < 1 f(x) = x 2 < x x R,x >= 1 f(x) = x 2 >= x We first need to decide the gap from which the rotation will increase very fast. Let s say that we would like to increase the rotation speed when the distance is greater or equal to 0.2 meters. This means that when we reach a distance greater than 0.2 meters, we must send a value greater than 1 to the square function. Actually, we just need to remap the values from 0 to 0.2 to between 0 and 1. A simple multiplication can do the job. 1/0.2 = 5 We just need to multiply our values by 5 before sending them to the square function. But we are not completely done because what we really want is a function that decreases the values smaller than x=0.2 and increases the values greater than x=0.2. f(x) = (x 5) (x 5) 16

17 Figure 13: The f(x)=(x*5)*(x*5) function Figure 13 shows the new square function. As you can see, we are far from what we want. Something is missing, and we can guess that we are looking for a function close to: Dividing by z should allow us to get what we are looking for. Here is how to find the value of z. What we know is: f(0.2) = 0.2 Then: Figure 14 shows the graph of this function. We can see that it is exactly what we are looking for. The new curve crosses the f(x)=x curve for x=

18 Figure 14: The f(x)=x*x*5 function Section Conclusion The result is important, but it s more important to understand how to get there. In this part, we have seen how to smooth the first version of motion controls with the Intel Perceptual Computing SDK. As you can see in this case, we applied a modified square function to the values returned by the SDK. In addition to smooth motion controls, such a function is very easy to compute and is definitely not a CPU consuming task. 6 Integration in Unity 6.1 Introduction In this part, we will explain how to implement this specific algorithm to control an object (a plane in this example) in Unity. This paper will not explain Unity s basics. We will just focus on the algorithm implementation with the SDK. 6.2 Project settings In Unity, the Intel Perceptual Computing SDK works as a plugin. The bad news is that plugin utilization is limited to Unity Pro. If you don t have Unity Pro, but want to give it a try and experiment with the SDK, you can get a free trial of Unity Pro for 30 days. The first thing to do is to create a folder named Plugins in your Assets directory. The Intel Perceptual Computing SDK plugin must be added in the directory: <project>/assets/plugins. Those files can be retrieved in: <SDKInstallationFolder>/framework/Unity/hellounity/Assets/Plugins. You can copy all the files. 18

19 6.3 PlaneMotion.cs script Now we need to create a C# script that we name PlaneMotion. This script must be attached to our plane. The main part of the script is contained in the Update() function. public class PlaneMotion : MonoBehaviour { //We must use the gesture mode (we will use gesture //matching to calibrate the camera) private PXCUPipeline.Mode mode = PXCUPipeline.Mode.GESTURE; //The UtilPipeline wrapper private PXCUPipeline pp; //We also define a speed. This parameter is public //and can be redefined easily in Unity. public float speed = 2.0f; //calibrated indicates whether or not the game //is calibrated public bool calibrated; //calibrationy holds the y position of the //reference point.this is the only value that //matters. private float calibrationy; void Start() { pp = new PXCUPipeline(); pp.init(mode); calibrated = false; void Update() { PXCMGesture.GeoNode mainhand; PXCMGesture.GeoNode secondaryhand; //Compute the rotation with the hand position if (!pp.acquireframe(true)) return; if (pp.querygeonode( PXCMGesture.GeoNode.Label.LABEL_BODY_HAND_LEFT PXCMGesture.GeoNode.Label.LABEL_HAND_MIDDLE, out mainhand) && pp.querygeonode( PXCMGesture.GeoNode.Label.LABEL_BODY_HAND_RIGHT PXCMGesture.GeoNode.Label.LABEL_HAND_MIDDLE, out secondaryhand)) { //We shouldn't need the following line but //I already encountered some troubles. //This function will be explained later. checkhands(ref mainhand, ref secondaryhand); 19

20 if (!calibrated) { //If the reference point is not set //we try to calibrate calibrate(ref mainhand); pp.releaseframe(); //Then we return return; else { //Even if the game is calibrated, the player //can ask for re-calibration calibrate(ref mainhand); else { //if one of the hands can't be retrieved //, the game is no more calibrated. calibrated = false; pp.releaseframe(); //If the game is not calibrated at //this point, we must return. if (!calibrated) return; //The next part retrieve the data that will //be used to compute the rotation angles. float mainhandy = mainhand.positionworld.y; float mainhandz = mainhand.positionworld.z; float secondaryhandy = secondaryhand.positionworld.y; float secondaryhandz = secondaryhand.positionworld.z; //Then we compute and apply the rotation angles controlroll(mainhandz, secondaryhandz); controlyaw(mainhandy, secondaryhandy); controlpitch(mainhandy, secondaryhandy); The checkhands(...) function checks that the hands are mapped in the right order. It is a very simple function that reverses the hand references if the mapping is not as expected. void checkhands(ref PXCMGesture.GeoNode mainhand, ref PXCMGesture.GeoNode secondaryhand){ if(mainhand.positionworld.x > secondaryhand.positionworld.x){ PXCMGesture.GeoNode temp = mainhand; mainhand = secondaryhand; 20

21 secondaryhand = temp; The Update() function defines the algorithm structure and includes some of the SDK calls. Let s look in detail at the calibration function. This function verifies if the player has one thumb up on either hand. If one thumb is up, we register the hand Y value and set the calibrated value to true. void calibrate(ref PXCMGesture.GeoNode mainhand){ PXCMGesture.Gesture datamain; PXCMGesture.Gesture datasecondary; //Thumb up on the first hand? if(pp.querygesture( PXCMGesture.GeoNode.Label.LABEL_BODY_HAND_PRIMARY, out datamain)){ if(datamain.label == PXCMGesture.Gesture.Label.LABEL_POSE_THUMB_UP){ calibrated = true; calibrationy = mainhand.positionworld.y; else if(pp.querygesture(//thumb up on the other? PXCMGesture.GeoNode.Label.LABEL_BODY_HAND_PRIMARY, out datasecondary)){ if(datasecondary.label == PXCMGesture.Gesture.Label.LABEL_POSE_THUMB_UP){ calibrated = true; calibrationy = mainhand.positionworld.y; The last part concerns the computation of the rotation factor for each axis. The code is very close to the explanations provided in Section 4. void controlroll(float mainhandz, float secondaryhandz){ float roll = mainhandz - secondaryhandz; //Here is the trick to smooth the moves roll *= Mathf.Abs (roll); roll *= sensibilityfactor; transform.rotatearoundlocal(transform.forward, roll); void controlyaw(float mainhandy, float secondaryhandy){ float yaw = mainhandy - secondaryhandy; //Here is the trick to smooth the moves yaw *= Mathf.Abs (yaw); yaw *= sensibilityfactor; transform.rotatearoundlocal(transform.up, yaw); void controlpitch(float mainhandy, float secondaryhandy){ float positiony = (mainhandy<secondaryhandy)? 21

22 mainhandy : secondaryhandy; float pitch = calibrationy - positiony; //Here is the trick to smooth the moves pitch *= Mathf.Abs (pitch); pitch *= sensibilityfactor; transform.rotatearoundlocal(transform.right, pitch); In this implementation, you can see that we use the absolute value instead of the raw value. The reason is simple we don t want to change the sign of the angle computed. As for, with z=5 We are doing the same thing in the code snippet: 1 pitch = Mathf.Abs ( pitch ) ; // pitch = x x 2 pitch = sensibilityfactor ; // pitch = 5 7 Conclusion This paper introduced a simple method to implement smooth motion controls with the Intel Perceptual Computing SDK in a flight simulator game. It also shows the method used to find this solution. The chosen solution has little impact on the CPU and can be adapted to other similar problems. We ve also seen that a simple function such as the square function does a very good job. In other situations, the Log, or square root, function can have interesting behaviors too. An important step when looking for an algorithm is to express the problem and the solution. Once the solution is expressed, it s easier to find a way to solve it. Here, expressing the solution was: I am looking for a function that decreases values smaller than 0.2 and increases values greater than 0.2. Notices INFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH INTEL PRODUCTS. NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS IS GRANTED BY THIS DOCUMENT. EXCEPT AS PROVIDED IN INTEL'S TERMS AND CONDITIONS OF SALE FOR SUCH PRODUCTS, INTEL ASSUMES NO LIABILITY 22

23 WHATSOEVER AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO SALE AND/OR USE OF INTEL PRODUCTS INCLUDING LIABILITY OR WARRANTIES RELATING TO FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT OF ANY PATENT, COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT. UNLESS OTHERWISE AGREED IN WRITING BY INTEL, THE INTEL PRODUCTS ARE NOT DESIGNED NOR INTENDED FOR ANY APPLICATION IN WHICH THE FAILURE OF THE INTEL PRODUCT COULD CREATE A SITUATION WHERE PERSONAL INJURY OR DEATH MAY OCCUR. Intel may make changes to specifications and product descriptions at any time, without notice. Designers must not rely on the absence or characteristics of any features or instructions marked "reserved" or "undefined." Intel reserves these for future definition and shall have no responsibility whatsoever for conflicts or incompatibilities arising from future changes to them. The information here is subject to change without notice. Do not finalize a design with this information. The products described in this document may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Contact your local Intel sales office or your distributor to obtain the latest specifications and before placing your product order. Copies of documents which have an order number and are referenced in this document, or other Intel literature, may be obtained by calling , or go to: Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors. Performance tests, such as SYSmark* and MobileMark*, are measured using specific computer systems, components, software, operations, and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products. Any software source code reprinted in this document is furnished under a software license and may only be used or copied in accordance with the terms of that license. Intel and the Intel logo are trademarks of Intel Corporation in the U.S. and/or other countries. Copyright 2013 Intel Corporation. All rights reserved. *Other names and brands may be claimed as the property of others. 23