Merging Physical and Virtual:

Transcription

1 Merging Physical and Virtual: A Workshop about connecting Unity with Arduino v1.0 R. Yagiz Mungan yagiz@purdue.edu Purdue University - AD41700 Variable Topics in ETB: Computer Games Fall 2013 September 30 Merging Physical and Virtual - 1

2 Contents 1. Introduction Unity Server Object to Rotate Trigger Zone Integrating the Scene Processing Arduino Interface (via Serial Port): Unity Interface (via TCP/IP): Arduino Software Hardware Integration Conclusions Merging Physical and Virtual - 2

3 1. Introduction This workshop is about creating mixed-reality interactive objects (games, art installations, gadgets, unique interfaces) by combining 3D gaming (Unity) and physical computing (Arduino). This workshop assumes basic understanding of Unity (interface and scripting), Processing, Arduino and programming or scripting in general. However, some basic instructions will be provided about the platforms. The workshop has been created and tested with the following versions of the software: Unity version o Unity can be downloaded from o C# will be used for scripting. Processing version and o Arduino library for Processing is also required. It can be downloaded from note that v2.x and v1.5.1 have different packages. Arduino version 1.0 o Software: o Hardware: o Arduino UNO is used. Unity and Arduino will be connected via Processing. Unity and Processing will talk to each other via TCP/IP and Processing and Arduino will talk to each other via firmata. TCP/IP Firmata Figure 1: Connections from Unity to Arduino Since we are using TCP/IP, Processing and Unity do not need to be in the same computer. They talk through internet. We will create a simple example with simple scripts, which means the system we will create will work. However, it will not be the most efficient or robust system. The system will work such that, we will use a potentiometer to rotate an object Unity and we will use a trigger in Unity to switch on and off the built in LED in Arduino UNO. The codes provided with this document will have multiple lines of debug code (println() in Processing and Debug.Log() in Unity) that can be enabled to debug and understand the signals that is being transmitted. Merging Physical and Virtual - 3

4 2. Unity On the Unity side, for this example, we will create the following items: A server! An object to rotate. A way to create a trigger in the game. The server will be a script that will be attached to an object. Make sure that the object is passed through all the scenes. The Main Camera or the character controller can be used. For the rotation, any object is sufficient. However, for this workshop, I will use a light source with a cookie (world map) attached to a textured (world map) sphere. For the trigger, I will use OnTriggerEnter and OnTriggerExit functions with a Box Collider. Before starting to write scripts, we need to create our project and a scene. Go to File -> New Project On the Project Wizard window, choose the location of the project and from the packages click on Character Controller.unityPackage. Once the project is ready, press CTRL+S or APPLE+S to save your scene 1. Figure 2: Creating a new project in Unity 1 A consistent folder structure is always recommended. For example: create a folder named Scenes and save the scene inside that folder. Merging Physical and Virtual - 4

5 Create at least a plane to have something where our character will stand. If you are not familiar with Unity, you should know that you can create primitive geometric objects inside Unity. To create a plane: GameObject -> Create Other -> Plane Figure 3: Creating an object in Unity The plane will be our extremely simple level. Now, we will create a First Person controller by using a prefab 2. Drag and drop First Person Controller from Project Window to Hierarchy window. Put it somewhere on the plane. On the top of the Inspector Window, change the tag of the object to Player. 2 Prefabs are template objects that are packaged and ready to use. Merging Physical and Virtual - 5

6 Figure 4: Unity Project Window When you have time, feel free to improve your scene in any way you like. Below image shows the scene from the First Person view. Figure 5: A fancy object to rotate Merging Physical and Virtual - 6

7 2.1. Server The server will be the recipient entity of the TCP/IP connection. For the server, we will use a C# 3 script and we will not go into the details of the script for this workshop. However, we will look at how it works in order to expand it to support more features. Create a C# script labeled TCPHost 4. External Libraries We need libraries additional to UnityEngine for the server. using UnityEngine; using System.Collections; using System; using System.Text; using System.Net; using System.Net.Sockets; using System.IO; Variables The first set of variables is required for the server and the second set is required for the game mechanics. The incoming_client object of class TcpClient will be the Processing that connects to the server, the netstream object of class NetworkStream will be the stream or the connection between the server and the server object of class TcpListener will be the server. As for the game, we need a game object, which we will rotate. This object is referenced as targetobjcet. We need a Boolean variable to control the LED at pin 13 on Arduino. And we need another Boolean for a simple improvement of the system, so that we only send data when the LED13 value changes. //be sure that the port is open and available public int portno = 8090; private TcpClient incoming_client; private NetworkStream netstream; private TcpListener server; private bool waiting; // Add variables as required // I will use a gameobject to rotate with the potentiometer public GameObject targetobjcet; // variable for LED13 in Arduino public bool LED13 = false; public bool IsLED13Changed = false; 3 In the case, where your C# script needs to reach a variable from a JavaScript/UnityScript script and vice versa, check Unity s documentation about the order in which codes are compiled: 4 C# scripts name must have the same name as the class name. Merging Physical and Virtual - 7

8 Start Function In the Start(), we initialize and start the server. For more details of the TcpListener object see the reference: // Use this for initialization void Start () waiting = false; server = new TcpListener(IPAddress.Any, portno); server.start(); Parse (string Parse) Function The Parse Function, processes the String received from client (Processing) and does the necessary actions. Here I have used a simple handshaking signal: Blank space is defined as the end of a message. Thus, the received data package is split from the blank spaces. The server uses the first part and disregards the rest. void Parse2(string toparse) //Debug.Log(toParse); //you can use multiple commands and use string.split() to split them //Using and an end signal is always good, here we are using blank space (" ") string[] values = toparse.split(' '); //Debug.Log("first is " + values[0]); //access the script attached to target object targetobjcet.getcomponent<rotator>().rotationangle = float.parse(values[0]); Update Function The Update is the heart of the server: It checks whether the server is busy or not. If it is not, the code looks for the next client and reads the data if it is available. After a successful read attempt, LED13 activation data is sent to the Processing patch. It is important to note that the server employs error caching in order to sustain functioning after any corrupted data or connection problems. The data s first suffixed with the end signal (single blank space) and then the new String is converted into an array of bytes. The array sent to the Processing patch through the netstream object. Merging Physical and Virtual - 8

9 void Update () string s; string[] values; if (server.pending()) incoming_client = server.accepttcpclient(); netstream = incoming_client.getstream(); waiting = true; while (waiting && netstream.dataavailable) try int numread = 0; byte[] tmpbuf = new byte[1024]; numread = netstream.read(tmpbuf, 0, tmpbuf.length); s = Encoding.ASCII.GetString(tmpbuf, 0, numread); s = s.replace("\n",""); values = s.split(';'); //print ("values"+values.tostring()); //for debugging if (values.length > 1) for (int i = 0; i < (values.length-1); i++) Parse2(values[i]); else Parse2(values[0]); //sending data //Test //Byte[] data = System.Text.Encoding.ASCII.GetBytes("HelloThere"); //netstream.write(data, 0, data.length); //Send only and only if the state of LED13 has changed if(led13!= IsLED13Changed) //send //again using space as end character! string TempSend = LED13.ToString(); TempSend += " "; //Debug.Log(TempSend); Byte[] data = System.Text.Encoding.ASCII.GetBytes(TempSend); netstream.write(data, 0, data.length); IsLED13Changed = LED13; //Called when netstream fails to read from the stream. catch (IOException e) waiting = false; netstream.close(); incoming_client.close(); //Called when netstream has been closed already. catch (ObjectDisposedException e) waiting = false; incoming_client.close(); Merging Physical and Virtual - 9

10 2.2. Object to Rotate Here we need an object. We will create a cube similar to creating a plane. To create a Cube go to: GameObject -> Create Other -> Cube Rename the Cube to something more specific and recognizable. Place the object somewhere. We need to attach a script to the object. The script will be used to link the data that the server received to the object. Let us create C# script and call the script Rotator : using UnityEngine; using System.Collections; public class Rotator : MonoBehaviour public float RotationAngle = 0; private Vector3 V3 = Vector3.zero; private float Conversion; public float ArduinoInputMax = 1023; //Arduino uses 10bits // Use this for initialization void Start () Conversion = 360/ArduinoInputMax; //scale the data read from Arduino to degrees // Update is called once per frame void Update () //rotate the object around Y-axis //with unit time (degree/second) V3.y = RotationAngle*Time.deltaTime*Conversion; //Debug.Log("v3y = "+V3.y.ToString()); transform.rotate(v3); The main idea of the script is to map the Arduino input to the rotation of the object. In Unity, the rotation of an object can be accessed from transform.rotate 5. The rotation is a vector with three elements that maps to the X, Y and Z-Axes. We will only rotate the Y-Axis. The data that is received from Arduino ranges from 0 to 1023 since the analog-to-digital converters have 10-bits. We need to convert this range to degrees before applying. In the Start function this conversion is done. In the Update, the script does not only scale the value with the degree-conversion rate but it also scales the value with Time.deltaTime. Scaling with the unit time-change allows us to have numerically correct change that is related with the real time rather than the number of frames. Thus, once scaled with Time.deltaTime, our object rotates X degrees per second rather than X degree per frame. 5 Merging Physical and Virtual - 10

11 2.3. Trigger Zone Creating a trigger zone in Unity can be simple. The easiest way to do is to create a Cube following: GameObject -> Create Other -> Cube Rename the object to something definitive. Scale the object so that it is easy to trigger. In the Inspector Window, find Box Collider and click on Is Trigger. This turns the object to a trigger zone such that objects that collide with the trigger zone can go through without reflection; however, the game engine sends a signal when the collision happens. Figure 6: Box Collider window in the Inspector Now create a script and label it TriggerActivation. When another game object enters or exits the trigger zone. Our script will look at that object s tag. If it is Player, it will make the Boolean variable, IsInside true when the player enters and false when the players exits. The Boolean will be sent to the TCPHost script via the variable TCPSendReference. Merging Physical and Virtual - 11

12 using UnityEngine; using System.Collections; public class TriggerActivation : MonoBehaviour public GameObject TCPSendReference; private bool IsInside = false; void OnTriggerEnter(Collider other) if(other.comparetag("player")) //Debug.Log("enter"); IsInside = true; TCPSendReference.GetComponent<TCPHost>().LED13 = IsInside; void OnTriggerExit(Collider other) if(other.comparetag("player")) //Debug.Log("exit"); IsInside = false; TCPSendReference.GetComponent<TCPHost>().LED13 = IsInside; Merging Physical and Virtual - 12

13 2.4. Integrating the Scene Before hitting Play, we need to attach the scripts and link the game objects to the reference variables. 1. Server First, attach the TCPHost script to the First Person Controller object by dragging and dropping it from Project View on to the object in Hierarchy View. In the Inspector View, set variable as follows: Figure 7: Setting up TCPHost script Here, the name of the object in my scene that I will rotate is World. Thus for the Target Object, choose the object that you created. 2. Rotating Object Attach the script, Rotator, to the object you have created. You do not need to change any of the variables. 3. Trigger Zone Attach the script, ActivationTrigger, to the trigger zone object that you have created. For the TCPSendReference variable drag and drop First Person Controller from Hierarchy View. Now, our scene should be ready and we will move on to Processing. Do not forget to save your scene! Merging Physical and Virtual - 13

14 3. Processing The Processing sketch is relatively simple it has two important parts: Arduino interface and Unity interface. We will read from the Analog 0 pin of the board, while writing to the Digital 13 pin of the board. On the Unity side, we will connect to a defined IP address through a defined port. Through this connection, we will send the data that is read from Analog 0 pin and the received data will carry the value that will control LED13 on Arduino UNO. Here is some reference information: Arduino Library 6 for Processing: Client object from Net Library for Processing: Here is some select information from the references: Arduino Library Arduino.list(): returns a list of the available serial devices. If your Arduino board is connected to the computer when you call this function, its device will be in the list. Arduino(parent, name, rate): create an Arduino object. Parent should be "this" (without the quotes); name is the name of the serial device (i.e. one of the names returned by Arduino.list()); rate is the speed of the connection ( for the v2 version of the firmware, for v1). Note that in the v2 library, the rate parameter is optional. pinmode(pin, mode): set a digital pin to input or output mode (Arduino.INPUT or Arduino.OUTPUT). digitalwrite(pin, value): writes Arduino.LOW or Arduino.HIGH to a digital pin. analogread(pin): returns the value of an analog input (from 0 to 1023). Client Object Methods available() : Returns the number of bytes in the buffer waiting to be read readstring(): Returns the buffer as a String write(): Writes bytes, chars, ints, bytes[], Strings 6 To install a library copy the extracted ZIP package, "arduino" folder, into the "libraries" sub-folder of your Processing Sketchbook. (You can find the location of your Sketchbook by opening the Processing Preferences. If you haven't made a "libraries" sub-folder, create one.) Merging Physical and Virtual - 14

15 3.1. Arduino Interface (via Serial Port): For the Arduino interface, we need to import the Arduino library and create an object of class Arduino. In the setup part, we need to configure the USB port and set the digital pin 13 accordingly. In the draw part, we will read the analog pin to variable dataout and set pin 13 according to the variable datain. //Libraries import cc.arduino.*; //this imports arduino as a class in processing import processing.serial.*; //processing will talk to arduino through serial port //Custom classes Arduino arduino; //let's have an object named arduino from the class, Arduino //Default classes int dataout; //from Arduino to Unity String datain = "False"; //from Unity to Arduino, initialized to False void setup() // Setup Arduino connection println(arduino.list()); //list the Arduinos connected to the computer arduino = new Arduino(this, Arduino.list()[0], 57600); //choose the first for Arduino arduino.pinmode(13, Arduino.OUTPUT); //set Arduino pin 13 as ouput void draw() if(datain.equals("true")) arduino.digitalwrite(13, Arduino.HIGH); if(datain.equals("false")) arduino.digitalwrite(13, Arduino.LOW); dataout = arduino.analogread(0); //read sensor, potentiometer data from analogpin 0 Merging Physical and Virtual - 15

16 3.2. Unity Interface (via TCP/IP): The Unity interface will use the processing.net library for the necessary objects. Unity will be the server and Processing will act as a client to the server. Thus, before running Processing, we need to run Unity first. We need to create an object of Client class. In the setup, we will map this client to a server with IP 7 and port 8 number. The connection is followed by a test message that can be tracked from the other hand. In draw, we read the data from the buffer, process it and pass it to the variable datain and we pass the data that we got from Arduino to Unity via the variable dataout. //Libraries import processing.net.*; //TCP/IP libraries //Custom object types Client myclient; //and an object named myclient //Default object types int dataout; //from Arduino to Unity String datain = "False"; //from Unity to Arduino, initialized to False String tempdatain; //temporary variable to parse received data void setup() // Setup Unity connection // Local host at port // Change IP if Unity is on a remote machine myclient = new Client(this, " ", 8090); myclient.write("connected to Unity!"); void draw() //See if there is something on the buffer if (myclient.available() > 0) //read it //println(myclient.readstring()); tempdatain = myclient.readstring(); //LED13 on/off String[] list = split(tempdatain, ' '); datain = list[0]; myclient.write(str(dataout)+" "); //send it to Unity is the local host (same computer), if you need to reach another computer use that computer s IP. 8 Make sure that the port is open. Merging Physical and Virtual - 16

17 3.3. Combined patch //Libraries import processing.net.*; //TCP/IP libraries import processing.serial.*; //processing will talk to arduino through serial port import cc.arduino.*; //this imports arduino as a class in processing //Custom object types Arduino arduino; //let's have an object named arduino from the class, Arduino Client myclient; //and an object named myclient //Default object types int dataout; //from Arduino to Unity String datain = "False"; //from Unity to Arduino, initialized to False String tempdatain; //temporary variable to parse received data void setup() // Setup Unity connection // Local host at port // Change IP if Unity is on a remote machine myclient = new Client(this, " ", 8090); myclient.write("connected to Unity!"); // Setup Arduino connection println(arduino.list()); //list the Arduinos connected to the computer arduino = new Arduino(this, Arduino.list()[0], 57600); //choose the first Arduino arduino.pinmode(13, Arduino.OUTPUT); //set Arduino pin 13 as ouput void draw() //See if there is something on the buffer if (myclient.available() > 0) //read it //println(myclient.readstring()); tempdatain = myclient.readstring(); //LED13 on/off String[] list = split(tempdatain, ' '); datain = list[0]; //If it is 1 turn on LED13 //println(datain+"lalal"); if(datain.equals("true")) //println("true here"); arduino.digitalwrite(13, Arduino.HIGH); if(datain.equals("false")) //println("false here"); arduino.digitalwrite(13, Arduino.LOW); dataout = arduino.analogread(0); //read sensor, potentiometer data from analogpin 0 //println(dataout); Merging Physical and Virtual - 17 myclient.write(str(dataout)+" "); //send it to Unity

18 4. Arduino The Arduino part is relatively easy: 4.1. Software We will only load Standard Firmata. Firmata allows us to talk to the Processing in real time through the serial port. See the image below to load Firmata. File -> Examples -> Firmata -> Standard Firmata Figure 8: Loading Standard Firmata Once StandarFirmata patch is loaded. Click on to upload the programming. Merging Physical and Virtual - 18

19 4.2. Hardware We will connect a potentiometer 9 to the board in order to create a variable input. Regular potentiometers have the three nodes: 1. Power reference: Connect this to 3V or 5V output of the board. 2. Ground reference: Connect this to the GDD output of the board. 3. Output reference: Connect this to the analog input (A0) of the board. Figure 9: Connecting the potentiometer 9 Potentiometer is a variable resistor. For this workshop any element that provides an analog input to Arduino can be used. Merging Physical and Virtual - 19

20 4. Integration Upload the StandardFirmata to Arduino Start the Unity scene Run the Processing patch Once the patch is up and running, return to Unity window. Unity Game Window works only when it is in focus. If another software (Arduino or Processing) is on the foreground, the game (thus the server) is paused. If the Processing patch is started before Unity, it will give errors; however, this can be improved. For troubleshooting, follow these steps (in any order): Make sure there is no mistake between the manual and your implementation. Make sure the port is open and the local host is accessible. Make sure that correct port is chosen for Arduino. Use the debug lines to see if the signals are correct and are received correctly. Merging Physical and Virtual - 20

21 5. Conclusions The documentation and the final Unity project and Processing can be found from the following link: Use this tool to create unique experiences. Merging Physical and Virtual - 21