Intelligent agents (TME285) Lecture 6,

Transcription

1 Intelligent agents (TME285) Lecture 6, D visualization and animation

2 Information Again, check the FAQ frequently! New items are tagged with the date at which they were added. For example, check carefully the items added (or modified) and In particular, see the item added yesterday (and also mentioned in yesterday s lecture), marked [ ].

3 Dialogues and dialogue items Just a note on nomenclature: Each agent contains a list of dialogues, i.e. instances of the Dialogue class, and each dialogue contains a list of dialogue items, i.e. instances of classes inherited from DialogueItem (e.g. InputItem, OutputItem etc.) Read carefully Chapter 3 in the compendium, and also go through the code (for the Hazel agent) in the agentmainform.

4 Dialogues and dialogue items In the case of Hazel seven different dialogues are defined. Examples: WakeUpDialogue: Contains a single dialogue item (an OutputItem) TimeDialogue: Contains two dialogue items (an InputItem and a TimeItem) WhoIsDialogue: Contains six different dialogue items (InputItem, MemorySearchItem, OutputItem, OutputItem, WaitItem, and OutputItem) etc. etc.

5 Checking how dialogues work.. Looking at the specification of the dialogues can help to understand a dialogue: The first dialogue item is added to this dialogue. InputItem (AD1): The user says the agent s name The second dialogue item is added to this dialogue. OutputItem (AD2): The agent responds. The dialogue is added to the agent s list of dialogues

6 Checking how dialogue items work.. However, in order to fully understand how individual dialogue items work, it is better to set a breakpoint at the respective Run() method, and then step through the code: After this point, step through (with F10), check variable values (e.g. targetid) etc.

7 Dialogues and dialogue items In some cases, I have received documents (for planning stage 2) in which the entire agent is generated as one very complex dialogue. It is much better (and also required) to implement a set of dialogues, one for each task, which can then be debugged and tested separately. When the agent has completed a given task, set both the context and ID to (the empty string). In that way, the agent will be ready to start any dialogue, depending on the input it receives.

8 Dialogues and dialogue items Example (Hazel s TimeDialogue): End of this dialogue set context and ID as => the agent is prepared to start any dialogue, depending on the input it receives.

9 Dialogues and dialogue items Again, do check carefully (and run through with breakpoints) the Hazel agent. It is a simple example, but it does illustrate the division of the agent s competences into separate dialogues. The 7 dialogues defined for the Hazel agent.

10 Today s learning goals After this lecture you should be able to Describe 3D rendering using OpenTK Define and compute the normal vector of a triangle Describe the concepts of lighting and shading Implement a 3D object derived from Object3D Describe and use the concept of nested 3D objects Set up and animate a simple 3D face Describe and use the FaceEditor user control

11 OpenGL and OpenTK 3D visualization requires a large number of matrix operations. There are software libraries specifically designed for this purpose, e.g. OpenGL and Direct3D. OpenTK is a C# wrapper for OpenGL (i.e. a class library that allows access to the more OpenGL).

12 3D rendering In 3D rendering one uses a sequence of matrix operations to generate a 3D view Model matrix: Deals with the transformations necessary to place an object at a given pose in the modeled world. View matrix: Deals with the transformations required to account for the camera position. (Often combined to one ModelView matrix) Projection matrix: Handles the perspective projection on the (2D) screen of the computer.

13 Orientation of axes In OpenGL, the x- and y-axes are in the plane of the screen, with the z-axis pointing towards the user. I have used a coordinate system (in the ThreeDimensionalVisualization library) where the x- and z-axis are in the plane of the screen, with the y-axis pointing into the screen, away from the viewer. y z y z OpenGL x ThreeDimensionalVisualization library x

14 Today s learning goals After this lecture you should be able to Describe 3D rendering using OpenTK Define and compute the normal vector of a triangle Describe the concepts of lighting and shading Implement a 3D object derived from Object3D Describe and use the concept of nested 3D objects Set up and animate a simple 3D face Describe and use the FaceEditor user control

15 Triangles and normal vectors In 3D rendering one normally builds objects from sets of triangles since triangles are always planar (=> simple!) so that their normal vector can easily be computed. The normal vector is important for lighting and shading, as will be shown below.

16 Triangles and normal vectors

17 Today s learning goals After this lecture you should be able to Describe 3D rendering using OpenTK Define and compute the normal vector of a triangle Describe the concepts of lighting and shading Implement a 3D object derived from Object3D Describe and use the concept of nested 3D objects Set up and animate a simple 3D face Describe and use the FaceEditor user control

18 Rendering objects Many aspects can be varied when rendering a 3D object: Position Orientation Color Lighting Shading Translucence Textures etc. etc.

19 Lighting If no light is present, an object will be rendered based on the Color property of the vertices. Interpolaton used if different vertices have different color.

20 Lighting Without light, however, 3D objects (with uniform color) appear flat. Consider, for example, a sphere:

21 Lighting If light is added, the 3D nature of the sphere becomes apparent:

22 Lighting Lighting requires: Three colors: Ambient, diffuse, and specular Normal vectors to trace the reflected light. Ambient color: background light Diffuse color: isotropic reflection of light Specular color: mirror-reflection of light (requires also a Shininess parameter). NOTE: See the Sphere3DApplication in the FaceVisualizationSolution!

23 Lighting Note: Whenever lighting is used, the vertex color is replaced by the ambient, diffuse, and specular colors: No light: use the vertex colors (here set uniformly to green). With light: Use the ambient, diffuse, and specular color

24 Shading The triangle normal determines the light reflection (and, therefore, the color). However, to get a more smooth variation of light over an object, one can interpolate the normal vectors. In that case, one uses the vertex normals instead of the triangle normals. The vertex normal is defined as the average triangle normal (vector) of all the triangles connected to a given vertex.

25 Shading

26 Shading Flat shading Smooth shading

27 More rendering options One can also choose not to show the surfaces of the triangles, but instead Vertices (i.e. the corners of the triangles) Wireframe (i.e. the sides of the triangles). See also the next slide and the Sphere3DApplication!

28 The Sphere3D application Shows how to render a sphere in various ways.

29 Today s learning goals After this lecture you should be able to Describe 3D rendering using OpenTK Define and compute the normal vector of a triangle Describe the concepts of lighting and shading Implement a 3D object derived from Object3D Describe and use the concept of nested 3D objects Set up and animate a simple 3D face Describe and use the FaceEditor user control

30 The ThreeDimensionalVisualization library This library contains code for generating and visualizing 3D objects, especially faces (needed for IPAs). It also contains a complex user control for generating a face. Important classes: Object3D Viewer3D Face FaceEditor

31 The Object3D class This is the base class, from which all other 3D objects are derived. Each Object3D instance contains A list of the object s vertices (where each vertex also contains its normal vector, used in smooth shading), A list of the vertex indices for each triangle, A list of normal vectors for each triangle, An optional list (object3dlist) of Object3D instances, thus allowing nested objects. For example, in face visualization, the eyes can be placed in the object3dlist of the face.

32 The Object3D class Moreover, each class derived from Object3D must have a Generate() method, which generates the specific shape for the object in question. An example will be given below.

33 The Object3D class Add to the stack of transformation matrices Set the object s position and orientation. Render as required. Calling this method again! Use the relative position and orientation! Remove from the stack of transformation matrices

34 The Object3D class The RenderSurfaces() method in turn calls the RenderTriangles() method, rendering the triangle surfaces: To do: Run the Sphere3D application, set a breakpoint in the RenderTriangles() method, and step through the code. Similar methods render the wireframe and the vertices, if requested.

35 Example: The Rectangle3D class Derived from Object3D

36 Example: Rectangle3D class Contains the necessary parameters for the 3D shape in question Indices of the first triangle Indices of the second triangle These three methods must be called at the end of the Generate() method for any class derived from Object3D.

37 Example: Rectangle3D class GenerateTriangleConnectionLists() Generates a list of all the triangles to which a given vertex is connected. Needed for computing vertex normals (see above). GenerateTriangleNormalVectors() Generates the triangle normals needed in flat shading. ComputeVertexNormalVectors() Uses the list obtained in the first method above to compute the vertex normal (= the average of the normal vectors of the triangles to which the vertex is connected).

38 Additional 3D objects In addition to the Rectangle3D, the library also contains classes for generating a sphere, a torus (or a sector thereof) etc. You may need to implement additional 3D object classes (in particular their Generate() method), depending on the level of complexity of your visualization. If so, then check the source code for the existing classes (e.g. Rectangle3D, which is the simplest one). Of course, you can ask us, at any time, how to implement additional 3D object classes!

39 Today s learning goals After this lecture you should be able to Describe 3D rendering using OpenTK Define and compute the normal vector of a triangle Describe the concepts of lighting and shading Implement a 3D object derived from Object3D Describe and use the concept of nested 3D objects Set up and animate a simple 3D face Describe and use the FaceEditor user control

40 The Viewer3D class Handles visualization of the 3D objects.

41 Rotation and translation To move and object relative to its current location, one uses the Move() method (in Object3D). Rotations around the coordinate axes are handled using RotateX(), RotateY(), and RotateZ(). Note that 3D rotation is a complex topic. The result will depend on the order of the rotations! Here, in most cases, we ll only need rotation around a single axis at a time (e.g. when the agent blinks!).

42 Faces In addition to simple objects (Sphere3D, Rectangle3D) etc., there is also a considerably more complex Face class. A horizontal slice through the face is a closed curve, here represented as a (composite) Bézier spline. The various horizontal slices are then joined to form triangles, which always contain two points from one slice and one point from either the slice below or the slice above.

43 Faces A slice through a face (the green plane...)...represented as a smooth (closed) Bézier curve. The yellow points are the support points for the splines. Note that the smooth curve can be sampled with any precision one can sample thousands of points, if one wishes to do so. (The yellow points just define the splines).

44 Faces: Visualization A simple 3D head can be modeled as a face (i.e. the shape of the face and the nose and ears), with added eyes, eye lids, eyebrows etc. A more advanced head model would also include a movable jaw. An even higher level of sophistication would be to model muscles and skin, attached to a rigid skull.

45 Faces: Visualization Structure of the face demonstrated in Lecture 1 and in Chapter 5: Face (Object3D) Object3DList: lefteyebulb (Sphere3D), lighteyebulb (Sphere3D), lefteyebrow (TorusSector3D), righteyebrow (TorusSector3D) lefteyebulb Object3DList : lefteyeiris (SphereSegment3D), lefteyepupil (SphereSegment3D), lefteyelid (SphereSegment3D) righteyebulb Object3DList : Same as for the left eye bulb, but with the prefix right (separate instances, of course).

46 Some code to help you... Load face (Generated with the FaceEditor application; see below). The Initialize() method in the FaceApplication Generate left eye bulb. Generate left eye iris. + More objects generated (not shown here) + Setting position and orientatin (not shown here)

47 Some code to help you Add iris, pupil, and lid in the object3dlist of the left eye and then add the left eye to the object3dlist of the face. With this nested structure, one can more easily move the face, or parts thereof. For example, if an eye is rotated, the iris, pupil, and lid will follow along (as they should).

48 Faces: Visualization Whenever an object is attached to another, use nesting (with the objectlist). Note that the attachment need not be rigid. For example, the agent can rotate its head while moving its gaze and blinking!

49 Faces: Visualization

50 Today s learning goals After this lecture you should be able to Describe 3D rendering using OpenTK Define and compute the normal vector of a triangle Describe the concepts of lighting and shading Implement a 3D object derived from Object3D Describe and use the concept of nested 3D objects Set up and animate a simple 3D face Describe and use the FaceEditor user control

51 Faces: Visualization Even with this simple structure, one can represent many facial expressions Neutral (alert) Sleepy Asleep Surprised Angry Frightened

52 Faces: Animation Here, animation has been implemented using two threads: Animation thread: Runs all the time (once animation is started), invalidating the Viewer3D at regular intervals (e.g. 25 times per second). The invalidation causes the Viewer3D to repaint itself. Specific motion thread (e.g. blink, nod etc.): Just changes the position or rotation (or both) of the object. When the object is repainted, the new pose (position and rotation) is used.

53 Faces: Animation...in the Viewer3D class

54 Faces: Animation

55 Today s learning goals After this lecture you should be able to Describe 3D rendering using OpenTK Define and compute the normal vector of a triangle Describe the concepts of lighting and shading Implement a 3D object derived from Object3D Describe and use the concept of nested 3D objects Set up and animate a simple 3D face Describe and use the FaceEditor user control

56 The FaceEditor application The FaceEditor application makes use of the highly complex FaceEditor user control that, in turn, makes use of Bézier splines, implemented in the MathematicsLibrary. With this user control, one can define, edit, save (in XML format), and load a Face object. The FaceEditor user control has two panels: One for 3D rendering, and one for showing horizontal slices through the face (see the next slide).

57 The FaceEditor user control Slice plane (green) [Left] and the corresponding close curve [Right]

58 The FaceEditor user control To edit a slice, use the mouse to grab one or several support points

59 The FaceEditor user control Then use the mouse to drag. Note that left-right symmetry is enforced. Use the mouse wheel to zoom and off-zoom in the 2D view (first click on the 2D panel). NOTE: Right-click with the mouse to clear the selection of points.

60 The FaceEditor user control Click on the 3D panel and use the arrow keys to move between slice planes Note: This does not change the vertical position of the planes in order to actually move a plane (in the z direction (up or down), click on Move up or Move down.

61 The FaceEditor user control Note the difference between Moving between planes (arrow keys) Actually moving a plane (Move up or Move down button) One can also remove a slice plane (click on the Remove slice button). insert a slice plane (click on the Insert slice button).

62 The FaceEditor user control An inserted slice is generated as the average (so to speak) between the slice below and the slice above: Slice above Slice below Inserted slice.

63 Today s learning goals After this lecture you should be able to Describe 3D rendering using OpenTK Define and compute the normal vector of a triangle Describe the concepts of lighting and shading Implement a 3D object derived from Object3D Describe and use the concept of nested 3D objects Set up and animate a simple 3D face Describe and use the FaceEditor user control