Visual Computing Meets AI:

Transcription

1 Visual Computing Meets AI: Intelligent Simulated Reality Philipp Slusallek Klaus Fischer, Rainer Jochem, Matthias Klusch, Werner Stefan Georg Demme, Hilko Hoffmann German Research Center for Artificial Intelligence (DFKI), Cluster of Excellence MMCI, and Saarland University

2 Overview Who am I? What is Computer Graphics? Research so far: Computer Graphics Research in SB Trends in Graphics Ongoing and Future Research Intelligent Simulated Reality Scientific Challenges Ongoing Work Some Outlook

3 Philipp Slusallek Study of Physics in Frankfurt and Tübingen (M.Sc.) PhD in Computer Graphics, Erlangen University Visiting Assistant Professor, Stanford University, US Full Professor for Computer Graphics, Saarbrücken First company (Architecture Visualization) Two Spin-Offs (Motama, intrace) Dean, University Senate, IT steering committee, Founding speaker, Competence Center CS, SB PI Ex.-Cluster Multimodal Computing and Interaction Sabbatical at Nvidia Research, Santa Clara Clara,, US Director DFKI-SB: Agents and Simulated Reality

4 Systems Oriented Graphics Research at University Graphics Lab Realtime Ray Tracing (OpenRT) Network Integrated Multimedia (NMM) Realtime Lighting Simulation Massive Models, Visualization Graphics Hardware Distributed Multimedia Middleware Future Rich-Media Applications Home-Entertainment, IPTV & HDTV Virtual-Reality 3D-Internet, 3D-GUI, Interaction Techniques

5 Image Synthesis / Rendering Global Operator Ray-Tracing/Visibility Rendering Equation L( x, ω o ) = Le ( x, ω o ) + Ω + = Le ( x, ω o ) + Ω + f r (ω o, x, ω i ) Li (x, ω i ) cos θ i dω i f r (ω o, x, ω i ) L ( y ( x, ω i ), ω i ) cos θ i dω i L = Le + TL = Le + TLe + TT L = = i T Le with T < 1 i= 0 Fredholm Equation of the 2nd kind L(x,w): Radiance [Watt /sr/m2] Challenge Solving complex, -dim. integrals with singularities Inherently recursive, massive FLOPS

6 Rasterization Primitive Operation of All Graphics Hardware!! Scan convert a single triangle at a time Ray casting with single origin and through fixed grid No recursion, no adaptivity, Sequentially Processes Every Triangle Individually No access to more than one triangle But most effects need access to the world: shadows, reflection, global illumination

7 Ray-Tracing Pipeline L ( x, ω o ) = Le ( x, ω o ) + Ω = Le ( x, ω o ) + Ω + + f r (ω o, x, ω i ) Li (x, ω i ) cos θ i dω i f r (ω o, x, ω i ) L ( y ( x, ω i ), ω i ) cos θ i dω i L = Le + TL = Le + TLe + TT L = = i T Le with T < 1 i= 0 Ray Generation Traversal Intersection Shading Framebuffer

8 1999 Some argue that in the very long term, rendering may best be solved by some variant of ray tracing tracing,, in which huge numbers of rays sample the environment for the eye eye s view of each frame. Real-Time Rendering, 1st edition, page 391, 1999

9 1999 Some argue that in the very long term, rendering may best be solved by some variant of ray tracing tracing,, in which huge numbers of rays sample the environment for the eye eye s view of each frame. And there will also be colonies on Mars, underwater cities, and personal jet packs. Real-Time Rendering, 1st edition, page 391, 1999

10 2006 Ray Tracing is now in the Main Stream Scientific American Issue August 2006

11 Ray-Tracing im Mainstream Bild der Wissenschaft, Jan'07 Spektrum der Wissenschaft, April'07

12 Today: Ray Tracing in Industry Industry giants competing for ray tracing IBM: CAS project to support RT on Cell Many-core, x86-based, SMT, wide SIMD (16x) Nvidia: G80 + Better use of parallelism, HW support,... Intel: Larrabee processor (2009/2010) Intel, Nvidia, IBM, Massive many-core, massive SMT, SIMT

13 Previous Work Realtime Ray Tracing Efficient, highly optimized software for Trace packets and frustums instead of individual rays Expose and use coherence to optimize for HW x86 (w/ and w/o SIMD), Itanium, Cell, GPUs,... Large compute clusters (up to 180 CPUs) Rendering library (OpenRT) Basis for spin-off company (intrace GmbH) Basis for many applications

14 Previous Work: Spatial Indices for Dynamic Scenes Find closest hit point along a ray Millions of pixel/frame, 60 frames/s Approach: Subdivide space (kd-tree) or objects (BVH) May have to rebuild/update index each frame Bounding KD-Trees (B-KD Trees) Merge of kd/bvh Well suited to HW Simple intersection Fast update

15 Past Research Results

16 Application in the Car Industry Headlight: 800K triangles, complex reflections and refractions Mercedes: Direct rendering of NURBS, accurate materials

17 VW Visualization Center

18 VW Visualization Center

19 VW Visualization Center

20 Measured Materials (BTFs)

21 Product Visualization at EADS

22 Applications: Realtime Lighting Simulation Illumination in Computer Graphics Local (no shadows) Direct (shadows) Global (with all interreflexions)

23 Application for Lighting Simulation Conference: 280K triangles, 104 area lights Wine glass: physically accurate caustics

24 Application in the Car Industry 250k Photons (3 fps) 25M Photons (11 fps) Switching on the headlight during simulation after accumulation (30s) real photograph

25 Large Model Visualization at Boeing CATIA Model of Boeing 777: 350 million triangles, 30 GB on disk, 2-3 fps on Dual-Opteron

26 Outdoor Environments with full Sky Illumination

27 Outdoor Environments with full Sky Illumination

28 Outdoor Environments with full Sky Illumination

29 Outdoor Environments with full Sky Illumination 12 ~90 10 Triangles

30 Ray Traced Games

31 Ray Tracing on Multi-Core Advantages: Disadvantage High-performance implementations are available Highly flexible environment Scales nicely with # of cores ((~10 ~10 Mrays/s per core core)) Need 30 cores for minimum requirements Not for the mass market any time soon But high-end systems are available Opteron-System (8 CPUs x Quad-Core) 32 cores

32 Ray Tracing on 8-core Cell: 1024 x1024 Scene Triangles Single-Cell Dual ERW Conference 280k Beetle 680k

33 Custom Hardware Implementation

34 D-RPU: High-Level Architecture [GH 2006] Ray-Generation Ray-Traversal Intersection Shading Framebuffer

35 D-RPU: Hardware Architecture FPGA and ASIC fully synthesized from ~8000 lines of high-level HWML description

36 GPU Architecture (Nvidia G80) G80 architecture diagram 128 scalar cores, 1,3 GHz

37 Future Trends in Graphics HW Many-Core Architectures Hundreds to thousands of cores Wide-SIMD SIMD: Single Instruction Multiple data Many cores driven by the same instructions Wide vector operations (SSE: 4, AVX: 8, LRB: 16, G80: 32) Programmable Graphics Less and less fixed-function HW blocks SW algorithms on general purpose processors Efficiency requires new programming models

38 Where are we going from here? Where are we going from here?

39 Structure of New DFKI Research Area Agents and Simulated Reality Philipp Slusallek Intelligent Simulated Reality Hilko Hoffmann Safe and Secure Software Werner Stephan Living Lab: Saarland Visualization Center Georg Demme Multiagent Systems K. Fischer/M. Klusch Computer Graphics Group Saarland University Multimodal Computing and Interaction (TP7) Cluster of Excellence

40 Joint Project: Intelligent Simulated Reality Reality Simulated Illumination, acoustics, traffic, character animation,... Intelligent Large-scale 3D models, highly detailed & realistic,... Multiagent systems, machine learning, speech,... Systems Oriented Research Incrementally build complete system from ground up Building block for NextGen 3D Internet

41 Long Term Goal: Simulated Reality Turing Test Provide an integrated system that provides Framework for Future 3D Internet applications Unified, standards-based semantic 3D world model Middleware for advanced simulation services Scalable, detailed, physically-correct, X3D-based,... Based on semantic web services and P2P communication Semantic information embedded in 3D world Interface to intelligent agents & behavior Formal and hybrid verification of dynamic systems Immersive presentation and interaction environment Indistinguishable from reality

42 Selected Applications Architecture & Outdoor Emerging market Huge and diverse data sets Simulation Design visualization, verification Traffic, crowds, flooding,... Illumination, acoustics,... Virtual heritage 3D dynamic fact base (Wikipedia)

43 Selected Applications Virtual Production Demonstrator Existing Market Simulation & Applications Car industry, suppliers,... Existing contacts (e.g. Avilus) Modular planning and design Functional validation & verification Timing and logistics Training In-house Smart Factory & Robotics Group

44 System Overview USER Realistic Realtime Rendering, 3D-GUI, Semantic Interaction, Tracking, and Immersive Visualization Multiagent Simulation Semantic World Model Traffic Simulation Human Simulation Lighting Simulation Acoustic Simulation P2P Middleware Ontology Services Animation Generator Sensor Networks Hybrid Verification

45 Why Develop a New System? We don't we build on existing tools & standards We will make those tools available OWLS-MX, WSMO-MX, SAWSDL-MX, OWLS-XPlan,... PIM4Agents, JACK/JADE, xaitment,... EA-prototype, DocTIP/FormalSafe, VSE,... X3D, OpenRT, RTfact, RTSG, NMM,... RTSG, RTfact, NMM, and others will be OpenSource Cooperations with industry (e.g. Avilus) and research We will use the tools ourselves Saarland Visualization Center (new building, 2010)

46 Why Develop a New System? Existing approaches CAD systems (CATIA, AutoCAD, Cinema4D,...) Visualization Systems (IBM DE, AVS, Amira,...) Simulation systems (Umbra, SCIRun, BALLView,...) VR systems (VDP, Lightning, VR Juggler,...) Game Engines (Unreal, Crystal Space, Ogre,...) None comes close to what we need Scalability, openness, generality, accuracy, realism, interactivity, performance, platform independence,... But integration (Cinema4D, Lightning, BallView,...)

47 Scientific Challenges Semantic and Service-Oriented Middleware Semantic service coordination for simulation modules Based on semantic service and scene descriptions Time-constrained, on-the-fly operation Peer-to-peer selection and communication Synchronized distributed computing Appropriate ontologies and semantic annotations Addition of spatial reasoning in semantic selections Integration into common platform

48 Scientific Challenges Agent-Based Software Systems Scalability and dealing with massive data sets Generic interfaces between agents and virtual worlds Describing environmental aspects to agents E.g. simulated agent perception Extensions to PIM4Agents abstract model Modeling reasoning (e.g. BDI) and reactive interface Exploration of common specification techniques Exploiting many-core computing Integration into the common platform

49 Scientific Challenges Hybrid System Verification Elimination Approach: Refinement & optimization Integration into DocTIP Integration into the common platform Translation of problem and claims into FOL Special fixed-point algorithm Which formal semantic data is needed Mapping of platform independent models Visualization of activity sequences and failure traces Modeling safety in the 3D model Evaluate using various application examples Mainly in production system demonstrator

50 Scientific Challenges Immersive Visualization and Interaction Modeling building blocks Scalable and flexible world model Go beyond rendering-optimized scene graphs Build upon X3D standard Novel indexing and access structures (beyond spatial) Distributed worlds Augmented with semantic, material, etc. information Attachments, connections, functionality Use NMM to integrate multimedia processing with 3D Explore novel 3D communication schemes Perception-based display and immersive & interaction

51 RTfact: Generic Concepts for Ray Tracing Goals Provide high-level flexibility AND performance Build efficient layers of abstraction Avoid SSE assembler Write algorithms once, reuse Approach Concepts describe data structures and algorithms C++ template programming maps to target platform Compiler merges code sections through inlining Highly optimized code generation

52 BALLView: Ray Tracing for Bioinformatics

53 From Client-Server to a Network-integrated MM Middleware Applications can run anywhere Even on a mobile device Can fully control entire MM flow-graph Any device and any processing All processing is fully synchronized Even across the network

54 Network-Integrated Multimeda Middleware (NMM) Flow graph for local and distributed media processing Host A Host B NMM {

55 Example: Cell and PS3 Efficient Many-Core Computing Joint Project with IBM Böblingen Many incoming HD video streams Flexible SW decoding and processing on SPEs Usage scenario for Consumer Electronics Optimized communication between cores Direct communication between SPEs Low latency, low memory bandwidth Directly applicable to Larrabee (Intel) as well Optimized inter-core transport via small L2 caches

56 Example: Remote Rendering and Video Wall Combining RTRT and NMM Rendering servers multiplex data to display clients Compression and streaming of video data Display client can also be a web browser Master Server Ray Tracing Client Display Client Ray Tracing Client Display Client

57 NMM: Integrating Multimedia and 3D Graphics Arbitrary Input Arbitrary Output Tone mapping, image processing, tracking,... Full Control Streaming, feedback, virtual devices: display walls,... Flexible Processing Streaming media, any local or remote devices,... Multimedia nodes are part of the X3D scene graph... and streaming 3D scene updates

58 Multi Agent Technology What are agents? Think about Ant populations Autonomously acting software modules Each one not too complicated Interacting with each other and the world Communication and negotiation Reactive and proactive (planning) behavior Where are they used? Modeling business processes Modeling behavior of virtual characters

59 Steel Mill Saarstahl Völklingen

60 Iron from blast furnace Preprocessing Steel Mill Saarstahl Völklingen Steelwork Steel converters Refinement Continuous Casting Further processing

61 Agent Technology at Saarstahl Völklingen User Planning Agent Device Agent Device Agent Device Agent Device Agent Batch Agent Batch Agent Batch Agent Batch Agent

62 Agent Technology in Simulated Reality Game-AI by xaitment GmbH Spin-Off of Multiagent Group Used as the basis multi-agent engine Integration and extension Adding of abstract modeling component (PIM4Agents) Scientific challenges: How to describe agents behavior? How do agents perceive the environment? How do agents interact with the environment? How to implement agents efficiently?

63 Agent Technology in Simulated Reality Convergence of Agents and Graphics Ray tracing for collision detection Parallel multi-agent systems Can reuse ray tracing, acoustics, lighting,... Semantic 3D world model But yet more irregular than ray tracing Agents observe the world around them Reuse of index data Streets, lanes, other cars, road conditions,... Rule base (control/data driven) Traffic rules, character of driver,...

64 Ongoing Work: City Along the River Engaging Citizens through VR

65 Ongoing work: Architectural Visualization University Campus

66 Apply Technology Ourselves: Saarland Visualization Center DFKI Living Lab Driven mainly by industry and other partnes Technology transfer Visualization services Research collaborations Meeting and demo facilities Forum for presenting research Outreach activities Window to the world

67 Collaborations Industry Intel, Nvidia, IBM, xaitment, Motama, Saarstahl,... Research DFKI Computer Graphics Group & M²CI (UdS) RTSG, RTfact, NMM, GPUs, MC methods, visualization,... Fraunhofer IAO & MPI Tübingen Carmina, Vital, i2home, FormalSafe/DocTIP, Shape, COIN,... Lighting, VR applications, interaction devices & perception DERI & Trinity College (Ireland) Semantic annotation and coordination & crowd computing

68 Conclusion Main goals of ISReal Methodology Integrate AI and graphics Build solid platform for research and industry Exploit this platform for new projects Build a working system quickly w/ existing technology Start using the system in projects to find limitations Direct research to overcome them Important basis for Future 3D Internet

69 Interested? Questions? DFKI Informatik Saarland