Software Architecture for Immersipresence

Transcription

1 Software Architecture for Immersipresence Alexandre R.J. François Computer Science Department ARJF 2006

2 Software Architecture Design, analysis and implementation of software systems Explicit system structure Improve the flexibility and comprehensibility of software systems (Parnas, 1972) Address modularization as a design issue (Parnas) Technical basis for design Provable properties Blue-prints for implementation Tools for analysis Project management Separation of concerns Planning: cost estimation, resource allocation

3 Immersipresence Vision of the Integrated Media System Center Combine real world with virtual world NSF ERC in Multimedia, est Experience immersion, presence Interact naturally Collaborate through shared virtual/augmented space Build systems capable of: Handling video, sound, haptics, etc. Real-time analysis/synthesis (immersion) Low latency (interaction)

4 Requirements for Immersipresence Interoperability Efficiency On-line, real-time, low latency Scalability Combine research from different fields/teams Performance evaluation and prediction General model for distributed asynchronous concurrent processing of data streams

5 SAI Principles Time Volatile vs. persistent data Asynchronous concurrent processing Architectural style [ICSE2004] High level abstractions Hybrid model or more general model?

6 Architectural Primitives Stream Processing unit (no state) Asynchronous parallel model Source Volatile data Cell persistent data Source Volatile data Cell Stream Shared repository of persistent data Pulse Synchronization structure (time stamp, duration) Active: volatile, flow down stream connections Passive: persistent (dynamic), held in sources

7 Constraints and Semantics Cell-cell: volatile data samples (stream) Cell-source: persistent data access At most one upstream cell Any number of downstream cells Process dependency Exactly one source to a given cell Any number of cells to a given source Stream (message passing) Source Concurrent access (shared repository) Processing model Active pulse triggers cell process Process may add to active pulse Process may modify passive pulse Cell Passive pulse: persistent data Active pulse: volatile data

8 Architectural Middleware Support architectural abstractions Pulse, source, cell, etc. Direct mapping from logical specification to code! Modular Flow Scheduling Middleware (MFSM) Open source project: mfsm.sourceforge.net C++, cross-platform (GNU compiler) Base library, functional modules, documentation, tutorials

9 MFSM Diagram

10 VisualSAI

11 CAMSHIFT Tracking Tracking algorithm: CAMSHIFT Continuous Adaptive Mean SHIFT (Bradski, 1998) Mean shift: iteratively find the mode in a probability density distribution (Comaniciu & Meer, 1997) Perceptual User Interface Real-time video processing

12 Example System Video input +input image CAMSHIFT tracker CAMSHIFT tracker node +new position Rendering +composite image Image display

13 Refinement 1 Video input Video input +input image +input image CAMSHIFT tracker CAMSHIFT tracker node Step I +new position Rendering +composite image Image display Color conversion +HSV image Step II Histogram node Step III Last known bounding box Back-projection +back-projection image CAMSHIFT +new bounding box, size and orientation Rendering Constrains +composite image Image display

14 Refinement 2 Video input Video input Video input +input image +input image +input image Color conversion Color conversion +HSV image +HSV image CAMSHIFT tracker CAMSHIFT tracker node Step I +new position Rendering +composite image Image display Step II Histogram node Step III Last known bounding box Back-projection +back-projection image +back-projection image CAMSHIFT +new bounding box, size and orientation Rendering Constrains Back-projection Histogram node +composite image Image display Box update Mean shift +new position Last known bounding box Moments +new bounding box, size and orientation Rendering +composite image Image display

15 General Tracking Pattern Video input +input image Color conversion +HSV image Back-projection Tracking source Predict +Predicted targets Tracked targets Update Histogram node +back-projection image Box update Mean shift +new position Last known bounding box Track +Tracked targets Moments +new bounding box, size and orientation Acquire +New targets Rendering +composite image Image display

16 Vision for Robot: Stevi 1 Stereo input and pre-processing +input left +input right Photo courtesy of ETRI Stereo video input Video input Video input interface left interface right Color transform right Color transform left +HSV right +HSV left Color data left Color data right +Color data left +Color data right Barrier Stereo multi-target tracking Barrier synchronization Predict targets Update targets Tracking source +Predicted targets Tracked targets Track targets Acquire targets +Tracked targets +Residual image left +Residual image right +New targets Visualization Render targets +Composite image left +Composite image right Image display left Image display right

17 MuSA.RT Music on the Spiral Array. Real Time Commands Spectator Pulsar Co-PI: Elaine Chew Camera control Pulsar Rendering Frames User input Display Spiral array camera and other process parameters MIDI input Events Pulsar Tonal data integration Event processing Performer

18 ESP Expression Synthesis Project Driver Co-PI: Elaine Chew Velocity update Pulsar Driver input Position computation Pulsar Pulsar Frames parameters for rendering and other processes Display Events MIDI event buffer current position and velocity MIDI output Visual rendering Pulsar MIDI event generation synthesizer

19 Distributed Game Project Net Receive Game server graph Net Send (Game Server) Decoding Encoding Physics (motion+collision) User input (player control) Consistency (referee) World update World description 25 students, 2 months Distributed development Real-time multiplayer gaming with database recording/replay Encoding Net Send (Clients) Net Send (DataBase) Database Player client graph Net Receive (Game server) Game Server Player Pulsar Renderer World update (Decoding) Parameters World description Spectator Client Player Client Image display

20 Summary SAI: Software Architecture for Immersipresence Applications: Design and analysis of complex software systems MFSM: Architectural middleware Patterns for synchronization Interactive music systems, Computer vision and graphics systems, Distributed Interactive games, etc. For more information:

21 SAI Properties (1) Model time explicitly in data and processing Model modularity Model concurrent execution (asynchronous) Separation of concerns Scalability Separate throughput and latency Model distributed computing

22 SAI Properties (2) Facilitate system design Facilitate system analysis Safety, liveness, etc. Facilitate distributed development Intuitive architectural style, based on data streams Unified processing model and unified data model Design patterns Fast integration Code reusability Facilitate system maintenance, modification and evolution Change in algorithm and in function

23 Acknowledgments and Disclaimer The research presented here was funded in part by: The Integrated Media Systems Center, an NSF Engineering Research Center, Cooperative agreement No. EEC The Advanced Research and Development Activity of the U.S. Government under contract No. MDA C-0036 Any opinions, findings and conclusions or recommendations expressed in this material are those of the author and do not necessarily reflect those of the funding agencies.