MPEG for Augmented Reality

Transcription

1 MPEG for Augmented Reality ISMAR, September 23, 2016 Workshop on Standards for Mixed and Augmented Reality Marius Preda, MPEG 3DG Chair Institut Mines TELECOM

2 Topics of my presentation Who is MPEG and why MPEG is doing AR MPEG ARAF design principles and the main features

3 Tidy City

4 Portal Hunt

5 Elements

6 ARQuiz

7 Augmented Books

8 Event LOOV Collecting virtual money in real world for buying real services and products Available on AppStore, AndroidStores and MyMultimediaWorld.com

9 Summer School (1 week) Games

10 What is common in these "games"? Based on MPEG ARAF Augmented Reality Application Format

11 MPEG Augmented Reality Why MPEG AR?

12 What is MPEG? A suite of ~130 ISO/IEC standards for: Coding/compression of elementary media: Audio (MPEG-1, 2 and 4), Video (MPEG-1, 2 and 4), 2D/3D graphics (MPEG-4) Transport MPEG-2 Transport, File Format, Dynamic Adaptive Streaming over HTTP (DASH) Hybrid (natural & synthetic) scene description, user interaction (MPEG-4) Metadata (MPEG-7) Media management and protection (MPEG-21) Sensors and actuators, Virtual Worlds (MPEG-V) Advanced User interaction (MPEG-U) Media-oriented middleware (MPEG-M) More ISO/IEC standards under development for Coding and Delivery in Heterogeneous Environments (incl.) 3DVideo

13 What is MPEG? A standardization activity continuing for 25 years, Supported by several hundreds companies/organisations from ~25 countries ~500 experts participating in quarterly meetings More than 2300 active contributors Many thousands experts working in companies A proven manner to organize the work to deliver useful and used standards Developing standards by integrating individual technologies Well defined procedures Subgroups with clear objectives Ad hoc groups continuing coordinated work between meetings MPEG standards are widely referenced by industry 3GPP, ARIB, ATSC, DVB, DVD-Forum, BDA, EITSI, SCTE, TIA, DLNA, DECE, OIPF Billions of software and hardware devices built on MPEG technologies MP3 players, cameras, mobile handsets, PCs, DVD/Blue-Ray players, STBs, TVs, Business friendly IPR policy established at ISO level

14 MPEG technologies related to AR: 1 st pillar 1992/ MPEG-1/2 (AV content) VRML MPEG-4 v.1 Part 11 - BIFS: -Binarisation of VRML -Extensions for streaming -Extensions for server command -Extensions for 2D graphics - Real time augmentation with audio & video Part 2 - Visual: - 3D Mesh compression - Face animation MPEG-4 v.2 Part 2 Visual - Body animation First form of broadcast signal augmentation

15 MPEG technologies related to AR: 1 st pillar MPEG-4 Part 16 - AFX: - A rich set of 3D graphics tools - Compression of geometry, appearance, animation A rich set of Scene and Graphics representation and compression tools MPEG-4 AFX 2 nd Edition: - Animation by morphing - Multi-texturing MPEG-4 AFX 3rd Edition - WSS for terrain and cities - Frame based animation MPEG-4 AFX 4th Edition - Scalable complexity mesh coding

16 MPEG technologies related to AR: 2 nd pillar x MPEG-V - Media Context and Control MPEG-U 1st Edition Advanced - Sensors and User Interface actuators - Interoperability between Virtual Worlds A rich set of Sensors and Actuators MPEG-V 2 nd Edition: - GPS - Biosensors - 3D Camera MPEG-H - 3D Video Compression CDVS of video + Feature-point based depth - 3D Audio descriptors for image recognition

17 MPEG technologies related to AR: 2 nd pillar MPEG-V Media Context and Control

18 MPEG technologies related to AR: 2 nd pillar MPEG-V Media Context and Control Sensors Light Ambient noise Temperature Humidity Distance Atmospheric pressure Position Velocity Acceleration Orientation Angular velocity Angular acceleration Force Torque Pressure Motion Intelligent camera type Multi Interaction point Gaze tracking Wind Global position Altitude Bend Gas Dust Body height Body weight Body temperature Body fat Blood type Blood pressure Blood sugar Blood oxygen Heart rate Electrograph EEG, ECG, EMG, EOG, GSR Weather Facial expression Facial morphology Facial expression characteristics Geomagnetic Actuators Light Flash Heating Cooling Wind Vibration Sprayer Scent Fog Color correction Initialize color correction parameter Rigid body motion Tactile Kinesthetic Global position command

19 Main features of MPEG AR technologies All AR-related data is available from MPEG standards Real time composition of synthetic and natural objects Access to Remotely/locally stored scene/compressed 2D/3D mesh objects Streamed real-time scene/compressed 2D/3D mesh objects Inherent object scalability (e.g. for streaming) User interaction & server generated scene changes Physical context Captured by a broad range of standard sensors Affected by a broad range of standard actuators

20 MPEG vision on AR Produce Download Authoring Tool Compression MPEG-4/MPEG-7/MPEG-21/ MPEG-U/MPEG-V MPEG Player ARAF

21 End to end chain Remote Real World Environment Remote Sensors & Actuators Local Sensors & Actuators Local Real World Environment Authoring Tools MPEG ARAF ARAF Browser User Media Servers Service Servers

22 MPEG-A Part 13 ARAF Three main components: scene, sensors/actuators, media A set of scene graph nodes/protos as defined in MPEG-4 Part 11 Existing nodes : audio, image, video, graphics, programming, communication, user interactivity, animation New standard PROTOs : Map, MapMarker, Overlay, Local & Remote Recognition, Local & Remote Registration, CameraCalibration, AugmentedRegion, Point of Interest Connection to sensors and actuators as defined in MPEG-V Orientation, Position, Angular Velocity, Acceleration, GPS, Geomagnetic, Altitude Local or/and remote camera sensor Flash, Heating, Cooling, Wind, Sprayer, Scent, Fog, RigidBodyMotion, Kinestetic Compressed media

23 MPEG-A Part 13 ARAF Scene: 73 XML Elements

24 MPEG-A Part 13 ARAF Local object recognition (&tracking)

25 MPEG-A Part 13 ARAF Local object recognition (&tracking) EXTERNPROTO LocImg [ exposedfield SFString videosource "" exposedfield MFString targetresources [] exposedfield MFString targetresourcestypes [] exposedfield SFBool enabled FALSE exposedfield SFInt32 maximumdelay 200 #milliseconds exposedfield SFInt32 optimaldelay 50 #milliseconds exposedfield MFVec2f recognitionregion [] eventout MFInt32 onrecognition eventout MFVec3f ontranslation eventout MFRotation onrotation eventout SFInt32 onerror ] org:mpeg:local_image_recongnition_registration

26 MPEG-A Part 13 ARAF Server side object recognition (&tracking)

27 MPEG-A Part 13 ARAF Local object recognition (&tracking) EXTERNPROTO RemImgProxy [ exposedfield SFString videosource "" exposedfield MFString processingserverurl [] exposedfield SFInt32 frameencodingtype 0 exposedfield SFInt32 processingtype 0 exposedfield MFString targetresources [] exposedfield MFString targetresourcestypes [] exposedfield SFBool enabled FALSE exposedfield SFInt32 maximumdelay 200 #milliseconds exposedfield SFInt32 optimaldelay 50 #milliseconds exposedfield MFVec2f recognitionregion [] eventout MFInt32 onrecognition eventout MFVec3f ontranslation eventout MFRotation onrotation eventout SFInt32 onerror ] org:mpeg:remote_image_recognition_registration

28 Conclusions ARAF is a modeling language able to express AR experiences ARAF V2 published in February 2016 by ISO ARAF Browser is Open Source ios, Android, WS, Linux distributed at ARAF MOOC available on Coursera