MPEG for Augmented Reality

Transcription

1 MPEG for Augmented Reality ISMAR, September 9, 2014, Munich Marius Preda, MPEG 3DG Chair InsAtut Mines TELECOM h7p:// augmented- reality- tutorial

2 What you will learn today Who is MPEG and why MPEG is doing AR MPEG ARAF design principles and the main features Create ARAF experiences: two exercises

3 Tidy City

4 Portal Hunt

5 Elements

6 ARQuiz

7 Augmented Books

8 Event LOOV Collecting virtual money in real world for buying real services and products Available on AppStore, AndroidStores and MyMulAmediaWorld.com

9 Summer School (1 week) Games

10 What is common in these "games"? Based on MPEG ARAF Augmented Reality ApplicaBon Format

11 MPEG Augmented Reality Why MPEG AR?

12 Answers to (some) of ChrisBne s (non- technical) quesbons Who is MPEG? What MPEG does successfully? Who are the members? IPR policy

13 What is MPEG? A suite of ~130 ISO/IEC standards for: Coding/compression of elementary media: Audio (MPEG- 1, 2 and 4), Video (MPEG- 1, 2 and 4), 2D/3D graphics (MPEG- 4) Transport MPEG- 2 Transport, File Format, Dynamic AdapBve Streaming over HTTP (DASH) Hybrid (natural & synthebc) scene descripbon, user interacbon (MPEG- 4) Metadata (MPEG- 7) Media management and protecbon (MPEG- 21) Sensors and actuators, Virtual Worlds (MPEG- V) Advanced User interacbon (MPEG- U) Media- oriented middleware (MPEG- M) More ISO/IEC standards under development for Coding and Delivery in Heterogeneous Environments (incl.) 3DVideo

14 What is MPEG? A standardizaaon acavity conanuing for 25 years, Supported by several hundreds companies/organisabons from ~25 countries ~500 experts parbcipabng in quarterly meebngs More than 2300 acbve contributors Many thousands experts working in companies A proven manner to organize the work to deliver useful and used standards Developing standards by integrabng individual technologies Well defined procedures Subgroups with clear objecbves Ad hoc groups conbnuing coordinated work between meebngs MPEG standards are widely referenced by industry 3GPP, ARIB, ATSC, DVB, DVD- Forum, BDA, EITSI, SCTE, TIA, DLNA, DECE, OIPF Billions of souware and hardware devices built on MPEG technologies MP3 players, cameras, mobile handsets, PCs, DVD/Blue- Ray players, STBs, TVs, Business friendly IPR policy established at ISO level

15 MPEG technologies related to AR: 1 st pillar 1992/ MPEG- 1/2 (AV content) VRML MPEG- 4 v.1 Part 11 - BIFS: - BinarisaAon of VRML - Extensions for streaming - Extensions for server command - Extensions for 2D graphics - Real &me augmenta&on with audio & video Part 2 - Visual: - 3D Mesh compression - Face animaaon MPEG- 4 v.2 Part 2 Visual - Body animaaon First form of broadcast signal augmentaaon

16 MPEG technologies related to AR: 1 st pillar MPEG- 4 Part 16 - AFX: - A rich set of 3D graphics tools - Compression of geometry, appearance, animaaon A rich set of Scene and Graphics representaaon and compression tools MPEG- 4 AFX 2 nd EdiAon: - AnimaAon by morphing - MulA- texturing MPEG- 4 AFX 3rd EdiAon - WSS for terrain and ciaes - Frame based animaaon MPEG- 4 AFX 4th EdiAon - Scalable complexity mesh coding

17 MPEG technologies related to AR: 2 nd pillar x MPEG- V - Media Context and Control MPEG- U 1st EdiAon Advanced - Sensors and User Interface actuators - Interoperability between Virtual Worlds A rich set of Sensors and Actuators MPEG- V 2 nd EdiAon: - GPS - Biosensors - 3D Camera MPEG- H - 3D Video Compression CDVS of video + Feature- point based depth - 3D Audio descriptors for image recogniaon

18 MPEG technologies related to AR: 2nd pillar MPEG- V Media Context and Control

19 MPEG technologies related to AR: 2 nd pillar MPEG- V Media Context and Control Sensors Light Ambient noise Temperature Humidity Distance Atmospheric pressure PosiBon Velocity AcceleraBon OrientaBon Angular velocity Angular accelerabon Force Torque Pressure MoBon Intelligent camera type MulB InteracBon point Gaze tracking Wind Global posibon AlBtude Bend Gas Dust Body height Body weight Body temperature Body fat Blood type Blood pressure Blood sugar Blood oxygen Heart rate Electrograph EEG, ECG, EMG, EOG, GSR Weather Facial expression Facial morphology Facial expression characterisbcs GeomagneBc Actuators Light Flash HeaBng Cooling Wind VibraBon Sprayer Scent Fog Color correcbon IniBalize color correcbon parameter Rigid body mobon TacBle KinestheBc Global posibon command

20 Main features of MPEG AR technologies All AR- related data is available from MPEG standards Real Ame composiaon of syntheac and natural objects Access to Remotely/locally stored scene/compressed 2D/3D mesh objects Streamed real- Bme scene/compressed 2D/3D mesh objects Inherent object scalability (e.g. for streaming) User interacaon & server generated scene changes Physical context Captured by a broad range of standard sensors Affected by a broad range of standard actuators

21 MPEG vision on AR Produce Download Authoring Tool Compression MPEG- 4/MPEG- 7/MPEG- 21/ MPEG- U/MPEG- V MPEG Player ARAF

22 End to end chain Remote Real World Environment Remote Sensors & Actuators Local Sensors & Actuators Local Real World Environment Authoring Tools MPEG ARAF ARAF Browser User Media Servers Service Servers

23 MPEG- A Part 13 ARAF Three main components: scene, sensors/actuators, media A set of scene graph nodes/protos as defined in MPEG- 4 Part 11 ExisBng nodes : Audio, image, video, graphics, programming, communicabon, user interacbvity, animabon New standard PROTOs : Map, MapMarker, Overlay, Local & Remote RecogniBon, Local & Remote RegistraBon, CameraCalibraBon, AugmentedRegion, Point of Interest ConnecAon to sensors and actuators as defined in MPEG- V OrientaBon, PosiBon, Angular Velocity, AcceleraBon, GPS, GeomagneBc, AlBtude Local or/and remote camera sensor Flash, HeaBng, Cooling, Wind, Sprayer, Scent, Fog, RigidBodyMoBon, KinesteBc Compressed media

24 MPEG- A Part 13 ARAF Scene: 73 XML Elements DocumentaAon available online: hep://wg11.sc29.org/augmentedreality/

25 Event LOOV, how it looks like?

26 MPEG- A Part 13 ARAF Exercises AR Quiz Augmented Book

27 MPEG- A Part 13 ARAF Exercises AR Quiz Augmented Book h7p://youtu.be/la- Oez0aaHE h7p://youtu.be/lxzubafpp- Y

28 MPEG- A Part 13 ARAF AR Quiz senng, preparing the medias images, videos, audios, 2D/3D assets GPS locaaon

29 MPEG- A Part 13 ARAF AR Quiz XML inspecbon h7p://bny.cc/mpegarquiz

30 MPEG- A Part 13 ARAF AR Quiz Authoring Tool go to Create / Augmented Reality

31 MPEG- A Part 13 ARAF Augmented Book senng images, audios

32 MPEG- A Part 13 ARAF Augmented Book XML inspecbon h7p://bny.cc/mpegaugbook

33 MPEG- A Part 13 ARAF Augmented Book Authoring Tool go to Create / Augmented Books

34 Conclusions ARAF Browser is Open Source ios, Android, WS, Linux distributed at ARAF V1 published early 2014 ARAF V2 in progress Visual Search (client side and server side) 3D Video, 3D Audio Connection to Social Networks Connection to POI servers

35 MPEG 3DG Report ARAF 2 nd Edition

36 MPEG 3DG Report ARAF 2 nd Edition, items under discussion 1. Local vs Remote recognition and tracking 2. Social Networks 3. 3D video 4. 3D audio

37 MPEG 3DG Report Server side object recognition: a real system* Client Server (2.1) Query image (1) Query descriptors DB descriptors [ExtracBon] Descriptors (3.1) (2.2) [DetecBon] Key points Binary Data (3.2) HTTP POST (binary descriptor + key points) (4) Decod e (6) Matching (7) (5.2) Decod e (5.1) Parse and display the answer HTTP Response (8 ) Error/no message Data as String (10) ID (8 ) Correspondin g InformaBon Descriptors, images and informabon [DB] (9) * Wine recognizer : GooT and IMT

38 MPEG 3DG Report Server side object recognition: ARAF version Processing Server URLs End- user Device ARAF Browser Processing Servers Corresponding media DB MAR Experience Creator + Content Creator opaonal: recognibon region Source (video URL) MAR Scene Video stream Video stream Video source ORB Media data Binary (base64) key points + descriptors Image RecogniBon DetecBon Libraries Library Large Image DB

39 MPEG 3DG Report Server side object recognition: ARAF version Discussions on: - Does the content creator specify the form of request (full image or descriptors) or the browser will take the best decision? - Is the server s answer formalized in ARAF?

40 ARAF can do this directly by programming the access to the SN service at the scene level MPEG 3DG Report ARAF Social Network Data in ARAF scene Scenario: display posts from SN in a geo-localized manner

41 MPEG 3DG Report ARAF Social Network Data in ARAF scene At minimum, user login to SN - at maximum : the MPEG UD

42 MPEG 3DG Report ARAF Social Network Data in ARAF scene Connect to an UD server to get all the necessary data

43 MPEG 3DG Report ARAF Social Network scenario Two categories of SNS Data Static data Name, photo, , phone number, address, sex, interest, Social Network related activity Reported location, SNS post title, SNS text, SNS media, SNS media Obtained from the UD server

44 MPEG 3DG Report ARAF 2 nd Edition introducing 3D Video Modeling of 3 AR classes for 3D video: 1. Pre-created 3D model of the environment, using visual search and other sensors to obtain camera position and orientation; 3D video used for handle occlusions 2. No a priori 3D model of the scene, depth captured in real-time and used to handle occlusions at the rendering step 3. No a priori model of the scene but created during AR experience (SLAM Simultaneous Location and Mapping)

45 MPEG 3DG Report ARAF introducing 3D Audio Spatialisation Recognition Use sounds from the real world to trigger events in an AR scene

46 MPEG 3DG Report ARAF 3DAudio : local spatialisation Mobile device ARAF Browser MAR Experience Creator + Content Creator ARAF file User locabon & direcbon + sound locabon Scene RelaBve sound locabon RelaBve sound locabon + (AcousBc scene) + audio source SpaBalized audio source Coordination mapping 3D Audio Engine PosiBon & orientabon sensor Sensed data Video/audio stream Synthesized audio stream Mixer Microphone Video/audio stream Camera

47 MPEG 3DG Report ARAF 3DAudio : remote spatialisation MAR Experience Creator + Content Creator ARAF file Processing Server URL Scene Mobile device ARAF Browser User locabon & direcbon + sound locabon RelaBve sound locabon Coordination mapping RelaBve sound locabon + Audio source + (AcousBc scene) SpaBalized audio source Proxy Server Detection 3D Audio Library Engine PosiBon & orientabon sensor Sensed data video/audio stream Synthesized audio stream Mixer Microphone Video/audio stream Camera

48 MPEG 3DG Report ARAF Audio recognition: local Mobile device ARAF Browser Target Resources MAR Experience Creator + Content Creator Target Resources or descriptors Source (microphone/audio URL) opaonal: detecbon window, sampling rate, detecbon delay Scene ID Mask Audio DetecBon DetecBon Library Microphone/audio stream Audio source

49 MPEG 3DG Report ARAF Audio recognition: local MAR Experience Creator + Content Creator Target Resources or descriptors Source (microphone/audio URL) opaonal: detecbon window, sampling rate, detecbon delay URL of Processing Server Scene Mobile device ARAF Browser ID Mask Microphone/audio stream Proxy Server Target Resources or descriptors + IDs + opbonal detecbon window, sampling rate, detecbon delay Audio Detection DetecBon Library Audio source

50 MPEG 3DG Report ARAF Audio recognition: local MAR Experience Creator + Content Creator Target Resources or descriptors Source (microphone/audio URL) opaonal: detecbon window, sampling rate, detecbon delay URL of Processing Server Scene Mobile device ARAF Browser Microphone/audio stream ID Mask Descriptor ExtracBon Descriptors Processing Server Target Resources or descriptors + IDs + opbonal detecbon window, sampling rate, detecbon delay Audio Detection DetecBon Library Audio source

51 MPEG 3DG Report ARAF joint meeting with 3DAudio Spatialisation Recognition The 3D audio renderer needs an API to get the user position and orientation It may be more complex to update in real time position and orientation of all the acoustic objects MPEG-7 has several tools for audio fingerprint Investigate the ongoing work on Audio synchronisation and check if it is suitable for AR