MP3 - then and now Breakthrough technology applications for the New Digital World

Transcription

1 MP3 - then and now Breakthrough technology applications for the New Digital World Karlheinz Brandenburg Technische Universität Ilmenau, Institute for Media Technology Fraunhofer Institute for Digital Media Technology IDMT Science Circle, Delhi, January 11 th, 2018 Fraunhofer 1

2 Fraunhofer-Gesellschaft Fraunhofer is the international leader of applied research 69 institutes and independent research units staff at more than 40 locations More than 2.1 billion euros annual research budget totaling; 1.9 billion euros from contract research Roughly two thirds of this sum is generated through contract research on behalf of industry and publicly funded research projects Roughly one third is contributed by the German federal and Länder governments in the form of base funding International cooperations Fraunhofer 2

3 Fraunhofer IDMT Foundation of the working group of the Fraunhofer IIS in Erlangen in 2000 Foundation of the Fraunhofer IDMT on January 1st, 2004 Director of the institute: Prof. Dr. Karlheinz Brandenburg 160 employees, apprentices, PhD students, interns and student assistants Head office in Ilmenau, branch lab in Oldenburg (Lower Saxony) Fraunhofer 3

4 Overview Introduction: The dream of high fidelity The mp3 story Basic research Standardization Internet marketing works How do standardization, patents and commercialization interact Newer examples from the world of digital media 3D-Audio, recognition methods, listening to machines Conclusions Fraunhofer 4

5 Early Audio Storage Media Copyright: Nadja Wallaszkovits Fraunhofer 5

6 Overview Introduction: The dream of high fidelity The mp3 story Basic research Standardization Internet marketing works How do standardization, patents and commercialization interact Newer examples from the world of digital media 3D-Audio, recognition methods, listening to machines Conclusions Fraunhofer 6

7 Early Years of Audio Coding Research in Erlangen Dreams in 1982: Dieter Seitzer: We should use ISDN to transmit music An anonymous patent examiner: There is no high fidelity music at 128 kbit/s There was coding of pictures, video, speech, but not of high quality audio signals Digital Audio Broadcast needed high quality, low bit-rate audio Start of work at Erlangen University and Fraunhofer IIS (and Duisburg, Murray Hill, Hannover, Munich, Tokyo other places at the same time) Fraunhofer 7

8 A Success Story of Audio Signal Processing: Audio Coding: mp3, AAC, AC-3 etc. Status today: More than 50% sold music via non-physical distribution mp3 decoders in phones, CD players, computers etc. More than 8 billion devices deployed The same for AAC: More than 8 billion devices deployed And: Yes, there are some other systems out there, too Fraunhofer 8

9 How does mp3 work? Wanted: Transmission of music at 1/10 th data rate Impossible without loss of information (we call this reduction of irrelevancy) The quality is judged by listening tests, our goal is no audible difference from the original signal The basic idea: perceptual coding The signal sink: Our ears Human ears are very sensitive, but we do not hear everything, loud sounds mask faint sounds Fraunhofer 9

10 Basics of High Quality Audio Coding The goal: transparent coding of music signals The source is not known in advance Use information about the sink, not the source The solution: Modeling of the masking threshold of the ear The quantization noise has to be kept below the masked threshold Fraunhofer 10

11 Psychoacoustics (Masked Threshold) 80 db 60 f =0,25 m f =1kHz f m m =4kHz L T ,02 0,05 0,1 0,2 0, khz Redrawn after Zwicker f T Fraunhofer 11 11

12 Demo: The 13 db-miracle (Johnston/Brandenburg 1989) Original signal Original + white noise, SNR = 13,6 db Original + noise at threshold, S/N = 13,6 db Difference (modulated white noise) Difference (noise at threshold) 12 Fraunhofer 12

13 LC-ATC 1987 Copyright: Fraunhofer IIS/ Kurt Fuchs Fraunhofer 13

14 Overview Introduction: The dream of high fidelity The mp3 story Basic research Standardization Internet marketing works How do standardization, patents and commercialization interact Newer examples from the world of digital media 3D-Audio, recognition methods, listening to machines Conclusions Fraunhofer 14

15 Standardization by ISO/IEC These are organizations which work under treaties between countries ISO: International Organization for Standardization IEC: International Electrotechnical Commission Binding force of standards: standards are no laws standards are mostly not binding Exceptions: Laws are referring to standards Violation of security standards claim for damages Fraunhofer 15

16 Standardization and First Products 1988: MPEG starts soliciting proposals for video Dec. 1988: Ooops, we need audio compression 1989 to 1991: Fierce competition Two consortia, one with IRT (Munich), one with Fraunhofer (Erlangen) Everybody has some advantage 1992: MPEG-1 audio gets three modes Layer 1: DCC Layer 2: DAB, DVB, CD-I Layer 3: Feeding audio over ISDN Fraunhofer 16

17 Overview Introduction: The dream of high fidelity The mp3 story Basic research Standardization Internet marketing works How do standardization, patents and commercialization interact Newer examples from the world of digital media 3D-Audio, recognition methods, listening to machines Conclusions Fraunhofer 17

18 Marketing the Internet Way 1994: Nearly no licensees for our technology 1994: Harald Popp and Dagfinn Bach visit the offices of a major record label in Munich 1995: Ricky Adar wants to start the Internet music distribution business, he asks us Do you know that you will destroy the music industry? 1995: World Space decides to use mp3, first CE application 1995: We start working on DRM technologies Fraunhofer 18

19 Marketing the Internet Way 1995: Winplay3 Demo, Name mp3 ( ) Registration codes are spread Macromedia licenses Layer : Students write other decoders First license to Microsoft 1997: A stolen mp3 encoder can be found widely mp3.com starts 2003: Apple announces the itunes music shop Fraunhofer 19

20 20 Years later Copyright: Fraunhofer IIS/ Kurt Fuchs Fraunhofer 20

21 Standards in Audio Coding 1992: MPEG Audio Layer 1/2/3 (Layer-3 = mp3) 1997: MPEG-2 part 7: Advanced Audio Coding (AAC) 2003 and following: HeAAC, extensions following : 3GPP EVS (Enhanced Voice Services) VoLTE 2015: MPEG-H 3D Audio Fraunhofer 21

22 Overview Introduction: The dream of high fidelity The mp3 story Basic research Standardization Internet marketing works How do standardization, patents and commercialization interact Newer examples from the world of digital media 3D-Audio, recognition methods, listening to machines Conclusions Fraunhofer 22

23 Standards and Patents What are patents for: Enable access to research results and technology (!!!) Standards are for cooperation, this enforces a number of rules There are clear rules who can participate (often delegates of countries) The results must be open to all companies who want to use them There may be patents which have to be licensed to exercise the standard. RAND rules (Reasonable And Non Discriminatory): Reasonable license fees I am not allowed to discriminate certain possible licensees These rules have been defended in court in many countries Fraunhofer 23

24 Why did Licensing Work as it did? Needed: The right business model RealAudio first had audio technology on PCs We partly followed their business model The grass-roots movement of students and others using music on PCs helped a lot CE companies had to follow There was a clear distinction: Encoder / Decoder PCs / all other devices Fraunhofer 24

25 Overview Introduction: The dream of high fidelity The mp3 story Basic research Standardization Internet marketing works How do standardization, patents and commercialization interact Newer examples from the world of digital media 3D-Audio, recognition methods, listening to machines Conclusions Fraunhofer 25

26 What is next? Immersive audio Virtual reality, augmented reality, mixed reality Analysis of media signals Let the machine help in finding my favorite music Help in retaining privacy if needed, security if needed Apply media technologies beyond the field of entertainment Fraunhofer 26

27 The Home Entertainment Center of Tomorrow Remembers my music: Each CD gets ripped just when I play it Helps to search for my favorite music Query by Humming Look for certain genres Searches the Internet Where can I listen to/buy this music? Helps to assemble my favorite music playlist What is similar, but new? Please play my favorite music for this time of the day. Fraunhofer 27

28 Query by Humming at Saturn-Markt, Munich Fraunhofer 28

29 3D Audio (Object based Audio) Immersive and stable audio reproduction Interactive dynamics room simulation Intuitive User interface Easy storage and transfer of produced audio content Open interface to many Hard- und Software platforms Copyright: Monika Rittershaus Fraunhofer 29

30 3D Audio for Entertainment Applications Open Air Clubs Cinema Planetaria Theater Virtual Reality Fraunhofer 30

31 Copyright: Bregenz Festival Thanks to our SpatialSound technology, the Bregenz Festival presents an impressive high-quality spatial sound. "There are no other installations comparable to 'Bregenz Open Acoustics' - not even in Berlin or the USA. Prof. Dr. Wolfgang Fritz, chief sound engineer of the Bregenz Festival Fraunhofer 31

32 Copyright: AUDI Cars will turn into rolling hifi studios using a fully integrated spatial sound system based on Wave Field Synthesis Fraunhofer 32

33 Video - Object Detection & Face Recognition Goal: detection of e.g. animals, persons, buildings, cars, etc. Approach: Based on deep learning (CNN, TensorFlow transfer learning) Fraunhofer 33

34 The Micro-Sleep Alerter recognizes when drivers are about to fall asleep at the wheel, and warns them by means of an audio signal. Fraunhofer 34

35 Industrial Media Applications Measurement of structure- borne sound air-borne sound ultrasonic sound in noise-free and noisy surroundings Added Value for Industrial Production Signal analysis of one-channel multi-channel signals Signal processing embedded in networked systems cloud Secure distribution storage analysis of measurement results Scalable methods supervised unsupervised deep learning on various sensor data Fraunhofer 35

36 Example: Process Monitoring in the Food Industry Fraunhofer 36

37 Example: Process Monitoring During Production Copyright ripley engineering: Fraunhofer 37

38 Process Monitoring During Production Sound sample: optimal welding process Sound sample: Displacement of the plates during the welding process Fraunhofer 38

39 Key Technologies for Digital Media Applications of the Future Our Expertise Signal analysis understand and facilitate the use of media data 3-D audio record, analyze, create, and play Acoustic condition monitoring listening, analyzing and secure processing for Industry 4.0 Adaptive media for education and entertainment User-friendly human-machine interaction Audio technology and communication systems for people with normal and impaired hearing Fraunhofer 39

40 Overview Introduction: The dream of high fidelity The mp3 story Basic research Standardization Internet marketing works How do standardization, patents and commercialization interact Newer examples from the world of digital media 3D-Audio, recognition methods, listening to machines Conclusions Fraunhofer 40

41 Conclusions The quest for the ultimate sound reproduction continues Some tasks we can consider solved Some applications of DSP are just entering the market place For some areas, we still lack basic knowledge Many new applications become possible due to advanced technologies 3D-Audio for entertainment from the concert hall to my car Analysis of audio and video signals Industrial applications for preventive maintenance and end-of-line testing It has been a privilege to work on topics, where pleasure (listening to music) and work blend in such a nice way. Fraunhofer 41