CS 498PS Audio Computing Lab. Audio Restoration. Paris Smaragdis. paris.cs.illinois.edu U NIVERSITY OF URBANA-CHAMPAIGN

Transcription

1 CS 498PS Audio Computing Lab Audio Restoration Paris Smaragdis paris.cs.illinois.edu

2 Today s lecture Audio restoration Gap filling Click removal Clip recovery 2

3 Missing data approach When restoring audio you often need to make up data e.g. fill-in network drops, replace scratches on disks, etc. How do we treat such problems? 3

4 General classification of cases Missing completely at random (MCAR) Missingness is really random E.g. dropping samples unpredictably Missing at random (MAR) Missingness of x depends on another variable E.g. buffer drops in network Not missing at random (NMAR) None of the above E.g. clipping 4

5 MCAR case Random samples are dropped? How do we fix this?

6 Take advantage of signal smoothness Use linear interpolation For all missing data x(t) replace with: x(t 1)+ x(t +1) 2 You can also see this as some sort of a filter applied only on the data that was missing 6

7 Result Looks (and sounds) pretty good Missing Samples Recovered signal

8 A slightly worse case MAR data on simple waveform How do we deal with this case?

9 Same thing Only this time we will extrapolate Our signal is predictable Use a bandpass filter centered on this tone s frequency Filter only the missing parts given the past samples This imposes the sinusoid s continuity 9

10 What will this do? Frequency (Hz) Time (sec) 1

11 Result Perfect reconstruction Of course, we could do this in easier ways (but it won t scale)

12 A less simple example Now the data is not as predictable How can we find the missing values?

13 Key observations Temporal structure The signal is somewhat periodic We could predict the future from the past How do we formalize this?

14 A predictive model Predict current sample from the past Using a weighted average of preceding samples x t = Autoregressive (AR) model a i x t 1 + e t 14

15 Rewriting the model Linear algebra notation A = e = A x a p! a 1 1! a p! a 1 1! " " # # # # # " "! a p! a 1 1! a p! a 1 1! a p! a

16 Filling gaps Model the input sequence as: e = A x = A U x u + K x k ( ) = A u x u + A k x k x u and x k are the unknown and known samples U and K are sample positioning matrices x = x 1 x 2 x 3 x 4,U x = U u x 1 x 3 = x 1 x 3,K x = K k x 2 x 4 = x 2 x 4 16

17 Find x u to minimize e Set the derivative to zero e e = 2e e x u Solve for unknown samples ( ) A u = x u = 2 A x u + A k x k ( ) 1 A u A k x k ˆx u = A u A u 17

18 Results Works pretty well! Input data blue/cyan real red restoration

19 Some real-life uses Filling in missing data from damaged CDs Minor scratches cause loss of data that needs to be made up This isn t for catastrophic damage, pretty much every CD has scratches Network dropouts When missing a small buffer, we can make up the data that way 19

20 But not for large windows! Input data blue/cyan real red restoration

21 Looking for a new idea We should move away from the sample level, and look at a coarser scale e.g. a time-frequency view 21

22 A simpler idea Find sections most similar to the gap edges Replace missing sections with their neighbors?? x 1 x 2 x 3?? x 6 x 7 x 8?? Iterate until all gaps are closed Match sections using a nearest neighbors scheme 22

23 In action First iteration 23

24 In action Second iteration 24

25 In action Third iteration 25

26 In action Fourth iteration 26

27 In action Final iteration Input Output 27

28 Some examples Out In Out In Speech Music Sound effects In Out 28

29 A model-based viewpoint Nearest-neighbor search was local We ignore global data structure Missing data should conform to a model of the input i.e. they shouldn t be outliers 29

30 NMF-based imputation Simple global approach Replace missing data with some values x = E{x} or x U(µ) or... u u Perform an NMF approximation of the data Replace missing data with NMF approximation Repeat! X W H, X,W,H ( ) u x u = W H 3

31 Why? Known data will dominate model Assuming they are enough! Each approximation will try to conform to the global model of the input i.e. made up values will not be random With each new iteration that conformance to global model will increase 31

32 Variations We don t have to use NMF e.g. our data might have negative values So you can use eigen-decompositions, probabilistic models, etc. Any global model would work Just make sure that it fits the data well 32

33 Example Learn from input and fill-in missing values Input NMF 33

34 Learning from outside Bandwidth expansion Learn model from other recordings Input NMF 34

35 Glitch removal What happens when we have lots of glitches in a track? E.g. vinyl scratches, network glitches, etc. Use missing data methods to replace glitches But how do we know where they are? 35

36 A simple example Signal with multiple clicks and glitches

37 Detecting abnormalities We can use an AR model once again x t = Prediction a i x t 1 + e t Current sample Prediction error The model The past Glitches are rare events, and will be poorly predicted Therefore e t will be large during a glitch 37

38 Prediction error And in action it does pretty well 6 Input signal Prediction error Glitch Threshold

39 Glitch removal Identify the location of problems spots Use prediction error peaks and treat their neighborhood as missing If there s clipping, look for clipped values Use aforementioned methods to fill in the data Remove glitchy region and use the AR model to replace the data 39

40 A real-world example Lots of glitches due to hardware issues 4

41 Finding the glitches Prediction error reveals trouble spots 41

42 Filling in the missing data No glitches and no gaps! 42

43 What about NMAR data? Samples are not missing at random i.e. they depend on the values of the signal Most common audio example of this is clipping Values outside than the dynamic range clip to fixed values Resulting effect is known as distortion 43

44 The toy case Clipping at extreme values How do we recover the clipped peaks? 44

45 What does clipping do? Clipping adds new harmonics in the signal Makes the spectrum busier Frequency (normalized) Frequency (normalized) Original Time (samples) Clipped Time (samples)

46 A plan Fill in values for the missing data such that the spectrum becomes more sparse Can be formalized as a linear optimization problem: y = IDFT(Y) minimize: Y such that: y(k)= x(k) y(u + ) c + k is index of unclipped samples u + is index of positive clipped samples c + is largest sample value (clip point) 46 y(u ) c u is index of negative clipped samples c is smallest sample value (clip point)

47 Result Does a good job in recovering clipped samples Original Clipped Recovered

48 Result 48 And produces the desired spectral effect Frequency (normalized) Frequency (normalized) Frequency (normalized) Original Time (samples) Clipped Time (samples) Recovered Time (samples)

49 With a more challenging case Speech with severe clipping Original sound Clipped sound Recovered sound 49

50 Results 5 Distortion harmonics are suppressed Frequency (normalized) Frequency (normalized) Frequency (normalized) Original Time (samples) Clipped Time (samples) Recovered Time (samples)

51 Recap Missing data techniques AR models Nearest neighbors, NMF Glitch removal Minimizing clipping effects 51

52 Reading A fantastic book on audio restoration (and general audio): 52