Convolutional neural networks contain strong priors for generating natural looking images. These priors enable image denoising, super resolution, and inpainting in an unsupervised manner. Previous attempts to demonstrate similar ideas in audio, namely deep audio priors, (i) use hand picked architectures such as harmonic convolutions, (ii) only work with spectrogram input, and (iii) have been used mostly for eliminating Gaussian noise. In this work we show that existing SOTA architectures for audio source separation contain deep priors even when working with the raw waveform. Deep priors can be discovered by training a neural network to generate a single corrupted signal when given white noise as input. A network with relevant deep priors is likely to generate a cleaner version of the signal before converging on the corrupted signal. We demonstrate this restoration effect with several corruptions: background noise, reverberations, and audio inpainting.
The first type of corruption we attempt to correct is denoising. We corrupt different types of sounds adding different types of noises to them. While these results are not competitive with SOTA denoising methods, they do show the existence of a deep prior inherent to the architecture which is the goal of this study.
Reverberations are acoustical noise appearing in enclosed spaces through multiple reflections of the sound on the walls and objects of a room. This task is inherently more difficult then the other denoising tasks we performed since the corruption itself has the same structure as the signal to be cleaned. As expected, our success here was more limited although a modest amount of dereverberation was obtained.
Speech and music signals are often subject to localized distortions, where the intervals of distorted samples are surrounded by undistorted samples. Examples may include noises or clicks, CD scratches, old recordings, packet loss in cellphones etc. In this work we masked a small (1-5 ms), non-silent, segment of a 2 second clip, and trained the network to recreate the input signal. The training loss ignored the network's output under the mask. Hence, the network is unobstructed in recreating this part of the signal and it's deep priors can come into view.
@article{turetzky2022deep,
title={Deep audio waveform prior},
author={Turetzky, Arnon and Michelson, Tzvi and Adi, Yossi and Peleg, Shmuel},
journal={arXiv preprint arXiv:2207.10441},
year={2022}
}