Phoneme-Based Distribution Regularization For Speech Enhancement

Existing speech enhancement methods mainly focus on the signal level similarity of the enhanced speech and the target. They do not pay attention to understanding the whole speech and context. Therefore, the recognizability and coherence of the enhanced speech are impaired. To address this problem, we propose a phoneme-based distribution regularization (PbDr) for speech enhancement, which aims to incorporate context information into speech enhancement network in a conditional manner to achieve better perceptual quality and better recognizability. As different phonemes always lead to different feature distributions in frequency, we propose to learn a parameter pair, i.e. scale and bias, through a phoneme classification vector to modulate the speech enhancement network. The modulation parameter pair not only includes frame-wise condition but also frequency-wise condition, which effectively map features to phoneme-related distributions. In this way, we explicitly regularize speech enhancement features by recognition vectors and the semantic information effectively helps to improve the recognizability and coherence of the enhanced speech. Experiments on public datasets demonstrate the effectiveness of PbDr in achieving both better perceptual quality and recognizability.