Low-latency electrolaryngeal speech enhancement based on FastSpeech2-based voice conversion and self-supervised speech representation

In this paper, we propose a low-latency sequence-to-sequence speech enhancement technique for electrolaryngeal (EL) speech. A low-latency EL speech enhancement technique based on CLDNN was previously proposed to enable laryngectomees to produce relatively naturally sounding speech compared to the original EL speech. However, the naturalness of the enhanced speech is still far from that of the natural speech due to insufficient modeling accuracy of an acoustic feature sequence and speech waveform caused by frame-wise conversion processing. To solve this problem, in this paper, we propose a low-latency sequence-to-sequence EL speech enhancement technique based on CLDNN-FastSpeech2-based VC. Moreover, to improve various factors such as naturalness, intelligibility, and robustness, we also propose the following techniques: 1) utilizing a self-supervised speech representation (SSL) and 2) randomly sampling the predicted and ground-truth features to alleviate prediction errors in variance adaptor. The experimental results demonstrate that the proposed method yields better performance in both objective and subjective evaluations compared to the conventional frame-based EL speech enhancement technique.