Addressing Accent Mismatch In Mandarin-English Code-Switching Speech Recognition

Automatic speech recognition systems suffer from accuracy degradation when code-switching (multiple languages are spoken in a single utterance) is encountered. This is especially common for non-native speakers where there is a mismatch between speech and acoustic model. In this paper, we experiment on Mandarin-English code-switching audio spoken by native Chinese speakers and evaluate three techniques to improve accuracy—data adaptation, individual senone modeling and lexicon enrichment. Our results show the recognition of accented speech improves up to 12% on various code-switching datasets. We also propose several metrics to measure code-switching recognition quality, not captured in typical word error rate (WER) measurement.