Oral Qualification Examination_Kun_Zhou

Editor's Notes

• #2: Good morning! I am PhD candidate from ECE department supervised by Prof. Li Haizhou. I am going to present my work on emotional voice conversion with non-parallel data during my first two years.
• #3: First I will give an introduction to emotional voice conversion and the related work. Next I will talk about my PhD research on this topic during these two years.
• #4: As we known, speech conveys information through linguistic, which refers to what we speak, and para-linguistic, which refers to how we speak. Emotional state is a para-linguistic attribute that can even reshape the meaning and understanding of the utterance. Emotional voice conversion is a technique which aims to change the emotional state of an utterance. In the meantime, we would like to preserve the linguistic content and the speaker identity. At run-time, for example, if give the system a happy utterance [Play demo], we can get a sad utterance with the same speaking content and the same speaker [Play demo].
• #5: We can use this technology for many applications, such as social robots and conversational agents. It makes the synthesized voice more emotional and closer to the human voice.
• #6: In emotional voice conversion, there are still many challenges. For example, how to train an emotional voice conversion framework only with nonparallel and limited data is a challenging topic. Besides, emotional prosody is also difficult to model. And it is still difficult for us to control the output emotion strength. The current frameworks are also lack of generalizability, which means if we test the framework with unseen emotion or unseen speakers, the performance may not be so good.
• #7: Here is the list of the publications during the first two years of my PhD study. They are all about emotional voice conversion. I have five conference papers published, and one journal paper submitted.
• #8: Next I will introduce the related work on this topic. Previous studies on emotional voice conversion follow the pattern “analysis-mapping”, as shown in this figure. During the training, imagine we have a source speech which is neutral [Demo], and a target speech which is angry [Demo]. we first extract the speech features from the source and target utterance. We mostly use the spectral features and fundamental frequency to study spectrum and prosody. Then we train a conversion model to learn a feature mapping between the source and target features. Therefore, how to learn a feature mapping function has been the main focus of emotional voice conversion.
• #9: According to the training data, EVC can be divided into two types: If the source and target speech are paired, which means if only the emotion is different, we call it as parallel data. But in real life, such data is expensive and difficult to collect. Therefore, our research has been focusing on how to find a feature mapping function with non-parallel data, which means we have a source speech which is neutral [Demo], but we don’t have a target speech with the same content. Instead, we have a target speech with different content [Demo]. And we try to only learn the difference of emotional style between source and target. Compared with parallel data, emotional voice conversion with non-parallel data is much more challenging but more suitable for real-life applications.
• #10: When it comes to the conversion, we have a feature mapping function that we learnt from the last training stage. We first extract the speech features from the source speech, and give these features to the feature mapping function as the input. Then we can get the converted features through the feature mapping. Next we need to reconstruct the speech waveform from the converted features. We call this step as “waveform generation”. The model we used for this step we call it as vocoder. The vocoder quality determines the quality of the output speech. But speech quality is not our focus, we more care about the emotional expression in the output speech. We want the output emotion is more intelligible and more similar with the target emotion. Therefore, our main focus is to train a good feature mapping. And the emotional expression of the speech mostly depends on the feature mapping function. If we want to get a better speech quality, all we need to do is just to train a better vocoder. Here are some speech samples generated by different vocoders. 1/ the first one is Reference audio [Demo]; 2/ the second one is synthesized by Griffin-Lim [Demo]; 3/ This one is WaveRNN [Demo]; 4/ This one is Parallel WaveGAN [Demo]; From these demos, the vocoder quality determines the speech quality. But my research is focus on the emotional expression rather than the speech quality.
• #11: In speech community, speaker voice conversion is a popular and well-studied research topic. Speaker voice conversion aims to convert the speaker identity and protect other prosodic attributes. Compared with speaker voice conversion, emotional voice conversion can be a more challenging task. On the one hand, emotion is much more subjective and difficult to describe. And emotional style also complex with multiple signal attributes. Prosody, such as intonation, speaking rate, and speech energy, all play an important role in emotional expression and perception.
• #12: During the first two years, I propose and develop these four different frameworks on this topic. They all aim to tackle the current challenges we talked about in the previous slide. For these four frameworks, we always focus on finding nonparallel and limited data solutions for emotional voice conversion.
• #13: The first work I talk about is cycle-gan based evc, which is published in Speaker odyssey 2020. In this work, we aim to focus on non-parallel training and F0 modelling. F0, which is fundamental frequency, is an essential part of the intonation, varying from syllables to utterances which makes it difficult to model. Therefore we propose to use continuous wavelet transform to model F0 over multiple time scales. And convert it together with spectral features with CycleGAN. Moreover, we also investigate different training strategies to get a better performance.
• #14: We use continuous wavelet transform to study F0 modelling in different time scales. As shown in this figure, we decompose the F0 into ten scales, and we assume the lower scales can capture the short-term variations, such as syllables and phonemes, and higher scales can capture the long-term variations, such as phrases and utterances.
• #15: During training, we have two pipelines: the first is spectral cyclegan, which is to learn the feature mapping of spectral features, and another one is prosody cyclegan, to learn the mapping of CWT- F0 features given by continuous wavelet transform. During conversion, these two models convert the spectral and F0 features from source to target emotion types.
• #16: We further conduct two listening tests to assess the final performance. The first figure shows , with continuous wavelet transform, the emotion similarity is improved, From the 2nd figure, we shows that the separate training of spectral and prosodic features outperforms the joint training. Here you can listen to some samples, if we have a source speech which is neutral [demo], we can convert it to angry [demo], sad [demo], and surprise [demo].
• #17: The second work I am going to talk about is speaker-independent EVC, which we publish on Interspeech 2020. As we known, emotional expression shares some common cues across individuals. For example, no matter who speaks with happy, the fundamental frequency usually have a higher mean and standard variance than sad. So in this work, we study the universal pattern of emotional expression across different speakers, thus we call it as speaker-independent emotional voice conversion. In the technical part, we propose a VAW-GAN-based architecture to enable non-parallel training, and study prosody modelling with continuous wavelet transform and F0 conditioning for encoder training.
• #18: Our proposed framework is based on VAW-GAN. It is a conditional variational auto-encoder followed by a discriminator. Compared with VAE, VAW-GAN can generate more realistic features. We further use VAW-GAN to study how to disentangle the emotional elements from speech. As the model input, spectral features contains speaker information which related to the speaker identity, phonetic information which is linguistic, and the prosodic information. We propose to provide emotion ID and F0 to the decoder, and the encoder can learn to discard emotion-related information and the latent code can be emotion-independent.
• #19: Our framework also consists of two pipelines, one for prosody and one for spectrum. For prosody pipeline, the encoder learns emotion-independent representation from CWT-F0 features and the generator learns to reconstruct prosody features with one-hot emotion label. The discriminator learns to judge whether the reconstructed features real or not.
• #20: The spectrum pipeline is similar with the prosody one. The only difference is we condition the generator not only with the emotion ID, but also the F0 values.
• #21: In experiments, we would like to validate our idea of 1/ CWT analysis on prosody modelling; 2/ F0 conditioning for encoder training; and 3/ performance with seen and unseen speakers. These two XAB preference test show the effectiveness of our proposed framework.
• #22: Next, I am going to talk about our work which published in this year ICASSP, EVC for seen and unseen emotions. Current emotional voice conversion frameworks represent each emotion with one-hot emotion label, but such representation only learn to remember a fixed set of emotions and may not be sufficient to describe different emotional styles. As we known, the emotional styles also can present subtle difference even with the same emotion category. In this work, we propose a one-to-many emotional style transfer framework. We use a pre-trained Speech Emotion Recognition model to describe different emotional styles. Our framework can work with non-parallel data and transfer the emotional style for both seen and unseen emotions.
• #23: There are three stages in our proposed framework. In the 1st stage, we train a SER model and use it to get deep emotional features for each utterances; Then we train the VAW-GAN framework with deep emotional features; the decoder learns to reconstruct the spectral features from the latent representation from the encoder, F0 and deep emotional features; In the last stage run-time conversion, if we give the framework the deep emotional features of either seen emotion or unseen emotion, the framework can reconstruct the speech features with the reference emotional style.
• #24: We further conduct two preference tests for speech quality and emotion similarity. Both of them validate the effectiveness of our proposed framework.
• #25: The last work I am going to talk about is our recent work published in this year Interspeech, limited data EVC. Previous work I talk about all convert the feature frame-by-frame, which means the speech duration always kept to be the same; but speech duration is an important factor in speech rhythm and it has been a missing point in these frame-based models. To convert the speech duration, one solution is to train a sequence-to-sequence framework, which can predict the speech duration with the attention mechanism. But to train a seq2seq framework, we need a large amount of training data, nearly tens of hours to achieve a good prediction. If the training data is not sufficient, the framework may not learn a good alignment and the final performance will be poor. But for emotional speech data, there is no such large-scale datasets. In this work, we are trying to build a seq2seq evc framework only requires a limited amount of emotional speech data, and can do both emotional voice conversion and emotional text-to-speech. Besides, it can jointly model spectrum, prosody, and duration; it can work with non-parallel training data and can do many-to-many conversion.
• #26: We propose a two-stage training for limited data evc: Style initialization and emotion training. During style initialization, we leverage available large TTS corpus which is all neutral data. The style encoder learns speaker. During the emotion training, we retrain the whole framework with limited amount of emotional speech data. The style encoder becomes an emotion encoder to learn the emotional style.
• #27: To validate our idea of two-stage training, we visualize the emotion embeddings derived from the style encoder and emotion encoder. From this figure, we observe that with emotion retraining, the emotion encoder can generate meaningful emotion representations and each emotion form separate clusters and there is a significant separation between different emotion types.
• #28: We choose two state-of-the-art emotional voice conversion frameworks CycleGAN and StarGAN for the baselines. We conduct emotion conversion from neutral to angry, happy, sad and surprise. Here are some speech samples. If we give a source neutral speech [demo], we can convert it to angry, it is cyclegan sounds like [demo], it is stargan[demo], it is our proposed [demo], it is the target[demo]. For the neutral to happy, it is the source[demo]…. From these speech samples, we can clearly feel that our proposed framework has a much better performance than the baselines, especially for the neutral-to-surprise.
• #29: Our framework also can do emotional text-to-speech. If we give an input text “clear than clear water” [demo], we can synthesis angry one[demo], sad[demo], surprise[demo], and happy[demo].
• #30: We further conduct listening experiments to evaluate the emotional expression. From this figure, we can observe that our framework significantly outperform the baselines in emotion similarity evaluation.
• #31: As a conclusion, in this presentation, we first introduce the emotional voice conversion and its theory and challenges. With these challenges, we introduce our work, which is CycleGAN based EVC, Speaker-independent EVC, EVC for seen and unseen emotions, and limited data EVC. For all the work, we provide the demo website and the codes are all available at github. As for future studies, we would like to these following topics, 1/ Emotional voice conversion with emotion strength control, which aims to control the output emotion strength, emotional interpolation for emotional voice conversion, which aims to convert the emotion in a continuous scale. Cross-lingual representations for emotional voice conversion, which aims to study a cross-lingual emotion representation across different languages.
• #32: It is the ending of this presentation. Thank you for listening!