Is it possible to train a Turkish text-to-speech model with English data

Abstract

Keywords: Natural language processing; nlp; speech processing; speech synthesis; text-to-speech; tts; artificial intelligence

Introduction

Interaction with machines becomes more interactive day by day. In the past, graphical interfaces were often preferred for entering commands to machines, but today more modern interaction methods can be preferred like voice control. To control a machine by voice, it is necessary to teach it to interpret the speech signal first. Then, we have to give the speaking ability to the machine to get a voice response from the machine. Researchers working in the field of natural language processing carry out these human-specific behaviors (listening, understanding, writing, speaking) to machines. Speech synthesis, i.e. the process of making machines speak, provides the artificial generation of a speech signal against text input. While these studies were initially done by adding phones to each other as a rule-based[1], they were later modeled statistically[2]. Today, speech synthesis studies can be done as naturally as human speech with deep neural network-based models trained with sufficient data. In addition to speech naturalness, another important issue in speech synthesis is to ensure the production of the sound on the CPU is synthesized very close to real-time. Because

in many products, performing speech synthesis on the device is an important factor to reduce server costs. In models developed with deep learning methods, the amount and quality of data have great importance. As stated in many publications, the performance of models developed with deep learning methods increases with the amount of data[3]. However, in some cases, finding or producing data can be quite costly. Especially in speech synthesis studies, there is a need for tens of hours of sound recordings and transcriptions of these recordings, which are language-specific and recorded in a studio environment without background noise. In some languages with data scarcity, it can be very difficult to carry out speech synthesis studies with deep learning methods. Turkish is one of the languages that is very difficult to find open data sources in this field. There are datasets created by volunteers vocalizing non-copyrighted books in languages such as English[4] [5], German[6] and Russian[7]. Since there is no such dataset for Turkish, it will be investigated whether the need for data can be eliminated by expressing the sources in other languages phonetically in Turkish.

*Corresponding author. *E-mail address: eengin.eergun@gmail.com Published by Yıldız Technical University Press, İstanbul, Turkey Copyright 2022, Yıldız Technical University. This is an open access article under the CC BY-NC license (http://creativecommons.org/licenses/by-nc/4.0/).

In this study, it will be discussed how to construct a Turkish speech synthesis model using acoustic data from an English dataset. For this, the LJSpeech dataset, which is frequently used in English speech synthesis studies, was used. This set contains approximately 24 hours of recordings of a female speaker. In this dataset, there are 13100 speech segments from 1 to 10 seconds in length. The texts corresponding to these segments will be phonetically translated into Turkish. Then the speech synthesis model will be trained with the updated dataset. In the sub-headings of the methodology section, it will be explained how to do these operations in order.

in Turkish. In non-phonetic languages such as English, words can be expressed with phonemes. To synthesize Turkish speech with English data, the words in the English dataset first must be phonetically translated into Turkish. This situation can be likened to writing the words spoken in English as they are heard directly in Turkish. In order to express the phones in Turkish, the steps shown in Figure 1 were followed. In this study, CMUDict[14] and LJSpeech were used. CMUDict is a dictionary that phonetically expresses more than 120000 words in English. LJSpeech is a 24-hour dataset used in speech synthesis studies.

Literature Review

Phonetical Conversion The CMUDict shown in Figure 1 presents English words and their phonetic representations. The phones in this phonetic dictionary were replaced with the Turkish expressions of the English phones as in Table 1, and as a result, CMUDict’ was created. For example, the word and phoneme sequence “ABANDONED [AH0 B AE1 N D AH0 N D]” in CMUDict is expressed as “ABANDONED [ı b n d ı n d]” in CMUDict’. Table 1 was used directly while performing this conversion.

In the speech synthesis process, the model tries to obtain a complex and non-linear signal such as voice from highly compressed primitive data such as text. In this case, there is a lot of information gap between input and output. To reduce this information gap, two different models are generally used together. The first model, called the synthesizer, converts text to basic representations of sound, called mel-spectrograms, and the second model, called vocoder, generates sound signals in the time domain with mel-spectrograms. Vocoders play a very important role in producing realistic sounds. In the early works, the Short Time Fourier Transform based Griffin-Lim[8] algorithm was used to obtain time domain signals from mel-spectrogram features. Robotic sound problems have been observed in the sounds synthesized with the Griffin-Lim algorithm. Since 2016, as a result of modeling vocoders with deep neural networks, more natural outputs have started to be obtained. With WaveNet[9], one of the pioneering works, artificial sounds that are almost indistinguishable from human speech began to be synthesized. Despite the naturalness of the synthesized sounds, it took a lot of time to produce the sound signals with this model. Due to its autoregressive nature, it also uses the calculations made in the previous stages in the next steps. Autoregressive models are very successful in producing natural sounds, but due to their inherent lack of parallelization, training and inference take a longer time than non-autoregressive models. Autoregressive[9] and non-autoregressive[10-13] methods are available for the synthesizer and vocoder models. In current studies, non-autoregressive models can synthesize sounds with comparable quality with autoregressive models in real time on the CPU.

Training of G2P CMUDict does not include all words in the LJSpeech dataset. It is necessary to obtain the Turkish phonetic representations of out-of-vocabulary words. To predict unknown words, a finite-state-transducer-based, n-gram Grapheme-to-Phoneme (G2P) model was trained with CMUDict’. In the training phase, The Montreal Forced Aligner[16] tool was used. This tool implements the following steps during the training. Firstly, a unigram aligner is constructed with a finite state transducer using graphemes and phonemes, then to maximize probabilities, Viterbi training is used until the model convergence. After that, the best probability is computed with the Viterbi algorithm. Then, alignments are encoded with a finite-state acceptor. Therefore, each transaction matches grapheme and phoneme pairs. Then, using encoded alignments a higher-order n-gram model is constructed and then smoothed with Kneser-Ney[17] method[18]. In the decoding phase again

Methodology

In order to train speech synthesis models, sounds and phoneme sequences expressing sounds are needed. Since Turkish is a phonetic language, words are expressed as they are read. That is, character sequences are constructed words

Table 1. English to Turkish Phonetic Conversion Table[15] (The phonemes shown with lowercase letters represent the Turkish pronunciation of the phonemes on the leftside.) AA0

Viterbi algorithm is used to find the most probable path for Turkish phoneme sequence against English word input. In Figure 2, first and second lines show a sound signal and its transcription, respectively. Third line shows the G2P model’s output against transcription input. Training of Synthesizer FastSpeech2[11] architecture, which can work in real time, is used as a synthesizer. It is difficult to synthesize realistic sound with direct text and voice pairs for this non-autoregressive network. For this, extra information is needed to establish a relationship between text and voice in the created network. In particular, the duration information to be obtained at the phone level is of great importance. The length of the audio signal to be synthesized is determined by the duration information. In order to extract the phone level duration, the audio signal must be aligned with the text. The result of the alignment process is shown in Figure 2. To do this, the Montreal Forced Aligner[16] tool was used. In addition to the duration information, other features such as pitch and energy of the relevant audio segment are extracted in the preprocessing step. During the training, the weights of these features are also updated within the network. In the inference phase, only text input is used and finally, the mel-spectrogram is obtained.

Training of Vocoder Vocoders are trained with mel-spectrogram and audio signals in the time domain. The network tries to predict the audio signals against the mel-spectrogram input. As a vocoder, Multiband-MelGAN[13] architecture, which is an optimized version of MelGAN[12] was preferred. Both of these networks use Generative Adversarial Network (GAN)[19] architecture as backbone. The generative network tries to synthesize sound with the mel-spectrogram input, while the discriminator network tries to understand whether the result obtained from the generative network is real or fake (synthesized). Multiband-MelGAN architecture has basically three different advantages over MelGAN. First, the number of receptive fields in the MelGAN architecture was increased, multi-resolution STFT loss was used instead of feature matching loss in order to better distinguish between the synthesized voice and the target voice, and finally, the ability to operate in multi-band instead of single band was gained. This network attempts to estimate the mel-spectrogram input and the signals in the time domain for subbands in parallel, and combines the final results. Thus, the inference time has been accelerated compared to the classical MelGAN. A very fast synthesis process is achieved with a real-time factor of approximately 0.03 on the CPU[13]. The training of the vocoder model is completed by training the discriminator network until it cannot distinguish between the synthesized sound and the target sound. To briefly summarize the operations performed, first of all, the LJSpeech dataset was updated by bringing it into Turkish phonetic order using the G2P model. Then Synthesizer and vocoder networks are trained. Now, all the requirements for speech synthesis are fulfilled. Mel-spectrograms are synthesized with the text given to the input of the synthesizer, then the outputs from this network are given directly to the vocoder and finally audio signals in the time domain are generated.

Ethics

Finding data for some tasks can be quite difficult. It is sometimes impossible to find publicly available data to use, especially in speech technologies. One of the biggest reasons for this is the law on the protection of personal data. There are also task specific challenging situations. For example, the data required for speech synthesis studies are usually created by professional sound artists in the studio environment. This affects both cost and availability of the data. Parallel to this situation, there is no suitable dataset to train or evaluate the Turkish TTS model. In this context, the main purpose of this work is to investigate whether the problem arising from the lack of data can be solved with the proposed method. In this study, a Turkish TTS model was trained by converting the transcription of the English LJSpeech dataset into phonetically in Turkish. Although there are no “ı, ü, ş, ö, ç” characters in English, there are phones corresponding to these characters. The trained TTS model can generate phones corresponding to the “ı, ü, ş, ö, ç” characters. Because phonetic conversion produced some words that are included these characters. The only problem is that there is no phone that can correspond to “ğ”. As a result of the study, it has been proven that a Turkish speech synthesis model can be produced using the English dataset. Interestingly, the synthesized sounds resemble the speech of an English tourist trying to speak in Turkish. However, in practice, using this model directly may not be the right approach. Because, it is not easy to understand generated sentences due to its heavy accent. The resulting model can be fine-tune with a smaller Turkish dataset to break accent problem. In this way, more realistic results can be obtained. Similar to the method used in phonetic conversion, it is considered to repeat the study in future by using International Phonetic Alphabet (IPA). Thus, it is foreseen that the phones will be better expressed and the British accent will be suppressed a little more.

There are no ethical issues with the publication of this manuscript.

Share and Cite