2023 Intelligent Computing and Control for Engineering and Business Systems (ICCEBS)

AI-Powered Real-Time Speech-to-Speech

Translation for Virtual Meetings Using Machine
Learning Models
2023 Intelligent Computing and Control for Engineering and Business Systems (ICCEBS) | 979-8-3503-9458-0/23/$31.00 ©2023 IEEE | DOI: 10.1109/ICCEBS58601.2023.10448600

Karunya S Jalakandeshwaran M Thanuja Babu

Department of Computer Science and Department of Computer Science and Department of Computer Science and
Engineering Engineering Engineering
Sri Sairam Engineering College Sri Sairam Engineering College Sri Sairam Engineering College
Chennai, India Chennai, India Chennai, India
[email protected] [email protected] [email protected]

Uma R
Department of Computer Science and
Engineering
Sri Sairam Engineering College
Chennai, India
[email protected]

Abstract—In our interconnected world, language diversity seamless dialogue, particularly in the realm of virtual
poses communication challenges, particularly in virtual meetings. The inability to effortlessly converse across
meetings. Our solution, a Real-Time Speech-to-Speech linguistic divides can significantly impede the
Translation system for Virtual Meetings, bridges these gaps. productivity and inclusivity of virtual meetings, hindering
It captures speech in one language, providing clear and
understandable translations in real-time during virtual
progress in both professional and personal contexts.
meetings. By seamlessly integrating Automatic Speech To confront this contemporary challenge head-on, we
Recognition (ASR), Machine Translation (MT), and embark on a journey to introduce a pioneering solution -
Text-to-Speech (TTS) components, this system transcends
the Real-Time Speech-to-Speech Translation system for
language barriers, enabling participants to engage
effortlessly and effectively in multilingual virtual
Virtual Meetings. This transformative project signifies a
interactions. It's more than text; it fosters spoken paradigm shift in communication technology, heralding a
interaction, revolutionizing cross-lingual communication in new era where linguistic disparities no longer impede the
virtual meetings. Applications abound, from enhancing free flow of ideas and collaboration within virtual meeting
global business negotiations to aiding virtual travelers and spaces. Our innovative system is meticulously designed to
connecting educators to broader international audiences of transcend the limitations of traditional text-based
diverse languages in virtual educational platforms. In an era translation by facilitating fluid and real-time spoken
where virtual communication is paramount, our project conversations during virtual meetings. It achieves this by
empowers meaningful connections, proving technology's
harnessing cutting-edge speech recognition and machine
remarkable ability to unite people and transcend language
barriers in virtual settings worldwide.
translation technologies, and evaluation metrics that are
precisely tailored for the virtual communication
Keywords – Language Barriers, Speech Recognition, landscape. Through this venture, we envision a world
Translation Technology, Machine Learning models, ASR, MT, where the boundaries of language no longer constrain
TTS, Cross-Lingual Communication, Virtual Meetings. virtual meetings, fostering a global community of
collaboration and understanding.
In the pages that follow, we will delve into the
I. INTRODUCTION intricate details of our Real-Time Speech-to-Speech
In an era of unprecedented digital connectivity and Translation system for Virtual Meetings, exploring its
global interactions, the significance of effective development, applications, and the profound impact it
communication transcends geographical boundaries. promises to have on the way we communicate within the
However, within this vast tapestry of interconnectedness, dynamic and interconnected sphere of virtual meetings.
language diversity often poses intricate barriers to

Authorized licensed use limited to: K K Wagh Inst of Engg Education and Research. Downloaded on August 20,2024 at 07:33:58 UTC from IEEE Xplore. Restrictions apply.
979-8-3503-9458-0/23/$31.00 ©2023 IEEE
 II. EXISTING SYSTEM meetings. It will capture spoken input from users in one
language during virtual meetings.
Current speech-to-speech translation systems primarily
rely on machine translation services and mobile D. Precise Machine Translation for Virtual Meetings
applications. These systems enable users to speak in one
language and receive real-time translations as shown in We will integrate advanced machine translation
Fig. 1. However, they often face challenges in terms of algorithms that accurately convert the spoken input into
accuracy, context-awareness, and seamless conversation the desired target language during virtual meetings. The
flow, especially in virtual meetings. They may also require system's strength lies in its precision and fluency, crucial
a constant internet connection for cloud-based translation for effective virtual communication.
services during virtual meetings.

E. Natural Speech Output in Virtual Meetings

Unlike traditional translation systems, ours will excel

in delivering translated content as clear and natural speech
during virtual meetings. This feature enables users to
engage in real-time conversations with ease during virtual
meetings.

F. Cross-platform Accessibility for Virtual Meetings

Fig. 1. Flowchart of Existing System
The system will be accessible across various virtual
Our project seeks to build upon these existing systems meeting platforms, ensuring convenience and widespread
by offering a more advanced and precise solution tailored usability.
explicitly for virtual meetings. We aim to enhance the
accuracy and context-awareness of translations, enabling G. Offline Mode for Virtual Meetings
seamless cross-lingual conversations even in virtual
settings with limited internet connectivity. Recognizing the importance of accessibility, our
system will include an offline mode for virtual meetings.
Users can continue utilizing its translation capabilities
III. PROPOSED SYSTEM even without a stable internet connection, making it
Our proposed Real-Time Speech-to-Speech valuable in remote or low-connectivity virtual meeting
Translation system for Virtual Meetings will build upon environments.
the latest advancements in speech recognition and
H. Customization Options for Virtual Meetings
machine translation technologies, specifically designed
for virtual communication. Here's an overview of its key Users will have the flexibility to customize the system
components: to specific domains or industries, tailoring it to their
unique virtual meeting needs.
A. User-friendly Interface for Virtual Meetings
I. Applications in Virtual Meetings
The system will feature a user-friendly interface
accessible to individuals participating in virtual meetings The proposed system's applications are diverse,
with varying levels of technological expertise. including facilitating international business
communication in virtual meetings, aiding virtual tourists,
B. Secure Authentication for Virtual Meetings enhancing virtual educational outreach to global
audiences, and fostering cross-lingual cultural exchange in
To ensure authorized access during virtual meetings, virtual settings.
the system will employ a robust authentication system,
requiring users to have unique login credentials for virtual Our proposed Real-Time Speech-to-Speech
sessions. Translation system for Virtual Meetings is poised to
revolutionize cross-lingual communication in the context
C. Real-time Speech Recognition in Virtual Meetings of virtual meetings. It represents a significant step toward
a world where language barriers no longer impede
The core of our system will encompass state-of-the-art meaningful interactions in virtual communication
speech recognition capabilities tailored for virtual environments.

Authorized licensed use limited to: K K Wagh Inst of Engg Education and Research. Downloaded on August 20,2024 at 07:33:58 UTC from IEEE Xplore. Restrictions apply.
 IV. LITERATURE SURVEY issues related to transparency and model understanding.
All referenced papers primarily focus on Moreover, our approach empowers NMT systems to
speech-to-speech translation across various languages. effectively handle lexical and structural constraints,
Hence, in this study, we explored the convergence of expanding their applicability to a wider range of
speech-to-speech translation and virtual meeting translation tasks. Through our project, we contribute to
platforms, enabling seamless multilingual communication. advancing NMT technology, making it more versatile and
interpretable, ultimately enhancing the quality of
[1] Prior studies have established the efficacy of word translations across various language pairs and domains.
embeddings in enhancing ASR and ST models, providing
valuable contextual and semantic information from textual [5] The paper explores training multilingual and
data. Our research extends this by integrating advanced multi-speaker text-to-speech (TTS) systems based on
deep learning techniques like transformer-based language families for Indian languages, addressing the
architectures (e.g., BERT and GPT) to further optimize challenges of linguistic diversity and data scarcity.
ASR and ST models. This approach aims to bridge the However, it primarily focuses on training TTS systems
gap between spoken and textual languages, resulting in and adaptation within language families. In our project,
reduced word error rates in ASR, improved translation we aim to extend this approach to real-time
metrics in ST, and ultimately more accurate and efficient speech-to-speech translation in virtual meetings, utilizing
spoken-to-textual language conversion. language family-based TTS models for natural and
contextually relevant speech synthesis. Additionally, we
[2] End-to-end speech translation remains a challenge will incorporate real-time translation capabilities to bridge
for syntactically distant language pairs due to language barriers in virtual meetings, creating a
long-distance reordering complexities. This study comprehensive communication solution. This holistic
pioneers an attention-based encoder-decoder model for approach distinguishes our project from existing research
English-Japanese language pairs with differing word and enhances the practicality of virtual meetings for
orders (SVO vs. SOV). To address the lack of parallel diverse language users.
speech-text data, text-to-speech synthesis (TTS) is
employed for data augmentation. The proposed model
incorporates transcoding and curriculum learning (CL) V. IMPLEMENTATION
strategies to guide the model, starting with ASR or MT
tasks and gradually transitioning to end-to-end speech Our Real-Time Speech-to-Speech Translation
translation. Results indicate significant performance system for Virtual Meetings aims to revolutionize
improvements compared to conventional cascade models, cross-lingual communication during virtual meetings. It
particularly for distant language pairs. integrates advanced technologies to capture, translate, and
produce speech, ensuring a seamless flow of
[3] Unsupervised Neural Machine Translation conversations across language barriers.
(UNMT) has achieved remarkable results, particularly for
language pairs like French-English and German-English, The system's initiation revolves around a
through methods like unsupervised bilingual word user-friendly interface developed with HTML5, CSS3,
embedding (UBWE) and cross-lingual masked language and JavaScript. It's responsive, ensuring compatibility
model (CMLM) pre-training. This paper empirically with various devices and screen sizes. User interface (UI)
explores the relationships between UNMT and and user experience (UX) principles are followed for
UBWE/CMLM, revealing that the quality of UBWE and intuitive navigation and accessibility features for a
CMLM significantly influences UNMT performance. To seamless user experience.Prioritizing the utmost privacy
address this, the paper introduces a novel UNMT structure and security, the authentication process verifies users'
with cross-lingual language representation agreement, identities, ensuring that only authorized individuals can
offering two approaches: UBWE agreement and CMLM participate in virtual meetings.
agreement. These methods, including regularization and
adversarial training, ensure the preservation of UBWE We harnessed the GigaSpeech dataset as a
and CMLM quality during UNMT training. Experimental foundational resource for training our system. Given the
results across several language pairs demonstrate dataset's extensive audio recordings, we initiated a data
substantial improvements over conventional UNMT. preprocessing pipeline to optimize it for further training
[4] Inspired by the limitations of existing neural our models. This pipeline involved segmenting the
machine translation (NMT) models in capturing alignment lengthy audio recordings into shorter, coherent fragments,
between input and output, our project introduces a typically spanning a few seconds to a minute each. This
valuable add-on to NMT technology. We propose an segmentation was vital to ensure that our training models
innovative approach that incorporates explicit phrase could efficiently handle real-time processing, a necessity
alignment into NMT models. This enhancement for seamless virtual meetings. In addition to segmentation,
significantly improves NMT's interpretability, addressing we addressed transcription peculiarities within the

Authorized licensed use limited to: K K Wagh Inst of Engg Education and Research. Downloaded on August 20,2024 at 07:33:58 UTC from IEEE Xplore. Restrictions apply.
 GigaSpeech dataset in order to enhance the reliability of model's parameters to minimize translation errors and
our models. This included the removal of non-speech maximize fluency.
elements such as laughter, disfluencies, and background In order to generate final vocal output in target
noise annotations. By eliminating these extraneous language, we integrated a Text-to-Speech (TTS)[5] model
elements, we crafted transcriptions that portrayed clean into our system. The TTS model is responsible for
and coherent speech. Furthermore, we applied rigorous converting the translated text generated by the MT model
text standardization techniques, effectively managing into natural and coherent speech in the target language.
variations in punctuation, capitalization, and formatting. We employ deep neural networks and generative
This standardization fostered consistency across the adversarial networks, to train our TTS model. This
dataset, facilitating robust model training. training process involves using the translated text from the
MT model as input and generating corresponding speech
waveforms as output. Fine-tuning and optimization are
performed to ensure that the synthesized speech is clear,
natural, and maintains appropriate intonation.

Fig. 2. Components of our translation system

Following the data processing phase, our system

proceeds to the model training stage. We used Automatic
Speech Recognition (ASR)[1], Machine translation
(MT)[2], and Text-to-Speech (TTS)[2] models as seen in
Fig. 2. In this step, we utilize the refined and segmented
GigaSpeech dataset to train our ASR, MT and TTS
models as seen in Fig. 3. The Automatic Speech
Recognition (ASR) serves as the initial component of our
system. It is trained to utilize speech recognition
technology models to convert spoken language into
textual transcripts. ASR takes the spoken input and
transcribes it into text in the source language. These
transcriptions serve as the foundation for the subsequent Fig. 3. Mechanism of Speech-to-Speech Translation
translation process. This model is designed to be highly
accurate, capturing not only the words spoken but also To optimize the performance of ASR, MT, and
nuances, accents, and variations in speech. We employ TTS models, extensive training iterations are conducted,
recurrent neural networks (RNNs)[1] and fine-tune them fine-tuning model parameters and adjusting
using the segmented audio data. hyperparameters. We employ evaluation metrics, such as
Word Error Rate (WER)[1] for ASR and Bilingual
Once ASR transcribes the spoken content into Evaluation Study (BLEU)[6] score for MT, to assess and
text, the Machine translation (MT) model is used to enhance the model's accuracy and fluency. For TTS, we
translate the source input to text in the desired output make use of Mean Opinion Score (MOS)[7] and
language. This model receives the transcribed texts Naturalness in order to assess the naturalness to ensure the
generated by the Automatic Speech Recognition (ASR) quality of the synthesized speech. This iterative training
component, which converts spoken language into text, approach ensures that our system becomes proficient in
with a strong emphasis on precision and fluency. This MT accurately transcribing and translating spoken language,
model goes beyond mere word-by-word translation, thus achieving high-quality, human-like speech synthesis.
considering contextual nuances, idiomatic expressions,
and language flow for precise and fluent translations. It We develop RESTful APIs using the Flask
understands the relationships between words, ensuring framework in Python to enable communication between
that the translated content retains clarity and sounds our translation system and the virtual meeting platform.
natural to native speakers. This nuanced approach is We design API endpoints to accept audio data from the
crucial for effective cross-lingual communication, virtual meeting's microphone feed, translate spoken
especially in the dynamic context of virtual meetings. To language during the virtual meeting, and then the API
train our MT model, we rely on parallel data, consisting of endpoints send back the translated audio to the virtual
source language transcripts and their corresponding meeting platform in real time. The data exchanged
translations in the targeted output language. This data between the virtual meeting platform and our system is
forms the basis for the model to learn the intricate patterns typically in a structured JSON format, allowing easy
and language structures necessary for accurate translation. parsing and interpretation. To ensure that translated
The training process involves iteratively optimizing the speech aligns perfectly with the ongoing conversation, we

Authorized licensed use limited to: K K Wagh Inst of Engg Education and Research. Downloaded on August 20,2024 at 07:33:58 UTC from IEEE Xplore. Restrictions apply.
 synchronize the audio streams. This involves of speech speed, pitch modulation, voice type, and the
timestamping audio data, so the translated content is ability to activate or deactivate translation features. Such
delivered at the right moment, maintaining a natural flexibility empowers users to adapt their virtual meeting
conversation flow. Our system is designed to support experience according to their individual needs and desires.
multiple languages. Through language detection
algorithms, it identifies the source language of the speaker In addition, our system generates textual closed
and translates it into the chosen target language. We use captions within the virtual meeting interface. This feature
language codes and recognition models to achieve is particularly advantageous for participants who prefer
accurate language detection. We integrate TTS engines reading translations or those with hearing impairments,
that generate natural-sounding speech in the target ensuring inclusivity and accessibility. The translated
language. Parameters such as pitch, speed, and voice type content seamlessly aligns with the ongoing discourse,
can be customized. reducing interruptions and preserving the natural flow of
conversation.This synchronization ensures the fluidity of
The virtual meeting platform's user interface is conversations.
extended to include our translation features. Participants
can select target languages, enable or disable translation, Offline mode feature enables participants to continue
adjust volume levels, and view closed captions of benefiting from translation capabilities in environments
translated content. To maintain a low-latency experience, with limited or no internet connectivity, relying on
we optimize our system for real-time processing. This pre-loaded translation models and TTS voices.
includes efficient data transmission, minimal processing
delays, and robust error handling. We implement an Ultimately, our system's outcomes culminate in
offline mode to handle situations with limited internet enriched virtual meeting interactions and collaborations.
connectivity. Participants can still use our system with Language diversity ceases to be a barrier, empowering
pre-loaded translation models and TTS voices. participants to engage confidently and proficiently,
transcending linguistic boundaries.
Through this implementation, our system ensures
that language diversity no longer hinders effective virtual
communication. By bridging linguistic gaps, it empowers VII. CONCLUSION
users to engage confidently in meaningful interactions and In the realm of virtual meetings, our Real-Time
collaborations within the virtual meeting space. Speech-to-Speech Translation system marks a
groundbreaking stride in breaking down language barriers.
It empowers seamless multilingual conversations, uniting
VI. RESULTS participants worldwide. This technology transcends
Upon seamless integration into virtual meetings, our borders, enabling effective global communication for
real-time speech-to-speech translation system yields a business, education, and cultural exchange. Our project
range of highly valuable outputs that significantly enhance underscores the pivotal role of technology in fostering
the virtual meeting experience. connections and meaningful interactions. In a world
where virtual meetings dominate, it ensures that language
Our system facilitates secure participation within the never hinders the exchange of ideas and collaboration
virtual realm, offering real-time translation of spoken among diverse participants.
content. As the participants converse in their native
languages, the system diligently transcribes, translates,
and articulates their words in the chosen target languages. VIII. FUTURE WORKS
This effortless cross-lingual communication ensures that Future work in Real-Time Speech-to-Speech
language differences do not hinder the effectiveness of Translation for Virtual Meetings includes enhancing
conference meetings. translation accuracy, expanding language support, and
optimizing resource usage. Efforts should target
Participants are ensured that they can engage in fluid adaptability to diverse accents, incorporating more
and comprehensible conversations by maintaining a high languages, and supporting real-time voice recognition for
degree of naturalness and fluency in the translated multiple speakers. Integrating sentiment analysis and
content. Unlike conventional text-based translations, our real-time subtitles for accessibility are promising
system generates spoken translations that sound clear and additions. Collaboration with virtual meeting platforms
coherent, mimicking human speech patterns. can make this technology widely accessible, transforming
virtual meetings into inclusive global forums, promising
Users are granted a spectrum of customization an even more seamless and inclusive virtual
options to tailor the translation process to their communication experience.
preferences. These include language selection, adjustment

Authorized licensed use limited to: K K Wagh Inst of Engg Education and Research. Downloaded on August 20,2024 at 07:33:58 UTC from IEEE Xplore. Restrictions apply.
 REFERENCES
[1] S.P.Chuang, A.H.Liu, T.W.Sung, and H.y.Lee, "Improving
Automatic Speech Recognition and Speech Translation via Word
Embedding Prediction," in IEEE/ACM Transactions on Audio,
Speech, and Language Processing, vol. 29, 2021.
[2] T.Kano, S.Sakti, and S.Nakamura,"End-to-End Speech Translation
With Transcoding by Multi-Task Learning for Distant Language
Pairs," in IEEE/ACM Transactions on Audio, Speech, and
Language Processing, vol. 28, 2020.
[3] H.Sun, R.Wang, K.Chen, M.Utiyama, E.Sumita, and
T.Zhao,"Unsupervised Neural Machine Translation With
Cross-Lingual Language Representation Agreement," in
IEEE/ACM Transactions on Audio, Speech, and Language
Processing, vol. 28, 2020.
[4] J.Zhang, H.Luan, M.Sun,F.Zhai, J.Xu, and Y.Liu,"Neural Machine
Translation With Explicit Phrase Alignment," in IEEE/ACM
Transactions on Audio, Speech, and Language Processing, vol. 29,
2021, pp. 1001-1011.
[5] A.Prakash and H.A.Murthy,"Exploring the Role of Language
Families for Building Indic Speech Synthesisers," in IEEE/ACM
Transactions on Audio, Speech, and Language Processing, vol. 31,
pp. 734-747, 19 December 2022.
[6] Hajar Chatoui and Oğuz Ata, “ Automated Evaluation of the
Virtual Assistant in Bleu and Rouge Scores” , in 2021 3rd
International Congress on Human-Computer Interaction,
Optimization and Robotic Applications (HORA).
[7] Babak Naderi and Sebastian Möller , “ Transformation of Mean
Opinion Scores to Avoid Misleading of Ranked Based Statistical
Techniques” in 2020 Twelfth International Conference on Quality
of Multimedia Experience (QoMEX).

Authorized licensed use limited to: K K Wagh Inst of Engg Education and Research. Downloaded on August 20,2024 at 07:33:58 UTC from IEEE Xplore. Restrictions apply.