#Paris Meeting 2018 - Presentation of @chist_era_AMIS
1 like86 views
The AMIS project aims to make multilingual video content accessible to broader audiences. It involves partners from France, Poland, and Spain developing technologies for video summarization, speech recognition, machine translation and cross-lingual opinion analysis. The goal is to take input videos in Arabic or French and output English summaries that capture the main ideas, as well as analyze opinion divergence between videos in different languages. The project has collected 300 hours of video data and developed systems for video summarization, Arabic speech recognition, and Arabic-English machine translation. Evaluation of the systems is ongoing along with further development and subjective user testing.
#Paris Meeting 2018 - Presentation of @chist_era_AMIS
- 1. Presentation of AMIS K. Sma¨ıli AMIS Presentation of AMIS April 11, 2018 1 / 22
- 2. Presentation of AMIS AMIS: Access Multilingual Information opinionS Starting date: December 2015 Duration: 36 months The consortium is composed of partners from three countries: France: University of Lorraine (LORIA), University of Avignon (LIA) Poland: University of Science and Technology Krak´ow (AGH) Spain: University of DEUSTO (Bilbao) AMIS Presentation of AMIS April 11, 2018 2 / 22
- 3. Presentation of partners AGH University of Science and Technology, Krak`ow - Poland AGH is expert on video content summarization. AGH leads several WP: Definition of the requirements and data video collection Video summarization and video content analysis Automatic Evaluation of the different components Dissemination AMIS Presentation of AMIS April 11, 2018 3 / 22
- 4. Presentation of partners DEUSTO University Bilbao Spain DEUSTO has skills in designing protocols of evaluation for people with special needs. DEUSTO team is composed by psychologists, engineers and linguists. The team is in charge of: End-user Evaluation Collecting social network data Protocol of tests and evaluation AMIS Presentation of AMIS April 11, 2018 4 / 22
- 5. Presentation of partners University of Avignon - LIA France LIA of university of Avignon (UA) is expert on information retrieval, text and audio summarization. LIA is in charge of: Segmentation of speech transcription. Text and audio Summarization. AMIS Presentation of AMIS April 11, 2018 5 / 22
- 6. Presentation of partners University of Lorraine - LORIA France LORIA is expert on speech recognition and machine translation. LORIA is the coordinator of AMIS and is responsible of few WP: Speech Recognition. Machine Translation. Comparison of opinions. AMIS Presentation of AMIS April 11, 2018 6 / 22
- 7. The key challenge and potential impact With the growth of information on internet, a new issue arises: How to access to a maximum of information? High educated people do not speak more than two or three languages, while the majority speaks only one, which makes this huge amount of information inaccessible. How to make the main idea presented in a video in a foreign language accessible and easy to understand by everyone? Accessing to information in foreign languages would permit to access to the other side of a story. Due to political, socio-cultural or religion reasons, divergence of opinions may exist within two medias from two different sources. AMIS Presentation of AMIS April 11, 2018 7 / 22
- 8. Main objectives Objective 1: Understanding the main idea of a video by summarizing Input: A video in Arabic or French. Output: A summary of the input video subtitled in English. This summary is supposed to capture the main idea of the source video. Objetcive 2: Cross-lingual opinion Analysis Input: An Arabic video will be compared to French or English video. Output: A review concerning the degree of divergence between the two videos . AMIS Presentation of AMIS April 11, 2018 8 / 22
- 9. Objective 1: Understanding by summarizing To reach the first objective several components are necessary. We developed several scenarios. AMIS Presentation of AMIS April 11, 2018 9 / 22
- 10. Available data and components necessary for the first version 300 hours of videos have been collected (100 hours for each language). Video summarization has been developed. Speech recognition system for Arabic, French and English have been developed. A machine translation system Arabic-English and French-English have been developed. Text segmentation system has been developed. Text summarization is being integrated in the framework. A protocole for subjective evaluation has been proposed. AMIS Presentation of AMIS April 11, 2018 10 / 22
- 11. Video summarization Detection of the anchor-person. Recognition of day and night shots and extraction of low-level video quality indicators. AMIS Presentation of AMIS April 11, 2018 11 / 22
- 12. Arabic Automatic Speech Recognition System ALASR: Arabic Loria Automatic Speech Recognition system. We use Kaldi to develop ALASR. Acoustic Material: 63 hours (Train: 52h, Dev: 6h, Test: 5h). Language Model: GigaWord corpus +315K words from the transcription of the acoustic training data. A vocabulary of 95K words with an average of 5.07 pronunciations for each entry. AMIS Presentation of AMIS April 11, 2018 12 / 22
- 13. Machine Translation We developed a statistical Arabic - English Machine Translation A training corpus of 9.7 Million of parallel sentences extracted from Union Nation corpus. A 4-gram language model trained on English. A vocabulary of 224K words. AMIS Presentation of AMIS April 11, 2018 13 / 22
- 14. Sentence boundary detection The source from which a text summary is created is the transcription of ASR system; this transcript does not contain any punctuation mark. In order to solve this problem, a module of sentence boundary detection has been developed. The developed SBD system uses mainly textual features and convolutional neural networks (CNN) to segment the transcripts. AMIS Presentation of AMIS April 11, 2018 14 / 22
- 15. In the following table, each component of the system has been evaluated. Compo R P WER BLEU Test Corpus Video Sum 0.13 0.36 X X 50 Seq ALASR X X 14.02 X 31K Sent MT X X X 0.39 3K Sent BSD 0.63 0.78 X X 12M Sent These results have been achieved on data not extracted from our video database except for the video summarization. AMIS Presentation of AMIS April 11, 2018 15 / 22
- 16. Evaluation on AMIS data ASR: Since there is no reference we used the transcription of Youtube and Euonews as reference. On a corpus test of 1300 sentences we get a WER of 36.5. MT: There is no reference, we translated the output of the ASR with Google and we considered it as a reference. The value of BLEU on a test set of 197 videos is 26.7. AMIS Presentation of AMIS April 11, 2018 16 / 22
- 17. Conclusion Subjective evaluation is under progress. Deployment of other architectures. Comparison of opinions. AMIS Presentation of AMIS April 11, 2018 17 / 22
- 18. Publications 1 E.L. Pontes, J.M. Torres-Moreno, A.C. Linhares (2016) Automatic Text Summarization with a Reduced Vocabulary Using Continuous Space Vectors. In: M´etais E., Meziane F., Saraee M., Sugumaran V., Vadera S. (eds) Natural Language Processing and Information Systems. NLDB 2016. Lecture Notes in Computer Science, vol 9612. Springer, pp 440-446 2 C.E. Gonz´alez-Gallardo, J.M. Torres-Moreno, A. Rend´on, G. Sierra, Efficient Social Network Multilingual Classification using Character, POS n-grams and Dynamic Normalization. In Proceedings of the 8th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management (IC3K 2016) - Volume 1: KDIR, pp 307-314 3 C.E. Gonz´alez-Gallardo, J.M. Torres-Moreno, A. Rend´on, G. Sierra, Social Network Multilingual Author Profiling using character and POS n-grams, Linguam´atica, 8(1):21-29, 2016. AMIS Presentation of AMIS April 11, 2018 18 / 22
- 19. Publications 1 M. Rouvier, LIA at SemEval-2017 Task 4 : An Ensemble of Neural Networks for Sentiment Classification. 2 M. Leszczuk, J. Derkacz, M. Grega, A. Kozbial, and K. Sma¨ıli ”Definition of Requirements for Accessing Multilingual Information and Opinions”, 10th International Conference on Multimedia and Network Information Systems - MISSI 2016, published in Advances Intelligent Systems and Computing, Springer 3 M.A. Menacer, O. Mella, D. Fohr, D.Jouvet, D.Langlois and K. Smaili ”An enhanced automatic speech recognition system for Arabic”, EACL - The Third Arabic Natural Language Processing Workshop, 2017 AMIS Presentation of AMIS April 11, 2018 19 / 22
- 20. Publications 1 K. Abidi and K. Smaili ” How to match bilingual Tweets?”, Sixth International Conference on Natural Language Processing (NLP 2017), Volume Editor(s): David Wyld et al., 2017 2 J. Derkacz, Mikola Leszczuk, Michal Grega, Arian Kozbial, Fernando Jorge Hernandez, Amaia Mendez Zorrilla, Garcia Zapirain Begona, Kamel Smaili, Definition of requirements for accessing multilingual information opinions, Multimedia Tools and Applications, Springer Verlag, 2017 3 M. Leszczuk, M. Grega, A. Ko´zbial, J. Gliwski, K. Wasieczko, K. Sma¨ıli (2017) Video Summarization Framework for Newscasts and Reports – Work in Progress. In: Dziech A., Czy˙zewski A. (eds) Multimedia Communications, Services and Security. MCSS 2017. Communications in Computer and Information Science, vol 785. Springer, Cham AMIS Presentation of AMIS April 11, 2018 20 / 22
- 21. Publications 1 M.A. Menacer, D. Langlois, O. Mella, D. Fohr, D. Jouvet, K. Sma¨ıli, Is statistical machine translation approach dead?, ICNLSSP 2017 – International Conference on Natural Language, Signal and Speech Processing, Dec 2017, Casablanca, Morocco. pp.44-48, 2017 2 D. Jouvet, D. Langlois, M.A. Menacer, D. Fohr, O. Mella, K. Sma¨ıli, About vocabulary adaptation for automatic speech recognition of video data, ICNLSSP 2017 – International Conference on Natural Language, Signal and Speech Processing, Dec 2017, Casablanca, Morocco. PP. 21-25, 2017 3 M.A. Menacer, D. Langlois, O. Mella, D. Fohr, D. Jouvet, K. Sma¨ıli, Development of the Arabic Loria Automatic Speech Recognition system (ALASR) and its evaluation for Algerian dialect, ACLing 2017 – 3rd International Conference on Arabic Computational Linguistics, Nov 2017, Dubai, United Arab Emirates. pp.1-8, 2017 AMIS Presentation of AMIS April 11, 2018 21 / 22
- 22. Publications 1 E.L. Pontes, J.M. Torres-Moreno, S. Huet, A.C. Linhares, A New Annotated Portuguese/Spanish Corpus for the Multi-Sentence Compression Task, LREC2018, Miyazaki, Japan, 2018 2 C.E. Gonz´alez-Gallardo and J.M. Torres-Moreno, Sentence Boundary Detection for French with Subword-Level Information Vectors and Convolutional Neural Networks, International Conference on Natural Language, Signal and Speech Processing, Dec 2017, Casablanca. 3 C.E. Gonz´alez-Gallardo, E. SanJuan and J.M. Torres-Moren, International Journal of Computational Linguistics and Applications, accepted 2017, to be published AMIS Presentation of AMIS April 11, 2018 22 / 22