Neural Machine Translation (D2L10 Insight@DCU Machine Learning Workshop 2017)
1 like•1,186 views
The document details a workshop presentation on neural machine translation at Dublin City University, focusing on the encoder-decoder model and its components. It discusses the stages of transforming input into output through one-hot encoding, word embeddings, and recurrent structures in neural networks. A sequence-to-sequence learning approach is emphasized, including the use of input triggers and how predicted outputs inform subsequent inputs.
![12
Step 1: One-hot encoding
Example: letters. |V| = 30
‘a’: xT
= [1,0,0, ..., 0]
‘b’: xT
= [0,1,0, ..., 0]
‘c’: xT
= [0,0,1, ..., 0]
.
.
.
‘.’: xT
= [0,0,0, ..., 1]](https://image.slidesharecdn.com/dlmmdcud2l10neuralmachinetranslation-170429103826/85/Neural-Machine-Translation-D2L10-Insight-DCU-Machine-Learning-Workshop-2017-12-320.jpg)
![13
Step 1: One-hot encoding
Example: words.
cat: xT
= [1,0,0, ..., 0]
dog: xT
= [0,1,0, ..., 0]
.
.
house: xT
= [0,0,0, …,0,1,0,...,0]
.
.
.
Number of words, |V| ?
B2: 5K
C2: 18K
LVSR: 50-100K
Wikipedia (1.6B): 400K
Crawl data (42B): 2M](https://image.slidesharecdn.com/dlmmdcud2l10neuralmachinetranslation-170429103826/85/Neural-Machine-Translation-D2L10-Insight-DCU-Machine-Learning-Workshop-2017-13-320.jpg)
Neural Machine Translation (D2L10 Insight@DCU Machine Learning Workshop 2017)
- 1. DEEP LEARNING WORKSHOP Dublin City University 27-28 April 2017 Xavier Giro-i-Nieto [email protected] Associate Professor Universitat Politecnica de Catalunya Technical University of Catalonia Neural Machine Translation Day 2 Lecture 10 #InsightDL2017
- 5. 5 Predecents of Neural Machine Translation Neco, R.P. and Forcada, M.L., 1997, June. Asynchronous translations with recurrent neural nets. In Neural Networks, 1997., International Conference on (Vol. 4, pp. 2535-2540). IEEE.
- 6. 6 Encoder-Decoder Representation or Embedding Cho, Kyunghyun, Bart Van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. "On the properties of neural machine translation: Encoder-decoder approaches." SSST-8 (2014).
- 7. 7 Encoder-Decoder Front View Side View Representation of the sentence Kyunghyun Cho, “Introduction to Neural Machine Translation with GPUs” (2015) Cho, Kyunghyun, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. "Learning phrase representations using RNN encoder-decoder for statistical machine translation." arXiv preprint arXiv:1406.1078 (2014).
- 8. Encoder
- 9. 9 Encoder in three steps Kyunghyun Cho, “Introduction to Neural Machine Translation with GPUs” (2015) (2) (3) (1) One hot encoding (2) Continuous-space Word Representation (word embedding) (3) Sequence summarization
- 10. 10 Step 1: One-hot encoding Example: letters. |V| = 30 ‘a’: x = 1 ‘b’: x = 2 ‘c’: x = 3 . . . ‘.’: x = 30
- 11. 11 Step 1: One-hot encoding Example: letters. |V| = 30 ‘a’: x = 1 ‘b’: x = 2 ‘c’: x = 3 . . . ‘.’: x = 30 We impose fake range ordering
- 12. 12 Step 1: One-hot encoding Example: letters. |V| = 30 ‘a’: xT = [1,0,0, ..., 0] ‘b’: xT = [0,1,0, ..., 0] ‘c’: xT = [0,0,1, ..., 0] . . . ‘.’: xT = [0,0,0, ..., 1]
- 13. 13 Step 1: One-hot encoding Example: words. cat: xT = [1,0,0, ..., 0] dog: xT = [0,1,0, ..., 0] . . house: xT = [0,0,0, …,0,1,0,...,0] . . . Number of words, |V| ? B2: 5K C2: 18K LVSR: 50-100K Wikipedia (1.6B): 400K Crawl data (42B): 2M
- 14. 14 Step 1: One-hot encoding ● Large dimensionality ● Sparse representation (mostly zeros) ● Blind representation ○ Only operators: ‘!=’ and ‘==’
- 15. 15 Step 2: Projection to word embedding Kyunghyun Cho, “Introduction to Neural Machine Translation with GPUs” (2015) siM Wi E The one-hot is linearly projected to a embedded space of lower dimension with matrix E for learned weights (=fully connected). K K si = Ewi
- 16. 16 Word embeddings Figure: Christopher Olah, Visualizing Representations Mikolov, Tomas, Ilya Sutskever, Kai Chen, Greg S. Corrado, and Jeff Dean. "Distributed representations of words and phrases and their compositionality." NIPS 2013 Step 2: Projection to word embedding
- 17. 17 Step 2: Projection to word embedding GloVe (Stanford)
- 18. 18 Step 2: Projection to word embedding GloVe (Stanford)
- 19. 19 Step 2: Projection to word embedding Figure: TensorFlow tutorial Bengio, Yoshua, Réjean Ducharme, Pascal Vincent, and Christian Jauvin. "A neural probabilistic language model." Journal of machine learning research 3, no. Feb (2003): 1137-1155.
- 20. 20 Step 2: Projection to word embedding Mikolov, Tomas, Ilya Sutskever, Kai Chen, Greg S. Corrado, and Jeff Dean. "Distributed representations of words and phrases and their compositionality." NIPS 2013 the cat climbed a tree Given context: a, cat, the, tree Estimate prob. of climbed Word2Vec: Continuous Bag of Words
- 21. 21 Step 2: Projection to word embedding Mikolov, Tomas, Ilya Sutskever, Kai Chen, Greg S. Corrado, and Jeff Dean. "Distributed representations of words and phrases and their compositionality." NIPS 2013 the cat climbed a tree Given word: climbed Estimate prob. of context words: a, cat, the, tree (It selects randomly the context length, till max of 10 left + 10 right) Word2Vec: Skip-gram
- 22. 22 Step 2: Projection to word embedding ● Represent words using vectors of dimension d (~100 - 500) ● Meaningful (semantic, syntactic) distances ● Dominant research topic in last years in NLP conferences (EMNLP) ● Good embeddings are useful for many other tasks
- 23. 23 Pre-trained Word Embeddings for 90 languages trained using FastText, on Wikipedia. Step 2: Projection to word embedding
- 24. 24 Step 3: Recurrence Kyunghyun Cho, “Introduction to Neural Machine Translation with GPUs” (2015) hT
- 25. 25 Step 3: Recurrence Sequence Figure: Cristopher Olah, “Understanding LSTM Networks” (2015) Activation function could be LSTM, GRU, QRNN, pLSTM...
- 26. Decoder
- 27. 27 Decoder Kyunghyun Cho, “Introduction to Neural Machine Translation with GPUs” (2015) The Recurrent State (zi ) of the decoder is determined by: 1) summary vector hT 2) previous output word ui-1 3) previous state zi-1 hT
- 28. 28 Decoder Kyunghyun Cho, “Introduction to Neural Machine Translation with GPUs” (2015) With zi ready, we can compute a score e(k) for each word k in the vocabulary with a dot product with the Recurrent State (zi ): RNN internal state Neuron weights for word k (output layer) hT
- 29. 29 Decoder A score e(k) is higher if neuron weights for word k (wk ) and the decoder’s internal state zi are similar to each other. Reminder: a dot product computes the length of the projection of one vector onto another. Similar vectors (nearly parallel) the projection is longer than if they are very different (nearly perpendicular) RNN internal state Neuron weights for word k (output layer)
- 30. 30 Decoder Bridle, John S. "Training Stochastic Model Recognition Algorithms as Networks can Lead to Maximum Mutual Information Estimation of Parameters." NIPS 1989 ...we can finally normalize to word probabilities with a softmax. Probability that the output word at timestep i (wi ) is word k Previous words Hidden state Given the score for word k...
- 31. 31 Decoder Kyunghyun Cho, “Introduction to Neural Machine Translation with GPUs” (2015) Once an output word sample ui is predicted at timestep i, the process is iterated… (1) update the decoder’s internal state zi+1 (2) compute scores and probabilities pi+1 for all possible target words (3) predict the word sample ui+1 ...
- 32. 32 Decoder Kyunghyun Cho, “Introduction to Neural Machine Translation with GPUs” (2015) More words for the decoded sentence are generated until a <EOS> (End Of Sentence) “word” is predicted. EOS
- 33. 33 Encoder-Decoder Kyunghyun Cho, “Introduction to Neural Machine Translation with GPUs” (2015) Representation or Embedding
- 35. 35 Encoder-Decoder: Training Training requires a large dataset of pairs of sentences in the two languages to translate. Cho, Kyunghyun, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. "Learning phrase representations using RNN encoder-decoder for statistical machine translation." AMNLP 2014.
- 36. 36 Encoder-Decoder: Seq2Seq Sutskever, Ilya, Oriol Vinyals, and Quoc V. Le. "Sequence to sequence learning with neural networks." NIPS 2014. The Seq2Seq variation: ● trigger the output generation with an input <go> symbol. ● the predicted word at timestep t, becomes the input at t+1.
- 37. 37 Encoder-Decoder: Seq2Seq Sutskever, Ilya, Oriol Vinyals, and Quoc V. Le. "Sequence to sequence learning with neural networks." NIPS 2014.
- 38. 38 Thanks ! Q&A ? Follow me at https://imatge.upc.edu/web/people/xavier-giro @DocXavi /ProfessorXavi