Generative adversarial text to image synthesis
5 likes6,256 views
The document discusses the methodology and results of using generative adversarial networks (GANs) for text-to-image synthesis, detailing the network architecture, training losses, and qualitative results. It highlights experiments on sentence interpolation and style transfer, revealing how the model achieves effective representation of text through embeddings. Conclusions are drawn on the capability of GANs to generate images that align with textual descriptions.
![Generative Adversarial Text to Image
Synthesis
Scott Reed, Zeynep Akata, Xinchen
Yan, Lajanugen Logeswaran
[GitHub] [Arxiv]
Slides by Víctor Garcia [GDoc]
Computer Vision Reading Group (30/09/2016)](https://image.slidesharecdn.com/03-generativeadversarialtexttoimagesynthesis-161007150809/85/Generative-adversarial-text-to-image-synthesis-1-320.jpg)
![GANs
Discriminator
D(·)
MAX → E[log(D(X))]
True
World
Fake
Generator
q(x) x
G(z) zx’](https://image.slidesharecdn.com/03-generativeadversarialtexttoimagesynthesis-161007150809/85/Generative-adversarial-text-to-image-synthesis-7-320.jpg)
![GANs
Discriminator
D(·)
MAX → E[log(D(X))] + E[ log(1 - D(G(Z))) ]
True
World
Fake
Generator
q(x) x
G(z) zx’](https://image.slidesharecdn.com/03-generativeadversarialtexttoimagesynthesis-161007150809/85/Generative-adversarial-text-to-image-synthesis-8-320.jpg)
![GANs
Discriminator
D(·)
MAX → E[log(D(X))] + E[ log(1 - D(G(Z))) ]
True
World
Fake
Generator
q(x) x
G(z) zx’](https://image.slidesharecdn.com/03-generativeadversarialtexttoimagesynthesis-161007150809/85/Generative-adversarial-text-to-image-synthesis-9-320.jpg)
![GANs
Discriminator
D(·)
True
World
Fake
Generator
q(x) x
G(z) zx’
MIN → E[ log(1 - D(G(Z))) ]](https://image.slidesharecdn.com/03-generativeadversarialtexttoimagesynthesis-161007150809/85/Generative-adversarial-text-to-image-synthesis-10-320.jpg)
Generative adversarial text to image synthesis
- 1. Generative Adversarial Text to Image Synthesis Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran [GitHub] [Arxiv] Slides by Víctor Garcia [GDoc] Computer Vision Reading Group (30/09/2016)
- 2. Index ● Introduction ● State of the Art ● Method ○ Network Architecture ○ Losses ● Experiments ○ Qualitative Results ○ Sentence interpolation ○ Style Transfer ● Conclusions
- 4. Index ● Introduction ● State of the Art ● Method ○ Network Architecture ○ Losses ● Experiments ○ Qualitative Results ○ Sentence interpolation ○ Style Transfer ● Conclusions
- 8. GANs Discriminator D(·) MAX → E[log(D(X))] + E[ log(1 - D(G(Z))) ] True World Fake Generator q(x) x G(z) zx’
- 9. GANs Discriminator D(·) MAX → E[log(D(X))] + E[ log(1 - D(G(Z))) ] True World Fake Generator q(x) x G(z) zx’
- 10. GANs Discriminator D(·) True World Fake Generator q(x) x G(z) zx’ MIN → E[ log(1 - D(G(Z))) ]
- 11. GANs with Join Distributions How do we generate the image from text?
- 12. GANs with Join Distributions How do we generate the image from text? Discriminator 1/0 f(x,t) f(x’,t)
- 13. GANs with Join Distributions Discriminator 1/0 Real Image + Text Gen. Image + Text Generator + Text
- 14. GANs with Join Distributions Discriminator 1/0 Real Image + Text Gen. Image + Text Generator + Text
- 15. Text Embeddding In order to represent the text in a vector... MIN WHERE
- 16. Text Embeddding In order to represent the text in a vector... MIN WHERE This is the recurrent text encoder
- 17. Index ● Introduction ● State of the Art ● Method ○ Network Architecture ○ Losses ● Experiments ○ Qualitative Results ○ Sentence interpolation ○ Style Transfer ● Conclusions
- 19. Losses - CLS log(D(x,t)) log(1-D(G(z,t))) True Image + True Text Fake Image + True Text Real Images match the text content?
- 20. Losses - CLS log(D(x,t)) log(1-D(G(z,t))) log(1-D(G(zi,tk))) True Image + True Text Fake Image + True Text True Image (i) + True Text (j) Unmatched
- 21. Losses - INT They train interpolating between different text embedding vector (t1~t2). So the generator learns to fill GAPS on the data manifold.
- 22. Index ● Introduction ● State of the Art ● Method ○ Network Architecture ○ Losses ● Experiments ○ Qualitative Results ○ Sentence interpolation ○ Style Transfer ● Conclusions
- 23. Qualitative Results - Birds
- 25. Disentangling style and content Generator. z + Text If ‘text’ is describing the content? What is ‘z’ describing?
- 26. Disentangling style and content Generator. z + Text If ‘text’ is describing the content? What is ‘z’ describing? Style → Pose, Background…, let’s extract ‘z’
- 27. Disentangling style and content z0 z1 z2 z3 z4 z5
- 28. Qualitative Results - Flowers
- 29. Qualitative Results - MSCOCO