【DL輪読会】Hierarchical Text-Conditional Image Generation with CLIP Latents
1 like53,414 views
2022/9/30 Deep Learning JP http://deeplearning.jp/seminar-2/
![DEEP LEARNING JP
[DL Papers]
http://deeplearning.jp/
Hierarchical Text-Conditional Image
Generation with CLIP Latents](https://image.slidesharecdn.com/dall-e2-221003050814-4b20a2bb/85/DL-Hierarchical-Text-Conditional-Image-Generation-with-CLIP-Latents-1-320.jpg)
![[DL輪読会]GLIDE: Guided Language to Image Diffusion for Generation and Editing](https://cdn.slidesharecdn.com/ss_thumbnails/glide2-220107030326-thumbnail.jpg?width=560&fit=bounds)
![SSII2021 [SS1] Transformer x Computer Visionの 実活用可能性と展望 〜 TransformerのCompute...](https://cdn.slidesharecdn.com/ss_thumbnails/ss1-01-210607043349-thumbnail.jpg?width=560&fit=bounds)
![SSII2021 [OS2-01] 転移学習の基礎：異なるタスクの知識を利用するための機械学習の方法](https://cdn.slidesharecdn.com/ss_thumbnails/os2-02final-210610091211-thumbnail.jpg?width=560&fit=bounds)
![[DL輪読会]Vision Transformer with Deformable Attention （Deformable Attention Tra...](https://cdn.slidesharecdn.com/ss_thumbnails/dl0114-220114032933-thumbnail.jpg?width=560&fit=bounds)
![[DL輪読会]相互情報量最大化による表現学習](https://cdn.slidesharecdn.com/ss_thumbnails/20190913iwasawa-190913002312-thumbnail.jpg?width=560&fit=bounds)
![[DL輪読会]Flow-based Deep Generative Models](https://cdn.slidesharecdn.com/ss_thumbnails/20190307-190328024744-thumbnail.jpg?width=560&fit=bounds)
![SSII2022 [SS1] ニューラル3D表現の最新動向〜 ニューラルネットでなんでも表せる？？ 〜](https://cdn.slidesharecdn.com/ss_thumbnails/ss1ssii2022hkatoneural3drepresentationhiroharukato-220607054619-fadc6480-thumbnail.jpg?width=560&fit=bounds)
![[DL輪読会]Learning Transferable Visual Models From Natural Language Supervision](https://cdn.slidesharecdn.com/ss_thumbnails/dlkobayashi0115-210115012308-thumbnail.jpg?width=560&fit=bounds)
![[DL輪読会]Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://cdn.slidesharecdn.com/ss_thumbnails/swintransformer-210514020542-thumbnail.jpg?width=560&fit=bounds)
Ad
More Related Content
What's hot (20)
Similar to 【DL輪読会】Hierarchical Text-Conditional Image Generation with CLIP Latents (15)
Ad
More from Deep Learning JP (20)
Ad
【DL輪読会】Hierarchical Text-Conditional Image Generation with CLIP Latents
- 1. DEEP LEARNING JP [DL Papers] http://deeplearning.jp/ Hierarchical Text-Conditional Image Generation with CLIP Latents
- 2. 書誌情報 タイトル : Hierarchical Text-Conditional Image Generation with CLIP Latents 著者 : Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, Mark Chen 組織 : OpenAI 会議 : arXiv (2022 April) URL : https://arxiv.org/pdf/2204.06125.pdf
- 4. モデルの構造 ➀ENCODER (CLIP MODEL) … 入力した画像とテキストをドット積の形でencode ➁PRIOR (DIFFUSION MODEL) … bipartite latent representation (zi; xT )を取得してdecoderに渡す ➂DECODER (DIFFUSION MODEL) … 生成画像を出力 ドット積𝑧𝑖・𝑧𝑡 𝑧𝑡 𝑧𝑖
- 5. ➀Encoder “CLIP Encoder” Learning Transferable Visual Models From Natural Language Supervision (2021) 画像とテキストのエンコードを個別で行い、そのドット積を出力
- 6. ➀Encoder “CLIP Encoder”の特徴 ➀学習に使用された画像データの量が大きい ②画像のエンコーディングで非常に優秀 ③一方、下流のV&Lタスクにおいてはイマイチな場合もあり、 他のモデルと組み合わせて使用することが推奨されている ・How Much Can CLIP Benefit Vision-and-Language Tasks? https://arxiv.org/pdf/2107.06383.pdf
- 7. ②Prior モデル➀ : Autregression Model (Transformer) Attention Is All You Need https://arxiv.org/pdf/1706.03762.pdf モデル② : Latent Diffusion Model An Image is Worth One Word_ Personalizing Text-to-Image Generation using Textual Inversion https://arxiv.org/pdf/2208.01618.pdf 通常の画像embeddingと、それを反転させたinverted embeddingを セットでDecoderに渡す どちらかを選択 画像分布の変化を高精度で追うことができる ・Diffusion Models Beat GANs on Image Synthesis https://arxiv.org/pdf/2105.05233.pdf ・GAN Inversion: A Survey https://arxiv.org/pdf/2101.05278.pdf
- 8. ③Decoder • モデル Diffusion Model ADMNets (Diffusion Models Beat GANs on Image Synthesis, https://arxiv.org/pdf/2105.05233.pdf) ・入力 (CLIP embedding, 反転済み CLIP embedding) ・出力 テキストに沿うように生成された画像
- 9. 実験➀ 画像操作 バリエーション 補間 A photo of a landscape in winter → a photo of a landscape in fall テキスト差分
- 12. 実験➃ DALL-E2とGLIDEとの比較 GLIDE vs DALL-E2 写実性 : 互角 妥当性: GLIDE 多様性 : DALL-E2 GLIDE DALL-E2 “A green vase filled with red roses sitting on top of table.” Human Evaluationの得票率 (vs GLIDE)
- 14. DALL-E2の弱点 DALL-E2 GLIDE “a red cube on top of a blue cube” 複雑な文の再現に対してはDALL-E2は弱い
- 16. 関連論文 • Hierarchical Text-Conditional Image Generation with CLIP Latents(DALL-E2) • Denoising Diffusion Probabilistic Models(採用したDiffusion Modelについて) • Learning Transferable Visual Models From Natural Language Supervision(CLIPモデルの提案) • How Much Can CLIP Benefit Vision-and-Language Tasks? (CLIPモデルをV&Lタスクに利用することに関する評価と提案) • GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models(本論文の比較対象となった生成モデル、Priorの損失関数はこれを参照) • An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion(Latent Diffusion Model) • LAFITE _ Towards Language-Free Training for Text-to-Image Generation(text-to-imageの他のモデル) • Zero-Shot Text-to-Image Generation(当初のDALL-Eモデル) • GAN Inversion: A Survey (inversionについて) • Diffusion Models Beat GANs on Image Synthesis (GANではなくDiffusion Modelを使うことを提案) • Classifier-Free Diffusion Guidance (diffusion guidanceの一つ)
- 17. GLIDEモデルの概要 Text Image Text Encoder : ADM Model Diffusion Models Beat GANs on Image Synthesis https://arxiv.org/pdf/2105.05233.pdf Image Encoder : Transformer Attention Is All You Need https://arxiv.org/pdf/1706.03762.pdf Noising : Diffusion Model Noised CLIP Denoising : Diffusion Model Decode Image … DALL-E2と共通 … DALL-E2と相違
- 18. CLIP Encoderに使われているモデル ・Image Encoder … Transformer AN IMAGE IS WORTH 16X16 WORDS: TRANSFORMERS FOR IMAGE RECOGNITION AT SCALE (2021) https://arxiv.org/pdf/2010.11929v2.pdf ・Text Encoder … Transformer Language Models are Unsupervised Multitask Learners https://d4mucfpksywv.cloudfront.net/better-language- models/language_models_are_unsupervised_multitask_learners.pdf