Generative

Artificial
Intelligence

Lecture-01
Dr. Rajiv Misra, Professor
Dept. of Computer Science & Engg.
Indian Institute of Technology,
Patna
[email protected]
 Introduction

Challenge: understand complex, unstructured inputs

Computational Speech
Computer Vision

Natural Language Processing Robotics

2
 Introduction

Richard Feynman: “What I cannot create, I do not understand”

Generative modeling: “What I understand, I can create”

3
 Generative Modeling: Computer Graphics
How to generate natural images with a computer?

High level Cube(color=blue, position=(x,y,z), size=…)

description Cylinder(color=red, position=(x’,y’,z’), size=..)

Generation (graphics) Inference (vision as

inverse graphics)

Raw sensory
outputs

Many of our models will have similar structure (generation + inference) 4

 Statistical Generative Models

Statistical generative models are learned from data

…
+
Data Prior Knowledge
(e.g., images of bedrooms) (e.g., physics, materials, ..)

Priors are always necessary, but there is a spectrum

Prior
Data Knowledge
This course Graphics 5
 Statistical Generative Models

A statistical generative model is a probability distribution p(x)

• Data: samples (e.g., images of bedrooms)
• Prior knowledge: parametric form (e.g., Gaussian?), loss function (e.g.,
maximum likelihood?), optimization algorithm, etc.

A probability
Image x distribution scalar probability p(x)
p(x)
It is generative because sampling from p(x) generates new images

6
Building a simulator for the data generating
process

Data simulator Data

Building a simulator for the data generating
process

Control signals Data simulator New datapoints

 Building a simulator for the data generating
process

Control signals New datapoints

Data simulator
Potential datapoints Probability values
 Building a simulator for the data generating
process

Control signals New datapoints

Data simulator
=Statistical model
Potential datapoints Probability values
 Building a simulator for the data generating
process

Control signals New datapoints

Data simulator
=Statistical model
=Generative model
Potential datapoints Probability values
 Data generation in the real world
Generate

Generative model
Stroke paintings to realistic images
of realistic images [Meng, He, Song, et al., ICLR 2022]

Generate
“Ace of Pentacles”

Generative model
of paintings Language-guided artwork creation
https://chainbreakers.kath.io @RiversHaveWings
Solving inverse problems with generative
models

Generate

Generative model
of medical images

Medical image reconstruction

[Song et al., ICLR 2022]
Outlier detection with generative models

High Low
probability probability

Generative model
of traffic signs

Outlier detection
[Song et al., ICLR 2018]
Progress in Generative Models of Images
GANs

Ian Goodfellow, 2019

15
 Progress in Generative Models of Images –
Diffusion Models

Song et al., Score-Based Generative Modeling through Stochastic Differential Equations, 2021 16
 Text2Image Diffusion Models

User input:

An astronaut riding a
horse

17
 Text2Image Diffusion Models

User input:

A perfect Italian meal

18
 Text2Image Diffusion Models

User input:

19
 Dalle3
A minimap diorama of a cafe adorned with indoor plants.
Wooden beams crisscross above, and a cold brew station stands
out with tiny bottles and glasses
 Progress in Inverse Problems
P(full image| mask)
P(high resolution | low resolution)

Menon et al, 2020 Liu al, 2018

P(color image| grayscale)

Antic, 2020
 Progress in Inverse Problems

User input:
Progress in Inverse Problems
Progress in Inverse Problems

Kawar et al., 2023

 Medical image reconstruction
Cross-sectional image

Sparse-view
computed
tomography
Sparse-view sinogram (CT)

Forward model is given by physical simulation

 WaveNet
Text to Speech
Generative model of speech signals
Parametric

Concatenative

WaveNet

Unconditional

Music

van den Oord et al, 2016c

26
 Diffusion Text2Speech

Generative model of speech signals

Betker, Better speech synthesis through scaling 2023

27
 Audio Super Resolution

Conditional generative model P(high-res signal | low-res audio signal)

Low res signal

High res audio signal

Kuleshov et al., 2017

28
 Language Generation

Radford et al., 2019

Demo from talktotransformer.com
P(next word | previous words)
 Machine Translation

Conditional generative model P( English text| Chinese text)

Figure from Google AI research

blog.
30
Code Generation

OpenAI Codex
 Video Generation

Suddenly, the walls of the embankment broke and there

was a huge flood
 Video Generation

a couple sledding down a snowy hill on a tire roman

chariot style
Video Generation
 Imitation Learning

Conditional generative model P(actions | past observations)

Janner et al., 2022 35

Li et al., 2017
Molecule generation
 Roadmap and Key Challenges
• Representation: how do we model the joint distribution of many random
variables?
• Need compact representation
• Learning: what is the right way to compare probability distributions?

• Inference: how do we invert the generation process (e.g., vision as inverse

graphics)?
• Unsupervised learning: recover high-level descriptions (features) from raw data

37
 Models to Study
• Fully observed likelihood-based models
• Autoregressive
• Flow-based models
• Latent variable models
• Variational learning
• Inference amortization
• Variational autoencoder
• Implicit generative models
• Two sample tests, embeddings, F-divergences
• Generative Adversarial Networks
• Energy Based Models
• Score-based Diffusion Generative Models
• Learn about algorithms, theory & applications
 Prerequisites

• Basic knowledge about machine learning from at least one of

CS 221, 228, 229, or 230.
• Basic knowledge of probabilities and calculus:
• Gradients, gradient-descent optimization, backpropagation
• Random variables, independence, conditional independence
• Bayes rule, chain rule, change of variables formulas
• Proficiency in some programming language, preferably Python,
required.
Thank You

Dr. Rajiv Misra, Professor

Dept. of Computer Science & Engg.
Indian Institute of Technology,
Patna
[email protected]