Denoising Diffusion Models for

Plug-and-Play Image Restoration

Yuanzhi Zhu

Supervisors: Dr. Kai Zhang, Jingyun Liang, Jiezhang Cao

Principal Investigator: Prof. Luc Van Gool

18/June/2023
Content

• Preliminaries

• Methods

• Results
 DDPM: Denoising Diffusion Probabilistic Models*
True data dist. :
Markov assumption
Forward process:

Reverse process:

*[2006.11239] Denoising Diffusion Probabilistic Models (arxiv.org) 3

DDPM: Denoising Diffusion Probabilistic Models
Forward Diffusion Process

Each Step

or
norm invariant
noise schedule controls the diffusion process

For arbitrary t

4
DDPM: Denoising Diffusion Probabilistic Models
Reverse Diffusion Process if is small enough, will also be Gaussian

match
Reverse when condition on
all three are forward processes

model with NN ☺

5
 DDPM: Denoising Diffusion Probabilistic Models
Negative Log Likelihood to Variational Lower Bound
Known

Explicit parameterization

two options†

Model The Noise (Residual)

Denoiser

† Covariance has analytical optimal form (Estimating the Optimal Covariance with Imperfect Mean in Diffusion Probabilistic Models) 6
DDPM: Denoising Diffusion Probabilistic Models

7
DDPM: Denoising Diffusion Probabilistic Models

8
 DDIM: Denoising Diffusion Implicit Models*
Reverse Process: deterministic given , with

Sampling:

Accelerated Generation Processes

[2010.02502] Denoising Diffusion Implicit Models (arxiv.org) 9

Content

• Preliminaries

• Methods

• Results
Plug-and-Play Image Restoration

Measurement (degradation model):

Reconstruction (Bayes' theorem):

Maximum A Posteriori (MAP) estimation:

11
Plug-and-Play Image Restoration
Substitute degradation model :

data term prior term

Introduce auxiliary variable z:

Lagrange multiplier:

Prior

Half Quadratic Splitting (HQS) algorithm:

Data

12
Plug-and-Play Image Restoration
by definition

degradation models

Inpainting

Deblurring

SR

Approximately
13
 Plug-and-Play Image Restoration
Previous Iterative Approaches:

• Empirically chosen schedules 

• Discriminative denoisers 

Introduce Diffusion Models: Sampling as Optimization

• Well-defined sampling schedules/trajectories ☺

• Generative prior ☺ But where does the generative power come from?

Image from: cszn/USRNet: Deep Unfolding Network for Image Super-Resolution (CVPR, 2020) (PyTorch) (github.com) 14
 Denoising Diffusion Models for Plug-and-Play Image Restoration

One iteration HQS → estimate

Calculate the predicted conditional noise

Finish one sampling step by adding noise back

In this page we use instead to avoid confusion 15

Denoising Diffusion Models for Plug-and-Play Image Restoration

HQS algorithm

unconditional

DiffPIR

Approximately

16
Denoising Diffusion Models for Plug-and-Play Image Restoration

But where does the generative power come from?

HQS algorithm
for denoising problem is not a “noisy image”

DiffPIR

fix

17
Denoising Diffusion Models for Plug-and-Play Image Restoration

Ours
DDIM

denoiser
… …

18
Denoising Diffusion Models for Plug-and-Play Image Restoration
t=1000 t=600 t=300 t=100

Can skip this part! 19

Ablation Study: Sampling Steps & Start Timestep

Effect of sampling steps Effect of start sampling timestep

20
Ablation Study: Effect of Hyperparameters
λ 0.1 1.0 10 100 1000 10000
ζ
0.0

• λ<1 → the noise is amplified

0.2

• λ>1000 → more unconditional

0.5

• ζ~1 → more blurry

0.8

1.0
21
Content

• Preliminaries

• Methods

• Results
Quantitative Results

23
Quantitative Results

24
Qualitative Results: Noisy 4x SR

25
 Qualitative Results: Noisy Motion Deblurring

*All methods used the same diffusion model as denoiser 26

Diverse Reconstruction: Inpainting

27
Diverse Reconstruction: Super Resolution

28
Thank You!
Additional Slides on Diffusion Models for IR
Sampling from the Posterior
 SDE-based Generative Models: A Unified Framework*

*[2011.13456] Score-Based Generative Modeling through Stochastic Differential Equations (arxiv.org) 31

SDE-based Generative Models: A Unified Framework
known Gaussian when if affine
Training Objective (DSM)

Discretizations

SDE Form Discrete Markov Chain SDE Expression

Variance Exploding (VE) SDE

(NCSN)

Variance Preserving (VP) SDE

(DDPM)

32
SDE-based Generative Models: A Unified Framework
Model: DDPM and SDE point of views

Score in score-based model is affine transformation of predicted noise in DDPM

Equivalent one step forward

Denoising score matching

Gaussian assumption

33
SDE-based Generative Models: A Unified Framework
Controllable Generation

time-dependent classifier
Bayesian (guidance term)

unconditional model

34
 ILVR: Conditioning Method for DDPM*

*[2108.02938] ILVR: Conditioning Method for Denoising Diffusion Probabilistic Models (arxiv.org) 35
 RePaint: Inpainting using Denoising Diffusion Probabilistic Models*

Same idea but different downstream tasks from ILVR

unconditional

*[2201.09865] RePaint: Inpainting using Denoising Diffusion Probabilistic Models (arxiv.org) 36

 Diffusion Posterior Sampling for General Noisy Inverse Problems*

General forward model

*[2209.14687] Diffusion Posterior Sampling for General Noisy Inverse Problems (arxiv.org) 37
Denoising Diffusion Models for Plug-and-Play Image Restoration

HQS as one diffusion step

38
 Diffusion Model Based Posterior Sampling for Noisy Linear Inverse Problems*

A itself is row-orthogonal

efficient computation via SVD

*[2211.12343] Diffusion Model Based Posterior Sampling for Noisy Linear Inverse Problems (arxiv.org) 39
 Denoising Diffusion Restoration Models (DDRM)*
An efficient, unsupervised
posterior sampling method

SVD

*[2201.11793] Denoising Diffusion Restoration Models (arxiv.org) 40

 Denoising Diffusion Restoration Models (DDRM)*
Linear inverse problem

forward
SVD

reverse

singular values

y null-space

DDRM final steps

generative part

*[2201.11793] Denoising Diffusion Restoration Models (arxiv.org) These equations in DDRM are in VESDE form 41
 Zero-Shot Image Restoration Using Denoising Diffusion Null-Space Model*

Decouple
range-space of A null-space of A

find a proper 𝑥ҧ that makes the null-space term

Reconstruction
is in harmony with the range-space term

Diffusion Models

*[2212.00490] Zero-Shot Image Restoration Using Denoising Diffusion Null-Space Model (arxiv.org) 42
Zero-Shot Image Restoration Using Denoising Diffusion Null-Space Model

43
 ΠGDM: Pseudoinverse-Guided Diffusion Models for Inverse Problems*

*Pseudoinverse-Guided Diffusion Models for Inverse Problems | OpenReview 44

On Equivalence of Diffusion Posterior Sampling Strategies

DPS

some coefficient
DDNM

45
Solving Image Restoration Tasks Iteratively (Traditional PnP Methods)

Image Restoration by Iterative Denoising and Backward Projections Plug-and-Play Image Restoration with Deep Denoiser Prior

What are the advantages of diffusion sampling framework? → well-defined path connecting two distributions
In our experiment all methods use the same diffusion model checkpoints → schedule is all you need!!
46
Sampling from Langevin Dynamics?

[2103.04715] Bayesian imaging using Plug & Play priors: when Langevin meets Tweedie (arxiv.org)

[1611.02862] The Little Engine that Could: Regularization by Denoising (RED) (arxiv.org)
An Interpretation Of Regularization By Denoising And Its Application With The Back-Projected Fidelity Term

47
 PnP Generative Networks: Conditional Iterative Generation of Images in Latent Space*

Metropolis-adjusted Langevin algorithm (MALA) sampler

Denoising AutoEncoder output as score

*[1612.00005] Plug & Play Generative Networks: Conditional Iterative Generation of Images in Latent Space (arxiv.org) 48