VQUNet: Vector �antization U-Net for Defending Adversarial

A�acks by Regularizing Unwanted Noise

Zhixun He Mukesh Singhal
University of California, Merced University of California, Merced
Merced, CA, U.S.A Merced, CA, U.S.A
[email protected] [email protected]

ABSTRACT model’s prediction performance on benign data when there is no

Deep Neural Networks (DNN) have become a promising paradigm adversarial attack.
when developing Arti�cial Intelligence (AI) and Machine Learn- Adversarial Training is one defense approach for DNN models
ing (ML) applications. However, DNN applications are vulnerable to mitigate the attacks, by mixing adversarial samples with benign
to fake data that are crafted with adversarial attack algorithms. data [4, 11] when training a DNN. Although it has shown progress
Under adversarial attacks, the prediction accuracy of DNN applica- in robustness improvement, there are two main drawbacks: (1) each
tions su�ers, making them unreliable. In order to defend against real sample can be crafted by various attack algorithms and it is
adversarial attacks, we introduce a novel noise-reduction procedure, di�cult to include all possible adversarial samples into training
Vector Quantization U-Net (VQUNet), to reduce adversarial noise samples; (2) there is a trade-o� between the diversity of adversarial
and reconstruct data with high �delity. VQUNet features a discrete samples and training time to achieve satisfactory generalization to
latent representation learning through a multi-scale hierarchical tackle all adversarial attacks. Another line of e�ort is to certify a
structure for both noise reduction and data reconstruction. The robustness threshold for DNN models [37], that given an adversarial
empirical experiments show that the proposed VQUNet provides sample, as long as the di�erence of itself from its benign counterpart
better robustness to the target DNN models, and it outperforms is within a !? -norm-ball, the certi�ed model will be resistant to
other state-of-the-art noise-reduction-based defense methods under the adversarial sample no matter what attack algorithm is used to
various adversarial attacks for both Fashion-MNIST and CIFAR10 create it. In practice, in order to simulate various adversarial attacks
datasets. When there is no adversarial attack, the defense method and the adversarial noise that covers the !? -norm-ball on benign
has less than 1% accuracy degradation for both datasets. samples, a random sampling process is used to create adversarial
samples during training.
CCS CONCEPTS Other than adversarial training, various other categories also
show progress in adversarial defense. (1) Gradient Obfuscation,
• Computing methodologies ! Machine learning algorithms;
which refers to mechanisms that inhibit the adversary to calculate
Arti�cial intelligence; Computer vision.
the adversarial noise. For example, Papernot et al. [36] dynamically
adjusts the activation function to reduce the magnitude of the
KEYWORDS gradient of the loss with respect to the model’s input such that
Adversarial Defense, Noise Reduction, Vector Quantization, U-Net the adversarial perturbation is reduced. (2) Detection, in which the
benign training data can maintain their consistency in their latent
1 INTRODUCTION space features, and this consistency can distinguish themselves
Modern DNNs have made their success in achieving excellent per- from the latent feature anomaly hidden in adversarial counterparts
formance on a wide range of tasks in various domains, e.g., images [8, 48]. (3) Adversarial Noise Reduction, which takes unknown
[14, 18, 27], audio [22, 43] and videos [7, 13], etc. However, it’s inputs to reconstruct outputs that are similar to the original benign
been widely observed that DNN-based approaches are sensitive data by reducing unwanted adversarial noise in the unknown inputs.
to samples perturbed by adversarial attacks [1, 6]. Such adversar- For example, generative neural networks, like, VAE [24, 25] and
ial samples can be indistinguishable from real data but they can GAN [10], are common paradigms for generating noise-reduced
cause DNN models to give wrong predictions. Because each benign samples by mapping inputs to the outputs from the desired data
sample can be used to craft various adversarial counterparts, it’s domain.
challenging for a defense mechanism to be able to handle all pos- VAE-based noise reduction defense methods [9, 15, 53] opti-
sible adversarial samples without introducing any decrease in a mize their performance by reducing the divergence between re-
constructed output and the training data. Although this objective
Permission to make digital or hard copies of all or part of this work for personal or allows VAE to generalize well to unseen data, VAEs with sequential
classroom use is granted without fee provided that copies are not made or distributed structure typically su�er from unwanted noise during the decoding
for pro�t or commercial advantage and that copies bear this notice and the full citation stage, e.g., incurring blur e�ect in the reconstructed images. GANs
on the �rst page. Copyrights for components of this work owned by others than ACM
must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, [2, 23, 45] demonstrated promising generative features, which can
to post on servers or to redistribute to lists, requires prior speci�c permission and/or a create quality images with high resolution. While both generative
fee. Request permissions from [email protected].
networks hold the promise of generating noise-reduced images
ICMVA ’24, March 12–15, 2024, Singapore
© 2024 Association for Computing Machinery. for defense purposes, it’s still a challenge to reconstruct samples
ACM ISBN 978-1-4503-XXXX-X/18/06. . . $15.00
https://doi.org/10.1145/1122445.1122456
ICMVA ’24, March 12–15, 2024, Singapore He and Singhal, et al.

as clean as original training data when the adversarial noise be- with a feedback loop provides multiple estimations that will be used
comes more noticeable. Also, the more complicated the dataset the in a dedicated neural network to restore the clean image. He et al.
more di�cult for the generative networks to maintain the data [15] improve the prediction accuracy on adversarial samples by
reconstruction quality (e.g., RGB images from Cifar10 versus gray- combining the prediction accuracy on noise-reduced images and a
scaled images from Fashion-MNIST), thus, the performance of noise- majority voting on multiple estimations through a Bayesian update
reduction-based methods vary for di�erent datasets [15, 16, 20, 21]. process.
In this paper, we present a new generative model for adversarial Multiple VAE-based noise reduction methods [9, 19, 29] attempt
noise reduction that combines multi-scale hierarchical representa- to use VAEs multiple times to further reduce the adversarial noise.
tion learning and discrete latent feature learning through Vector However, there is a caveat to reusing the same de-noising network
Quantization (VQ), to reconstruct high-�delity images and achieve multiple times. If the �rst pass introduces an unwanted e�ect in the
high tolerance to adversarial noise. In particular, this work makes output, e.g., a blur e�ect, such e�ect can be ampli�ed when passing
the following contributions: the output into the de-noising network for a second time. A di�erent
• We introduce Vector Quantization U-Net (VQUNet), a novel attempt enforces a cycle consistency loss when training two de-
generative model that maintains the data �delity after noise noising generative networks simultaneously [16]. In this process,
reduction, by leveraging multi-scale hierarchical latent fea- one generative network generates images to be in a di�erent domain
ture learning and quantized feature learning. VQUNet fur- from the original domain, and then a second generative network
ther improves the robustness of data reconstruction under restores the generated outputs back to the original images’ domain.
various adversarial attacks and it outperforms existing state-
of-the-art adversarial defense methods, especially when the 2.2 Generative Networks on Adversarial
adversarial perturbation is high. Defense
• Via rigorous experiments, we show that VQUNet outper- Although adversarial noise-reduction-based methods can mitigate
forms state-of-the-art adversarial noise reduction defense adversarial attacks, special care should be given to the data recon-
methods across a wide spectrum of adversarial noise levels struction process, because it is important for the DNN model to
and various adversarial attacks on two benchmark datasets keep the same prediction accuracy for benign inputs after the de-
(CIFAR10 [26] and Fashion-MNIST [46]). The de-noising noising process. Maintaining the �delity of the reconstructed data
process through VQUNet does not sacri�ce the prediction while e�ectively reducing the adversarial noise is still a challenge.
accuracy on clean data after they are processed by the de- VAE [24, 25, 51] and GAN [10, 34, 45] are two of the most pop-
noising network. The proposed method incurs a minimal ular generative network paradigms for generating samples. VAE
impact on the original deep learning model’s performance features a two-step process to reconstruct the data. The inputs
when there is no adversarial attack. are �rst encoded into latent feature space, and then the encoded
• Through extensive experimental study, we further examine vectors are decoded back to the original data space [52]. This encod-
the individual components and provide insight into their ing/decoding feature is attractive for defense purposes as the en-
e�ectiveness on the overall performance against adversarial coding process can suppress the adversarial noise [9, 15, 29, 32, 53].
attacks. However, the decoder may struggle to recreate high-�delity data
if the encoding process discards too much information, and it can
2 RELATED WORK introduce unwanted noise in the reconstructed images.
2.1 Adversarial Noise Reduction As opposed to data encoding/decoding, GANs aim at generat-
Adversarial noise-reduction-based methods aim to clean the input ing outputs that are similar to those from the original domain, by
before they are predicted by a deep learning model [47]. The clean- training a generator and a discriminator simultaneously. The dis-
ing process signi�cantly reduces the e�ectiveness of adversarial criminator attempts to distinguish the actual training samples from
attacks on the target models [1]. For example, during the JPEG com- the generated samples, and the generator tries to generate outputs
pression/decompression process [30], by quantizing the images’ that can confuse the discriminator [33]. The relaxation from the
frequency domain, a certain level of adversarial perturbation can be data encoding/decoding gives GANs �exibility and diversity in
their network structure and training schemes [2, 16, 20, 21, 23, 34].
�ltered. Other than reducing noise for the entire image, adversarial
For example, U-net [40] designs their feature extraction and data
noise reduction can also be applied only to a high-attention region
in an image using a wavelet-based image de-noising process [12]. reconstruction path to be symmetric at di�erent depth levels as
Sun et al. [42] tackle adversarial noise by projecting an input to opposed to traditional linear expansion.
a feature space using a "Sparse Transformation Layer", and approxi-
mating the projected space to that of natural images. In comparison, 2.3 Quantized Vector as Latent Features
Defense-GAN [41] tries to �nd a latent space feature that can be Multiple studies [3, 19, 30, 48] have indicated that data compression
used by the generator to reconstruct the image as close as the test- has a regularizing e�ect of reducing model sensitivity for small
ing input, and then this latent feature will be used as the seed to noise to mitigate adversarial attacks. Input discretization is one of
generate the de-noised version of the input. the compression techniques, which refers to the process of convert-
Yuan et al. [49] mitigate the adversarial noise pattern by process- ing continuous values into di�erent non-overlapping buckets [3].
ing an input image through an orthonormal transformation and Di�erent discretization strategies have been studied, such as, evenly
a deadzone-based activation function. Then a generative network quantizing the compression feature [30], adjusting the quantization
VQUNet: Vector �antization U-Net for Defending Adversarial A�acks by Regularizing Unwanted Noise ICMVA ’24, March 12–15, 2024, Singapore

process dynamically [38], and discretizeing the feature vector into a VQUNet converts each natural feature vector into one of the learned
vector dictionary [42, 44]. The proposed method learns a codebook vectors in the codebook.
that consists of multiple quantized vectors, and each of the original
feature vectors will be mapped to one of the vectors in the codebook 3.1 VQUNet Structure
based on the similarity of the original vector and the vectors in the
codebook. The structure of VQUNet is illustrated in Figure 1a, where the
The input quantization process has a regularization e�ect on input is an image, and the output is its noise-reduced version. For
adversarial noise, which is partially due to the fact that a certain simplicity, we annotate the encoding, decoding, and quantization
level of unwanted adversarial noise will be discarded when com- process with ⇢3 , ⇡3 , and &3 respectively, where 3 represents their
pressing input to a smaller dimension. However, there is a trade-o� depth level. The single arrows with labels indicate the data �ow in
between the compression rate and the data reconstruction quality. Figure 1a, and the details of the structure of ⇢3 , ⇡3 are illustrated
If too much information is discarded, the reconstruction process in Figure 1b and Figure 1c respectively.
will struggle to generate samples with high �delity to the inputs, The vertical direction in Figure 1a is the encoding process, which
especially when the dataset becomes more complicated, for exam- extracts more global shared latent features as the forward pass
ple, images from CIFAR10 are more challenging to be reconstructed moves deeper vertically. A common practice for latent feature ex-
as opposed to those from Fashion-MNIST or MNIST. traction is to apply a sequence of Convolutional Layers to the
The vector quantization model [44] features a learned codebook feature outputs. However, when the forward pass is too deep, the
that contains up to 1-D vectors. Toward the end of the encoding network struggles to use the highly abstract latent features for data
process, each of the natural feature vectors will be mapped to one of reconstruction. Therefore, we depart from this design by applying
the vectors in the codebook. The non-linear mapping is decided by: residual blocks to the encoding process, which helps maintain the
1) calculating the Euclidean distance between the natural feature information from earlier layers, avoiding the information degrada-
vector and the quantized vectors in the codebook; 2) the quantized tion problems, as shown in Figure 1b.
vector with the closest distance will be selected as the replacement Each encoded feature vector from the encoder block will be
for the natural feature vector. Formally [39], with the input data, G, mapped to one of the quantized vectors in the codebook, where
to quantize the natural feature vectors ⇢ (G)8 , the quantized feature the quantized vector with the closest Euclidean distance from the
& (⇢ (G)8 ) is calculated as below: feature vector will be selected. Note that each depth level has its
own quantization process and unique codebook for the quantiza-
& (⇢ (G)8 ) = 4: where : = arg min ⇢ (G)8 49 2 (1) tion process. The new set of quantized vectors selected from the
9 codebook at each depth level will be used as the input for the next
level’s encoding process.
where : 2 1, 2, · · · , (K is the total number of quantized vectors At the deepest level after all encoding processes, the latent fea-
in the codebook), and 8 2 1, 2, · · · , " where " is the total number tures have been transformed to the dimension of 2 ⇥ 2 ⇥ 512. From
of natural feature vectors in ⇢ (G). The quantized feature vectors there, the decoding process starts to expand the highly abstract
& (⇢ (G)) will be used for data reconstruction during the forward information, which includes two stages: (1) the features �rst go
pass. The training of the codebook will be further explained in through a convolutional block and two residual blocks; (2) then a
section 3.2. ConvTranspose process expands the size with a stride of two; as
shown in Figure 1c.
3 METHOD The decoded features will be concatenated to the quantized fea-
In this work, we introduce a new adversarial noise reduction net- tures from the immediate previous depth level. Then these hybrid
work, Vector Quantization U-Net (VQUNet), an e�ective noise features are expanded again by the next decoding block as the data
reduction model to regularize both adversarial and clean data. pipeline shows in Figure 1a. Toward the end, the size of the output
VQUNet enhances deep learning models’ robustness against vari- reaches that of the original data, and the output’s channel number
ous adversarial attacks. VQUNet features a U-shaped latent feature will be adjusted accordingly to be consistent with the domain of
forward pass and hierarchical vector quantization to capture the the original data.
correlation between high-level latent features and low-level infor-
mation, such as texture, localized features, and geometry of objects. 3.2 Training
The combination of the information from di�erent hierarchies of n o#
the network allows the decoding blocks to propagate the local tex- For the # images - = G (8 ) , the VQUNet regularizes the input
8=1
ture features toward the end of the forward pass to yield a more to its noise-reduced counterpart - ⇠ : 5+ &* # 4C : - ! - ⇠ . The
n o#
precise recreation of the input.
adversarial version of the data is denoted as - 03E = G 03E (8 ) .
During the forward pass, VQUNet applies non-linear mapping 8=1
from each encoding block’s natural outputs to discrete codes at Reconstruction Loss. For the data reconstruction, the loss to
multiple hierarchical levels. The discrete codes are selected from a guide the network to recreate data that are similar to the clean
learnable codebook, where each natural feature vector is mapped samples is the mean square error between the inputs and network
to the most similar discrete code. In comparison, the classic VAEs outputs:
[9, 24] compress latent code (the prior ? (I) under the Bayesian h i
2
theory) into a continuous multi-dimensional unit ball, while the !A42>=BC = EG 2- G 5+ &* # 4C (G)) (2)
ICMVA ’24, March 12–15, 2024, Singapore He and Singhal, et al.

(a) The structure and data �ow of VQUNet, where ⇢ denotes the encoding block, (b) The encoding (c) The decoding
& denotes the quantization block and ⇡ denotes the decoding block. The depth block, denoted as block, denoted as
level is indicated by 3, which controls the parameters in ⇢ and ⇡ blocks. “E" in Figure 1a. “D" in Figure 1a.

Figure 1: The overall structure of VQUNet, and the detailed structure of encoder block and decoder block. The three-digit pair
in Conv. and ConvTrans., such as (32/3/1), in Figure 1b and 1c represents (output �lter number / kernel size / strides )

The encoding process is denoted as 5⇢ : G ! 0. 03 is used to @3⇤ = 03 + >3 . @3⇤ will be used as the input for the layer that comes
denote the output from an encoding block at depth level 3. The immediately after the quantization block at depth 3.
vector quantization process is denoted as 5& : 0 ! @, and the Note that @3⇤ holds the same value as @3 . However, by treat-
quantized vector at depth level 3 is denoted as @3 . Note that the ing >3 as constant and using @3⇤ in the forward-pass instead of @3 ,
implementation of the codebook is through the Embedding Layer the computational path from 03 to @3⇤ becomes linear. This linear
in Tensor�ow, in which a 2-D matrix is created as the codebook relationship makes the back-propagation path from @3⇤ to 03 di�er-
parameters. The size of a 2-D matrix is ⇥ ⌧, where is the total entiable in the network. Thus, the parameters in Encoder blocks
number of the 1-D quantized vector in the codebook, and each can be optimized with the gradient of !A42>=BC during training.
quantized vector has ⌧ entries. The �rst step of the quantization Optimizing the quantized vectors in the codebook w.r.t. !A42>=BC
process is to compare an input vector against all the discrete is done by normal back-propagation, because the quantized vectors,
vectors, such that the index of the most similar discrete vector can @3 , already form a continuous connection to the layers that come
be found. Then this index is used in Embedding Layer’s look-up immediately after them.
process to output the corresponding discrete vector. Encoder&Quantization Loss. Because there is no constraint
However, this look-up operation that compares the input vector on the value range for 03 and @3 , as training goes, 03 and @3 can
against discrete vectors and then �nds the index of the most end up having large di�erences in their elements’ value magnitude.
similar discrete vector, is a discrete process. Because of the discrete This value range divergence can make it di�cult for the training
nature of the comparison operation, in the Tensor�ow computation to converge. For example, if the values of the elements in 03 are
graph of the neural network, there is not a di�erentiable path much bigger than that of @3 , then during the discrete mapping, a
during gradient back-propagation from the Embedding Layer to small change in 03 can cause a very di�erent set of quantized codes
its previous layer. In such case, the gradient of !A42>=BC w.r.t. the to be selected between each training step. Therefore, two extra loss
parameters of those layers before the Embedding Layer (e.g., the functions are imposed to bring the value range of 03 and @3 closer.
parameters in encoding blocks) will not be available, for example, First, to guide the parameters of the encoder to be closer to the
Tensor�ow assigns “None" or “Zero" to the gradient of loss w.r.t. codebook’s value range, the intermediate quantized vectors @3⇤ will
those variables because they are regarded as unconnected variables. be reused. The mean square error between @3⇤ and 03 is used as
To bypass the non-di�erentiable path issue above, an interme- the loss function to guide the encoder’s outputs to be closer to the
diate variable is used to bridge the encoding blocks’ parameters values of the quantized vectors in the codebook. The encoder loss
to the Embedding Layer, such that the encoders’ parameters are is de�ned as:
connected to the rest of the computation graph during the gradient h i
2
back-propagation. First, 03 and @3 are already available during each !⇢ = EG 2- @3⇤ 03 (3)
forward pass, and a new variable >3 is created in the computation
Second, in order for the quantized vectors in the Embedding Layer
graph and de�ned as >3 = @3 03 . The value of >3 is treated as
to form a similar value range to the encoder’s outputs, another
constant, and then a new quantized vector is created and de�ned as
intermediate variable, 03⇤ , is created in the computation graph for
training. The variable 03⇤ is used to hold the value of the encoder’s
VQUNet: Vector �antization U-Net for Defending Adversarial A�acks by Regularizing Unwanted Noise ICMVA ’24, March 12–15, 2024, Singapore

outputs, 03 . Then the mean square error between 03⇤ and @3 is attack algorithms on the target model, which is a Wide Residual Net-
used as the quantization loss to guide the training of the quantized work (with parameters 34?C⌘ = 28 and : = 10), and the other 50%
vectors in the codebook, denoted as !& : are created with 3 trained DenseNet [17] with di�erent structures.
h i When creating adversarial samples using di�erent attack algo-
2
!& = EG 2- @3 03⇤ (4) rithms, we keep most of the parameters to their default values,
except the noise perturbation level Y. For classi�cation accuracy,
The overall loss to guide the training of VQUNet is as follows:
the target model being tested is a Wide-Residual-Network [50].
! = U ⇤ !A42>=BC + V ⇤ !⇢ + !& (5)
where U and V are two weight parameters that can be adjusted to 4.2 Accuracy Degradation from Noise
give di�erent priority to the individual loss. During the empirical Reduction Process
study, the value of U and V a�ect the convergence speed of !A42>=BC , Note that the noise reduction defense methods �lter every input
!⇢ , and !& only around the �rst 30 epochs, but after that, the before the de-noised output is used by the target model, even when
convergence of loss does not vary much for di�erent values of U or there is no adversarial attack. We use “�ltered” and “un�ltered” to
V. Therefore, for the rest of the experiments, U = 1 and V = 1 are distinguish the data that have been processed by the noise reduction
used. networks as opposed to those data that have not. We call those data
that have not been modi�ed by an attack algorithm as “benign data”,
4 EXPERIMENTS and those that are crafted by an attack algorithm as “adversarial
4.1 Experimental Setup samples”.
It is important to see if the �ltering process would change the
Once VQUNet is trained, the de-noised version of data, - ⇠ , shall be
target model’s original performance, hence, we tested the target
used to retrain the target deep learning model. During the testing
model’s accuracy on the benign data before and after the �ltering
phase, VQUNet acts as an adversarial noise �lter for the inputs, and
process, and the result is shown in Table 1.
then the de-noised outputs are fed into the target deep learning
Compared with the other three defense methods in Table 1,
model for testing. In our experiments, two datasets are used for
“VQUNet” is the only one that has < 1% accuracy degradation for
evaluation: CIFAR10 [26] and Fashion-MNIST [46]. Each sample
both CIFAR10 and Fashion-MNIST datasets, which means “VQUNet”
in CIFAR10 is a 32 ⇥ 32 ⇥ 3 RGB-colored image, and each sample
incurs the least alteration to the target model’s original perfor-
belongs to 1 of the 10 categories of objects. Individual sample in
mance.
Fashion-MNIST is a 32 ⇥ 32 ⇥ 1 gray-scaled image, and there are also
10 categories of fashion objects in the dataset. CIFAR10 contains
model’s acc on benign CIFAR10: 94.03%
50,000 training samples and 10,000 testing samples, and Fashion-
Methods acc (�ltered benign) acc degradation
MNIST contains 60,000 training samples and 10,000 testing samples.
VQUNet 93.08% -0.95%
For comparison, the proposed method and three other existing
adversarial noise reduction methods are evaluated: the proposed Cycle_GAN 91.78% -2.25%
VQUNet, Defense-VAE [29], High-Frequency Loss VAE (FHL_VAE) HFL_VAE 86.51% -7.52%
[15], and Defense-CycleGAN (CycleGAN) [16]. Following a com- Defense_VAE 78.84% -15.19%
mon practice for adversarial defense evaluation, all defense methods model’s acc on benign Fashion-MNIST data: 94.31%
will be tested under four di�erent adversarial attacks: Fast Gradient Methods acc (�ltered benign) acc degradation
Sign Method (FGSM) [11], Basic Iterative Method (BIM) [28], Pro- VQUNet 93.39% -0.92%
jected Gradient Descent (PGD) [31], Carlini and Wagner Method Cycle_GAN 93.13% -1.18%
(CW) [5]. All noise reduction networks will only be trained once and HFL_VAE 92.81% -1.50%
their performance is evaluated against all four adversarial attacks Defense_VAE 92.83% -1.48%
for a wide range of noise levels Y. Note that there is a post-training Table 1: The target model’s accuracy before and after the
add-on feature that can be attached to most of the noise reduction noise reduction process of di�erent defense methods on CI-
process to further boost the target model’s accuracy as mentioned FAR10 (top) and Fashion-MNIST (bottom).
in [15], but to test the data reconstruction properties, the experi-
ments did not integrate the add-on feature but just focus on the
noise reduction properties.
The implementation of generating adversarial samples for all
four di�erent attacking algorithms is through Cleverhans [35] pack- 4.3 Defense against White/Black-Box Attacks
age. White-box attack refers to the case when the adversary has The comparison of the four noise reduction defense methods against
full access to the target models’ information, e.g., parameters, struc- various adversarial attacks on CIFAR10 and Fashion-MNIST datasets
ture, and hyper-parameters, to generate adversarial samples. For is shown in Figure 2. Note that when there is no defense, the target
a black-box attack, in comparison, the attacker does not have any model’s accuracy drops below 10% under all adversarial attacks for
knowledge about the target network, but the attacker can test the both datasets.
target model through trial and error, such as collecting the results CIFAR10. From the result shown in Figure 2 (top), “VQUNet”
using adversarial inputs computed from other networks. In the maintains signi�cantly higher accuracy than the other defense
experiment, 50% of the testing samples are created using white-box methods across the entire adversarial noise levels. As adversarial
ICMVA ’24, March 12–15, 2024, Singapore He and Singhal, et al.

Figure 2: Comparison of di�erent defense methods’ performance under various adversarial attacks on CIFAR10 (top) and
Fashion-MNIST (bottom)

noise perturbation increases, “VQUNet” shows a �atter curve in ac- latent features. To study the regularization e�ect from VQ, we
curacy compared to other methods, which indicates that “VQUNet” compared VQUNet to a second generative network called “non-
has better robustness against adversarial noise. VQUNet", which is nearly identical to the VQUnet except that there
Fashion-MNIST. Defense performance of di�erent defense meth- is no VQ block in it.
ods on Fashion-MNIST is shown in Figure 2 (bottom). We observe We �rst compared the averaged ;1 di�erence between the origi-
that compared to other methods, “VQUnet" demonstrates a �atter nal images and the reconstructed images from the generative net-
accuracy curve as the noise perturbation level increases. This in- works when they were under FGSM attack. The results for both
dicates that “VQUnet" has a better resistance against adversarial VQUNet and non-VQUNet are shown in Figure 3. As the adversarial
noise for the Fashion-MNIST dataset. Among the four methods in perturbation increases, the reconstructed images from VQUNet
our experiment, “VQUnet" is the only one that maintains at least demonstrated less divergence from the original images compared
90% accuracy under all adversarial attacks across the entire noise to those of the non-VQUNet.
spectrum. To take a closer look, our experiments also examine the behavior
of encoding blocks for both VQUnet and non-VQUNet when they
4.4 Regularization from Vector Quantization are under FGSM attack. Figure 4 shows the ;1 di�erence between
the encoding blocks’ output before and after the network is under
FGSM attack at various depth levels (d=1, d=2, and d=3) for both of
the VQUNet and non-VQUNet. The value deviation of the encoders’
output before and after the attack for the non-VQUNet is much
larger than that of VQUNet, which means the encoding features in
the non-VQUNet are much more susceptible to adversarial noise
than those in VQUNet. This helps explain why the VQUNet achieves
better �delity in image reconstruction as shown in Figure 3.
Pre_VQ v.s. Post_VQ. Note that during the VQ process, the
pre_vq features are mapped to post_vq features through the VQ
layer. In order to see the regularizing e�ect of VQ on the adversarial
noise, we compared the ;1 di�erence of the features before and
after the adversarial attack (FGSM) for both pre_vq vectors and
post_vq vectors at di�erent depth levels, as shown in Figure 5. It’s
observed that both pre_vq and post_vq vectors show an increase
in the divergence of their features before and after the attack as
the adversarial noise level becomes larger. But for each depth level,
Figure 3: The ;1 di�erence between benign images and the
the ;1 di�erence in post_vq vectors is noticeably smaller than that
reconstructed images that are generated from VQUNet and
of pre_vq. In other words, the adversarial noise has less negative
non-VQUNet for various adversarial (FGSM) noise levels.
impact on the post_vq vectors after the VQ process, compared to
VQ v.s. Non-VQ. One of the main motivations for using vector those of pre_vq vectors. Comparing di�erent depth levels, both
quantization is to regularize the impact of adversarial noise on
VQUNet: Vector �antization U-Net for Defending Adversarial A�acks by Regularizing Unwanted Noise ICMVA ’24, March 12–15, 2024, Singapore

Figure 6: The percentage change of the indices of selected

discrete codes before and after the FGSM attack.
Figure 4: The ;1 di�erence of pre_vq vectors before and after
FGSM attack at various depth levels.
5 CONCLUSION
pre_vq and post_vq at deeper depth show less susceptibility to We introduced a new noise-reduction model, VQUNet, for adver-
adversarial noise compared to those at shallower depth. sarial defense. The features include a hierarchical structure for
high-�delity image reconstruction and a vector quantization mech-
anism that helps regularize the impact from the adversarial noise
for better data reconstruction. We showed that under various ad-
versarial attacks and a wide range of noise perturbation levels,
VQUNet outperforms other state-of-the-art defense methods on
both CIFAR10 and Fashion-MNIST datasets with a noticeable mar-
gin. In addition, when there is no adversarial attack, only VQUNet
controls its accuracy degradation < 1% from the target model’s
original accuracy performance compared to other methods.

REFERENCES
[1] Naveed Akhtar, Ajmal Mian, Navid Kardan, and Mubarak Shah. 2021. Advances
in adversarial attacks and defenses in computer vision: A survey. IEEE Access 9
(2021), 155161–155196.
[2] Andrew Brock, Je� Donahue, and Karen Simonyan. 2018. Large scale GAN
training for high �delity natural image synthesis. arXiv preprint arXiv:1809.11096
(2018).
[3] Jacob Buckman, Aurko Roy, Colin Ra�el, and Ian Goodfellow. 2018. Thermometer
Figure 5: L1 di�erence of features before and after the FGSM encoding: One hot way to resist adversarial examples. In International Conference
attack for pre_vq and post_vq vectors at various depth levels. on Learning Representations.
[4] Nicholas Carlini, Guy Katz, Clark Barrett, and David L Dill. 2018. Ground-truth
adversarial examples. openreview.net (2018).
Code Index Change under Adversarial Attack. Note that the [5] Nicholas Carlini and David Wagner. 2017. Towards evaluating the robustness
VQ process maps the encoder blocks’ outputs to a new set of discrete of neural networks. In 2017 ieee symposium on security and privacy (sp). IEEE,
39–57.
codes, where each code has its unique index in the codebook. To [6] Anirban Chakraborty, Manaar Alam, Vishal Dey, Anupam Chattopadhyay, and
study the e�ect of adversarial noise on the discretization process, we Debdeep Mukhopadhyay. 2018. Adversarial attacks and defences: A survey. arXiv
preprint arXiv:1810.00069 (2018).
compared the how much percentage change of the selected codes’ [7] Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew
indices before and after the adversarial attack (FGSM). Figure 6 Zisserman. 2019. Temporal cycle-consistency learning. In Proceedings of the
shows the percentage change of selected codes’ indices at di�erent IEEE/CVF Conference on Computer Vision and Pattern Recognition. 1801–1810.
[8] Reuben Feinman, Ryan R Curtin, Saurabh Shintre, and Andrew B Gardner. 2017.
depth levels (d=1, d=2, and d=3) of VQUNet when it’s under the Detecting adversarial samples from artifacts. arXiv preprint arXiv:1703.00410
FGSM attack. From the observation, the VQ layer at a deeper level (2017).
maintains its original codes’ indices relatively better than that at a [9] Partha Ghosh, Arpan Losalka, and Michael J Black. 2019. Resisting adversarial
attacks using gaussian mixture variational autoencoders. In Proceedings of the
shallower depth. As the adversarial noise increases, the change of AAAI Conference on Arti�cial Intelligence, Vol. 33. 541–548. Issue 01.
indices rises more rapidly at the beginning, but this change slows [10] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley,
Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial
down as the adversarial noise level becomes larger. This means that nets. Advances in neural information processing systems 27 (2014).
the VQ process shows more susceptibility to noise perturbation [11] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. 2014. Explaining and
when the noise level is small. However as the adversarial level harnessing adversarial examples. arXiv preprint arXiv:1412.6572 (2014).
[12] Puneet Gupta and Esa Rahtu. 2019. Ciidefence: Defeating adversarial attacks by
becomes larger, the VQ process maintains a similar selection of the fusing class-speci�c image inpainting and image denoising. In Proceedings of the
discrete codes when it’s under the attack. IEEE/CVF International Conference on Computer Vision. 6708–6717.
ICMVA ’24, March 12–15, 2024, Singapore He and Singhal, et al.

[13] GM Harshvardhan, Mahendra Kumar Gourisaria, Manjusha Pandey, and Sid- [37] Aditi Raghunathan, Jacob Steinhardt, and Percy Liang. 2018. Certi�ed defenses
dharth Swarup Rautaray. 2020. A comprehensive survey and analysis of genera- against adversarial examples. arXiv preprint arXiv:1801.09344 (2018).
tive models in machine learning. Computer Science Review 38 (2020), 100285. [38] Adnan Siraj Rakin, Jinfeng Yi, Boqing Gong, and Deliang Fan. 2018. Defend deep
[14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015. Delving deep neural networks against adversarial examples via �xed and dynamic quantized
into recti�ers: Surpassing human-level performance on imagenet classi�cation. activation functions. arXiv preprint arXiv:1807.06714 (2018).
In Proceedings of the IEEE international conference on computer vision. 1026–1034. [39] Ali Razavi, Aaron van den Oord, and Oriol Vinyals. 2019. Generating diverse
[15] Zhixun He and Mukesh Singhal. 2021. Adversarial Defense Through High high-�delity images with vq-vae-2. In Advances in neural information processing
Frequency Loss Variational Autoencoder Decoder and Bayesian Update With systems. 14866–14876.
Collective Voting. In 2021 17th International Conference on Machine Vision and [40] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. U-net: Convolutional
Applications (MVA). IEEE, 1–7. networks for biomedical image segmentation. In International Conference on
[16] Zhixun He and Mukesh Singhal. 2022. Defense-CycleGAN: A Defense Mechanism Medical image computing and computer-assisted intervention. Springer, 234–241.
Against Adversarial Attacks Using CycleGAN to Reconstruct Clean Images. In [41] Pouya Samangouei, Maya Kabkab, and Rama Chellappa. 2018. Defense-gan:
2022 3rd International Conference on Pattern Recognition and Machine Learning Protecting classi�ers against adversarial attacks using generative models. arXiv
(PRML). 173–179. https://doi.org/10.1109/PRML56267.2022.9882238 preprint arXiv:1805.06605 (2018).
[17] Gao Huang, Zhuang Liu, and Kilian Q. Weinberger. 2016. Densely Connected [42] Bo Sun, Nian-hsuan Tsai, Fangchen Liu, Ronald Yu, and Hao Su. 2019. Adversarial
Convolutional Networks. CoRR abs/-1608.06993 (2016). arXiv:1608.06993 http: Defense by Strati�ed Convolutional Sparse Coding. 2019 IEEE/CVF Conference on
//arxiv.org/abs/1608.06993 Computer Vision and Pattern Recognition (CVPR) 00 (2019), 11439–11448. https:
[18] Abdul Jabbar, Xi Li, and Bourahla Omar. 2021. A survey on generative adversarial //doi.org/10.1109/cvpr.2019.01171
networks: Variants, applications, and training. ACM Computing Surveys (CSUR) [43] Guangzhi Sun, Yu Zhang, Ron J Weiss, Yuan Cao, Heiga Zen, Andrew Rosenberg,
54, 8 (2021), 1–49. Bhuvana Ramabhadran, and Yonghui Wu. 2020. Generating diverse and natural
[19] Xiaojun Jia, Xingxing Wei, Xiaochun Cao, and Hassan Foroosh. 2019. Comde- text-to-speech samples using a quantized �ne-grained VAE and autoregressive
fend: An e�cient image compression model to defend adversarial examples. In prosody prior. In ICASSP 2020-2020 IEEE International Conference on Acoustics,
Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. Speech and Signal Processing (ICASSP). IEEE, 6699–6703.
6084–6092. [44] Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. 2017. Neural
[20] Lingyun Jiang, Kai Qiao, Ruoxi Qin, Linyuan Wang, Wanting Yu, Jian Chen, Discrete Representation Learning. arXiv:1711.00937 [cs.LG]
Haibing Bu, and Bin Yan. 2020. Cycle-consistent adversarial GAN: The integration [45] Zhengwei Wang, Qi She, and Tomas E Ward. 2021. Generative adversarial
of adversarial attack and defense. Security and Communication Networks 2020 networks in computer vision: A survey and taxonomy. ACM Computing Surveys
(2020). (CSUR) 54, 2 (2021), 1–38.
[21] Guoqing Jin, Shiwei Shen, Dongming Zhang, Feng Dai, and Yongdong Zhang. [46] Han Xiao, Kashif Rasul, and Roland Vollgraf. 2017. Fashion-MNIST:
2019. Ape-gan: Adversarial perturbation elimination with gan. In ICASSP 2019- a Novel Image Dataset for Benchmarking Machine Learning Algorithms.
2019 IEEE International Conference on Acoustics, Speech and Signal Processing arXiv:1708.07747 [cs.LG]
(ICASSP). IEEE, 3842–3846. [47] Han Xu, Yao Ma, Hao-Chen Liu, Debayan Deb, Hui Liu, Ji-Liang Tang, and Anil K
[22] Andrej Karpathy, George Toderici, Sanketh Shetty, Thomas Leung, Rahul Suk- Jain. 2020. Adversarial attacks and defenses in images, graphs and text: A review.
thankar, and Li Fei-Fei. 2014. Large-scale video classi�cation with convolutional International Journal of Automation and Computing 17, 2 (2020), 151–178.
neural networks. In Proceedings of the IEEE conference on Computer Vision and [48] Weilin Xu, David Evans, and Yanjun Qi. 2017. Feature squeezing: Detecting
Pattern Recognition. 1725–1732. adversarial examples in deep neural networks. arXiv preprint arXiv:1704.01155
[23] Tero Karras, Samuli Laine, and Timo Aila. 2019. A style-based generator ar- (2017).
chitecture for generative adversarial networks. In Proceedings of the IEEE/CVF [49] Jianhe Yuan and Zhihai He. 2020. Ensemble generative cleaning with feedback
Conference on Computer Vision and Pattern Recognition. 4401–4410. loops for defending adversarial attacks. In Proceedings of the IEEE/CVF Conference
[24] Diederik P Kingma and Max Welling. 2013. Auto-encoding variational bayes. on Computer Vision and Pattern Recognition. 581–590.
arXiv preprint arXiv:1312.6114 (2013). [50] Sergey Zagoruyko and Nikos Komodakis. 2016. Wide Residual Networks. CoRR
[25] Diederik P Kingma, Max Welling, et al. 2019. An introduction to variational abs/1605.07146 (2016). arXiv:1605.07146 http://arxiv.org/abs/1605.07146
autoencoders. Foundations and Trends® in Machine Learning 12, 4 (2019), 307–392. [51] Junhai Zhai, Sufang Zhang, Junfen Chen, and Qiang He. 2018. Autoencoder and
[26] Alex Krizhevsky, Vinod Nair, and Geo�rey Hinton. 2009. CIFAR-10 (Canadian its various variants. In 2018 IEEE International Conference on Systems, Man, and
Institute for Advanced Research). onlie (2009). http://www.cs.toronto.edu/~kriz/ Cybernetics (SMC). IEEE, 415–419.
cifar.html [52] Lei Zhou, Chunlei Cai, Yue Gao, Sanbao Su, and Junmin Wu. 2018. Variational
[27] Alex Krizhevsky, Ilya Sutskever, and Geo�rey E Hinton. 2012. Imagenet classi�ca- Autoencoder for Low Bit-rate Image Compression.. In CVPR Workshops. 2617–
tion with deep convolutional neural networks. In Advances in neural information 2620.
processing systems. 1097–1105. [53] Jinlin Zhu, Guohao Peng, and Danwei Wang. 2020. Dual-Domain-Based Adver-
[28] Alexey Kurakin, Ian Goodfellow, and Samy Bengio. 2016. Adversarial examples sarial Defense With Conditional VAE and Bayesian Network. IEEE Transactions
in the physical world. arXiv preprint arXiv:1607.02533 (2016). on Industrial Informatics 17, 1 (2020), 596–605.
[29] Xiang Li and Shihao Ji. 2019. Defense-vae: A fast and accurate defense against
adversarial attacks. In Joint European Conference on Machine Learning and Knowl-
edge Discovery in Databases. Springer, 191–207.
[30] Zihao Liu, Qi Liu, Tao Liu, Nuo Xu, Xue Lin, Yanzhi Wang, and Wujie Wen. 2019.
Feature distillation: Dnn-oriented jpeg compression against adversarial examples.
In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
IEEE, 860–868.
[31] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and
Adrian Vladu. 2017. Towards deep learning models resistant to adversarial attacks.
arXiv preprint arXiv:1706.06083 (2017).
[32] Dongyu Meng and Hao Chen. 2017. Magnet: a two-pronged defense against
adversarial examples. In Proceedings of the 2017 ACM SIGSAC conference on
computer and communications security. 135–147.
[33] Achraf Oussidi and Azeddine Elhassouny. 2018. Deep generative models: Survey.
In 2018 International Conference on Intelligent Systems and Computer Vision (ISCV).
IEEE, 1–8.
[34] Zhaoqing Pan, Weijie Yu, Xiaokai Yi, Asifullah Khan, Feng Yuan, and Yuhui
Zheng. 2019. Recent progress on generative adversarial networks (GANs): A
survey. IEEE Access 7 (2019), 36322–36333.
[35] Nicolas Papernot, Fartash Faghri, Nicholas Carlini, Ian Goodfellow, Reuben Fein-
man, Alexey Kurakin, Cihang Xie, Yash Sharma, Tom Brown, Aurko Roy, et al.
2016. Technical report on the cleverhans v2. 1.0 adversarial examples library.
arXiv preprint arXiv:1610.00768 (2016).
[36] Nicolas Papernot, Patrick McDaniel, Xi Wu, Somesh Jha, and Ananthram Swami.
2016. Distillation as a defense to adversarial perturbations against deep neural
networks. In 2016 IEEE Symposium on Security and Privacy (SP). IEEE, 582–597.