IEEE COMPUTER MAGAZINE 1

Learn to Unlearn: Insights into Machine Unlearning

Youyang Qu, Member, IEEE, Xin Yuan, Member, IEEE, Ming Ding, Senior Member, IEEE,
Wei Ni, Senior Member, IEEE, Thierry Rakotoarivelo, Senior Member, IEEE,
and David Smith, Senior Member, IEEE
Machine Learning (ML) models have been shown to potentially leak sensitive information, thus raising privacy concerns in
ML-driven applications. This inspired recent research on removing the influence of specific data samples from a trained ML
model. Such efficient removal would enable ML to comply with the “right to be forgotten” in many legislation, and could also
address performance bottlenecks from low-quality or poisonous samples. In that context, machine unlearning methods have been
proposed to erase the contributions of designated data samples on models, as an alternative to the often impracticable approach of
retraining models from scratch. This article presents a comprehensive review of recent machine unlearning techniques, verification
mechanisms, and potential attacks. We further highlight emerging challenges and prospective research directions (e.g. resilience
and fairness concerns). We aim for this paper to provide valuable resources for integrating privacy, equity, and resilience into ML
arXiv:2305.07512v2 [cs.LG] 26 Oct 2023

systems and help them “learn to unlearn”.

Index Terms—Machine Unlearning, Exact Unlearning, Approximate Unlearning, Verification, Attacks

I. I NTRODUCTION from scratch to more sophisticated techniques that selectively

prune or adjust the model’s knowledge. These approaches not
I N recent years, the proliferation of data and computational
resources has led to the widespread adoption of Machine
Learning (ML) models across various domains. From health-
only ensure compliance with data regulations but also pave the
way for more responsible, flexible, and adaptive AI systems
care to finance, these models play an instrumental role in that can evolve with changing data landscapes.
decision-making processes and predictive analytics. However, Machine unlearning can primarily be categorised into “ex-
in the ever-changing landscape of data and user requirements, act” and “approximate” unlearning paradigms. Exact machine
static ML models, which remain unaltered after their initial unlearning ensures the complete erasure of specific data from
training, become inadequate and ineffective. Consequently, a a model, rendering the model as if the data had never been
new paradigm emerges featuring a shift towards more flexible introduced. Conversely, approximate unlearning aims for more
ML models. efficient removal, albeit less precise, ensuring the model be-
On a separate front of the recent worldwide legislation on haves similarly to one that has never seen the deleted data,
the rights of data privacy, it becomes necessary to retrain ML though without guaranteeing exact equivalency.
models on datasets that have been modified. For instance, While both approaches offer promising pathways to ad-
trained ML models should be modified when data points must dress data privacy and model adaptability, the dichotomy
be removed or amended [1] due to privacy concerns, resilience between them introduces intricate challenges in verification
requirements, bias reduction, or uncertainty quantification [2]. and potential vulnerabilities to adversarial attacks. As these
In order to exclude certain data samples from a trained methodologies become foundational in modern AI systems,
ML model, a new concept called machine unlearning has the imperative for research into their verification becomes
been proposed recently to efficiently re-train an ML model paramount. Ensuring the integrity and security of unlearning
without significantly sacrificing the ML performance. This processes is not just a technical necessity but a mandate for
concept opens up an alternative avenue to the traditional way the ethical deployment of AI in sensitive and dynamic data
of retraining the ML model entirely. environments.
Machine unlearning is an emerging frontier in the realm With the growing significance of machine unlearning, This
of artificial intelligence and machine learning, focusing on paper strives to provide a concise yet comprehensive overview
the ability of models to forget or remove specific data or of the current landscape in this field. Our goal is to offer a
knowledge. At its core, this research topic addresses concerns systematic perspective on the ongoing advancements in this
related to data privacy, system updates, and model robustness. area and to highlight its critical role in the constantly evolving
Traditional machine learning systems are designed to accumu- machine learning ecosystem.
late and retain information, but with the increasing scrutiny of The main contributions of this article are summarized as
data privacy and the dynamic nature of data sources, there’s follows.
a pressing need for systems that can adapt by ’unlearning’
outdated or undesired information. The proposed approaches • A comprehensive survey of existing research on exact
in machine unlearning vary, ranging from retraining models and approximate machine unlearning, with a taxonomy
presented in Table I.
Youyang Qu, Xin Yuan, Ming Ding, Wei Ni, Thierry Rakotoarivelo, • A thorough examination of the verification of machine
and David Smith are with Data61, Commonwealth Scientific and Industrial unlearning and relevant attacks.
Research Organization, Australia. Email: {youyang.qu, xin.yuan, ming.ding,
wei.ni, thierry.rakotoarivelo, smith.david}@data61.csiro.au. • Experimental results that provide valuable insights into
the effectiveness and cost of machine unlearning.
IEEE COMPUTER MAGAZINE 2

In this study, we focused on representative research that

encompasses both exact and approximate machine unlearning
techniques, in addition to delving into associated attacks
and verification methods. To ensure a thorough analysis of
recent advancements in the field, we selected research papers
spanning from 2015 through 2022, from top-ranked peer-
reviewed publication venues in the IEEE and ACM databases.

II. F ROM LEARNING TO UNLEARNING

A. Motivation
Privacy concerns, security, and performance optimization
underpin the need for machine unlearning in the broader Fig. 1. Diagram of a general scenario for machine unlearning
landscape of machine learning models and their applications.
Firstly, from a privacy protection perspective, the Euro-
pean Union (EU) implemented the General Data Protection on a model’s parameters. Every data point that a machine
Regulation (GDPR) in 2016, which is widely recognized learning model processes induces specific modifications to its
as the most stringent legislation regarding data privacy and parameters, and recognizing these nuances becomes pivotal in
security globally [3]. This regulation enforces obligations on the unlearning process.
organizations worldwide that collect or use data relating to Several algorithms have been proposed to facilitate this
individuals within the EU. It recognizes novel data privacy intricate process. The straightforward method involves the
rights to grant individuals greater autonomy over their data, complete removal of undesired data followed by retraining,
including the right to erasure (i.e., right to be forgotten). ensuring the model is purged of the specific knowledge.
In 2018, California passed the California Consumer Privacy Another approach is weight adjustment, where the influence
Act (CCPA)1 , providing consumers with substantial control of the unwanted data is counteracted by directly tweaking the
over the personal information that businesses collect about model’s weights, negating the need for complete retraining.
them. The “right to delete data” defined in the CCPA is the Leveraging regularization methods, some algorithms impose
equivalent of the right to erasure in the GDPR. Recently, Cali- penalties on the model for retaining specific data, coaxing it
fornia established the California Privacy Rights Act (CPRA)2 , to forget. Differential unlearning, a more recent technique, fo-
also known as “CCPA 2.0”. This Act extensively expands and cuses on understanding the changes in model weights induced
modifies the CCPA, rendering it comparable to the GDPR. The by particular data points during training and then reversing
CPRA came into effect in January 2023. Over the same period, these changes to achieve unlearning.
the Australian Privacy Act Review Report3 has recommended Supporting these algorithms are several mechanisms that
adding the “right to be forgotten” to Australian legislation. enhance the efficiency and reliability of the unlearning process.
Secondly, in the realm of cybersecurity, machine unlearn- Model versioning, for instance, maintains snapshots of various
ing is crucial for poisoning attack recovery. This involves model states, allowing for quick reversion to a prior state when
the removal of maliciously injected, poisoned data samples, unlearning becomes necessary. Another significant develop-
which could otherwise adversely affect the model’s predictive ment is the integration of data sanitization layers within the
accuracy or behaviour. training process. These layers continuously screen and filter
Lastly, from a performance optimization perspective, ma- out data that’s flagged for unlearning. Additionally, checkpoint
chine unlearning aids in eliminating the influence of low- systems, which store periodic model states, offer rapid rollback
quality data samples. These may have been incorporated options, enabling efficient unlearning without trudging through
during the training process and could have degraded the overall the entire training data again.
performance of the model. Hence, unlearning these specific In essence, as the digital landscape becomes increasingly
instances can enhance the model’s efficiency and effective- dynamic, machine unlearning, underpinned by these foun-
ness [4], [5], [6]. Therefore, machine unlearning emerges as dational concepts, algorithms, and mechanisms, ensures our
an essential aspect of managing and improving ML models in models remain adaptable and up-to-date.
contemporary data-driven applications.
III. E XACT M ACHINE U NLEARNING
B. Overview of Machine Unlearning Exact ML methods aim at eliminating the influence of the
Fig. 1 illustrates the general scenario for Machine unlearn- particular segment/proportion of data to be removed com-
ing. At its core, it is the art of selective forgetting. It revolves pletely. Generally, some degree of retraining is necessary.
around the principle of inverse operations, which aims to The major challenges encountered with this approach involve
mathematically reverse or negate the effect of certain data retraining efficiency and the subsequent performance of the
1 https://oag.ca.gov/privacy/ccpa
retrained model.
2 https://oag.ca.gov/privacy/ccpa The pioneering study in the domain of machine unlearning
3 https://www.ag.gov.au/rights-and-protections/publications/privacy-act- was conducted by Cao et al. [4], marking the initial introduc-
review-report tion of this concept. Their research entailed the proposal of
IEEE COMPUTER MAGAZINE 3

a heuristic methodology that allowed for the transformation algorithm and does not extend to other ML algorithms.
of an ML algorithm into a summation format, effectively In conclusion, there has been considerable investigation into
neutralizing the influence of data lineage. The non-uniform exact machine unlearning from diverse perspectives. Till now,
distribution of unlearning requests in practical applications a majority of the research has centered on rapid retraining
was also considered within their study. However, it merits techniques, with minimal or ideally no compromise in the
acknowledgment that not every ML algorithm can undergo performance of the model. Theoretically, the removed data
conversion into the summation format, and the data lineage can be validated through the retrained model, given that the
varying in granularity levels might not always be addressed. primary objective of exact machine unlearning is to physically
To enhance the precision of machine unlearning techniques, eliminate the influence of the excised data without disrupting
Bourtoule et al. [7] introduced an alternative method referred the retrained model.
to as SISA. This approach aims to ameliorate the universal- Discussion: Although exact machine unlearning techniques
ity and performance of the retrained model. The acronym have made substantial advancements towards actualizing the
SISA signifies “Sharded, Isolated, Sliced, and Aggregated right to erasure, researchers have pinpointed emerging chal-
training”, and it realizes machine unlearning via incremental lenges from two distinct viewpoints. Firstly, exact machine
unlearning on partitioned data. This technique involves the unlearning invariably induces a deterioration in model per-
categorization of data into several isolated shards, followed by formance, albeit occasionally the decay may be marginal.
further segmenting of the data within each shard. Subsequent Secondly, the prospect of privacy leakage is a concern. For
to this, incremental learning is implemented, and parameters instance, should a data entity named “Alice” be eliminated and
are archived. Upon receiving a new unlearning request, SISA a discrepancy is detected between the original and the retrained
navigates back to the relevant data slice and initiates retraining model, a hostile entity could infer that the observed difference
from that point. Given that the remaining model parameters is attributable to the removal of “Alice”. Consequently, this
have already been preserved, the process of aggregation be- could instigate an inference attack, aggravating the privacy
comes straightforward and efficient. Nonetheless, SISA could risk.
potentially experience a drop in performance in each com-
ponent model when datasets are not sufficiently large, or the
IV. A PPROXIMATE M ACHINE U NLEARNING
learning task is particularly intricate.
“DeltaGrad” [2], an exact machine unlearning methodology, To mitigate the challenges of exact machine unlearning,
emphasizes expedited retraining, as shown in Fig. 2. It accom- approximate machine unlearning methods have been devel-
plishes this by training the data designated for deletion in a oped, which try to mask the difference between models before
counteractive manner – that is, by maximizing the loss rather and after data removal while simultaneously optimizing model
than minimizing it, as is typical in traditional ML scenarios. performance.
The resulting reversed model is subsequently integrated with A prominent method in the realm of approximate machine
the original model, which is preserved beforehand. Through unlearning is the “certified-removal” approach as proposed by
the application of DeltaGrad, the performance of the model Guo et al. [5]. Their research illustrates a mechanism to ex-
can be well-sustained, given certain constraints on the propor- punge data points from 𝐿 2 -regularized linear models, provided
tion of data to be removed (1% as suggested in the DeltaGrad these models have been trained using a differentiable convex
publication). Nevertheless, although DeltaGrad is compatible loss function. The approach deployed by them utilizes a New-
with stochastic gradient descent ML algorithms, it lacks the ton step on the model parameters, which proficiently eradicates
capability to manage mini-batch sizes. the influence of the data point under deletion. Additionally,
The gradient descent method of machine learning is de- the residual is obscured by introducing random perturbations
picted in Fig. 2a. The red arrows descend towards a target into the training loss, thereby preventing potential privacy
point, minimizing the loss function. This is the traditional violations. Despite its groundbreaking nature, the proposed
optimization process of training an ML model. In contrast, methodology has certain restrictions. Firstly, implementing
machine unlearning, as shown in Fig. 2b, follows a different Newton’s optimization method requires the inversion of the
trajectory. The blue arrows ascend towards a target point, Hessian matrix, a task that can pose considerable challenges.
maximizing the loss function. This process seeks to emphasize Secondly, the technique is not adaptable to models with non-
and amplify the errors associated with the designated data convex losses. Lastly, the data-dependent bound does not
for deletion, ultimately removing their influence from the ML quite succeed in accurately estimating the gradient residual
model. norm, which underlines the need for further enhancements and
Rather than general-purpose machine unlearning method- rigorous scrutiny.
ologies, focused strategies tailored to specific ML algorithms To explore approximate machine unlearning for intricate
are currently being devised. To illustrate, Brophy et al. [8] models, notably deep neural network (DNN) models, Golatkar
presented “DaRE”, a technique specifically engineered for the et al. [9] introduced a forgetting Lagrangian to accomplish
random forest algorithm. Even though a minor performance selective forgetting in DNNs. Building upon this concept,
reduction in the model is observed, DaRE is capable of the authors devised a scrubbing method capable of erasing
swiftly retraining, enhancing efficiency by 2 to 4 orders of information from trained weights without the need to access
magnitude compared to completely starting anew. However, the original training data or mandate a complete retraining of
the application of DaRE is limited solely to the random forest the network. Moreover, they developed a computable upper
IEEE COMPUTER MAGAZINE 4

(a) Gradient Descend (Loss Minimization) (b) Gradient Ascent (Loss Maximization)
Fig. 2. Machine Learning (Loss Minimization) v.s. Machine Unlearning (Loss Maximization)

bound to measure the amount of residual information, a value In the field of approximate machine unlearning, there are
that can be efficiently computed for DNNs. Nevertheless, solutions purpose-built for specific ML models. For example,
when it pertains to forgetting without assuming prior training, Ginart et al. [11] have developed machine unlearning tech-
research has revealed that even slight perturbations during the niques that are custom-designed for K-means. The primary
crucial learning phase can engender substantial variations in objective of these methods is to strike an equilibrium between
the eventual solution. efficiency and model performance. It should be noted that
The aforementioned two research contributions imple- these methods are constrained to eliminating merely one
mented differential privacy mechanisms to disturb the retrained record per iteration. Moreover, the authors formulated four
model parameters, raising the inquiry of whether machine fundamental principles for the design of efficient machine
unlearning can be considered a special instance of differential unlearning methodologies. These guiding principles include
privacy within the field of machine learning. Sekhari et al. [10] linearity, laziness, modularity, and quantization.
tackled this matter by undertaking an exhaustive exploration Discussion: In summary, approximate machine unlearning
of generalization in machine unlearning, with the objective places a higher premium on preserving model performance
of achieving commendable performance on new data points. while introducing additional privacy safeguards to the retrained
In contrast to earlier research, the algorithms deployed in model. However, it confronts several challenges, including
this study do not require the unlearning algorithm to have the intricacy of verifying the implementation of machine
access to the training data during the deletion of samples. unlearning methodologies. In fact, based solely on output ob-
This study delineates a clear distinction between differential servations, it becomes nearly impossible to distinguish whether
privacy and machine unlearning; however, it does contain a the unlearning process has been enacted or not. Furthermore,
few limitations. Specifically, it does not provide dimension- since the majority of approximate machine unlearning models
dependent information-theoretic lower bounds on the deletion initiate with the model rather than the data, it prompts ques-
capacity and is unable to handle non-convex loss functions. tions regarding the congruity of this method with the “right to
In contrast to the perturbation-based methodologies, Nguyen be forgotten”, as stipulated in privacy laws and regulations.
et al. [6] proposed a machine unlearning technique for varia-
tional Bayesian machine learning, a significant sub-discipline
of machine learning. They accomplished this by integrating
Bayesian theory into the model’s prior and posterior knowl-
edge, thereby extending machine unlearning into the Bayesian
domain. Though employing stochastic methodologies, the au-
thors approached the issue from a probabilistic theory perspec-
tive, as opposed to the randomized noise injection method em-
ployed in differential privacy. This unlearning technique does
not compromise the model’s performance and is exclusively
applicable to this specific segment of ML models.
IEEE COMPUTER MAGAZINE 5

TABLE I
TAXONOMY OF M ACHINE U NLEARNING

Reference Dataset Methodology Performance Summary Security & Privacy

Cao et al. Collected Converting
[4] from algorithms into a • Efficiency: The idea of forgetting systems that can • Real-world data pollution
Facebook summation form restore privacy, security, and usability by attacks developed against
completely and quickly erasing data lineage systems and algorithms
for efficiently • Accuracy: The accuracy performance depends on
forgetting data the percentage of the deletion data
• Limitations: Not all ML algorithm supports
lineage
summation form, granularity issues

Bourtoule Purchase, SISA stands for

et al. [7] SVHN Sharded, Isolated, • Efficiency: The incremental unlearning mode of • Can well-resist membership
Sliced, and SISA is efficient inference attacks
• Accuracy: A combination of transfer learning and • Vulnerable to poisoning
Aggregated SISA yields a slight decrease in accuracy (˜2%) attacks
training with improved retraining time.
• Limitations: Hard to Verify and cannot deal with
Exact Unlearning

small datasets

Wu et al. MNIST, Storing the

[2] RCV1, original gradients, • Efficiency: Can quickly (2-6 times quicker) retrain • The storage of all
HIGGS retaining the machine learning models based on stochastic inter-media model
gradient ascend parameters pose a threat to
deleted data by • Accuracy: Negligible changes in performance in model inversion attacks
gradient ascend, response to small changes in the data • Gradient ascent for deletion
• Limitations: Extra storage, mini-batch not samples may lead to risks
and then
supported, requiring convexity of ML models to other similar records not
integrating to be deleted

Brophy et Adult, Machine

al. [8] HIGGS, unlearning for • Efficiency: Swift removal of training instances for a • Privacy leakage potential
Bank, etc. random forests novel variant of random forests called DaRE, with before and after deletion
a speed advantage of 2˜4 orders
• Accuracy: Slight performance degradation (less than
1%) based on in handling sequences of deletions on
13 binary classification datasets from the real world
• Limitations: Only applicable to Random Forest

Guo et al. MNIST Training a

[5] differentiable • Efficiency: Utilize a Newton step to effectively • Proposes to mask the
convex loss remove the influence of the removed data point on residual by randomly
Approx. Unlearning

the model parameters perturbing the training loss

function in a • Accuracy: Indistinguishable performance when the to prevent adversaries from
certified manner deleting sample size is within a certain range extracting information from
• Limitations: Difficult to invert the Hessian matrix, small residual
that is from
non-convex losses not supported, large disparity
L2-regularized
linear models
Golatkar et MNIST, Defining a
al. [9] CIFAR-10, forgetting • Efficiency: The selective forgetting method can • Propose a scrubbing
Lacuna-10 Lagrangian to improve the efficiency by careful design procedure that removes
• Accuacy: If the pretraining assumption is not information from the
achieve selective considered while forgetting, even minor trained weights without
forgetting for perturbations during the initial critical learning accessing the raw training
deep neural phase can result in significant variations in the final data or re-training from
solution scratch
networks • Limitation: Requires pre-training to guarantee
convergence
IEEE COMPUTER MAGAZINE 6

Reference Dataset Methodology Performance Summary Security & Privacy

Sekhari et Experi- Generalization in
al. [10] mental machine • Efficiency: Explore the number of samples that can • Do not rely on the
evaluation unlearning for be unlearned while preserving performance availability of training data
• Accuracy: Comparable performance of during sample deletion,
not performing well generalization in machine unlearning for distinguishing it from prior
provided on unseen data performing well on unseen data work
• Limitations: Fail to provide dimension-dependent • Show a strict separation
information-theoretic lower bound on the deletion between DP and machine
capacity, cannot deal with non-convex loss unlearning
functions

Ginart et MNIST, Machine

Approximate Unlearning

al. [11] CellType, unlearning for • Efficiency: Introduce and characterize the problem • Maybe vulnerable to
Botnet K-means while of efficient data deletion in machine learning membership inference
• Accuracy: Propose two solutions to achieve deletion attacks by manipulating the
synthesizing 4 efficiency in k-means clustering, offering theoretical center of K-means
general guarantees and strong empirical performance • The models are vulnerable
• Limitations: Can only unlearn one single data point to data sample
engineering
at one time, an assumption that one model addition/deletion
principles corresponds to one dataset

Nguyen et Airline, Machine

al. [6] Moon, unlearning for • Efficiency: Extend machine unlearning to efficient • New vulnerabilities of
Fashion variational and feasible Variational Bayesian Unlearning model fraud attacks
• Accuracy: Comparable performance after evaluating
MNIST Bayesian ML unlearning methods on Bayesian models, sparse
Gaussian process, logistic regression
• Limitations: vulnerable to attacks, only applicable
to variational inference frameworks of Bayesian
machine learning

Sommer et MNIST, Probabilistic

al. [12] CIFAR-10, verification of • Verification Efficiency: Acceptable verification • High risk of backdoor
Fashion machine efficiency by identifying the pre-injected backdoor injection for attacks instead
• Verification Accuracy: Superior accuracy based on of verification
MNIST, unlearning the verification performance using multiple datasets
etc. leveraging the • Verification Limitations: Constraints on data
backdoor samples, conflicting backdoor patterns, potential
vulnerability to backdoor detection mechanisms
Verification & Attacks

Chen et al. MNIST, Proposed member

[13] CIFAR-10, inference attacks • Attack Efficiency: Can efficiently launch the • Priavcy-breacching attack
Adult, etc. on machine proposed member inference attack to identify the existence of
• Attack Success Ratio: The attack can easily breach a record
unlearning the privacy of ML model-related unlearning but not • Quantify the privacy risks
the data-related unlearning like SISA in machine unlearning by
• Limitations: Not applicable to examining membership
data-manipulation-based unlearning methods, inference attacks
performance drops if the unlearned/updated samples
represent more than 0.2%

Marchant MNIST, Poisoning attacks

et al. [14] Fashion- on certified • Attack Efficiency: Requires multiple rounds of • Significant threats to
MNIST, machine poisoning operation with a balance of the attacker’s certified machine unlearing
objective optimality with the computation time and • Poisoning strategy can
etc. unlearning the feasibility of long-term attack execution evolve with unlearning
• Attack Success Ratio: Consider different factors process
such as white-box and grey-box attacks, various
perturbation geometries and bounds, etc
• Limitations: Only targets on a particular category
of machine unlearning models, not work well if
off-the-shelf anomaly detection models have been
deployed
IEEE COMPUTER MAGAZINE 7

V. V ERIFICATION AND ATTACKS OF U NLEARNING These gaps not only leave pressing questions unanswered but
A LGORITHMS also hint at the potential for fresh perspectives that might
Concurrent with the design of novel unlearning methods, reshape our current understanding. Our series of experiments,
mechanisms for systematically verifying data erasure and at- presented in this survey, are not merely a reiteration of
tacks on machine unlearning have been developed. During our known concepts but an endeavour to introduce novel ideas
survey, we discovered only one research contribution related to and concepts that have been largely unexplored in contem-
the former [12], with a handful of other contributions related porary discourse. By weaving these empirical findings into
to the latter [13], [14]. our review, we aim to break the traditional paradigm of
Regarding the verification mechanism, Sommer et al. [12] survey papers, offering readers not just a synthesis of existing
have conducted research on probabilistic verification of ma- knowledge but also tangible evidence of recent advances. This
chine unlearning. In cases where data needs to be erased, integration of experiments serves a dual purpose: it provides a
data owners intentionally implant a backdoor within the data more comprehensive view of machine unlearning and ensures
prior to transmitting it to the data user (for instance, Internet an engaging read by interspersing theoretical discussions with
titan corporations). Once the data user claims to have deleted practical insights.
the data, the data owner can corroborate the erasure by Specifically, our experiments are based on Deltagrad [2],
checking the previously inserted backdoor. In addition, there which is considered to be one of the most prominent and
exist several strategies aiding in the verification of machine precise machine unlearning techniques. As previously men-
unlearning, encompassing but not limited to data sampling tioned, the key approach employed in this study is “reverse
verification, synthetic data verification, and bias analysis. For learning”. To elaborate further, Deltagrad amplifies the loss of
instance, bias analysis involves assessing the effect of data the excluded data samples in the model training, instead of
elimination on the bias inherent in the ML model. This could using the conventional method that minimizes the loss.
entail gauging the fairness and accuracy of the model before
and after data removal or examining the data distribution A. Experiment Design
to discern the influence of data removal on the represen- Our evaluation procedure of machine unlearning is as fol-
tation of various groups. The mechanism has demonstrated lows. Firstly, we explore the relationship between the accuracy
its effectiveness across an array of datasets and networks. of the model and the percentage of deleted data samples
Nevertheless, a few limitations include restrictions on data in the independent and identically distributed (IID) scenario.
samples, conflicting backdoor patterns, risks associated with Secondly, we investigate the same correlation in the non-IID
backdoor injection, and potential susceptibility to backdoor situation. Thirdly, we assess the time consumption associated
attack detection mechanisms. with the proportion of the deleted data samples.
Membership inference is a type of privacy-oriented attack For our experiments, we used the original “Deltagrad”4
proposed against ML models [15], and it bears relevance to implementation and further extended it with new features to
machine unlearning. In their recent work, Chen et al. [13] accommodate the evaluation of non-IID deletion and accuracy
unveiled a novel method for executing membership inference of each class of different datasets5 . These evaluations and
attacks within the context of machine unlearning. This ap- related results were not considered in the original “Deltagrad”
proach allows them to ascertain whether a particular sample article. All presented results are the mean values over 20
was part of the training set of the original model, thereby rounds of experiments, with error bars showing the corre-
illuminating unforeseen privacy risks tied to machine unlearn- sponding standard deviations.
ing. However, the success rates of the attack decrease under
specific conditions, such as the application of the SISA method B. Experiment Environments and Settings
and when more than 0.2% of data samples require deletion. We conducted our experiments on a Windows-based ma-
Beyond the attacks discussed earlier, poisoning attacks are chine equipped with an Intel Xeon KVM [email protected] and
a prevalent form of assault in ML applications, inclusive of 16G of memory. Our model was a two-layer neural network
machine unlearning. Marchant et al. [14] have conceived an consisting of 300 hidden ReLU neurons, which we trained on
innovative poisoning attack that accounts for various factors. the MNIST dataset. The MNIST dataset comprises 10 labels,
These include white-box and grey-box attacks, diverse per- and contains 60,000 images for training and 10,000 images for
turbation geometries and limits, optimization of the attacker’s testing. Each image is composed of 28 × 28 features (pixels)
objective harmonized with computational time, and the fea- and represents a single digit ranging from 0 to 9. We applied
sibility of sustaining the attack over an extended duration. L2 regularization with a rate of 0.001 and utilized a decaying
Nonetheless, this attack is tailored specifically for a unique learning rate strategy. Specifically, we set the learning rate to
class of machine unlearning models, specifically, the certified- 0.2 for the first half of the training iterations and reduced it to
removal [5] and its variants. Its performance may deteriorate 0.1 for the remaining iterations. We also utilized deterministic
if standard anomaly detection models have been implemented. gradient descent as our optimization algorithm.
Regarding our hyperparameter configuration, we set 𝑇0 = 5
VI. E VALUATION AND E XPERIMENTAL R ESULTS as the period of explicit gradient updates, and 𝑗 0 = 10 as the
In the developing field of machine unlearning, there are 4 https://github.com/thuwuyinjun/DeltaGrad

discernible gaps and ambiguities in the existing literature. 5 https://github.com/DrQu89757/Extened code from DeltaGrad
IEEE COMPUTER MAGAZINE 8

(a) IID (b) non-IID

Fig. 3. Evaluation of accuracy in IID and non-IID settings

length of the initial “burn-in”. Specifically, for the two-layer

DNN, we set 𝑇0 = 2, which is even smaller, and we use the
first quarter of the iterations as the “burn-in”. The history size
m is set to 2 for all of our experiments.
In the following experiments, we will assess the model’s
performance by varying the deletion ratios, which are set
to 0.001, 0.005, 0.01, 0.05, 0.1, 0.2, and 0.3, respectively.
Furthermore, we study the effect of machine unlearning on
data samples deleted in a non-IID manner.

C. Experimental Results
This subsection presents our primary evaluation results,
including accuracy evaluation in both IID and non-IID settings,
as well as the inversion of time consumption. The models
examined in this evaluation are as follows:
• Baseline Training: training a model using the whole Fig. 4. Evaluation of time consumption
dataset (in Red color).
• Retraining from scratch: retraining a model using the
updated dataset after deletion (in green color).
• Deltagrad: retraining a model using the deleted samples
“unlearn” models is quite significant. As shown in Fig. 3b,
(in blue). the accuracy degradation of Deltagrad is from around 0.8735
1) Evaluation of accuracy in an IID setting to 0.7921 (over 8%) when the deletion ratio increases to 30%.
In order to evaluate the accuracy of the model under an IID This is mainly caused by the significant absence of label “4”.
setting, we randomly removed data samples using a uniform Noticeably, the classification accuracy of Label “9” increases
distribution. Fig. 3a illustrates that the testing accuracy of all from 82.9% (deletion ratio: 0.001) to 88.3% (deletion ratio:
three models remains consistent throughout this process. As 0.3).
a result, we can conclude that the baseline model’s accuracy
is not affected by this form of data deletion. However, for 3) Evaluation of time consumption
the other two models, the degradation in testing accuracy is
limited, decreasing from approximately 0.8735 to 0.8725 (less In Fig. 4, the time consumption v.s. deletion ratio in an IID
than 0.1%). setting is displayed. In particular, to show the crossover point,
2) Evaluation of accuracy in a non-IID setting we evaluate two more deletion ratio values, which are 0.4 and
To evaluate the model accuracy in a non-IID setting, we 0.5. With the increase in the deletion ratio from 0.001 to 0.3,
deleted more samples with the label “4” than the other labels. the time consumption of “Re-training from scratch” decreases
In more detail, 30% of the deleted samples are with the from 42.19s to 29.15s. Meanwhile, the time consumption of
label “4”, while the remaining 70% of the deleted samples “Deltagrad” increases from 12.53s to 21.22s. At last, there will
have a uniform distribution over the 9 labels without “4”. exist a cross-over point. It is worth mentioning that the time
In this non-IID setting, the performance degradation of two consumption trend in a non-IID setting is almost the same.
IEEE COMPUTER MAGAZINE 9

VII. I NSIGHTS AND C HALLENGES - Stratified Unlearning: Similar to stratified sampling, one
This section shares some insights on the state of machine could look into stratified unlearning, where data is removed
unlearning based on our analysis and evaluation, and discusses in a way that maintains the original distribution of the dataset
further challenges and potential approaches. as much as possible.
- Meta-Learning: Future research could explore meta-
learning, where models are trained not just on the task at hand
A. Insights but also on how to forget or unlearn effectively.
Training from scratch or not? Given the negligible - Dynamic Learning Rates: Adjusting learning rates dynami-
degradation of accuracy comparing training-from-scratch with cally during the unlearning process based on the nature of the
Deltagrad in the IID setting, we can conclude that the machine data being removed could be beneficial. This might help in
unlearning methods should be selected according to other handling the complexities arising from non-IID data removal.
performance metrics such as computational complexity. Broader implications for models that frequently undergo
Non-IID machine unlearning is challenging. It has been updates and deletions, especially in evolving and dynamic
widely acknowledged that non-IID data distribution in dis- environments:
tributed learning paradigms, such as federated learning, leads - Continuous Re-evaluation: Models in dynamic environ-
to suboptimal model performance. In the context of unlearn- ments should be re-evaluated regularly. This is crucial because,
ing, the removal of non-IID samples also results in a tremen- with every unlearning and retraining cycle, the model’s per-
dous performance loss, which must be taken into consideration formance may change unpredictably due to the non-IID nature
in practice. of updates.
Unlearning may lead to Unfairness. When data samples - Scalability Concerns: As models grow in size and com-
are erased, particularly in a non-IID manner, the classification plexity, the overhead of frequently updating and deleting
accuracy of certain labels can significantly degrade while the becomes a challenge. Ensuring that the unlearning process is
accuracy of others may improve. This phenomenon would efficient and scalable is crucial.
cause unfairness across different classes, which cannot be re- - Privacy Concerns: In some cases, unlearning is driven by
vealed by traditional ML performance metrics such as average the need to remove sensitive or personal information. With
accuracy. non-IID data, ensuring complete removal becomes complex,
Unlearning calls for novel and more generalized loss leading to potential privacy breaches.
function formulation methods. The majority of the present - Model Robustness: If a model is trained on a diverse set
machine unlearning methods rely on maximizing a quadratic of data and then frequently updated with non-IID data or has
loss function using Newton’s method. Nonetheless, in practical non-IID deletions, its robustness can be compromised. This
scenarios, various forms of loss functions exist. Exploring the can make the model more susceptible to adversarial attacks or
uses of these alternatives may lead to the design of more produce unexpected outputs in unfamiliar scenarios.
adaptable machine unlearning mechanisms in the future.
VIII. S UMMARY
B. Challenges In summary, machine unlearning holds great promise for
Conceptual reasons why non-IID data can complicate addressing concerns of privacy and equity in machine learning.
unlearning: In this short survey article, we have presented a comprehen-
- Data Heterogeneity: In a non-IID setting, certain data sive examination of current machine unlearning techniques,
points or classes may be overrepresented in the data that are including both exact and approximate unlearning methods, and
to be unlearned. This can skew the model’s understanding, compared various techniques. The merits and shortcomings
particularly if the unlearning process inadvertently strips away of each method have been discussed, and our experimental
vital information about underrepresented classes. evaluations have demonstrated that machine unlearning can
- Model Stability: When training on IID data, models remove sensitive data or biases from a trained model without
often converge to a stable set of weights. With non-IID data, significantly impairing its performance, particularly in IID
especially during unlearning, the model’s weights can oscillate settings. However, there exist significant hurdles to overcome,
or diverge due to the uneven distribution of the deleted data. such as developing robust methods for non-IID sample dele-
This can make the unlearning process unpredictable. tion and evaluating the effectiveness of machine unlearning
- Loss Landscape: Non-IID data can introduce sharp, non- techniques against potential attacks. We believe that further
convex regions in the loss landscape. When unlearning, nav- research in this field will continue to advance the state-of-
igating this complex terrain can lead to getting stuck in sub- the-art in machine unlearning, allowing for the development
optimal local minima or experiencing larger jumps in loss. of more robust and trustworthy machine-learning systems that
Potential mitigations and areas of future research to meet the needs of diverse stakeholders.
address this issue:
- Regularization Techniques: Implementing regularization R EFERENCES
methods, like dropout or L1/L2 regularization, may help in
[1] E. Chien, C. Pan, and O. Milenkovic, “Efficient model updates for
ensuring that the model does not overfit to the non-IID nature approximate unlearning of graph-structured data,” in International Con-
of the data being removed. ference on Learning Representations, 2023.
IEEE COMPUTER MAGAZINE 10

[2] Y. Wu, E. Dobriban, and S. Davidson, “Deltagrad: Rapid retraining

of machine learning models,” in International Conference on Machine
Learning. PMLR, 2020, pp. 10 355–10 366.
[3] P. Voigt and A. Von dem Bussche, “The EU general data protection
regulation (GDPR),” A Practical Guide, 1st Ed., Cham: Springer Inter-
national Publishing, vol. 10, no. 3152676, pp. 10–5555, 2017.
[4] Y. Cao and J. Yang, “Towards making systems forget with machine
unlearning,” in 2015 IEEE Symposium on Security and Privacy. IEEE,
2015, pp. 463–480.
[5] C. Guo, T. Goldstein, A. Hannun, and L. Van Der Maaten, “Certified
data removal from machine learning models,” in Proceedings of the 37th
International Conference on Machine Learning, 2020, pp. 3832–3842.
[6] Q. P. Nguyen, B. K. H. Low, and P. Jaillet, “Variational bayesian un-
learning,” Advances in Neural Information Processing Systems, vol. 33,
pp. 16 025–16 036, 2020.
[7] L. Bourtoule, V. Chandrasekaran, C. A. Choquette-Choo, H. Jia,
A. Travers, B. Zhang, D. Lie, and N. Papernot, “Machine unlearning,”
in 2021 IEEE Symposium on Security and Privacy (SP). IEEE, 2021,
pp. 141–159.
[8] J. Brophy and D. Lowd, “Machine unlearning for random forests,” in
International Conference on Machine Learning. PMLR, 2021, pp.
1092–1104.
[9] A. Golatkar, A. Achille, and S. Soatto, “Eternal sunshine of the spotless
net: Selective forgetting in deep networks,” in Proceedings of the
IEEE/CVF Conference on Computer Vision and Pattern Recognition,
2020, pp. 9304–9312.
[10] A. Sekhari, J. Acharya, G. Kamath, and A. T. Suresh, “Remember
what you want to forget: Algorithms for machine unlearning,” Advances
in Neural Information Processing Systems, vol. 34, pp. 18 075–18 086,
2021.
[11] A. Ginart, M. Guan, G. Valiant, and J. Y. Zou, “Making ai forget you:
Data deletion in machine learning,” Advances in neural information
processing systems, vol. 32, 2019.
[12] D. M. Sommer, L. Song, S. Wagh, and P. Mittal, “Athena: Probabilistic
verification of machine unlearning,” Proceedings on Privacy Enhancing
Technologies, vol. 3, pp. 268–290, 2022.
[13] M. Chen, Z. Zhang, T. Wang, M. Backes, M. Humbert, and Y. Zhang,
“When machine unlearning jeopardizes privacy,” in Proceedings of the
2021 ACM SIGSAC Conference on Computer and Communications
Security, 2021, pp. 896–911.
[14] N. G. Marchant, B. I. Rubinstein, and S. Alfeld, “Hard to forget:
Poisoning attacks on certified machine unlearning,” in Proceedings of
the AAAI Conference on Artificial Intelligence, vol. 36, no. 7, 2022, pp.
7691–7700.
[15] R. Shokri, M. Stronati, C. Song, and V. Shmatikov, “Membership
inference attacks against machine learning models,” in 2017 IEEE
Symposium on Security and Privacy (SP), 2017, pp. 3–18.