Received: 5 September 2023

DOI: 10.1049/stg2.12161

REVIEW
- -
Revised: 1 February 2024 Accepted: 20 February 2024

- IET Smart Grid

Smart grid public datasets: Characteristics and associated

applications

Emran Altamimi1 | Abdulaziz Al‐Ali2 | Qutaibah M. Malluhi1,2 | Abdulla K. Al‐Ali1

1
Department of Computer Science and Engineering, Abstract
Qatar University, Doha, Qatar
The development of smart grids, traditional power grids, and the integration of internet of
2
KINDI Center for Computing Research, Qatar things devices have resulted in a wealth of data crucial to advancing energy management
University, Doha, Qatar
and efficiency. Nevertheless, public datasets remain limited due to grid operators' and
Correspondence
companies' reluctance to disclose proprietary information. The authors present a
Emran Altamimi. comprehensive analysis of more than 50 publicly available datasets, organised into three
Email: [email protected] main categories: micro‐ and macro‐consumption data, detailed in‐home consumption data
(often referred to as non‐intrusive load monitoring datasets or building data) and grid data.
Funding information Furthermore, the study underscores future research priorities, such as advancing synthetic
Qatar National Research Fund, Grant/Award data generation, improving data quality and standardisation, and enhancing big data
Number: NPRP12C‐33905‐SP‐66; Qatar National
Library management in smart grids. The aim of the authors is to enable researchers in the smart and
power grid a comprehensive reference point to pick suitable and relevant public datasets to
evaluate their proposed methods. The provided analysis highlights the importance of
following a systematic and standardised approach in evaluating future methods and directs
readers to future potential venues of research in the area of smart grid analytics.

KEYWORDS
building management systems, data analysis, power consumption, power grids, SCADA systems, smart metres,
smart power grids

1 | INTRODUCTION infrastructures (AMI) and the use of smart metres in the SG.
The data generated from IoT devices in the SG is characterised
Smart grids (SGs) are intelligent electric network models that by its enormous volume, wide varieties, varying sampling rate,
incorporate the actions of all connected end users, including veracity, and value range [5]. These data can be grouped into
internet of things (IoT) devices [1]. This infrastructure enables three categories: customer data, grid data, and external data.
seamless communication between users and grid operators, Customer data refers to any type of information about a
supporting various applications, such as self‐healing, automa- customer, such as energy consumption and other related data.
tion of the power grid, and integration of distributed energy Examples include non‐intrusive load monitoring (NILM)
resources (DER) [2]. SGs generate a massive, constant stream datasets and smart metering data. Table 1 summarises the
of data from various sources, such as customer data, grid data, consumer data categories and their respective features.
and external data [3]. The power system has become signifi- Grid data include all information about the electricity grid,
cantly more complex with the integration of DER, electric such as specifications for generation plants and DER, the
vehicles, and demand response (DR) techniques [4]. Advanced distribution grid, the transmission grid, electrical substations,
data analytics algorithms are required to process this data and energy storage, and supervisory control and data acquisition
derive valuable insights for SG operations and services. (SCADA) system data, which refer to data coming from a wide
IoT devices play an important role in the data generation range of sensor types (e.g. wide‐area measurement systems,
process, as seen in the incorporation of advanced metering intelligent electronic devices, power quality analysers, and pole

[Correction added on 16 October 2024, after first online publication. The references 53, 56, 57, 104, 105, 106 and 107, and their citations are deleted from the article.]

-
This is an open access article under the terms of the Creative Commons Attribution License, which permits use, distribution and reproduction in any medium, provided the original work is
properly cited.
© 2024 The Authors. IET Smart Grid published by John Wiley & Sons Ltd on behalf of The Institution of Engineering and Technology.

IET Smart Grid. 2024;1–28. wileyonlinelibrary.com/journal/stg2 1

 25152947, 0, Downloaded from https://ietresearch.onlinelibrary.wiley.com/doi/10.1049/stg2.12161 by Universitätsbibliothek Erlangen- Nürnberg, Wiley Online Library on [23/10/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
2
- ALTAMIMI ET AL.

TABLE 1 The categories and features related to consumer data.

Data categories for consumer data Features

Energy consumption Energy consumption in a particular time interval. Might be collected from appliances/plug
loads/phase loads or the aggregated load at a house level/building level or neighbourhood
level.

Electrical measurements Voltage, current, power, power factor. Might be collected from appliances/plug loads/phase
loads, or the aggregated load at a house level.

Metre data Status, ID, circuit ID and section (metre's location within the grid), manufacturer, installation
date, reprogramming messages, service points, GIS data

Customer account data Contracted power (maximum power contracted), type and status billing information (e.g., late
payments), pricing rates, fraud history, price, peak load, and load factor

Financial data Market and billing data

mounted auto‐recloser). SCADA data includes grid assess- datasets, NILM datasets, and grid datasets. This review
ments, voltage, current, power factor, alarm data such as un- emphasises publicly available data and facilitates the identi-
solicited openings, details pertaining to repaired faults, control fication of relevant datasets for specific research questions
commands, circuit outages, transmission loss, network quality or analyses, addressing the challenge of selecting the most
data measurement flows data, set points, and event logs. The suitable data sources in a rapidly evolving field.
use of this comprehensive dataset extends to various applica- � Analyses the features and characteristics of SG datasets,
tions, such as system control, monitoring, alarm processing, elucidating their applications and relevance in various
protection, and event management. Table 2 summarises the research contexts, including IoT‐based energy management
sensors, types of measurement, and purpose of the data. solutions. A comparative analysis of the features, strengths,
External data sources include regional meteorological and and weaknesses of various datasets is presented, enabling
weather data. Geographic information systems (GIS) and researchers to make informed decisions when selecting
temperature data are often used in research. The integration of appropriate data sources for their studies.
IoT devices in the SG enables various applications on both the � Examines the preprocessing methodologies, feature engi-
grid side, such as DR, and the consumer side, including home neering techniques, and evaluation procedures employed by
energy management systems (EMSs), ambient assisted living researchers fostering a deeper understanding of best prac-
(AAL), and appliance anomaly detection. tices in the field. This also aims to mitigate potential pitfalls
The vast variety and immense number of datasets and data in the utilisation and handling of diverse datasets, promoting
sources, the diverse number of applications that can be based a more robust and rigorous approach to research in IoT‐
on them, and the scarce availability of studies conducted on driven SG systems.
publicly available datasets in comparison to studies conducted
on private datasets stifles research in the field. This research
work aims to address this issue by reviewing the characteristics, 1.2 | Previous work
issues, and applications of existing public datasets in detail.
Such work will help researchers identify research gaps and In this section, we examine notable literature reviews and
directions. surveys in the domain, providing a concise overview of the
existing knowledge in this field.
The work in ref. [11] investigates the SG architecture for
1.1 | Contributions the study of software reliability engineering. The article cites
and discusses the characteristics of 15 datasets, which can be
The rapid transition of the power sector towards more sus- used for reliability engineering, and divides them into three
tainable and efficient smart grid systems, enhanced by the main categories: Loss of loading probability, power distribu-
integration of IoT technologies, has resulted in a complex and tion, and hardware. However, the article does not offer a
data‐rich environment. This work is motivated by the pressing detailed analysis of these datasets and their characteristics.
need to guide researchers through this intricate landscape. By The comprehensive study by ref. [12] presents 13 con-
offering a comprehensive review and comparative analysis of sumer datasets and their characteristics while exploring deep
smart grid datasets, we seek to simplify dataset selection for learning techniques applied to load analysis, forecasting and
specific applications. This paper makes several important management systems. The challenges associated with imple-
contributions to the field of electrical grid research, data menting deep learning techniques are discussed as well as
analysis, which are concisely outlined as follows: potential solutions to enhance performance. Furthermore, the
authors identify five open research issues concerning the future
� Offers an extensive and systematic review of data sources in of SGs. In a related review paper, ref. [13] focuses on data
the electrical grid domain, encompassing smart metre analytics applications of smart metre data, featuring 10 datasets
 25152947, 0, Downloaded from https://ietresearch.onlinelibrary.wiley.com/doi/10.1049/stg2.12161 by Universitätsbibliothek Erlangen- Nürnberg, Wiley Online Library on [23/10/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
ALTAMIMI ET AL.
- 3

TABLE 2 The devices, measurements and applications relating to SCADA systems.

Source of data Measurements and features Applications

Wide area monitoring system and Voltage, current, power, phase angle and harmonics Validate models and identify parameters, ensure dynamic
phasor measurement units [6] stability, state estimation, and system control and
protection (e.g., controlled‐island protection) [7]

Remote terminal unit Local collection points for collecting sensor reports. Delivering commands to control relays

Digital fault recorder Records and classifies faults (ex. Power swings, frequency Faults recording and classification
fluctuations, and time of fault)

Fibre bragg grating sensor Wavelength shift Overheating, sag, vibration, and galloping prediction

Hall effect sensor Voltage and magnetic field Speed detection, current sensing, proximity switching, and
positioning

Power quality analyser [8] Voltage and current levels, power factor, frequency, Record power parameters and power interruptions such as
waveform distortions, harmonics, flicker, phase under/over voltage, sags, swells, and noise
imbalance, voltage sags and swells, transient events,
outages, crest factor, energy consumption, load
patterns, interharmonics and inrush current.

Sagometer Temperature Line sagging

Transformer sensors Voltage, current, temp, partial discharge, load tap changer Preventive maintenance
values, oil pressure, tripped situations discharge
ground and short circuit current, etc [9]

High voltage line temperature Temperature Preventive maintenance

Intelligent electronic devices Status changes in substation and outgoing feeders Relay protection

Capacitor sensors Voltage, current, volt‐amps reactive, and harmonic Capacitor's bank control and monitoring
monitoring

Pole mounted auto reclosers Pick up events details Fault diagnosis and prognosis [10]

Magnetoresistive sensor Modulation, frequency, I, P, and energy Electromagnetic interference monitoring in substations

with general characteristics (e.g. number of records, frequency provide examples of research efforts that demonstrate the
and duration) and corresponding references. Although both practical application of these datasets. In contrast, the authors
reviews contain useful information, neither delves into exten- in ref. [18] focus on publicly available test distribution net-
sive detail about these public datasets, which would be bene- works with features in the United States, characterising them
ficial for researchers seeking suitable datasets for their studies. and identifying their use cases. Although providing valuable
Iqbal et al. [14] provide a comprehensive review of 42 information, its scope is limited to public grid datasets with US
NILM datasets, detailing their characteristics and statistical features, leaving a broader perspective unexplored.
information. However, the authors do not discuss NILM ap- A comparison of the review articles and the contribution of
plications or reference research articles that utilised these our article is provided in Table 3.
datasets. In contrast, the study in ref. [15] reviews several
NILM datasets and their characteristics, while also mentioning
the types of NILM approaches they permit, such as event‐ 1.3 | Methodology
based or event‐less methods. Despite these insights, the re-
view does not elaborate on how the datasets were used or the The methodology followed is to construct three different
specific techniques that were applied. comprehensive search strings for each data types. We used six
In the work of ref. [16], the models of the tools and the major search libraries namely IEEE Xplore, ScienceDirect,
datasets that can be used to operationalise local energy com- Wiley Online Library, SpringerLink, MDPI, and ACM digital
munities in practice were reviewed. The reviewed use cases are library. The search string used for macro and micro‐level
of interest to stakeholders but do not specify particular ap- consumption data is ((“smart metre” OR “energy consump-
plications of the data. The mentioned datasets consist of tion” OR “system level” OR “substation”) AND (“smart grid”
demand‐side data and climate‐related data, with specified OR “power grid”) AND (“public dataset” OR “publicly
characteristics. However, the specific uses of these datasets available”) AND (“dataset”)) and it returned a total of 275
were not referenced. articles. For the second type, which is, detailed in‐home con-
The review paper [17] discusses publicly available distri- sumption data we used the search string ((“Buildings” OR
bution and transmission grid datasets, detailing their charac- “Non‐intrusive load monitoring” OR “NILM”) AND (“public
teristics and intended usage. However, the work does not dataset” OR “publicly available”)) which returned 250 articles.
 25152947, 0, Downloaded from https://ietresearch.onlinelibrary.wiley.com/doi/10.1049/stg2.12161 by Universitätsbibliothek Erlangen- Nürnberg, Wiley Online Library on [23/10/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
4
- ALTAMIMI ET AL.

TABLE 3 Comparison of current work with existing survey papers.

Review
article Scope of datasets reviewed Depth of analysis Applications and research gaps highlighted
[11] Focused on datasets for software Limited analysis of dataset characteristics No detailed discussion on applications or
reliability in SG methodologies

[12] Consumer datasets with a focus on Moderate detail on dataset characteristics Some discussion on challenges and solutions in deep
deep learning applications learning for SGs

[13] Smart metre data analytics applications General characteristics of datasets reviewed Lacks depth in dataset usage and application‐specific
details

[14] 42 NILM datasets Detailed statistical information of datasets Lacks discussion on NILM applications or specific
research articles

[15] NILM datasets with a focus on event‐ Moderate detail on dataset characteristics Limited elaboration on dataset employment and
based or event‐less methods techniques

[16] Tools and datasets for local energy Specifies dataset characteristics but not in‐depth Lacks detail on specific data applications
communities

[17] Public distribution and transmission Detailed characteristics and intended usage The intended applications for each dataset are not
grid datasets discussed

[18] US‐featured public grid datasets Characterisation of datasets and use cases Limited to US datasets and lacks broader perspective

Our Comprehensive review covering smart Extensive analysis of dataset features, characteristics, Detailed discussion of preprocessing methodologies,
work metre, NILM and grid datasets and applications in smart and power grids feature engineering, evaluation procedures, and
identification of research gaps

Finally, for grid datasets we used the following search string

((“grid dataset” OR “test system” OR “benchmark grid” OR
“Representative grid” OR “Generic grid”) AND (“smart grid”
OR “power grid”)) which returned 501 articles. In addition to
the public datasets published, the search strings return research
articles that utilised public datasets to evaluate their proposed
approaches. Since datasets are still relevant regardless of the
time they were released we opted to keep all articles regardless
of the year the articles were published. Lastly, irrelevant articles
are excluded based on the title and abstract of the article. The
new numbers of articles for Macro and Micro‐level datasets,
detailed in‐home consumption datasets, and grid datasets were
192, 149 and 320 articles respectively. The datasets used for FIGURE 1 Road map of the paper.
evaluation by the articles that remained in the final set were
then extracted. The dataset characteristics and associated ap- improve grid efficiency and long‐term viability by providing
plications were then extracted from the datasets' meta data and valuable information on energy consumption, electrical mea-
articles that utilised the datasets for various applications. surements (e.g. voltage, current, power factors) [19, 20], metre‐
The rest of this paper is organised as follows, Sections 2–4 related issues, and outage data. Metre data management sys-
discuss micro and macro consumption data (which includes tems maintain records about each metre, such as its status,
both smart metre and system‐level data), detailed in‐home manufacturer, installation date, malicious behaviour, and
consumption data and grid data, respectively. Each section dis- reconfiguration data, as well as circuit installation locations and
cusses the applications, public datasets, and reviews the literature service point details. Service points represent the interface be-
on the most popular datasets in each category. Sections 5 and 6 tween the utility supply and a site's wiring system. Furthermore,
highlight data issues and conclude the paper. A graphical rep- customer account information, including contracted power,
resentation of the structure of the paper is shown in Figure 1. type, status, irregularities history, and billing data, can be lever-
aged for load forecasting by clustering similar customers [21]. A
comprehensive summary of this information can be found in
2 | MACRO AND MICRO‐LEVEL Table 1. Pablo et al. concluded in their work on 311K customers
CONSUMPTION DATA in Uruguay that complementary customer information and geo‐
localisation complement the consumption signal and are rele-
Smart metre data is the most commonly utilised in SG analytics, vant features [22]. While smart metre data provides granular
with a wide range of applications. They typically record data at insights and allows for nuanced interventions and measures,
intervals of 10–60 min. Smart metre data can significantly system‐level or macro‐data carries its own significance. Macro
 25152947, 0, Downloaded from https://ietresearch.onlinelibrary.wiley.com/doi/10.1049/stg2.12161 by Universitätsbibliothek Erlangen- Nürnberg, Wiley Online Library on [23/10/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
ALTAMIMI ET AL.
- 5

data provides a holistic view of consumption patterns in larger important when dealing with missing values, considering the
sections of the grid and are critical for high‐level planning, rate of missing values and the cause of failure.
management, and forecasting [23]. Conversely, micro or smart‐ In ref. [45], the authors used Prophet by Facebook to
metre data offers detailed load profiles of individual house- define anomalies and evaluated classification models in the
holds, presenting opportunities for customised energy efficiency Ausgrid residential dataset. The best performance was achieved
strategies and DR programs. using the Random Forest classifier. In ref. [46, 47], the authors
However, despite differences in scale and granularity, many used modified generative adversary networks (GANs) and
applications, such as load forecasting, anomaly detection, and removing variational autoencoder‐based techniques to impute
load management, incorporate an overlap between macro and missing values and anomaly detection, respectively, on the
micro data. For example, while load forecasting at the micro GEFCom 2014 dataset [48].
level informs individual household energy management stra-
tegies, at a macro level, it aids in power generation planning
and grid stability measures. The methodologies developed for 2.1.2 | Non‐technical loss detection
these applications can often be applied interchangeably be-
tween the two scales, although with adjustments to account for Non‐technical loss detection focuses on identifying discrep-
the inherent differences. ancies between energy injected and electricity paid for. It is
Therefore, given the considerable overlap in applications closely related to anomaly detection but evaluates users' load
and in order to maintain coherence and efficiency in our profiles in the same neighbourhood or previous benevolent
presentation, we have elected to group both the smart metre profiles to detect anomalies [30]. The only publicly available
(micro‐level) and system‐level (macro) datasets under the known labelled dataset for this purpose is the State Grid
umbrella of “Macro and Micro‐Level Consumption Data”. Corporation of China (SGCC) dataset [49]. Another approach
This arrangement streamlines the discussion, eliminates focuses on detecting abnormal behaviours in a private manner,
redundancy, and underscores the interconnected nature of data such as the work in ref. [50].
analysis at different scales within the context of the smart grid.
This section discusses the most popular applications and
public datasets and their characteristics. 2.1.3 | Load profiling

Load profiling aims to understand users' or groups' typical pat-

2.1 | Consumption data applications terns of electricity use, which is valuable for DR programs and
prospective load forecasting [32]. Load profiling helps to better
Yi Wang et al. categorised the applications of smart metre data comprehend socio‐demographic factors and target potential
into three main categories; load analysis, load forecasting [24] consumers for DR programs. Another research direction ex-
and load management [25–27]. Load analysis applications plores the development of privacy‐preserving techniques and
include bad data detection [28, 29], non‐technical loss detec- integrity assurance mechanisms for load profiling in SGs, with
tion [30, 31] and load profiling [32–34]. Other applications the goal of safeguarding sensitive smart metre data and main-
include data compression [35], privacy [36–38] and outage taining the accuracy of outsourced data analytics processes [51].
management [39].

2.1.4 | Load forecasting

2.1.1 | Bad data detection
Load forecasting is essential in the electric power industry for
Bad data detection, or anomaly detection, is a crucial pre- operations, planning, pricing, procurement, and hedging de-
processing technique that improves data quality and the ac- cisions. Load forecasting can be long‐term or short‐term, with
curacy of models and analytics by handling missing values or different use cases for each. Preprocessing techniques in load
correcting/removing outlier data. Smart metre data are time forecasting include smoothing and imputation, feature extrac-
series data, so existing techniques for time series data can be tion and selection, and clustering [52]. Various techniques are
applied. However, traditional short‐term load forecasting used, such as artificial neural networks, time series analysis,
(STLF) methods for imputing bad data have limitations [40]. bottom‐up approaches, SVM, and regression.
Probabilistic approaches also face challenges in determining
optimal rejection thresholds, especially for large datasets
[41–43]. 2.1.5 | Load management
There is a lack of publicly available datasets with labelled
ground‐truth fine‐grained anomalies for the SG context, Load management can provide better and more personalised
except for EnerNOC [44], which has a limited number of services by collecting sociodemographic information. Key as-
anomalies. Anomaly detection work is typically divided into pects include customer base load estimation and tariff design.
two steps: defining or injecting synthetic anomalies and Customer base load estimation evaluates the effectiveness of
implementing an anomaly detection technique. Imputation is DR programs by estimating load profiles without the program.
 25152947, 0, Downloaded from https://ietresearch.onlinelibrary.wiley.com/doi/10.1049/stg2.12161 by Universitätsbibliothek Erlangen- Nürnberg, Wiley Online Library on [23/10/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
6
- ALTAMIMI ET AL.

The literature is categorised into similar‐day methods, applied on 1122 of the consumers as part of an experiment
regression‐based methods, and morning‐consumption‐ carried out over the year 2013. The data set consists of the
adjustment methods [53]. New approaches using high‐ following:
frequency data, such as clustering‐based methods, improve
performance. Tariff design, on the other hand, is essential in 1) Energy consumption (in kWh) sampled from smart‐metres
balancing consumer response and utility provider profits. at 30 min frequency for each consumer. Data were
Clustering consumers is an important first step, followed by collected for a total of 12 months during the experiment
solving optimisation problems based on each cluster's load (i.e. when the dynamic time‐of‐use tariffs was in effect), in
profiles [54]. The real‐time price determination problem aims addition to 6 months before and 2 months after the
to maximise profits for SG retailers. Price bidding in the SG experiment period.
plays a crucial role in demand side management by allowing 2) Appliance survey that includes information such as number
consumers to participate in electricity markets actively. By of appliances, physical parameters of the household (e.g.
submitting price bids for electricity use, consumers can influ- insulation, number of rooms etc.) and basic details of the
ence the market price of electricity, encouraging energy savings occupants (e.g. number of occupants, age categories etc.).
and peak load reduction. This interactive process not only Data include 990 records from the group that opted for the
empowers consumers but also helps stabilise the grid by dynamic time‐of‐use tariffs, and 1870 from the group that
aligning energy usage with real‐time supply and demand did not.
conditions. 3) Attitudes survey to assess the change in consumption
behaviour of the group that opted for the experiment, such
as the factors that made them more likely to change their
2.2 | Smart metre and system‐level datasets behaviour. Seven hundred fourteen records were received.

In this section, we present a comprehensive review of all public The privacy of consumers was preserved by doing the
smart metre and/or region datasets identified in the literature following:
to the best of our knowledge at the time of writing this article,
examining their characteristics, features, associated applica- 1) Identifying information such as names, locations, and ad-
tions, and privacy considerations. The information is sum- dresses was omitted.
marised in Tables 4 and 5. Table 4 summarises the datasets 2) ID keys were generated randomly.
commonly used in the literature for certain applications. 3) The surveys were manually checked for any inadvertent
inclusion of personal details.

2.2.1 | Low Carbon London With the help of historical data prior to the implementation
of DR programs, some baseline load estimation algorithms are
Low Carbon London (LCL) dataset [55] is an open dataset that developed on this dataset [53]. The high frequency data also led
involved 5567 consumers. Dynamic time‐of‐use tariffs was to works in long and STLF as well [68, 69].

TABLE 4 Applications and corresponding public datasets.

Application Datasets
Load forecasting Low Carbon London (LCL) [55], PecanStreet [56], UMass smart* [57], ausgrid distribution
network [58], customer behaviour trials (CBT) [59], AEMO [60], ERCOT [61], building
data genome project [62], energy market authority of Singapore [63], EnerNOC [44],
GEFCom2012 [64]

Demand side management, price bidding, and power market Ausgrid distribution network [58], customer behaviour trials (CBT) [59], ISO new england
design [65], energy market authority of Singapore [63], ERCOT [61]

Solar panel generation and net demand forecasting Ausgrid distribution network [58]

Equipment failure modelling and voltage regulation Ausgrid distribution network [58], UMass smart* [57]

Descriptive analysis and building characteristics Building data genome project [62], energy market authority of Singapore [63]

Energy storage research EnerNOC [44], PecanStreet [56], AEMO [60]

Anomaly detection and concept drift aware algorithms UCI ElectricityLoadDiagrams20112014 [66], customer behaviour trials (CBT) [59]

Web‐of‐things studies and appliances management PecanStreet [56]

NILM, NIOM, and data compression research PecanStreet [56], UMass smart* [57]

SGCC dataset [67] Energy theft detection [67]

Renewable energy effects and solar panel simulation AEMO [60]

 25152947, 0, Downloaded from https://ietresearch.onlinelibrary.wiley.com/doi/10.1049/stg2.12161 by Universitätsbibliothek Erlangen- Nürnberg, Wiley Online Library on [23/10/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
ALTAMIMI ET AL.
- 7

2.2.2 | PecanStreet � Solar‐TK: contains solar energy generation data from 81

homes in the US.
The PecanStreet dataset [56] is supported by the Pecan Street � Solar panel: includes 50 rooftop solar panels energy gener-
experiment in Austin, TX. Pecan Street's research network is the ation data at a 1‐min interval.
world's only real‐world electricity‐gas‐water testbed. It covers � SunDance: includes 1 year's data of 100 solar sites in North
over 1000 homes without renewable energy sources, 250 homes America in 2015. Net metre, solar generation, and weather
with solar panels, and 65 owners of electric vehicles. The energy data were collected at a frequency of 1 sample per hour.
generated and used in each home is monitored in real time, down � Physical‐Black‐Box Model: This dataset includes weather
to the circuit level. Energy is used, generated, and stored in high and normalised solar generation data to build the physical
resolution at a frequency of a reading per second to per minute black‐box model. The files also include code to model
(both at the whole‐home level and the individually monitored shading effects.
appliance circuits level). The experiment aims to understand the
effects of modern technologies such as EMSs and DR programs Applications of this dataset include a privacy‐preserving
such as time‐of‐use pricing. architecture developed in ref. [75] while still meeting the util-
The studies done using this dataset are numerous and span ities' needs to achieve a net metering goal. The solution uses
across various applications; however, this dataset is best suited the concept of Zero‐Knowledge proofs and provides crypto-
for: Web‐of‐Things studies (due to the diversity and sparsity of graphic guarantees for the integrity, authenticity, and accuracy
the collected data) [70], NILM [71], appliance scheduling and of payments, while permitting changeable pricing without
management (since individual appliances were monitored) [72], disclosing the power measurements acquired throughout a
design controllers for solar panel energy storage devices [73], as billing period. NILM and NIOM algorithms development can
well as customer baseline load estimation [74]; due to availability be done on this dataset, because of the availability of circuit
of solar panel loads and DR programs that affect consumption as and appliance level consumption data, as well as the ground
in the LCL dataset. For privacy preservation, PecanStreet au- truth for detecting occupancy status [76]. Solar panel data can
thors mentioned that they implement “commercially reasonable also be used as auxiliary data for distribution grid management
security measures” for privacy and data protection. However, no algorithms such as voltage regulation as in ref. [77]. The high
details were mentioned on their privacy policy websites. frequency polling of data per minute also prompted some
researchers to study data compression algorithms such as the
work done in ref. [78].
2.2.3 | UMass smart*

The UMass Smart* [57] dataset is a collection consisting of the 2.2.4 | Ausgrid distribution network: residential
following 9 subsets: and substations

� DeepRoof dataset: Satellite images of building roofs and the The Ausgrid distribution network records and publishes four
planar segmentation of each. types of datasets [58]:
� Apartment dataset: Aggregated energy consumption of 114
single family apartments for the period of 2014–2016, � Electricity consumption: Ausgrid has grouped the yearly
together with their associated weather data. Readings were residential and non‐residential electricity consumption data
sampled once per minute. by local government areas (LGA), total of 32 areas, in its
� Home dataset (2017 release): The aggregated and individual distribution system. High‐voltage customers and supply
circuit consumption of 7 households collected at a per services such as public lighting and bus shelters are not
minute interval over multiple years. included in these data.
� NIOM (Non‐intrusive‐occupancy‐monitoring) dataset: � Solar panels and electricity consumption: A sample of 300
aggregated consumption at minute level for a 3 week period solar customers from Ausgrid's electricity network area was
for two households of two occupants each, with the ground randomly selected, all of whom were billed on the domestic
truth occupancy status. tariff and possessed a gross metered solar system
� Home dataset (2013 release): This dataset focuses on depth throughout the duration from 1 July 2010 to 30 June 2013.
instead of breadth. That is, only three houses were moni- To compile the data, metre reading processes were
tored. However, the data included information about con- employed to obtain a comprehensive dataset of actual
sumption (per circuits and aggregate, individual metres, electricity consumption and production at half‐hour in-
dimmable and non‐dimmable switches), two electrical phase tervals for the selected customers during the specified
data (voltage and frequency), environmental data (indoor period. Customers who fell at the extremes of household
and outdoor), oven and door status, energy generation data consumption and solar generation performance during the
(solar panels, wind, and battery voltage), and motion de- first year of the study were excluded. Solar homes with
tector data. The dataset also includes micro‐grid dataset of rooftop solar systems connected to the grid through a gross
443 homes over a single‐day period. metering configuration account for 2657 of the monthly
 8

TABLE 5 Data‐sets characteristics.

-

Dataset Geographic location Number of consumers Data duration Data frequency Dataset characteristics Privacy considerations
Low Carbon London 5567 12 months during the 30 min Energy consumption from Omitted identifying
London (LCL) experiment, 6 months smart metres, appliance information, randomly
before and 2 months after survey, attitudes survey generated ID keys, manual
check of surveys for
personal details

PecanStreet Austin, TX Over 1000 homes without Real‐time monitoring (1‐min) High‐resolution, from a reading Monitors energy usage, No specific details provided
renewable energy, 250 homes per second to per minute generation, and storage at
with solar panels, and 65 the whole‐home level and
electric vehicle owners individually monitored
appliance circuits level

UMass smart United States (specific Varies depending on the subset Aggregated and individual Privacy‐preserving architecture
locations vary energy consumption, circuit‐ using zero‐knowledge
depending on the level data, solar generation proofs.
subset) data, weather data,
occupancy status, voltage
and frequency data,
environmental data, oven
and door status, motion
detector data, shading effect
modelling code

Ausgrid distribution Ausgrid's distribution 32 LGAs, 300 solar customers, Yearly data: 2010–2013; monthly Yearly, half‐hourly intervals for Electricity consumption, solar Identifying information
network: system, Australia 2657 solar homes, 4064 non‐ data: 2007–2014; substation solar customers, 15‐min generation, substation load removed
Residential and solar homes, and 180 data: since 2005, regularly intervals for substations, and profiles, and power outage
substations substations updated; past outages: quarterly for past outages records
Published every 3 months

Customer Ireland 5375 households 18 months Electricity consumption every Split into benchmarking and test Not mentioned
behaviour half an hour period; four different groups
trials (CBT) with different time of use
tariffs; includes a survey of
143 questions about
household characteristics

The State grid China 42,372 (3615 thieves and 38,757 1 January 2014–31 October Daily consumption readings Labelled for malicious activities Not mentioned
corporation of honest consumers) 2016 (1035 days) (electricity theft)
China (SGCC)
dataset

ISO new englanda New England, USA Not specified Since 2003 Hourly System‐level hourly load data, Not mentioned
temperature data, location
regional prices, market
clearing prices, interchanges
with other power systems
for 9 zones
ALTAMIMI
ET AL.

25152947, 0, Downloaded from https://ietresearch.onlinelibrary.wiley.com/doi/10.1049/stg2.12161 by Universitätsbibliothek Erlangen- Nürnberg, Wiley Online Library on [23/10/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
 ALTAMIMI
ET AL.

T A B L E 5 (Continued)

Dataset Geographic location Number of consumers Data duration Data frequency Dataset characteristics Privacy considerations
a
AEMO [60] 5 Australian states 5 states Since 1998 Half‐hourly Aggregated demand data, Select profile values may be
electricity price data manipulated if deemed
sensitive.

ERCOTa [61] Texas, USA 4 regions Since 2001 Hourly Market and grid information Requires submission request for
access

GEFCom2012 [64] United States 20 zones 1 July 2003–30 June 2008 Hourly Temperature data from 11 Not specified
weather stations, holiday
data

Building data University campuses 507 whole (non‐residential) February 2014––April 2016 Hourly Electrical metre data, gross floor Not specified
genome (mostly) buildings size, primary use type,
project [62] meteorological information

Energy market Singapore Not specified Since 2004 30‐min intervals System demand data, historical Not specified
authority of market prices
Singaporea [63]

EnerNOC [44] Not specified 100 commercial/industrial sites 2012 5‐min intervals Energy consumption data Anonymised measurement
values, identifying
information removed

UCI Electricity Portugal 370 customers 2011–2014 4 readings per hour Residential and commercial Not specified
LoadDiagrams buildings and consumers
20112014 [66]
a
The dataset only includes system‐level data.
- 9

25152947, 0, Downloaded from https://ietresearch.onlinelibrary.wiley.com/doi/10.1049/stg2.12161 by Universitätsbibliothek Erlangen- Nürnberg, Wiley Online Library on [23/10/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
 25152947, 0, Downloaded from https://ietresearch.onlinelibrary.wiley.com/doi/10.1049/stg2.12161 by Universitätsbibliothek Erlangen- Nürnberg, Wiley Online Library on [23/10/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
10
- ALTAMIMI ET AL.

electricity data points. The Ausgrid Distribution Network reported in the study, 82% reported making some changes in
also provides monthly electrical data. Data are provided for their consumption patterns and 74% reported drastic changes
the period from 1 January 2007 to 31 December 2014 and, in their households. The trial reported noticeable drastic
as a result, it includes periods of household electricity changes in 38% of the consumers.
consumption prior to the installation of the solar system.
Furthermore, a data set of 4064 non‐solar homes is pro-
vided for the same time period to compare electricity con- 2.2.6 | The state grid corporation of China
sumption patterns between the two datasets.
� Ausgrid substation data: Since 2005, Ausgrid has provided SGCC [67] released the daily electricity consumption of 42,372
public access to the load profiles of approximately 180 zone consumers in the period from 1 January 2014 to 31 October
substations through their website, with regular updates that 2016 for a total of 1035 days (with a consumption reading per
ensure the data set remains current. Each entry in the day). The dataset is also labelled for malicious activities for a
dataset contains the year, zone substation name, date, and total of 3615 thieves. The other 38,757 consumers are labelled
corresponding data unit, followed by a full day's worth of as honest consumers. Labelling electricity theft acts as the
measurements at 15‐min intervals. ground truth to evaluate models.
� Past outages: Power supply interruptions that affect 50 or
more customers and last for more than 5 min are recorded
in the database and published quarterly. The dataset contains 2.2.7 | Independent system operator New Englad
information on the start time, average duration of the
outage in minutes, number of consumers affected, and its Every month since 2003, the independent system operator
potential cause. The data are organised by LGA as is done (ISO) New England publishes [65] system‐level hourly load
for the electricity consumption subset. data, as well as corresponding temperature data, regional
location prices, market clearing prices and interchanges with
Energy management solutions such as storage and DER other power systems for 9 different zones. The market data
scheduling and customer baseline load estimation can be allows for studies in power market design [81], price bidding
implemented in the solar panels and electric consumption (res- [82], as well as price forecasting.
idential) dataset. Electricity consumption, solar panel genera-
tion, and net demand forecasting can also be implemented in this
dataset. For the Ausgrid substation and past outages datasets, 2.2.8 | Australian energy market operator
aggregated load forecasting, and demand side management (e.g.
planning of charging infrastructure [79] and electrical distribu- The Australian Energy Market Operator (AEMO) [60] serves
tion system planning), and modelling equipment failure (e.g. as the main entity responsible for overseeing the management
power transform failures and retirement statistics [80]) are the and operation of electricity and gas networks, as well as price
most common applications. Customers in these datasets have determination, in five states in Australia. This organisation
been deidentified and do not represent a statistically significant maintains a comprehensive dataset that includes aggregated
sample of residential customers in the Ausgrid network area, nor demand data and electricity price data for these states, with
have they been subjected to detailed occupancy checks. temporal granularity provided at a half‐hourly rate. However, it
should be noted that beginning in November 2021, the reso-
lution of these data experienced a significant enhancement,
2.2.5 | Customer behaviour trials with the frequency of data points increasing to every 5 min.
Data have been updated and available since 1998. The research
Customer behaviour trials (CBT) dataset [59] consists of 5375 done on the datasets focused mostly on STLF. However, some
households electricity consumption data recorded every half an descriptive analysis work was also done. For example, in ref.
hour for the span of 18 months. The data was collected by the [83], the effects of wind and solar panel generation on
Commission for Energy Regulation of Ireland. The objective wholesale electricity prices were studied. The authors in ref.
of the CBT dataset is to evaluate smart‐metre technology time‐ [84] used the dataset to design the optimal battery capacity of
of‐use tariffs and different demand side strategies. Therefore, solar panels. The authors also simulated the hourly generated
the data was divided into two phases: the benchmark period solar panel power and made it public [85].
(6 months) and the test period (12 months). In the trial, four
different groups were assigned different time of use tariffs. A
survey (of 143 questions) on household characteristics is also 2.2.9 | Electric reliability council of Texas
included. The survey aims to depict the socio‐demographic
characteristics of the household; employment status, house- The Electric Reliability Council of Texas (ERCOT) [61] is an
hold size, age, and the social class. Given that consumers were ISO responsible for overseeing the state's electrical trans-
incentivised to change their behaviours through demand‐side mission and distribution network, serving over 25 million
strategies, the authors believe that the dataset is a good customers. Since its inception in 2001, ERCOT has managed
benchmark to develop. Concept drift aware algorithms. As the deregulated wholesale electricity market and has provided
 25152947, 0, Downloaded from https://ietresearch.onlinelibrary.wiley.com/doi/10.1049/stg2.12161 by Universitätsbibliothek Erlangen- Nürnberg, Wiley Online Library on [23/10/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
ALTAMIMI ET AL.
- 11

various datasets to the public, including real‐time and day‐ have been anonymised. However, the values were shifted on a
ahead market data, transmission and generation data, and linear scale to ensure consistency of the comparison over time
renewable energy data. These datasets encompass energy pri- and across sites. The data set has been used in the design of
ces, demand, and generation capacity for the entire ERCOT energy storage systems [93, 94] and load forecasting [95].
region, divided into four load zones. Access to ERCOT
datasets requires a submission request through their website.
These datasets have been utilised for various purposes, such as 2.2.14 | UCI ElectricityLoadDiagrams20112014
STLF and price forecasting (e.g. ref. [86]).
The ElectricityLoadDiagrams20112014 is a real‐world dataset
from Portugal [66]. The dataset has a resolution of 4 readings
2.2.10 | Global energy forecasting competition per hour from 2011 to 2014 for 370 customers. The dataset
2012 (GEFCom2012) includes residential and commercial buildings and consumers.

The Global Energy Forecasting Competition 2012 (GEF-

Com2012) [64] is a hierarchical load forecasting contest for a 2.3 | Discussion
utility located in the United States with 20 zones (from 1 July
2003 to 30 June 2008). The dataset includes temperature data While datasets from the UK, US, Australia, Ireland, and Portugal
from 11 weather stations and the holidays of that time period. provide valuable insights into energy consumption, the majority
The authors in ref. [64] reviewed the winning solutions. originate from developed nations in the Northern Hemisphere.
This overrepresentation may limit the applicability of research
outcomes to the diverse energy landscapes of developing
2.2.11 | The building data genome project countries, particularly those in the Southern Hemisphere, where
different economic, infrastructural, and climatic conditions
The Building Data Genome Project [62] consists of 507 whole prevail. The geographic concentration of these datasets suggests
(non‐residential) building electrical metres data from February potential limitations in the generalisability of research findings.
2014 to April 2016, most of which come from buildings on The distinct energy consumption patterns, regulatory frame-
university campuses. The dataset also includes different works, and customer behaviours specific to the United States
distinctive meta‐data such as gross floor size, primary use type, may not be directly transferrable to other global contexts. This
and meteorological information. The dataset was developed limitation underscores the need for a more diverse compilation
primary to test various algorithms and feature extraction of datasets that encapsulate the variegated nature of energy
techniques. Such use cases include load forecasting, load systems across different regions and cultures to truly harness the
shape/profile clustering, and synthetic load data creation [87] universal applicability of smart grid analytics. The issue is
and inference of buildings' characteristics [88]. particularly relevant in developing countries, where infra-
structural, economic, and policy differences shape distinct en-
ergy dynamics. Emerging markets often prioritise expanding
2.2.12 | Energy market authority of Singapore energy access, diverging from patterns found in more developed
nations. Consequently, a more inclusive dataset collection is
The Energy Market Authority of Singapore publishes numerous imperative for globally relevant smart grid analytics.
statistics pertaining to the grid operation [63], notably the system Additionally, the analysis of dataset utilisation across
demand data polled at 30‐min intervals since the beginning of different applications as demonstrated in Figure 2 reveals a
2004 and historical market prices. The datasets were used for pronounced emphasis on load forecasting, particularly within
reliability analysis [89, 90], descriptive analysis (for example, system‐level datasets that benefit from frequent updates. This
analysis of customers responding to socioeconomic de- trend aligns with the critical role that load forecasting plays in
terminants [91]) and demand‐side bidding [92]. the operational planning and reliability of the electrical grid.
Load forecasting's predominance in the literature is indicative
of its foundational importance in grid management and the
2.2.13 | EnerNOC value placed on accurate and timely predictions.
Furthermore, the synthesis of the dataset characteristics
EnerNOC [44] collected 5‐min energy consumption data for and their respective applications into a coherent framework
2012, for 100 commercial/industrial sites. EnerNOC is the presents an opportunity for a targeted approach to dataset
only dataset that we are aware of that has labelled anomaly utilisation. Table 6, which delineates the most popular datasets
data, which can be used as the ground truth for developing for each application, serves as a practical guide for researchers
anomaly detection algorithms. However, further examination and practitioners in the field. By identifying the datasets most
revealed that the number of anomalies in the dataset is arguably suited to specific applications, this summary aids in the effi-
negligible (no more than 11 instances out of more than cient allocation of analytical efforts and resources.
100,000 readings). For privacy, the real measurement values In conclusion, the analysis of smart metre and system‐level
and identifying information such as geolocations and floor area datasets highlights the centrality of certain applications in
 25152947, 0, Downloaded from https://ietresearch.onlinelibrary.wiley.com/doi/10.1049/stg2.12161 by Universitätsbibliothek Erlangen- Nürnberg, Wiley Online Library on [23/10/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
12
- ALTAMIMI ET AL.

FIGURE 2 The count of articles that utilised the public datasets for particular applications.

TABLE 6 The most common public dataset for each application.

Application Most common datasets

Load forecasting ISO New England, ElectricityLoadDiagrams20112014 and Australian energy market operator

Load profiling The building data genome project, pecan street and LCL

Energy theft State grid corporation of China (SGCC), ausgrid and customer behaviour trials (CBT)

Anomaly detection Pecan street, ISO New England and ausgrid

Load management ISO New England, Australian energy market Operator and electric reliability council of Texas
(ERCOT)

DER solutions Pecan street, ausgrid and Australian energy market operator

EV Pecan street, Australian energy market operator and LCL

smart grid analytics and the geographic concentration of electrical measurements taken at a very high sampling rate at
dataset origins. The field stands to benefit from an expansion the plug load, individual circuits in the house, and/or the main
of data sources that better represent the global diversity of line. Data may also include environmental measurements (e.g.
energy systems and from leveraging the specialised utility of temperature), auxiliary data, and information about events such
each dataset. This dual approach can enhance both the breadth as occupancy status (i.e. how many occupants are inside at any
and depth of insights in smart grid analytics, fostering ad- given time) and switches. The availability of labelled power
vancements that are both innovative and inclusive. events allows for event‐based approaches in energy disaggre-
gation, in contrast to event‐less approaches when power events
are not labelled. The datasets might also include information
3 | DETAILED IN‐HOME on the weather both inside and outside the building. This high
CONSUMPTION DATA frequency is different from smart metre datasets, which typi-
cally take measurements every 10–30 min, with most of the
NILM systems provide an efficient way to monitor multiple commercial smart metre's sampling at less than 1 Hz. Gao
appliances without the need for submonitoring, hence the et al. [97] suggested a 4 KHz threshold for a feasible and
name NILM. This section focuses on datasets that enable such reliable classification of appliances in energy disaggregation.
systems which are commonly referred to as buildings' datasets Higher sampling frequencies of the electrical measure-
[96] or NILM datasets [15]. NILM datasets contain data from ments enable features such as transient information, voltage‐
 25152947, 0, Downloaded from https://ietresearch.onlinelibrary.wiley.com/doi/10.1049/stg2.12161 by Universitätsbibliothek Erlangen- Nürnberg, Wiley Online Library on [23/10/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
ALTAMIMI ET AL.
- 13

current trajectories, electrical noise, and (power, reactive po- allowing early detection of minor failures and more efficient
wer, and distortion power) trajectories. Although low sampling maintenance strategies.
frequencies can be used to achieve some NILM applications,
transient analysis cannot be performed, limiting overall per-
formance and the range of applications that can be used. 3.1.4 | Ambient assisted living
Voltage, current, and power variables are the features that are
most important in low‐sampling‐frequency NILM datasets, AAL focuses on products and services that improve the lives
with reactive power being the distinctive feature that is most of elderly adults and promote their physical independence.
frequently used in research. NILM systems can facilitate AAL without the need for
If the aggregate metering of electricity consumption is not obtrusive monitoring [101].
available and only the measurements of individual appliances is
available, the data lack a ground truth for evaluating and testing
energy dis‐aggregation models. Therefore, the dataset is used 3.1.5 | Appliance anomaly detection
only for training, while the evaluation is performed on other
datasets. The naive method of aggregating all the appliances Detecting anomalous appliances using NILM techniques is
does not serve as a ground truth since most of the appliances more cost‐effective and practical than using individualised
in the house are not monitored. If the appliances events are metres per appliance [102, 103]. However, further development
labelled, then such datasets might be used for event is needed to improve the effectiveness of NILM‐based
classification. anomaly detection [104, 105].
This section discusses the most popular applications and
public datasets in NILM datasets.
3.2 | Detailed in‐home consumption public
datasets
3.1 | Detailed in‐home consumption data
applications The NILM datasets have been extensively reviewed in the
literature, for example, the authors in ref. [106] provided a
NILM datasets are primarily used for developing algorithms to comprehensive review of 29 existing open datasets, in terms of
disaggregate total consumption into individual appliances. The settings (residential or otherwise), measurement level (whole
output of energy disaggregation systems can be used for premises, individual appliances, and/or individual circuits),
purposes such as reducing energy consumption, preventing electrical and auxiliary measurements, time period, event labels
appliance failures, forecasting SG consumption peaks, and availability and file format. The authors in ref. [16] reviewed 22
monitoring daily living activities [98]. open datasets providing the country, the number of house-
holds/sites in the dataset and the sampling rate. In the work of
ref. [15], 26 datasets were reviewed that provide the same in-
3.1.1 | Energy disaggregation formation as the work of ref. [106], in addition to the country
of origin. A critical review of all NILM datasets was published
The energy disaggregation process typically involves three in ref. [14] in 2021, in which 42 datasets were comprehensively
stages: Event detection, feature extraction, and load identifi- reviewed. The datasets were divided into high‐frequency, low‐
cation [99]. Event detection captures appliance state transi- frequency, and synthetic datasets. Providing the same charac-
tions, while feature extraction uses steady‐state, transient, and teristics mentioned in refs. [15, 16, 106] in addition to the name
non‐traditional event detection approaches to extract relevant and number of appliances measured.
features [100]. Table 7 reviews (24) NILM datasets with respect to their
measurement levels and frequency, measured quantities and
sampling rate, and the applications for the datasets.
3.1.2 | Energy management system
EMS combines hardware and software to monitor and control 4 | GRID DATA
energy consumption and generation within a home, helping
consumers save on utility bills while maintaining comfort Electrical grid data prove invaluable for examining typical grid
levels. DR solutions incentivise customers to actively control operating conditions and analysing grid behaviour during
their energy demand based on market prices. failures and disturbances. Furthermore, it facilitates the
investigation of microgrids in islanding conditions, where the
microgrid is disconnected from the main grid, as well as the
3.1.3 | Condition‐based maintenance integration of renewable energy sources. The electrical grid
encompasses power generation, transmission, and distribution
Condition‐based maintenance monitors equipment conditions components, and grid data in the literature enables the
and performs maintenance tasks based on equipment status, emulation of electrical measurements and sensors using various
 25152947, 0, Downloaded from https://ietresearch.onlinelibrary.wiley.com/doi/10.1049/stg2.12161 by Universitätsbibliothek Erlangen- Nürnberg, Wiley Online Library on [23/10/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
14
- ALTAMIMI ET AL.

TABLE 7 Measurement levels and frequency, measured quantities and the sampling rate, and the applications for 24 NILM datasets.

Measured
Dataset Measurement levels quantities Labelled events Applications
REDD [107] AGG (15 KHz), IC P, V, I, S STLF, load disaggregation
(0.5 Hz), IA (1 Hz)

UK‐ AGG (1 Hz), IC, IA P, V, I, S Energy dissaggregation, behaviour analytics

DALE (per 6 s)
[108]

BLUED [109] AGG, IC, IA (all at P, V, I, Q State transitions Event detection

12 KHz)

AMPds [110] AGG, IC, IA (all at 1 P, V, I, Q, f, STLF, load disaggregation

sample per minute) phi, Pt

ECO [111] AGG, IA (both at 1 Hz) P, V, I, Q Occupancy status Occupancy detection, energy dissaggregation

Tracebase IA (per 10 s) P Steady and transient state analysis of appliances,

[112] energy dissaggregation.

HES [113] IA (per 2 min) P Behaviour and descriptive analytics

iAWE [114] AGG, IC, IA (all at 1 Hz) P, V, I, Q Only 1 day, time‐stamped data Supervised energy dissaggregation techniques

GREEND IA (1 Hz) P Energy dissaggregation, EMS

[115]

PLAID‐I [116] IA (30 KHz) V, I Load identification at plug level

REFIT [117] AGG, IA (both per 8 s) P Energy dissaggregation, behaviour and descriptive
analytics

COMBED AGG, IC (both at per 30 s) P, I Energy dissaggregation for commercial building

[118]

DRED [119] AGG (per 1 min.), P Occupancy (room level per 1 min.) Location aware energy dissaggregation
IA (1 Hz)

WHITED IA (44.1 KHz) V, I High frequency energy dissaggregation

[120]

HFED [121] IA (10 and 5 MHz) Electro‐ Energy dissaggregation generalisation Developing energy dissaggregation models that are
magnetic beyond lab settings able to generalise beyond lab settings
interference

SUSTDataED AGG (12.8 KHz), IA V, I State transitions and occupancy energy dissaggregation and occupancy detection
[122] (50 Hz) measurements

ACS‐fx [123] IA (per 10 s) P, V, I, phi Low frequency load identification

COOLL [124] IC (12 KHz) V, I 20 variations of dictinct energy Energy dissaggregation

consumption profiles of 12
appliances.

Dataport [125] AGG, IC (both at 1 Hz and P, I Energy dissaggregation

per min.)

BLOND [126] AGG (50 and 250 KHz), P, Q High frequency energy dissaggregation
IA (6.4 and 50 KHz)

RAE [127] AGG, IC (both at 1 Hz) P, V, I, Q Energy dissaggregation

BERDS [128] AGG, IA (per 20 s) P, Q, S Energy dissaggregation

EEUD [129] AGG (per 1 min.) P Simulating and analysing electricity consumption
for residential building

I‐ AGG (per 1 min.) P, V, I, pf, f Occupancy status per 10 min Occupancy detection, energy dissaggregation
BLEND
[130]

Abbreviations: I, current; P, power; pf, the power factor; phi, the phase angle; Pt, total power; Q, reactive power; S, apparent power; V, voltage.
 25152947, 0, Downloaded from https://ietresearch.onlinelibrary.wiley.com/doi/10.1049/stg2.12161 by Universitätsbibliothek Erlangen- Nürnberg, Wiley Online Library on [23/10/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
ALTAMIMI ET AL.
- 15

tools. It is worth noting that researchers often employ inter- 4.1.2 | State estimation
changeable terms when referring to grid datasets, such as
network, case, system, and grid. There are several terms for State estimation determines the state of the power grid from
grid datasets that are not always used consistently in the imperfect measurements, used for online applications like se-
literature, due to a lack of standardisation [17]: curity analysis, anomaly detection, and fault diagnosis, or off-
line purposes like planning [157]. With the advent of the SG,
� Test systems: A simple grid built for the purpose of state estimation is increasingly important for distribution
demonstrating a single problem or performing basic vali- grids [158].
dation or testing. Synthetic [131, 132] or real grids [133, 134]
are named test systems. IEEE case 9 [135] and ICPSs [136–
139] are examples of test systems. 4.1.3 | Power flow analysis
� Benchmark grids: Grids where the aim is to compare and
evaluate different algorithms. For example, the CIGRE Power flow analysis examines the flow of power in a networked
systems [140] and the authors of ref. [141] presented system, analysing steady‐state operations of power systems and
benchmark systems. However, it is worth noting that the optimising power flow for efficiency [159].
IEEE test cases are typically used as a benchmark (e.g. for
power flow analysis), which highlights the issue of the
interchangeable use of grid terms. 4.1.4 | Reliability and stability analysis
� Representative grids: Are grids that represent real grids and/
or a set of grids that share similar characteristic (e.g. rural Reliability analysis studies the life cycle of components and the
grids). Such grids bridge the gap between technical findings system level, while stability studies examine the steady state and
and real‐world grids [142]. transient stability of power grids [160].
� Generic grids: The work in ref. [143, 144] used the term
generic to refer to a grid where different parameters can be
tweaked to generate various grids. However, the term was 4.2 | Transmission and distribution grids
synonymous with representative grid in the work of
ref. [145]. The transmission grid is responsible for delivering the load
� Synthetic grids: Grids that are neither models of real grids over long distances from a generating site to electrical sub-
nor derived from a real‐world grid. stations, while the distribution grid is responsible for delivering
energy to consumers. The authors of ref. [161, 162] classify the
This section discusses the most popular applications and data collected from the grid into:
public datasets for grid data.
� Standard equipment (e.g. transformers, switch gears, circuit
breakers, storage batteries, transmission cables, and ccables)
4.1 | Applications � Technical parameters (e.g. transformers and capacitor rat-
ings, voltage levels, and number of buses)
Grid datasets are used for various applications such as plan- � Cost and maintenance data
ning, stability analysis, reliability analysis, state estimation, and � GIS data of the power lines, service points, and buildings
power flow analysis [146]. The SG paradigm has expanded � Substations data and locations
research opportunities in the effective integration of DER and � Parcel use category (e.g. residential)
storage devices within the power grid, focusing on assessing
the impact of incorporating these elements and evaluating their There are several works that reviewed available grid data-
potential to reduce generation costs, smooth power generation sets. The work in ref. [17] has reviewed steady‐state distribu-
curves, and maintain sustainable service reliability for tion grid datasets highlighting the intended use case. The
users [147]. authors in ref. [146] reviewed the IEEE and CIGRE bench-
mark test systems, highlighting the applications done on each.
A review of distribution test systems in the United States is
4.1.1 | Planning presented in ref. [18]. The authors analysed IEEE test systems,
Pacific Northwest National Laboratory test systems, Electric
Power system planning faces challenges such as generation Power Research Institute representative systems, and the Pa-
expansion planning (GEP) and transmission expansion plan- cific Gas and Electric Company (PG&E) grids. The IEEE PES
ning (TEP), which involve determining the ideal combination Working Group on Cascading Failure [163] provides a
of technology, location, and building time for new generation comprehensive review of test systems providing the intended
units and power lines [148]. Both GEP and TEP are formu- use case and technical details on the test grids. This section
lated as optimisation problems with constraints such as the provides a comprehensive concise summary of the most
electricity market, congestion, uncertainties, and other con- popular grid datasets along side the intended use cases and
siderations [149–156]. popular applications for these datasets.
 25152947, 0, Downloaded from https://ietresearch.onlinelibrary.wiley.com/doi/10.1049/stg2.12161 by Universitätsbibliothek Erlangen- Nürnberg, Wiley Online Library on [23/10/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
16
- ALTAMIMI ET AL.

The Test Feeder Working Group originally released five control schemes. Modifying the test systems to allow for
test feeders: IEEE 4, 13, 34, 37, and 123 bus test feeders. Test different analyses is possible [146].
feeders are synonymous with test systems with the exception In 2010 a 8500‐bus test feeder was published to represent
that test feeders have only one power source while test systems a full‐size distribution system [180], still allowing for the same
incorporate multiple power sources. They were intended to intended use cases in Table 8. The test system was also used in
benchmark power flow algorithms, however, various analysis time series load modelling [181] and DER integration in the
and research was conducted on the five test feeders originally SG [182].
released [164]. The test feeders are not representative of large Three test feeders and systems were published to tackle
and complex distribution grids and were small to medium specific scenarios and to subvert common assumptions.
radial feeders. In 2010, a sixth test feeder, called the IEEE Table 9 summarises the test feeders and systems with the
Comprehensive Test Feeder, was added to model various intended use case and common applications.
components of the grid and transformers in particular [165]. Texas A&M university hosts several datasets on their
The feeders are comprised of overhead lines and underground website [191] for electric grid test cases that cover a variety of
cables, voltage regulators, shunt capacitors, and various degrees systems and scenarios, and are crucial in different power sys-
of load unbalance [164]. Table 8 summarises the intended use tem analyses. These datasets do not contain Critical Energy
cases of the original feeders and other prominent applications Infrastructure Information (CEII), making them widely
in the datasets. accessible for research purposes.
Since then, several benchmark test systems have been made Among the datasets are the latest synthetic electric grid
public to serve as a standardised dataset to test various cases of 2023, which include a smaller self‐contained island test
methods and algorithms [179]. All IEEE 9, 14, 30, 39, 57, 118, case for the Hawaiian island of Oahu with a synthetic 138/
300, and Reliability Test Systems (RTS)‐24 and RTS‐73 test 69 kV transmission network. For larger‐scale scenarios, there
systems allow for power flow, state estimation, and planning are datasets such as the Texas Synthetic Grid, which covers the
studies. However, only IEEE RTS‐24/73 allows for reliability ERCOT portion of Texas with a 6717‐bus transmission
analysis and IEEE 39 for stability analysis and development of network, and the Combined East‐West US Grid, representing a

TABLE 8 The original IEEE test systems and their respective intended use case and common applications.

IEEE original test systems Intended use case and common applications
4‐bus Transformer modelling testing. State estimation [166] and step‐voltage regulators [167].

13‐bus Testing power flow convergance in unbalanced systems. Optimal capacitor placement [168],
control of renewable energy batteries in microgrid [169], and islanding detection in
microgrids [170].

34‐bus A test system that requires voltage regulators to comply with ANSI voltage standards. Optimal
distributed generator placement [171] and optimal placement of storage systems [172].

37‐bus Capability of software to solve for the less common three‐wire delta systems. Power flow analysis
with DER [173], distributed generators for providing reactive power [174], and micro‐grid
small signal analysis [175]

123‐bus Minimising voltage drops with voltage regulators and shunt capacitor. Power flow analysis in
unbalanced systems, operational planning for self‐healing action [176], stochastic reactive
power management in microgrids with renewable energy [177]

CTF Capability of software to solve for a variety of components in one system. Distributed generation
applications [178]

TABLE 9 Test feeders and systems, highlighted characteristic, and the intended use case and common applications.

Test feeders and

systems Highlighted characteristic Intended use case and applications
Neutral‐earth‐voltage test The neutral conductor is not reduced by Kron reduction [183] Study neutral voltages in case of connection failures. Harmonic
feeder because the neutral voltage is above zero. analysis [184] and load modelling [185]

Low voltage network test A low voltage highly meshed system that represents typical urban Tests software capability to handle highly meshed systems.
system areas. The system is also referred to as 342‐bus LVNTS. Economic dispatch with DER integration [186] and planning
of communications systems [187].

European low voltage test Represents a typical feeder in Europe and the first feeder to Tests software capability to solve for various test feeders. State
feeder [188] operate at 50 Hz. estimation with DER integration [189] and optimal sizing
and placement of renewable energy batteries [190].
 25152947, 0, Downloaded from https://ietresearch.onlinelibrary.wiley.com/doi/10.1049/stg2.12161 by Universitätsbibliothek Erlangen- Nürnberg, Wiley Online Library on [23/10/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
ALTAMIMI ET AL.
- 17

synchronously intertied model of the US portion of the eastern 5 | DISCUSSION

and western interconnects.
Datasets also exist for the ARPA‐E Performance‐based After evaluating a comprehensive number of the most popular
Energy Resource Feedback, Optimisation, and Risk Manage- public datasets and the work done on them, several aspects of
ment (PERFORM) program. This program aims to optimise the discussion were identified. This section covers these as-
grid management as the penetration of variable renewable re- pects, as well as research gaps and future research directions.
sources continues to increase. For this program, specific cases For example, for data availability and synthetic data, we iden-
such as the 6717‐bus Texas Case and 24,000‐bus Midwest Case tified that the generation of synthetic data oriented to privacy
were created. preserving data could solve the problem of data availability,
Additional datasets from 2021 to 2023 include synthetic allowing realistic data analysis on otherwise private data [192].
transmission and distribution test cases, such as the Full Texas In terms of privacy preservation, we have identified two main
Synthetic Transmission and Distribution Test Case, and the categories of techniques, which are either ’consumer‐oriented’
150‐bus Synthetic Transmission and Distribution Test Case [193] or ’utility‐oriented’ [194]. Analysing the impact of ’con-
based on Travis County, Texas. These datasets also include sumer‐oriented’ privacy techniques on the utility of the datasets
restoration data and scenarios, as well as an associated natural is an interesting future work. Moreover, to the best of our
gas pipeline network. knowledge, there is no work that aims at identifying consumers
Other notable datasets come from the ARPA‐E GridData that practice such privacy preserving techniques. Regarding
program, which offers synthetic electric grid models. These data quality, since the number of popular public datasets is
models are designed to be statistically and functionally similar fairly low, an interesting research direction is to develop at this
to actual electric grids, ensuring the confidentiality of CEII. early stage a toolkit to unify consumer datasets in terms of
Examples include a 200‐bus synthetic grid on the footprint of format, data exploration, preprocessing techniques, and feature
Central Illinois, a 500‐bus synthetic grid on the footprint of engineering techniques similar to the toolkit developed for
South Carolina, and a 2000‐bus synthetic grid on the footprint NILM datasets [195].
of Texas, among others. Moreover after analysing the public datasets and the liter-
Datasets for competitions like the GO Competition ature, we noticed two relevant and prominent issues:
Challenge 1, and literature‐based power flow test cases such as
IEEE bus systems and the Kundur Two‐Area System, are also 1. Energy theft detection (and more broadly anomaly detec-
included. Moreover, for stability analysis and control, the tion) and EV detection and load forecasting predominantly
dataset provides small signal stability test cases, such as the rely on synthetic or private datasets. The preference for
Three Machines Infinite Bus Benchmark System, the Brazilian non‐public datasets is largely due to privacy concerns and
Seven Bus System, and the New England 68‐Bus Test System. the proprietary nature of the data. In energy theft detection,
All of these cases include feasible AC power flow solutions, the data involves sensitive user information and operational
and some have additional parameters or models for analyses details from utility companies, which are legally protected
such as transient stability, geomagnetic disturbance analysis, and competitively sensitive. Similarly, for EV forecasting,
energy economic study, and more. They have been developed private companies hold detailed charging data that is
to improve the situational awareness of current system oper- commercially valuable and often kept confidential.
ating conditions and to support various studies and research in 2. Newly emerging challenges, such as the detection of
power systems. unauthorised crypto mining, suffer from a lack of public
datasets. The surge in cryptocurrency mining poses chal-
lenges to smart grid management. Unauthorised mining
4.3 | Power generation operations and rapid technological advancements in this
field hinder the collection of accurate and up‐to‐date
Electricity generation can be divided into two categories: datasets. One study estimates the energy consumption to
centralised and distributed generation. Centralised generation be between 120 and 240 billion kilowatt‐hours yearly [196].
refers to the generation of electricity through large‐scale pro- This level of consumption suggests a significant impact on
duction plants and the distribution of that electricity to con- grid resources, yet the lack of detailed data impedes
sumers, whereas distributed generation refers to the generation comprehensive analysis and grid optimisation efforts.
of electricity on a much smaller scale, typically by individuals Notably, during Texas's energy conservation periods, such
using renewable energy sources. as the 2022 summer heatwave, mining activities demon-
In both centralised and distributed generation, the data strated demand flexibility [197]. This behaviour indicates a
collected is identical. It includes load demand, historical power potential adaptive load management strategy, but a detailed
measurements, capacity, generating unit, cost, performance, dataset is critical for evaluating the feasibility and reliability
ramp‐rate limit, operating zones, and carbon dioxide emissions of such an approach.
data. These data are used in the management of both the
power generation side and the microgrid side. Power genera- These trends highlight a gap in available resources for re-
tors are modelled on both transmission and distribution grids searchers, emphasising the need for a collaborative effort to
(in the case of DER). establish data‐sharing protocols that can balance privacy,
 25152947, 0, Downloaded from https://ietresearch.onlinelibrary.wiley.com/doi/10.1049/stg2.12161 by Universitätsbibliothek Erlangen- Nürnberg, Wiley Online Library on [23/10/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
18
- ALTAMIMI ET AL.

commercial value, and research needs to support advance- communicated. Data aggregation, quantisation, and differ-
ments in smart grid technologies. ential privacy techniques [37, 209–211] all fall into this
category. For example, the effect of data granularity on
privacy was studied in ref. [212]. However, more sophisti-
5.1 | Data availability and synthetic data cated privacy‐aware techniques are required to ensure the
aggregation of private data [213]. Secure multi‐party
One of the major issues facing SG data analytics is the lack of computation coupled with homomorphic encryption [214],
public datasets available, which can be attributed to the and secret sharing [215] are considered powerful candidates
reluctance of energy providers to publish their data. Privacy, to achieve privacy aware data aggregation.
security, and political issues all contribute to this issue [198]. � Demand shaping and scheduling: In this category, smart‐
Aside from the privacy concerns posed by energy disaggre- metre values are not modified or obfuscated. Instead, bat-
gation discussed in previous sections, geographical location of teries, appliance scheduling, and renewable sources hide
consumers can be compromised by solar panels generation energy usage within the house and hinder privacy‐intrusive
data as in ref. [199]. The lack of data availability and a standard attacks, such as NILM. In these cases, smart metres mea-
benchmark is more prevalent in the findings of a 2019 sys- sure perturbed usage after using the battery and renewable
tematic mapping study of 358 articles in SG data analytics sources. As such, locally installed batteries and renewable
[200]. Their findings revealed that 70% of the articles were sources could provide total household demand and privacy
conducted on private datasets, 26% used publicly available is absolutely ensured. Table 10 illustrates the four main
datasets, 15% synthetically generated the data, and the categories and exemplary articles.
remaining 4% used a combination of public and private
datasets. Without a standardised large set of public datasets, the Security is another critical issue in the SG. Recent pub-
issue of reproducibility is expected to persist. As a result, there lished work in ref. [226] provides a comprehensive review of
has been an interest in developing sophisticated techniques to AMI security vulnerabilities in SG in the three layers: hardware,
synthesise SG data, and, in particular, energy consumption data and communication layers. The identified countermea-
data, either at an aggregate level or appliance level (i.e. the case sures fall into three main categories:
of NILM data). There is a lack of focus on synthesising other
categories of data such as market data. These types of data are � Data encryption: Encryption is critical to preserving confi-
abundant and made public by grid operators because they are dentiality and privacy at the data layer. The techniques here
necessary information for ISO and consumers. Grid data, on focus on encrypting the data before communicating them to
the other hand, is mostly synthetic since they are considered the utility with minimal computational and communication
critical information for grid operators. Synthetic data genera- overhead [227, 228].
tion, especially using data‐driven approaches, also gives rise to � Authentication mechanisms: Authentication is critical to
opportunities for grid operators to allow realistic data analytics verify the sources of messages in the SG and to prevent
without sacrificing their customer's privacy. GANs were first impersonation attacks [229, 230].
introduced to generate synthetic data in the work of ref. [201] � Intrusion detection systems (IDS): IDSs are a critical second
in 2018 and since then several other works have utilised GANs line of defence for detecting security breaches in critical
to generate time series data [87, 202–204]. The results of these infrastructure. Recent works in IDS for AMI include
efforts suggest that GANs are a promising research direction. [231–233].
However, simply using GANs is not enough to conceal privacy,
as they are susceptible to membership inference attacks [192]. For data encryption and authentication mechanisms, the
work is typically evaluated using simulations on any energy
consumption dataset to measure the computational and
5.2 | Privacy and security communication overheads. On the other hand, IDS are eval-
uated on popular datasets that are not specific to the SG. An
With higher sampling rate readings, the analysis of smart metre unpopular solution is to develop testbeds and simulations such
data on energy consumption patterns can be used to determine as in ref. [233]. Developing an IDS dataset in the context of the
household occupancy and other more detailed sensitive in- SG or evaluating the effectiveness of IDS trained on typical
formation about the household. The serious nature of the IDS datasets in the context of the SG is a necessary research
privacy issues that smart metres raise has been shown to be a direction.
barrier to the widespread implementation of smart metres in On the basis of the above, we argue that more focus should
some countries [205–207]. be put forward on understanding the impact of demand
The work in ref. [208] reviewed the existing literature on shaping and load scheduling approaches to preserve privacy on
smart metres privacy and categorised the techniques into two the electrical utility. From a management perspective. These
broad techniques: techniques might, for example, induce uncertainties similar to
NTL leading to poor utilisation of resources and poor tariff
� Data manipulation: In this category, the high‐resolution data design [234]. From a data analytics perspective, such tech-
is manipulated from the consumer's end before being niques could potentially disrupt the efficacy of load forecasting
 25152947, 0, Downloaded from https://ietresearch.onlinelibrary.wiley.com/doi/10.1049/stg2.12161 by Universitätsbibliothek Erlangen- Nürnberg, Wiley Online Library on [23/10/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
ALTAMIMI ET AL.
- 19

T A B L E 10 Demand shaping and load scheduling categories.

Categories Explanation Ref.

Demand shaping: Batteries A battery (physical or virtual) used for energy consumption can be charged and discharged to [193, 216–218]
obfuscate the fine grain consumption data of the house, thus preserving privacy.

Demand shaping: Renewable energy These techniques obfuscate energy consumption with batteries; however, renewable energy [216, 219–221]
generation must also be modelled.

Demand shaping: Heating and cooling Since cooling and heating have high consumption, scheduling them in a specific way would be [222–224]
able to obfuscate the consumption of smaller appliances and provide more privacy

Load scheduling Scheduling appliances to make non‐intrusive load monitoring more difficult [225]

or energy theft detection models. Another research direction is and innovative anomalies where the errors are propagated
to consider techniques that identify consumers that practice throughout the time series in the system.
such privacy‐preserving practices, to limit their possible Real power systems also suffer significantly from noise
problematic impact on energy management and data analytics. [239], especially after the introduction of powerline commu-
nication technologies (PLCs) that support higher data rate
transmission (also called high data rate narrowband 3–500 kHz
5.3 | Data quality PLC systems). These new technologies are desirable because
they can be built on the existing power systems, however they
In the SG context, missing values, outliers, and noisy data (i.e. are designed for one‐way communication and not the two‐way
logical errors or inconsistent data) are the three most common communication necessary for SG applications [240]. The noise
data quality issues [235]. Several solutions to each of these present in these systems affect very high‐frequency electrical
problems were suggested by existing work. measurement devices such as PMU devices. The noise of
Regarding missing values, most datasets do not report voltage and current measurements of the phases (e.g. in NILM
missing data, forcing data analysts to manually detect and datasets) at 60 or 120 Hz is negligible and can be ignored.
manage them. For time‐series forecasting applications, data Data quality issues can extend to several other dimensions,
replacement (also called imputation) is typically required to namely contextual, representational, and accessibility. Contex-
preserve the integrity and pattern of the data. In general, the tual quality are several characteristics of the data that must be
approaches to replace missing data are categorised as present in certain applications but not others. Such qualities are
interpolation‐based and prediction‐model‐based algorithms. record time (the time it took for the data to be available after it
The former being used for a few missing data points, while the happened in actual time), sampling rate, and quantity of the
latter for longer periods. However, since accurate time series data. The representational qualities simply refer to how well a
data are necessary to train forecasting models, researchers dataset follows the format and structure of similar datasets, as
mostly opt to omit a certain portion or timeframe in the data well as interpretability of notations. This issue was found to be
(e.g. the whole day or similar omission criteria). Although this a significant hurdle for data analytics [241]. The last dimension
is a common and straightforward way to deal with missing is accessibility, in particular availability, which is one of the
data, it omits a portion of the available data, which may lead to most prevalent issues in the SG context, as some datasets are
bias in common statistical analysis (e.g. linear regression) [236]. more readily available to researchers than others. For example,
The work in ref. [237] outlines an industry‐recognised rec- some datasets require extra procedures such as login creden-
ommended practice for imputing faulty or missing smart metre tials and/or licencing.
data. Periods less than 2 h are often imputed by using linear In light of these issues, we argue that more effort should be
interpolation to the adjacent data. For times longer than put to develop toolkits to standardise the datasets as a future
2 hours, the standard technique is to develop daily load profiles research direction. In terms of formatting, for example, the
based on previously verified historical data of ’like weekdays' Ausgrid dataset [58] for electricity consumption combines
and ’like days'. Holidays or other exceptional cases are often three consumption categories in the same Excel data sheet,
addressed individually. It is important to note that dealing with while the LCL contains only one. A toolkit with a unified API
missing values is not always necessary. For example, in ref. would make the repeatability of studies much more feasible.
[237] when creating a representative load pattern of a cluster of Another issue that could be addressed by the toolkits is data
consumers, the average of the available data points in a given preprocessing, since most work on energy consumption utilises
point stamp is taken. similar preprocessing techniques. Feature engineering is
In outliers detection (anomalous data detection), most another possible extension of such toolkits. For example, a
work utilise the two‐standard‐deviations rule as a preprocess- toolkit can facilitate the extraction of time‐related features (e.g.
ing step for their respective application. According to ref. peak hours) or apply simple clustering techniques to help with
[238], there are two types of outliers that should be taken into data exploration; clustering daily consumption profiles helps
account when dealing with time series data: isolated anomalies identify common, uncommon, and anomalous consumption
or events where the error is local to a certain set of data points habits [242].
 25152947, 0, Downloaded from https://ietresearch.onlinelibrary.wiley.com/doi/10.1049/stg2.12161 by Universitätsbibliothek Erlangen- Nürnberg, Wiley Online Library on [23/10/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
20
- ALTAMIMI ET AL.

5.4 | Big data in the smart grid specific knowledge that may have been introduced during this
stage. By addressing both the trade‐offs inherent in dataset
The key steps to handle and use big data are data acquisition, design and the need for clear documentation of preprocessing
storage, analysis, and operational integration. The work in ref. techniques, researchers can work towards developing more
[243] reviewed data management for SGs and its technical robust and generalisable energy disaggregation models.
requirements, the tools, and the necessary steps to integrate big
data solutions in the SG context. The authors highlighted three
main issues: standards and interoperability; lack of infrastruc- 5.6 | Challenges in preprocessing and
ture to be able to fully utilise the big data; and privacy, integrity, evaluation
authentication, and security. Furthermore, the authors of ref.
[244] discussed several challenges in the area of big data ana- In this subsection, we discuss the challenges and limitations
lytics, including data indexing and time synchronisation. The faced in current approaches to data pre‐processing, post‐
two broad categories of applications in big data are smart processing, model evaluation, and generalisability in the
metre big data and PMU big data. Smart metre big data ap- context of electrical grid data analysis.
plications are related to energy management such as load Most literature does not mention data preprocessing steps
forecasting, profiling, DR, baseline estimation. The CBT such as data cleaning and dealing with missing values despite
dataset is a common public dataset used for this area of being a crucial step known to boost performance. These steps
research due to its large volume (167 million data rows) [245]. are presumably taken, but not mentioned. Not explicitly stating
PMU big data are used for state estimation, transmission grid the preprocessing procedure harms replicability of the work as
visualisation, and SG reliability and stability. Simulations are there are several preprocessing procedures that can be fol-
commonly used to generate PMU data [246]. lowed. The authors could have introduced bias and/or domain
knowledge in the data, which may have enhanced the perfor-
mance of their models.
5.5 | Detailed in‐home consumption We have also observed a lack of post‐processing tech-
datasets niques, which we believe is a potential future work to explore
due to its promise to enhance performance (especially reducing
Currently available detailed in‐home consumption datasets, or false positives [247]) and mitigate common typical biases
as commonly referred to as buildings datasets or NILM especially in energy disaggregation. For example, the authors in
datasets, fall into two categories: laboratory measurements and ref. [248] discovered that disaggregation techniques typically
data from the actual environment. Available laboratory mea- overestimate or underestimate disaggregated loads and pro-
surements include data from individual devices, although these posed a technique that ensures that the disaggregated loads
data are of very little use for overall benchmark tests because sum up to approximately the true aggregate consumption.
real‐world datasets contain measurements where multiple de- Similarly, the authors of ref. [249] discovered bias when dealing
vices are active concurrently. However, assigning reference data with appliances that operate on multi‐states (e.g. dishwashers
in real‐world scenarios presents difficulties: and washing machines). Models typically produce several
sporadic activations for such appliances.
1) The synchronisation of references and measured data; that Another preprocessing issue observed in the literature is
is, a label should correspond to a pattern shift in the data the arbitrary exclusion of some data and without justification,
that corresponds to the labelled pattern. A further which threatens the validity of the models. For example, some
requirement is that all data streams must be in sync with houses in the REDD datasets include very few events. These
one another. houses were mostly excluded due to the effect they have on
2) The absence or excess of events, and the number of “on” training. The issue is not specific to the REDD dataset, as each
and “off” cycles for each device. model has its own setbacks that can be revealed if tested on
3) The probability distribution of the devices, as well as the more houses. To this end, we recommend using techniques
lengthy measurement cycles containing a correspondingly such as leave‐one‐house‐out cross‐validation for a more
large volume of data that contain a small number of events. complete evaluation in future work. Different authors also
select the appliances and a number of appliances that they will
While NILM datasets face trade‐offs between covering a train and test on without justification.
large number of houses or focusing on a more extensive set of There is no clear justification and/or consensus for the
appliances and measurements, an equally important aspect that is selection of the training and testing split. Some train their
often overlooked in the literature is the preprocessing of data. model for 5 days and test only on one, while others follow a
Ensuring data quality and dealing with missing values is a crucial different evaluation strategy. This makes it difficult to compare
step in the development of effective energy disaggregation and evaluate models, not to mention that models will be more
models, as it can significantly improve their performance. Un- likely to overfit the test data and perform better but have lower
fortunately, the lack of transparency regarding preprocessing generalisability. Some models also train and test on the same
procedures in many studies makes it difficult to replicate results house, while others train on a house and test on another, which
and assess the true impact of any potential bias or domain‐ means the former has lower generalisability.
 25152947, 0, Downloaded from https://ietresearch.onlinelibrary.wiley.com/doi/10.1049/stg2.12161 by Universitätsbibliothek Erlangen- Nürnberg, Wiley Online Library on [23/10/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
ALTAMIMI ET AL.
- 21

The authors also define steady states and transient states addressing privacy concerns like membership inference
differently. For example, a steady‐state power signal must not attacks.
fluctuate more than a certain threshold and must last for a � Advancing Privacy Preservation and Security: Future
period of time. In probabilistic models, such assumptions research can focus on exploring the impact of privacy
extend to the appliances (average, maximum, minimum, and preservation techniques, particularly demand shaping and
duration of power consumed). While necessary, this poses a load scheduling, on SG data utility and energy management.
trade‐off as follows: a more strict (i.e. high threshold) defini- Another avenue is to develop methods to identify con-
tion will eliminate noise; however, this may lead to not being sumers using privacy‐preserving techniques because these
able to detect small appliances consumption (they will still be consumers may affect utilities data analytics.
considered in the steady state). To better illustrate this point, � Improving Data Quality and Standardisation: Re-
imagine a kettle that consumes 10 W, if the steady‐state searchers in the future may address SG data quality issues by
threshold was, for example, 20 W then the kettle's consump- creating comprehensive toolkits for data standardisation and
tion will be considered noise and will not be detected. A more preprocessing, including feature engineering and clustering.
lenient definition (or a lower threshold) will allow for small A key focus may be on unifying dataset formats and
appliances to be detected, however, poorer performance be- structures for better data exploration and better analytics
comes inevitable. In probabilistic models, a more “diverse” accuracy.
assumption on the appliances (e.g., picking appliances that � Big Data Management and Analytics in SG: Investigate
have a high difference in their average consumption) will allow the integration of big data solutions in SG, addressing
for better distinction between the appliances and better per- challenges in data management, standards, interoperability,
formance overall. However, this will require handpicking of and infrastructure development. Emphasise improving data
appliances and is thus not practical. acquisition, storage, analysis, and operational integration.
We believe that more attention must be paid to developing � Detailed In‐Home Consumption Datasets and Pre-
models with genralisability and transferability in mind [250]. processing Techniques: Future research may aim at
This can be evaluated by training and testing on different improving in‐home consumption datasets by refining pre-
datasets or on different houses. Comparing the same model processing methods for handling data synchronisation,
with different datasets poses several challenges. First is the event detection, and large data volumes. Standardising pre-
different percentage of missing data in the datasets; some loads processing steps is crucial for enhancing study replicability
that are not sub‐metered and their consumption data become and minimising biases. This effort includes better strategies
missing. The second is the scarcity of fully labelled NILM for data cleaning and handling missing values. There's also a
datasets. The last challenge is the different characteristics of the need for more inclusive datasets covering diverse appliances
datasets, such as the type and sampling rate of the measure- and conditions to foster robust, generalisable energy disag-
ments, and the different formats. gregation models.
The learned models are affected by the sampling rate of
their associated dataset. Data preprocessing techniques that By addressing these areas, future research can significantly
can capture most of the features at lower sampling rates while contribute to the advancement of SG data analytics, ensuring
still maintaining high performance are a promising future more efficient, secure, and reliable data management systems.
research direction.
Another notable issue is associated with the use of metrics
that favour classifying high‐power consumption devices. Such 7 | CONCLUSION
metrics do not capture information about how well the model
performs in low‐power devices. It is argued, however, that such Power grids generate huge volumes of data and specifically in
information is valuable since low‐power appliances are typi- the SG context, where various types of data originate from
cally what the user has the greatest control over. several sources and typically at higher sampling rates. In
addition to enabling safe operation of the grid itself, such data
enable a wide variety of applications. Despite their high utility,
6 | FUTURE WORK the availability of public real‐world smart grid datasets is very
limited. In this work we reviewed over 50 public datasets in the
This section provides future research directions highlighting smart grid context, categorising them into three main cate-
key areas that require further exploration and development in gories; Consumers' data, NILM data, and Grid data. Each
the field of SG data analytics. category can enable for a distinct set of applications. After
considering the characteristics of the individual datasets, 14 of
� Synthetic Data Generation: Future research can focus on their most popular applications were discussed, as well as
developing privacy‐preserving synthetic data generation numerous other less popular applications. Several findings are
techniques for SG, particularly for market and grid data. discussed and highlighted throughout this contribution. In the
Another potential avenue is investigating advanced synthetic end, we present a discussion of some prevalent issues that
data generation methods such as the use of GANs while motivate potential future research and development directions.
 25152947, 0, Downloaded from https://ietresearch.onlinelibrary.wiley.com/doi/10.1049/stg2.12161 by Universitätsbibliothek Erlangen- Nürnberg, Wiley Online Library on [23/10/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
22
- ALTAMIMI ET AL.

Ultimately, this review provides a comprehensive survey of protection—a review. Energy Rep. 8, 699–709 (2022). https://doi.org/
public datasets in smart and power grid research, with the aim 10.1016/j.egyr.2021.11.217
of improving reproducibility and serving as a key reference for 8. Florencias‐Oliveros, O., et al.: Real‐life power quality sags. IEEE
Dataport (2017)
researchers developing applications in this domain. 9. Wang, Y., et al.: Power system disaster‐mitigating dispatch platform
based on big data. In: 2014 International Conference on Power System
AUT HO R CO NT R I BU T I ON S Technology, pp. 1014–1019. IEEE (2014)
The initial manuscript draft was prepared by the first author, 10. Wang, X., et al.: Automatic analysis of pole mounted auto‐recloser data
who was also responsible for the primary data collection and for fault prognosis to mitigate customer supply interruptions. In: 2014
49th International Universities Power Engineering Conference (UPEC),
analysis. All co‐authors provided intellectual oversight and pp. 1–6. IEEE (2014)
contributed significantly to the research design and method- 11. Chren, S., et al.: Reliability data for smart grids: where the real data can be
ology. They played an active role in refining the manuscript found. In: 2018 Smart City symposium prague (scsp), pp. 1–6. IEEE
through rigorous critique, constructive feedback, and sub- (2018)
12. Elahe, M.F., Jin, M., Zeng, P.: Review of load data analytics using deep
stantive editing. Additionally, they provided expert guidance in
learning in smart grids: open load datasets, methodologies, and appli-
the framing and structuring of the manuscript, ensuring its cation challenges. Int. J. Energy Res. 45(10), 14274–14305 (2021).
academic rigour and integrity. https://doi.org/10.1002/er.6745
13. Wang, Y., et al.: Review of smart meter data analytics: applications,
ACKN OW LE DG E ME N T S methodologies, and challenges. IEEE Trans. Smart Grid 10(3),
This publication is supported in part by grant NPRP12C‐ 3125–3148 (2018). https://doi.org/10.1109/tsg.2018.2818167
14. Iqbal, H.K., et al.: A critical review of state‐of‐the‐art non‐intrusive load
33905‐SP‐66 from the Qatar National Research Fund. The
monitoring datasets. Elec. Power Syst. Res. 192, 106921 (2021). https://
findings achieved herein are solely the responsibility of the doi.org/10.1016/j.epsr.2020.106921
authors. 15. Pereira, L., Nunes, N.: Performance evaluation in non‐intrusive load
Open Access funding provided by the Qatar National monitoring: datasets, metrics, and tools—a review. Wiley Interdiscip.
Library. Rev. 8(6), e1265 (2018). https://doi.org/10.1002/widm.1265
16. Kazmi, H., et al.: Towards data‐driven energy communities: a review of
open‐source datasets, models and tools. Renew. Sustain. Energy Rev.
CON FL ICT OF I N T ER E ST STAT E M EN T 148, 111290 (2021). https://doi.org/10.1016/j.rser.2021.111290
The authors declare that they have no financial or personal 17. Meinecke, S., Thurner, L., Braun, M.: Review of steady‐state electric
relationships with other people or organisations that could power distribution system datasets. Energies 13(18), 4826 (2020).
inappropriately influence or bias their work. https://doi.org/10.3390/en13184826
18. Postigo Marcos, F.E., et al.: A review of power distribution test feeders
in the United States and the need for synthetic representative networks.
DATA AVA IL AB I LI T Y S TAT E M EN T Energies 10(11), 1896 (2017). https://doi.org/10.3390/en10111896
No new data were generated or analysed in this study. All 19. Kabalci, E., Kabalci, Y.: From Smart Grid to Internet of Energy. Ac-
datasets reviewed are publicly available and are cited within the ademic Press (2019)
article. 20. Sun, Q., et al.: A comprehensive review of smart energy meters in
intelligent energy networks. IEEE Internet Things J. 3(4), 464–479
(2015). https://doi.org/10.1109/jiot.2015.2512325
PE RM ISSI O N T O R E PROD U CE M A T ER I A LS 21. Humeau, S., et al.: Electricity load forecasting for residential customers:
None. exploiting aggregation and correlation between households. In: 2013
Sustainable Internet and ICT for Sustainability (SustainIT), pp. 1–6.
OR CI D IEEE (2013)
Emran Altamimi https://orcid.org/0000-0002-2902-663X 22. Massaferro, P., Di Martino, J.M., Fernández, A.: Ntl detection: overview
of classic and dnn‐based approaches on a labeled dataset of 311k
customers. In: 2021 IEEE Power & Energy Society Innovative Smart
RE FE R ENC ES Grid Technologies Conference (ISGT), pp. 1–5. IEEE (2021)
1. Mattioli, R., Moulinos, K.: Communication Network Interdependencies 23. Saleh, S., Pijnenburg, P., Castillo‐Guerra, E.: Load aggregation from
in Smart Grids. EUA FNAI Security, Ed., EU: ENISA, (2015) generation‐follows‐load to load‐follows‐generation: residential loads.
2. Geisler, K.: The Relationship Between Smart Grids and Smart Cities. IEEE Trans. Ind. Appl. 53(2), 833–842 (2016). https://doi.org/10.
IEEE Smart Grid Newsletter (2013) 1109/tia.2016.2626261
3. Marinakis, V.: Big data for energy management and energy‐efficient 24. Edwards, R.E., New, J., Parker, L.E.: Predicting future hourly residential
buildings. Energies 13(7), 1555 (2020). https://doi.org/10.3390/ electrical consumption: a machine learning case study. Energy Build. 49,
en13071555 591–603 (2012). https://doi.org/10.1016/j.enbuild.2012.03.010
4. Siebert, L.C., et al.: An agent‐based approach for the planning of distri- 25. Kavousian, A., Rajagopal, R., Fischer, M.: Determinants of residential
bution grids as a socio‐technical system. Energies 13(18), 4837 (2020). electricity consumption: using smart meter data to examine the effect of
https://doi.org/10.3390/en13184837 climate, building characteristics, appliance stock, and occupants’
5. Zhou, K., Fu, C., Yang, S.: Big data driven smart energy management: behavior. Energy 55, 184–194 (2013). https://doi.org/10.1016/j.energy.
from big data to big insights. Renew. Sustain. Energy Rev. 56, 215–225 2013.03.086
(2016). https://doi.org/10.1016/j.rser.2015.11.050 26. Bai, Y., Zhong, H., Xia, Q.: 2016 IEEE Power and Energy Society
6. Stewart, E., Liao, A., Roberts, C.: Open μpmu: A Real World Reference General Meeting (PESGM). IEEE. pp. 1–5 (2016)
Distribution Micro‐phasor Measurement Unit Data Set for Research 27. Mahmoudi‐Kohan, N., et al.: A three‐stage strategy for optimal price
and Application Development (2016) offering by a retailer based on clustering techniques. Int. J. Electr. Power
7. Chintakindi, R., Mitra, A.: Wams challenges and limitations in load Energy Syst. 32(10), 1135–1142 (2010). https://doi.org/10.1016/j.
modeling, voltage stability improvement, and controlled island ijepes.2010.06.011
 25152947, 0, Downloaded from https://ietresearch.onlinelibrary.wiley.com/doi/10.1049/stg2.12161 by Universitätsbibliothek Erlangen- Nürnberg, Wiley Online Library on [23/10/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
ALTAMIMI ET AL.
- 23

28. Huang, H., et al.: False data separation for data security in smart grids. 50. Yao, D., et al.: Energy theft detection with energy privacy preservation
Knowl. Inf. Syst. 52(3), 815–834 (2017). https://doi.org/10.1007/ in the smart grid. IEEE Internet Things J. 6(5), 7659–7669 (2019).
s10115‐016‐1019‐8 https://doi.org/10.1109/jiot.2019.2903312
29. Mateos, G., Giannakis, G.B.: Load curve data cleansing and imputation 51. Yang, H., et al.: Pipc: privacy‐ and integrity‐preserving clustering anal-
via sparsity and low rank. IEEE Trans. Smart Grid 4(4), 2347–2355 ysis for load profiling in smart grids. IEEE Internet Things J. 9(13),
(2013). https://doi.org/10.1109/tsg.2013.2259853 10851–10861 (2022). https://doi.org/10.1109/jiot.2021.3125674
30. Chuwa, M.G., Wang, F.: A review of non‐technical loss attack models 52. Kuster, C., Rezgui, Y., Mourshed, M.: Electrical load forecasting models:
and detection methods in the smart grid. Elec. Power Syst. Res. 199, a critical systematic review. Sustain. Cities Soc. 35, 257–270 (2017).
107415 (2021). https://doi.org/10.1016/j.epsr.2021.107415 https://doi.org/10.1016/j.scs.2017.08.009
31. Zhao, Q., Chang, Z., Min, G.: Anomaly detection and classification of 53. Sun, M., et al.: Clustering‐based residential baseline estimation: a
household electricity data: a time window and multilayer hierarchical probabilistic perspective. IEEE Trans. Smart Grid 10(6), 6014–6028
network approach. IEEE Internet Things J. 9(5), 3704–3716 (2021). (2019). https://doi.org/10.1109/tsg.2019.2895333
https://doi.org/10.1109/jiot.2021.3098735 54. Yusta, J., et al.: Optimal electricity price calculation model for retailers in
32. Yang, S.L., Shen, C.: A review of electric load classification in smart grid a deregulated market. Int. J. Electr. Power Energy Syst. 27(5‐6), 437–447
environment. Renew. Sustain. Energy Rev. 24, 103–110 (2013). https:// (2005). https://doi.org/10.1016/j.ijepes.2005.03.002
doi.org/10.1016/j.rser.2013.03.023 55. Low carbon london (lcl) dataset. [Online]. https://data.london.gov.uk/
33. Chicco, G.: Overview and performance assessment of the clustering dataset/smartmeter‐energy‐use‐data‐in‐london%20‐households
methods for electrical load pattern grouping. Energy 42(1), 68–80 56. Dataport. (2021). [Online]. https://www.pecanstreet.org/dataport/
(2012). https://doi.org/10.1016/j.energy.2011.12.031 57. Weibel, T.: Smart* Data Set for Sustainability. [Online]. https://traces.
34. Wang, Y., et al.: Load profiling and its application to demand response: a cs.umass.edu/index.php/smart/smart
review. Tsinghua Sci. Technol. 20(2), 117–129 (2015). https://doi.org/ 58. Ausgrid datasets. [Online]. https://www.ausgrid.com.au/Industry/Our‐
10.1109/tst.2015.7085625 Research/Data‐to‐share
35. Unterweger, A., Engel, D., Ringwelski, M.: The effect of data granularity 59. C. for Energy Regulation (CER): Cer smart metering project ‐ electricity
on load data compression. In: DA‐CH Conference on Energy Infor- customer behaviour trial, 2009‐2010. Irish Soc. Sci. Data Arch. (2012).
matics, pp. 69–80. Springer (2015) s N : 0 0 1 2 ‐ 0 0 . [ O n l i n e ] . h t t p s : / / w w w. u c d . i e / i s s d a / d a t a /
36. Rottondi, C., Verticale, G., Krauss, C.: Distributed privacy‐preserving commissionforenergyregulationcer/
aggregation of metering data in smart grids. IEEE J. Sel. Area. Com- 60. Australian energy market operator (aemo) dataset. [Online]. https://
mun. 31(7), 1342–1354 (2013). https://doi.org/10.1109/jsac.2013. aemo.com.au/en/energy‐systems/electricity/national‐electricity‐%
130716 20market‐nem/data‐nem
37. Khwaja, A.S., et al.: Smart meter data obfuscation using correlated 61. Electric reliability council of Texas (ercot) dataset. [Online]. http://
noise. IEEE Internet Things J. 7(8), 7250–7264 (2020). https://doi.org/ www.ercot.com/
10.1109/jiot.2020.2983213 62. Miller, C., Meggers, F.: The building data genome project: an open,
38. Shateri, M., et al.: Privacy‐cost management in smart meters with public data set from non‐residential building electrical meters. Energy
mutual‐information‐based reinforcement learning. IEEE Internet Proc. 122, 439–444 (2017). https://doi.org/10.1016/j.egypro.2017.
Things J. 9(22), 22389–22398 (2022). https://doi.org/10.1109/jiot. 07.400
2021.3128488 63. Energy Market Authority of singapore. (2022). [Online]. https://www.
39. Wang, X., et al.: Detection and isolation of false data injection attacks in ema.gov.sg/Statistics.aspx
smart grids via nonlinear interval observer. IEEE Internet Things J. 64. Hong, T., Pinson, P., Fan, S.: Global Energy Forecasting Competition
6(4), 6498–6512 (2019). https://doi.org/10.1109/jiot.2019.2916670 2012, pp. 357–363 (2014)
40. Peppanen, J., et al.: Handling bad or missing smart meter data through 65. Energy, Load, and Demand Reports. ISO New England, (2012)‐pre-
advanced data imputation. In: 2016 IEEE Power & Energy Society sent. [Online]. https://www.iso‐ne.com/isoexpress/web/reports/load‐
Innovative Smart Grid Technologies Conference (ISGT), pp. 1–5. and‐demand/‐/tree/dmnd‐rt‐hourly‐sys
IEEE (2016) 66. Trindade, A.: Electricityloaddiagrams20112014. [Online]. https://
41. Barnett, V., Lewis, T.: Evolution by gene duplication. In: Outliers in archive.ics.uci.edu/ml/datasets/ElectricityLoadDiagrams20112014
Statistical Data, 3rd ed Wiley, Hoboken, NJ, USA (1994) 67. HenryRDlab: The State Grid Corporation of china (Sgcc) Dataset.
42. Buzzi‐Ferraris, G., Manenti, F.: Outlier detection in large data sets. https://github.com/henryRDlab/ElectricityTheftDetection
Comput. Chem. Eng. 35(2), 388–390 (2011). https://doi.org/10.1016/j. 68. Laurinec, P., et al.: Density‐based unsupervised ensemble learning
compchemeng.2010.11.004 methods for time series forecasting of aggregated or clustered electricity
43. Markou, M., Singh, S.: Novelty detection: a review—part 1: statistical consumption. J. Intell. Inf. Syst. 53(2), 219–239 (2019). https://doi.org/
approaches. Signal Process. 83(12), 2481–2497 (2003). https://doi.org/ 10.1007/s10844‐019‐00550‐3
10.1016/j.sigpro.2003.07.018 69. Laurinec, P., Lucká, M.: Interpretable multiple data streams clustering
44. Enernoc open data. [Online]. https://open‐enernoc‐data.s3. with clipped streams representation for the improvement of electricity
amazonaws.com/anon/index.html consumption forecasting. Data Min. Knowl. Discov. 33(2), 413–445
45. Jaiswal, R., et al.: Anomaly detection in smart meter data for preventing (2019). https://doi.org/10.1007/s10618‐018‐0598‐2
potential smart grid imbalance. In: 2021 4th Artificial Intelligence and 70. Venkatesh, J., et al.: Scalable‐application design for the iot. IEEE Softw.
Cloud Computing Conference, pp. 150–159 (2021) 34(1), 62–70 (2017). https://doi.org/10.1109/ms.2017.4
46. Zhang, W., et al.: Solargan: multivariate solar data imputation using 71. Singh, S., Majumdar, A.: Deep sparse coding for non–intrusive load
generative adversarial network. IEEE Trans. Sustain. Energy 12(1), monitoring. IEEE Trans. Smart Grid 9(5), 4669–4678 (2017). https://
743–746 (2020). https://doi.org/10.1109/tste.2020.3004751 doi.org/10.1109/tsg.2017.2666220
47. Zheng, R., et al.: Load forecasting under data corruption based on 72. Mocanu, E., et al.: On‐line building energy optimization using deep
anomaly detection and combined robust regression. Int. Trans. Electr. reinforcement learning. IEEE Trans. Smart Grid 10(4), 3698–3708
Energy Syst. 30(7), e12103 (2020). https://doi.org/10.1002/2050‐7038. (2018). https://doi.org/10.1109/tsg.2018.2834219
12103 73. Henri, G., Lu, N.: A supervised machine learning approach to control
48. Hong, T., et al.: Probabilistic Energy Forecasting: Global Energy energy storage devices. IEEE Trans. Smart Grid 10(6), 5910–5919
Forecasting Competition 2014 and beyond, pp. 896–913 (2016) (2019). https://doi.org/10.1109/tsg.2019.2892586
49. State grid corporation of China: electricity theft dataset. [Online]. 74. Xuan, Z., et al.: Pv‐load decoupling based demand response baseline
https://github.com/henryRDlab/ElectricityTheftDetection load estimation approach for residential customer with distributed pv
 25152947, 0, Downloaded from https://ietresearch.onlinelibrary.wiley.com/doi/10.1049/stg2.12161 by Universitätsbibliothek Erlangen- Nürnberg, Wiley Online Library on [23/10/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
24
- ALTAMIMI ET AL.

system. IEEE Trans. Ind. Appl. 56(6), 6128–6137 (2020). https://doi. evaluation of deployment cases. J. Energy Storage 8, 226–234 (2016).
org/10.1109/tia.2020.3014575 https://doi.org/10.1016/j.est.2016.08.011
75. Molina‐Markham, A., et al.: Private memoirs of a smart meter. In: 95. Tajeuna, E.G., Bouguessa, M., Wang, S.: A network‐based approach to
Proceedings of the 2nd ACM Workshop on Embedded Sensing Sys- enhance electricity load forecasting. In: 2018 IEEE International Con-
tems for Energy‐Efficiency in Building, pp. 61–66 (2010) ference on Data Mining Workshops (ICDMW), pp. 266–275. IEEE
76. Jin, M., Jia, R., Spanos, C.J.: Virtual occupancy sensing: using smart (2018)
meters to indicate your presence. IEEE Trans. Mobile Comput. 16(11), 96. Himeur, Y., et al.: Building power consumption datasets: survey, tax-
3264–3277 (2017). https://doi.org/10.1109/tmc.2017.2684806 onomy and future directions. Energy Build. 227, 110404 (2020).
77. Kekatos, V., et al.: Voltage regulation algorithms for multiphase power https://doi.org/10.1016/j.enbuild.2020.110404
distribution grids. IEEE Trans. Power Syst. 31(5), 3913–3923 (2015). 97. Gao, J., et al.: A feasibility study of automated plug‐load identification
https://doi.org/10.1109/tpwrs.2015.2493520 from high‐frequency measurements. In: 2015 IEEE Global Conference
78. Eichinger, F., et al.: A time‐series compression technique and its on Signal and Information Processing (GlobalSIP), pp. 220–224. IEEE
application to the smart grid. VLDB J. 24(2), 193–218 (2015). https:// (2015)
doi.org/10.1007/s00778‐014‐0368‐8 98. He, J., et al.: An efficient and accurate nonintrusive load monitoring
79. Li, C., et al.: Data‐driven planning of electric vehicle charging infra- scheme for power consumption. IEEE Internet Things J. 6(5),
structure: a case study of Sydney, Australia. IEEE Trans. Smart Grid 9054–9063 (2019). https://doi.org/10.1109/jiot.2019.2926815
12(4), 3289–3304 (2021). https://doi.org/10.1109/tsg.2021.3054763 99. Hart, G.: Nonintrusive appliance load monitoring. Proc. IEEE 80(12),
80. Martin, D., et al.: Investigation into modeling Australian power trans- 1870–1891 (1992). https://doi.org/10.1109/5.192069
former failure and retirement statistics. IEEE Trans. Power Deliv. 33(4), 100. Zoha, A., et al.: Non‐intrusive load monitoring approaches for dis-
2011–2019 (2018). https://doi.org/10.1109/tpwrd.2018.2814588 aggregated energy sensing: a survey. Sensors 12(12), 16838–16866
81. Krishnamurthy, D., Li, W., Tesfatsion, L.: An 8‐zone test system based (2012). https://doi.org/10.3390/s121216838
on iso new england data: development and application. IEEE Trans. 101. Ruano, A., et al.: Nilm techniques for intelligent home energy man-
Power Syst. 31(1), 234–246 (2015). https://doi.org/10.1109/tpwrs. agement and ambient assisted living: a review. Energies 12(11), 2203
2015.2399171 (2019). https://doi.org/10.3390/en12112203
82. Wang, W., Yu, N.: A machine learning framework for algorithmic 102. Ganu, T., et al.: Socketwatch: an autonomous appliance monitoring
trading with virtual bids in electricity markets. In: 2019 IEEE Po- system. In: 2014 IEEE International Conference on Pervasive
wer & Energy Society General Meeting (PESGM), pp. 1–5. IEEE Computing and Communications (PerCom), pp. 38–43. IEEE (2014)
(2019) 103. Pereira, W., Ferscha, A., Weigl, K.: Unsupervised detection of unusual
83. Csereklyei, Z., Qu, S., Ancev, T.: The effect of wind and solar power behaviors from smart home energy data. In: International Conference on
generation on wholesale electricity prices in Australia. Energy Pol. 131, Artificial Intelligence and Soft Computing, pp. 523–534. Springer (2016)
358–369 (2019). https://doi.org/10.1016/j.enpol.2019.04.007 104. Rashid, H., et al.: Can non‐intrusive load monitoring be used for
84. Sharma, V., Haque, M.H., Aziz, S.M.: Energy cost minimization for net identifying an appliance’s anomalous behaviour? Appl. Energy 238,
zero energy homes through optimal sizing of battery storage system. 796–805 (2019). https://doi.org/10.1016/j.apenergy.2019.01.061
Renew. Energy 141, 278–286 (2019). https://doi.org/10.1016/j.renene. 105. Rashid, H., et al.: Evaluation of non‐intrusive load monitoring algo-
2019.03.144 rithms for appliance‐level anomaly detection. In: ICASSP 2019‐2019
85. Sharma, V., Haque, M.H., Aziz, S.M.: Pv generation and load profile IEEE International Conference on Acoustics, Speech and Signal Pro-
data of net zero energy homes in south Australia. Data Brief 25, 104235 cessing (ICASSP), pp. 8325–8329. IEEE (2019)
(2019). https://doi.org/10.1016/j.dib.2019.104235 106. Ahajjam, M.A., et al.: Mored: a moroccan buildings’ electricity con-
86. Luo, S., Weng, Y.: A two‐stage supervised learning approach for elec- sumption dataset. Energies 13(24), 6737 (2020). https://doi.org/10.
tricity price forecasting by leveraging different data sources. Appl. 3390/en13246737
Energy 242, 1497–1512 (2019). https://doi.org/10.1016/j.apenergy. 107. Kolter, J.Z., Johnson, M.J.: Redd: a public data set for energy disag-
2019.03.129 gregation research. In: Workshop on Data Mining Applications in
87. Fekri, M.N., Ghosh, A.M., Grolinger, K.: Generating energy data for Sustainability (SIGKDD), San Diego, CA, vol. 25, pp. 59–62 (2011).
machine learning with recurrent generative adversarial networks. En- Citeseer
ergies 13(1), 130 (2019). https://doi.org/10.3390/en13010130 108. Kelly, J., Knottenbelt, W.: The UK‐dale dataset, domestic appliance‐level
88. Miller, C., Meggers, F.: Mining electrical meter data to predict principal electricity demand and whole‐house demand from five UK homes. Sci.
building use, performance class, and operations strategy for hundreds Data 2(1), 1–14 (2015). https://doi.org/10.1038/sdata.2015.7
of non‐residential buildings. Energy Build. 156, 360–373 (2017). 109. Anderson, K., et al.: Blued: a fully labeled public dataset for event‐based
https://doi.org/10.1016/j.enbuild.2017.09.056 non‐intrusive load monitoring research. In: Proceedings of the 2nd
89. Koh, L., et al.: Impact of energy storage and variability of pv on power KDD Workshop on Data Mining Applications in Sustainability
system reliability. Energy Proc. 33, 302–310 (2013). https://doi.org/10. (SustKDD), vol. 7, pp. 1–5. ACM New York (2012)
1016/j.egypro.2013.05.071 110. Makonin, S., et al.: Ampds: a public dataset for load disaggregation and
90. Shu, Z., Jirutitijaroen, P.: Non‐sequential simulation methods for reli- eco‐feedback research. In: 2013 IEEE Electrical Power & Energy
ability analysis of power systems with photovoltaic generation. In: 2010 Conference, pp. 1–6. IEEE (2013)
IEEE 11th International Conference on Probabilistic Methods Applied 111. Beckel, C., et al.: The eco data set and the performance of non‐intrusive
to Power Systems, pp. 703–709. IEEE (2010) load monitoring algorithms. In: Proceedings of the 1st ACM Conference
91. Loi, T.S.A., Le Ng, J.: Analysing households’ responsiveness towards on Embedded Systems for Energy‐Efficient Buildings, pp. 80–89 (2014)
socio‐economic determinants of residential electricity consumption in 112. Reinhardt, A., et al.: On the accuracy of appliance identification based
Singapore. Energy Pol. 112, 415–426 (2018). https://doi.org/10.1016/ on distributed load metering data. In: 2012 Sustainable Internet and ICT
j.enpol.2017.09.052 for Sustainability (SustainIT), pp. 1–9. IEEE (2012)
92. Srinivasan, D., et al.: Game‐theory based dynamic pricing strategies for 113. Zimmermann, J.‐P., et al.: Household Electricity Survey: A Study of
demand side management in smart grids. Energy 126, 132–143 (2017). Domestic Electrical Product Usage, pp. 213–214. Intertek Testing &
https://doi.org/10.1016/j.energy.2016.11.142 Certification Ltd (2012)
93. Yoshida, Y., Figueroa, H.P., Dougal, R.A.: Use of time series load data 114. Batra, N., et al.: It’s different: insights into home energy consumption in
to size energy storage systems. In: 2018 IEEE Green Energy and Smart India. In: Proceedings of the 5th ACM Workshop on Embedded Sys-
Systems Conference (IGESSC), pp. 1–6. IEEE (2018) tems for Energy‐Efficient Buildings, pp. 1–8 (2013)
94. Günter, N., Marinopoulos, A.: Energy storage for grid services and 115. Monacchi, A., et al.: Greend: an energy consumption dataset of
applications: classification, market review, metrics, and methodology for households in Italy and Austria. In: 2014 IEEE International
 25152947, 0, Downloaded from https://ietresearch.onlinelibrary.wiley.com/doi/10.1049/stg2.12161 by Universitätsbibliothek Erlangen- Nürnberg, Wiley Online Library on [23/10/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
ALTAMIMI ET AL.
- 25

Conference on Smart Grid Communications (SmartGridComm), pp. 137. Iwamoto, S., Tamura, Y.: A load flow calculation method for ill‐
511–516. IEEE (2014) conditioned power systems. IEEE Trans. Power Apparatus Syst. 4,
116. Gao, J., et al.: Plaid: a public dataset of high‐resoultion electrical 1736–1743 (1981). https://doi.org/10.1109/tpas.1981.316511
appliance measurements for load identification research: demo abstract. 138. Zollenkopf, K.: Load‐flow calculation using loss‐minimisation tech-
In: Proceedings of the 1st ACM Conference on Embedded Systems for niques. Proc. Inst. Electr. Eng. 115(1), 121–127 (1968). IET. https://
Energy‐Efficient Buildings, pp. 198–199 (2014) doi.org/10.1049/piee.1968.0019
117. Murray, D., et al.: A Data Management Platform for Personalised Real‐ 139. Stott, B., Alsac, O.: Fast decoupled load flow. IEEE Trans. Power
Time Energy Feedback (2015) Apparatus Syst. 3, 859–869 (1974). https://doi.org/10.1109/tpas.1974.
118. Batra, N., et al.: A comparison of non‐intrusive load monitoring methods 293985
for commercial and residential buildings. arXiv preprint arXiv:1408.6595 140. Barsali, S., et al.: Benchmark Systems for Network Integration of
(2014) Renewable and Distributed Energy Resources (2014)
119. Uttama Nambi, A.S., Reyes Lua, A., Prasad, V.R.: Loced: location‐aware 141. Dickert, J., Domagk, M., Schegner, P.: Benchmark low voltage distri-
energy disaggregation framework. In: Proceedings of the 2nd Acm bution networks based on cluster analysis of actual grid properties. In:
International Conference on Embedded Systems for Energy‐Efficient 2013 IEEE Grenoble Conference, pp. 1–6. IEEE (2013)
Built Environments, pp. 45–54 (2015) 142. Scheidler, A., et al.: Der Integration Study for the German State of
120. Kahl, M., et al.: Whited‐a worldwide household and industry transient Hesse–Methodology and Key Results (2019)
energy data set. In: 3rd International Workshop on Non‐intrusive Load 143. Garske, S., Schloemer, G., Hofmann, L.: Evaluation of reactive po-
Monitoring, pp. 1–4 (2016) wer management strategies and grid loss characteristics based on
121. Gulati, M., Ram, S.S., Singh, A.: An in depth study into using emi generic distribution grid models. In: NEIS 2017; Conference on
signatures for appliance identification. In: Proceedings of the 1st ACM Sustainable Energy Supply and Energy Storage Systems, pp. 1–6.
Conference on Embedded Systems for Energy‐Efficient Buildings, pp. VDE (2017)
70–79 (2014) 144. Adamczyk, A., et al.: Generic 12‐bus test system for wind power inte-
122. Ribeiro, M., et al.: Sustdataed: a public dataset for electric energy gration studies. In: 2013 15th European Conference on Power Elec-
disaggregation research. In: Proceedings of the ICT for Sustainability, tronics and Applications (EPE), pp. 1–6. IEEE (2013)
pp. 14–16 (2016) 145. Kraiczy, M., Stetz, T., Braun, M.: Parallel operation of transformers with
123. Gisler, C., et al.: Appliance consumption signature database and on load tap changer and photovoltaic systems with reactive power
recognition test protocols. In: 2013 8th International Workshop on control. IEEE Trans. Smart Grid 9(6), 6419–6428 (2017). https://doi.
Systems, Signal Processing and their Applications (WoSSPA), pp. org/10.1109/tsg.2017.2712633
336–341. IEEE (2013) 146. Peyghami, S., et al.: Standard test systems for modern power system
124. Picon, T., et al.: Cooll: controlled on/off loads library, a public dataset analysis: an overview. IEEE Ind. Electron. Mag. 13(4), 86–105 (2019).
of high‐sampled electrical signals for appliance identification. arXiv https://doi.org/10.1109/mie.2019.2942376
preprint arXiv:1611.05803 (2016) 147. Xu, G., et al.: Toward integrating distributed energy resources and storage
125. Holcomb, C.: Pecan street inc.: a test‐bed for nilm. In: International devices in smart grid. IEEE Internet Things J. 4(1), 192–204 (2016)
Workshop on Non‐intrusive Load Monitoring, Pittsburgh, PA, USA 148. Hemmati, R., Hooshmand, R.‐A., Khodabakhshian, A.: Comprehensive
(2012) review of generation and transmission expansion planning. IET Gener.
126. Kriechbaumer, T., Jacobsen, H.‐A.: Blond, a building‐level office Transm.Distrib. 7(9), 955–964 (2013). https://doi.org/10.1049/iet‐gtd.
environment dataset of typical electrical appliances. Sci. Data 5(1), 1–14 2013.0031
(2018). https://doi.org/10.1038/sdata.2018.48 149. Leou, R.‐C.: A multi‐year transmission planning under a deregulated
127. Makonin, S., Wang, Z.J., Tumpach, C.: Rae: the rainforest automation market. Int. J. Electr. Power Energy Syst. 33(3), 708–714 (2011).
energy dataset for smart grid meter data analysis. Data 3(1), 8 (2018). https://doi.org/10.1016/j.ijepes.2010.11.020
https://doi.org/10.3390/data3010008 150. Akbari, T., et al.: Towards integrated planning: simultaneous trans-
128. Maasoumy, M., et al.: Berds‐berkeley energy disaggregation data set. In: mission and substation expansion planning. Elec. Power Syst. Res. 86,
Proceedings of the Workshop on Big Learning at the Conference on 131–139 (2012). https://doi.org/10.1016/j.epsr.2011.12.012
Neural Information Processing Systems (NIPS), pp. 1–6 (2013) 151. Gu, Y., McCalley, J.D., Ni, M.: Coordinating large‐scale wind integration
129. Johnson, G., Beausoleil‐Morrison, I.: Electrical‐end‐use data from 23 and transmission planning. IEEE Trans. Sustain. Energy 3(4), 652–659
houses sampled each minute for simulating micro‐generation systems. (2012). https://doi.org/10.1109/tste.2012.2204069
Appl. Therm. Eng. 114, 1449–1456 (2017). https://doi.org/10.1016/j. 152. Rahmani, M., et al.: Efficient method for ac transmission network
applthermaleng.2016.07.133 expansion planning. Elec. Power Syst. Res. 80(9), 1056–1064 (2010).
130. Rashid, H., Singh, P., Singh, A.: I‐blend, a campus‐scale commercial and https://doi.org/10.1016/j.epsr.2010.01.012
residential buildings electrical energy dataset. Sci. Data 6(1), 1–12 153. Zhao, J.H., et al.: Flexible transmission network planning considering
(2019). https://doi.org/10.1038/sdata.2019.15 distributed generation impacts. IEEE Trans. Power Syst. 26(3),
131. Rui, H., Arnold, M., Wellssow, W.H.: Synthetic medium voltage grids for 1434–1443 (2010). https://doi.org/10.1109/tpwrs.2010.2089994
the assessment of smart grid techniques. In: 2012 3rd IEEE PES Inno- 154. Cedeño, E.B., Arora, S.: Performance comparison of transmission
vative Smart Grid Technologies Europe (ISGT Europe), pp. 1–8. IEEE network expansion planning under deterministic and uncertain condi-
(2012) tions. Int. J. Electr. Power Energy Syst. 33(7), 1288–1295 (2011).
132. Seack, A., Kays, J., Rehtanz, C.: Generating low voltage grids on the basis https://doi.org/10.1016/j.ijepes.2011.05.005
of public available map data. CIRED Workshop: Rome, Italy. 11, (2014) 155. Feng, Y., Ryan, S.M.: Scenario construction and reduction applied to
133. Han, X., et al.: Real‐time measurements and their effects on state stochastic power generation expansion planning. Comput. Oper. Res.
estimation of distribution power system. In: IEEE PES ISGT Europe 40(1), 9–23 (2013). https://doi.org/10.1016/j.cor.2012.05.005
2013, pp. 1–5. IEEE (2013) 156. Pereira, A.J., Saraiva, J.T.: Generation expansion planning (gep)–a long‐
134. Bracale, A., et al.: Active Management of Distribution Networks with term approach using system dynamics and genetic algorithms (gas).
the Atlantide Models (2012) Energy 36(8), 5180–5199 (2011). https://doi.org/10.1016/j.energy.
135. Anderson, P.M., Fouad, A.A.: Power System Control and Stability. John 2011.06.021
Wiley & Sons (2008) 157. Huang, Y.‐F., et al.: State estimation in electric power grids: meeting new
136. Tripathy, S., et al.: Load‐flow solutions for ill‐conditioned power sys- challenges presented by the requirements of the future grid. IEEE
tems by a Newton‐like method. IEEE Trans. Power Apparatus Syst. 10, Signal Process. Mag. 29(5), 33–43 (2012). https://doi.org/10.1109/msp.
3648–3657 (1982). https://doi.org/10.1109/tpas.1982.317050 2012.2187037
 25152947, 0, Downloaded from https://ietresearch.onlinelibrary.wiley.com/doi/10.1049/stg2.12161 by Universitätsbibliothek Erlangen- Nürnberg, Wiley Online Library on [23/10/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
26
- ALTAMIMI ET AL.

158. Ahmad, F., et al.: Distribution system state estimation‐a step towards 179. Dugan, R.C., et al.: Roadmap for the ieee pes test feeders. In: 2009
smart grid. Renew. Sustain. Energy Rev. 81, 2659–2671 (2018). https:// IEEE/PES Power Systems Conference and Exposition, pp. 1–4 (2009)
doi.org/10.1016/j.rser.2017.06.071 180. Dugan, R., Arritt, R.: The Ieee 8500‐node Test Feeder. Electric Power
159. Ghaddar, B., Marecek, J., Mevissen, M.: Optimal power flow as a Research Institute, Palo Alto (2010)
polynomial optimization problem. IEEE Trans. Power Syst. 31(1), 181. Schneider, K.P., Fuller, J.C., Chassin, D.P.: Multi‐state load models for
539–546 (2015). https://doi.org/10.1109/tpwrs.2015.2390037 distribution system analysis. IEEE Trans. Power Syst. 26(4), 2425–2433
160. Zhang, B., Wang, M., Su, W.: Reliability analysis of power systems in- (2011). https://doi.org/10.1109/tpwrs.2011.2132154
tegrated with high‐penetration of power converters. IEEE Trans. Power 182. Arritt, R.F., Dugan, R.C.: Distribution system analysis and the future
Syst. 36(3), 1998–2009 (2020). https://doi.org/10.1109/tpwrs.2020. smart grid. IEEE Trans. Ind. Appl. 47(6), 2343–2350 (2011). https://
3032579 doi.org/10.1109/tia.2011.2168932
161. Krishnan, V., et al.: Validation of synthetic us electric power distribution 183. Penido, D., et al.: Solving the single‐circuit nev test case using the
system data sets. IEEE Trans. Smart Grid 11(5), 4477–4489 (2020). current injection full‐Newton power flow. In: 2008 IEEE Power and
https://doi.org/10.1109/tsg.2020.2981077 Energy Society General Meeting‐Conversion and Delivery of Electrical
162. Mateo, C., et al.: Building large‐scale us synthetic electric distribution Energy in the 21st Century, pp. 1–7. IEEE (2008)
system models. IEEE Trans. Smart Grid 11(6), 5301–5313 (2020). 184. Variz, A., et al.: Harmonic analysis of the power distribution neutral‐to‐
https://doi.org/10.1109/tsg.2020.3001495 earth voltage (nev) test case using four‐wire three‐phase harmonic
163. Ieee pes cascading failure working group. [Online]. https://site.ieee. current injection method. In: 2009 IEEE Power & Energy Society
org/pes‐cascading/publications/ General Meeting, pp. 1–7. IEEE (2009)
164. Kersting, W.H.: Radial distribution test feeders. IEEE Trans. Power 185. Horton, R., et al.: Effect of line modeling methods on neutral‐to‐earth
Syst. 6(3), 975–985 (1991). https://doi.org/10.1109/59.119237 voltage analysis of multi‐grounded distribution feeders. In: 2011 IEEE/
165. Kersting, W.: A comprehensive distribution test feeder. In: IEEE PES PES Power Systems Conference and Exposition, pp. 1–6. IEEE (2011)
T&D 2010, pp. 1–4. IEEE (2010) 186. Yuan, Z., Hesamzadeh, M.R.: Hierarchical coordination of tso‐dso
166. De Oliveira‐De Jesus, P.M., Quintana, A.R.: New formulation for dis- economic dispatch considering large‐scale integration of distributed
tribution system state estimation. In: 2012 VI Andean Region Inter- energy resources. Appl. Energy 195, 600–615 (2017). https://doi.org/
national Conference, pp. 3–6. IEEE (2012) 10.1016/j.apenergy.2017.03.042
167. Mojumdar, R.R., Arboleya, P., González‐Morán, C.: Step‐voltage regu- 187. Yang, T., et al.: Optimal planning of communication system of cps for
lator model test system. In: 2015 IEEE Power & Energy Society distribution network. J. Sens. 2017, 1–10 (2017). https://doi.org/10.
General Meeting, pp. 1–5. IEEE (2015) 1155/2017/9303989
168. Eajal, A.A., El‐Hawary, M.: Optimal capacitor placement and sizing in 188. [Online]. https://cmte.ieee.org/pes‐testfeeders/resources/
unbalanced distribution systems with harmonics consideration using 189. Ni, F., et al.: Uncertainty analysis of aggregated smart meter data for
particle swarm optimization. IEEE Trans. Power Deliv. 25(3), state estimation. In: 2016 IEEE International Workshop on Applied
1734–1741 (2010). https://doi.org/10.1109/tpwrd.2009.2035425 Measurements for Power Systems (AMPS), pp. 1–6. IEEE (2016)
169. Gottwalt, S., et al.: Modeling and valuation of residential demand 190. Khalid Mehmood, K., et al.: Optimal sizing and allocation of battery
flexibility for renewable energy integration. IEEE Trans. Smart Grid energy storage systems with wind and solar power dgs in a distribution
8(6), 2565–2574 (2016). https://doi.org/10.1109/tsg.2016.2529424 network for voltage regulation considering the lifespan of batteries. IET
170. Matic‐Cuka, B., Kezunovic, M.: Islanding detection for inverter‐based Renew. Power Gener. 11(10), 1305–1315 (2017). https://doi.org/10.
distributed generation using support vector machine method. IEEE 1049/iet‐rpg.2016.0938
Trans. Smart Grid 5(6), 2676–2686 (2014). https://doi.org/10.1109/tsg. 191. Electric grid test cases. (2023), [Online] https://electricgrids.engr.tamu.
2014.2338736 edu/electric‐grid‐test‐cases/ Accessed 19 July 2023
171. Hedayati, H., Nabaviniaki, S.A., Akbarimajd, A.: A method for place- 192. Del Grosso, G., Pichler, G., Piantanida, P.: Privacy‐preserving synthetic
ment of dg units in distribution networks. IEEE Trans. Power Deliv. smart meters data. In: 2021 IEEE Power & Energy Society Innovative
23(3), 1620–1628 (2008). https://doi.org/10.1109/tpwrd.2007.916106 Smart Grid Technologies Conference (ISGT), pp. 1–5. IEEE (2021)
172. Nick, M., Cherkaoui, R., Paolone, M.: Optimal allocation of dispersed 193. Farokhi, F., Sandberg, H.: Fisher information as a measure of privacy:
energy storage systems in active distribution networks for energy bal- preserving privacy of households with smart meters using batteries.
ance and grid support. IEEE Trans. Power Syst. 29(5), 2300–2310 IEEE Trans. Smart Grid 9(5), 4726–4734 (2017). https://doi.org/10.
(2014). https://doi.org/10.1109/tpwrs.2014.2302020 1109/tsg.2017.2667702
173. Dall’Anese, E., Zhu, H., Giannakis, G.B.: Distributed optimal power 194. Buescher, N., et al.: Two is not enough: privacy assessment of aggre-
flow for smart microgrids. IEEE Trans. Smart Grid 4(3), 1464–1475 gation schemes in smart metering. Proc. Priv. Enhancing Technol.
(2013). https://doi.org/10.1109/tsg.2013.2248175 2017(4), 198–214 (2017). https://doi.org/10.1515/popets‐2017‐0045
174. Zin, A.A.M., et al.: Two circular‐updating hybrid heuristic methods for 195. Batra, N., et al.: Nilmtk: an open source toolkit for non‐intrusive load
minimum‐loss reconfiguration of electrical distribution network. IEEE monitoring. In: Proceedings of the 5th International Conference on
Trans. Power Syst. 28(2), 1318–1323 (2012) Future Energy Systems, pp. 265–276 (2014)
175. Rasheduzzaman, M., Mueller, J.A., Kimball, J.W.: Reduced‐order small‐ 196. Menati, A., Lee, K., Xie, L.: Modeling and analysis of utilizing crypto-
signal model of microgrid systems. IEEE Trans. Sustain. Energy 6(4), currency mining for demand flexibility in electric energy systems: a
1292–1305 (2015). https://doi.org/10.1109/tste.2015.2433177 synthetic Texas grid case study. IEEE Trans. Energy Mark. Pol. Regul.
176. Meng, F., et al.: Dual passive harmonic reduction at dc link of the 1(1), 1–10 (2023). https://doi.org/10.1109/tempr.2022.3230953
double‐star uncontrolled rectifier. IEEE Trans. Ind. Electron. 66(4), 197. Menati, A., et al.: High resolution modeling and analysis of crypto-
3303–3309 (2018). https://doi.org/10.1109/tie.2018.2844840 currency mining’s impact on power grids: carbon footprint, reliability,
177. Hobbs, B.F., Rijkers, F.A.: Strategic generation with conjectured trans- and electricity price. Adv. Appl. Energy 10, 100136 (2023). [Online].
mission price responses in a mixed transmission pricing system‐part i: https://doi.org/10.1016/j.adapen.2023.100136
formulation. IEEE Trans. Power Syst. 19(2), 707–717 (2004). https:// 198. Hu, J., Vasilakos, A.V.: Energy big data analytics and security: challenges
doi.org/10.1109/tpwrs.2003.821628 and opportunities. IEEE Trans. Smart Grid 7(5), 2423–2436 (2016).
178. Zhao, S., Meng, X., Song, X.: Increasing maximum penetration of https://doi.org/10.1109/tsg.2016.2563461
distributed generation by voltage regulation in smart distribution grid. 199. Chen, D., et al.: Sunspot: exposing the location of anonymous solar‐
In: 2015 5th International Conference on Electric Utility Deregulation powered homes. In: Proceedings of the 3rd ACM International Confer-
and Restructuring and Power Technologies (DRPT), pp. 1894–1898. ence on Systems for Energy‐Efficient Built Environments, pp. 85–94
IEEE (2015) (2016)
 25152947, 0, Downloaded from https://ietresearch.onlinelibrary.wiley.com/doi/10.1049/stg2.12161 by Universitätsbibliothek Erlangen- Nürnberg, Wiley Online Library on [23/10/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
ALTAMIMI ET AL.
- 27

200. Rossi, B., Chren, S.: Smart grids data analysis: a systematic mapping Acoustics, Speech and Signal Processing (ICASSP), pp. 2687–2691.
study. IEEE Trans. Ind. Inf. 16(6), 3619–3639 (2019). https://doi.org/ IEEE (2019)
10.1109/tii.2019.2954098 221. Isikman, A.O., et al.: Power scheduling in privacy enhanced microgrid
201. Zhang, C., et al.: Generative adversarial network for synthetic time se- networks with renewables and storage. In: 2016 13th IEEE Annual
ries data generation in smart grids. In: 2018 IEEE International Con- Consumer Communications & Networking Conference (CCNC), pp.
ference on Communications, Control, and Computing Technologies for 405–410. IEEE (2016)
Smart Grids (SmartGridComm), pp. 1–6. IEEE (2018) 222. Farokhi, F., Sandberg, H.: Fisher information privacy with application to
202. Li, J., et al.: Energy data generation with wasserstein deep convolutional smart meter privacy using hvac units. In: Privacy in Dynamical Systems,
generative adversarial networks. Energy 257, 124694 (2022). https:// pp. 3–17. Springer (2020)
doi.org/10.1016/j.energy.2022.124694 223. Sun, Y., Lampe, L., Wong, V.W.: Smart meter privacy: exploiting the
203. El Kababji, S., Srikantha, P.: A data‐driven approach for generating potential of household energy storage units. IEEE Internet Things J.
synthetic load patterns and usage habits. IEEE Trans. Smart Grid 11(6), 5(1), 69–78 (2017). https://doi.org/10.1109/jiot.2017.2771370
4984–4995 (2020). https://doi.org/10.1109/tsg.2020.3007984 224. Chen, D., et al.: Combined heat and privacy: preventing occupancy
204. Chen, Y., et al.: Model‐free renewable scenario generation using detection from smart meters. In: 2014 IEEE International Conference
generative adversarial networks. IEEE Trans. Power Syst. 33(3), on Pervasive Computing and Communications (PerCom), pp. 208–215.
3265–3275 (2018). https://doi.org/10.1109/tpwrs.2018.2794541 IEEE (2014)
205. Erdemir, E., Gündüz, D., Dragotti, P.L.: Smart meter privacy. In: Pri- 225. Liu, E., Cheng, P.: Achieving privacy protection using distributed load
vacy in Dynamical Systems, pp. 19–41. Springer (2020) scheduling: a randomized approach. IEEE Trans. Smart Grid 8(5),
206. Naperville Smart Meter V. City of Naperville. p. 521, (2018) 2460–2473 (2017). https://doi.org/10.1109/tsg.2017.2703400
207. Cuijpers, C., Koops, B.‐J.: Smart metering and privacy in europe: lessons 226. Shokry, M., et al.: Systematic survey of advanced metering infrastructure
from the Dutch case. In: European Data Protection: Coming of Age, security: vulnerabilities, attacks, countermeasures, and future vision.
pp. 269–293. Springer (2013) Future Generat. Comput. Syst. 136, 358–377 (2022). https://doi.org/
208. Farokhi, F.: Review of results on smart‐meter privacy by data manipu- 10.1016/j.future.2022.06.013
lation, demand shaping, and load scheduling. IET Smart Grid 3(5), 227. Alsharif, A., et al.: Epda: efficient and privacy‐preserving data collection
605–613 (2020). https://doi.org/10.1049/iet‐stg.2020.0129 and access control scheme for multi‐recipient ami networks. IEEE
209. Ács, G., Castelluccia, C.: I have a dream!(differentially private smart Access 7, 27829–27845 (2019). https://doi.org/10.1109/access.2019.
metering). In: International Workshop on Information Hiding, pp. 2900934
118–132. Springer (2011) 228. Ibrahem, M.I., et al.: Pmbfe: efficient and privacy‐preserving moni-
210. Sandberg, H., Dán, G., Thobaben, R.: Differentially private state esti- toring and billing using functional encryption for ami networks. In: 2020
mation in distribution networks with smart meters. In: 2015 54th IEEE International Symposium on Networks, Computers and Communica-
Conference on Decision and Control (CDC), pp. 4492–4498. IEEE tions (ISNCC), pp. 1–7. IEEE (2020)
(2015) 229. Naseer, H., Bhutta, M.N.M., Alojail, M.A.: A key transport protocol for
211. Kserawi, F., Al‐Marri, S., Malluhi, Q.: Privacy‐preserving fog aggrega- advance metering infrastructure (ami) based on public key cryptography.
tion of smart grid data using dynamic differentially‐private data In: 2020 International Conference on Cyber Warfare and Security
perturbation. IEEE Access 10, 43159–43174 (2022). https://doi.org/ (ICCWS), pp. 1–5. IEEE (2020)
10.1109/access.2022.3167015 230. Lee, Y., Hwang, E., Choi, J.: A unified approach for compression and
212. Eibl, G., Engel, D.: Influence of data granularity on smart meter privacy. authentication of smart meter reading in ami. IEEE Access 7,
IEEE Trans. Smart Grid 6(2), 930–939 (2014). https://doi.org/10. 34383–34394 (2019). https://doi.org/10.1109/access.2019.2903574
1109/tsg.2014.2376613 231. Huang, C.: Forest management and resource monitoring based on ami
213. Merad‐Boudia, O.R., Senouci, S.M.: An efficient and secure multidi- intrusion detection algorithm and artificial intelligence. J. Ambient
mensional data aggregation for fog‐computing‐based smart grid. IEEE Intell. Hum. Comput., 1–15 (2021)
Internet Things J. 8(8), 6143–6153 (2020). https://doi.org/10.1109/jiot. 232. Yao, R., et al.: Intrusion detection system in the advanced metering
2020.3040982 infrastructure: a cross‐layer feature‐fusion cnn‐lstm‐based approach.
214. Keoh, S.L., Ang, Y.H., Tang, Z.: A lightweight privacy‐preserved spatial Sensors 21(2), 626 (2021). https://doi.org/10.3390/s21020626
and temporal aggregation of energy data. In: 2015 11th International 233. Sun, C.‐C., et al.: Intrusion detection for cybersecurity of smart meters.
Conference on Information Assurance and Security (IAS), pp. 1–6. IEEE Trans. Smart Grid 12(1), 612–622 (2020). https://doi.org/10.
IEEE (2015) 1109/tsg.2020.3010230
215. Danezis, G., et al.: Smart meter aggregation via secret‐sharing. In: 234. Sajid, Z., Javaid, A.: A stochastic approach to energy policy and man-
Proceedings of the First ACM Workshop on Smart Energy Grid Se- agement: a case study of the Pakistan energy crisis. Energies 11(9), 2424
curity, pp. 75–80 (2013) (2018). https://doi.org/10.3390/en11092424
216. Giaconi, G., Gündüz, D., Poor, H.V.: Smart meter privacy with renewable 235. Chen, W., et al.: Data quality of electricity consumption data in a smart
energy and an energy storage device. IEEE Trans. Inf. Forensics Secur. grid environment. Renew. Sustain. Energy Rev. 75, 98–105 (2017).
13(1), 129–142 (2017). https://doi.org/10.1109/tifs.2017.2744601 https://doi.org/10.1016/j.rser.2016.10.054
217. Li, S., Khisti, A., Mahajan, A.: Information‐theoretic privacy for smart 236. Allison, P.D.: Missing Data. Sage publications (2001)
metering systems with a rechargeable battery. IEEE Trans. Inf. Theor. 237. Uniform Business Practices for the Retail Energy Market. Edison
64(5), 3679–3695 (2018). https://doi.org/10.1109/tit.2018.2809005 Electric Institute, (2000)
218. Kserawi, F., Malluhi, Q.M.: Privacy preservation of aggregated data 238. Choy, K.: Outlier detection for stationary time series. J. Stat. Plann.
using virtual battery in the smart grid. In: 2020 IEEE 6th International Inference 99(2), 111–127 (2001). https://doi.org/10.1016/s0378‐3758
Conference on Dependability in Sensor, Cloud and Big Data Systems (01)00081‐7
and Application (DependSys), pp. 106–111 (2020) 239. Korki, M., Hosseinzadeh, N., Moazzeni, T.: Performance evaluation of a
219. Giaconi, G., Gündüz, D.: Smart meter privacy with renewable energy narrowband power line communication for smart grid with noise
and a finite capacity battery. In: 2016 IEEE 17th International Work- reduction technique. IEEE Trans. Consum. Electron. 57(4), 1598–1606
shop on Signal Processing Advances in Wireless Communications (2011). https://doi.org/10.1109/tce.2011.6131131
(SPAWC), pp. 1–5. IEEE (2016) 240. Nassar, M., et al.: Cyclostationary noise modeling in narrowband
220. Erdemir, E., Dragotti, P.L., Gündüz, D.: Privacy‐cost trade‐off in a powerline communication for smart grid applications. In: 2012 IEEE
smart meter system with a renewable energy source and a rechargeable International Conference on Acoustics, Speech and Signal Processing
battery. In: ICASSP 2019‐2019 IEEE International Conference on (ICASSP), pp. 3089–3092. IEEE (2012)
 25152947, 0, Downloaded from https://ietresearch.onlinelibrary.wiley.com/doi/10.1049/stg2.12161 by Universitätsbibliothek Erlangen- Nürnberg, Wiley Online Library on [23/10/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
28
- ALTAMIMI ET AL.

241. Radhakrishnan, A., Das, S.: Quality assessment of smart grid data. In: International Workshop on Non‐intrusive Load Monitoring, pp. 34–38
2018 20th National Power Systems Conference (NPSC), pp. 1–6. IEEE (2020)
(2018) 248. He, K., et al.: Post‐processing for event‐based non‐intrusive load
242. Alhussein, M., Aurangzeb, K., Haider, S.I.: Hybrid cnn‐lstm model for monitoring. In: 4th International Workshop on Non‐intrusive Load
short‐term individual household load forecasting. IEEE Access 8, Monitoring, pp. 1–4 (2018)
180544–180557 (2020). https://doi.org/10.1109/access.2020.3028281 249. Kong, W., et al.: A practical solution for non‐intrusive type ii load
243. Daki, H., et al.: Big data management in smart grid: concepts, re- monitoring based on deep learning and post‐processing. IEEE Trans.
quirements and implementation. J. Big Data 4(1), 1–19 (2017). https:// Smart Grid 11(1), 148–160 (2019). https://doi.org/10.1109/tsg.2019.
doi.org/10.1186/s40537‐017‐0070‐y 2918330
244. Bhattarai, B.P., et al.: Big data analytics in smart grids: state‐of‐the‐art, 250. Pinto, G., et al.: Transfer learning for smart buildings: a critical review of
challenges, opportunities, and future directions. IET Smart Grid 2(2), algorithms, applications, and future perspectives. Adv. Appl. Energy 5,
141–154 (2019). https://doi.org/10.1049/iet‐stg.2018.0261 100084 (2022). https://doi.org/10.1016/j.adapen.2022.100084
245. Wang, Y., et al.: Clustering of electricity consumption behavior dynamics
toward big data applications. IEEE Trans. Smart Grid 7(5), 2437–2447
(2016). https://doi.org/10.1109/tsg.2016.2548565
246. Shyam, R., et al.: Apache spark a big data analytics platform for smart How to cite this article: Altamimi, E., et al.: Smart grid
grid. Proc. Technol. 21, 171–178 (2015). https://doi.org/10.1016/j. public datasets: characteristics and associated
protcy.2015.10.085
applications. IET Smart Grid. 1–28 (2024). https://doi.
247. Khazaei, M., Stankovic, L., Stankovic, V.: Evaluation of low‐complexity
supervised and unsupervised nilm methods and pre‐processing for org/10.1049/stg2.12161
detection of multistate white goods. In: Proceedings of the 5th