Adaptive Weighted Fuzzy Clustering Algorithm for

Load Profiling of Smart Grid Customers

Yun Lu, Tiankui Zhang, Zhimin Zeng
Beijing Key Laboratory of Network System Architecture and Convergence,
School of Information and Communication Engineering,
Beijing University of Posts and Telecommunications, Beijing, 100876, China
Email: {luyun, zhangtiankui, zengzm}@bupt.edu.cn

Abstract—With the rapid development of the smart grid, a (FCM) [9] and Fuzzy K-Means [10]) and model-based algo-
large volume of smart meter data are collected in the form of rithms (such as Finite Mixture Modeling (FMM) [11], Self-
time series, which is called load profiles. This paper investigates Organizing Map (SOM) [12] and Support Vector Machines
the load profile clustering of smart grid customers, which is
significant for many applications. An adaptive weighted fuzzy (SVM) [13]). Partition algorithm is more widely used because
clustering algorithm is proposed to cluster load profiles, where of its less complexity and faster convergence speed compared
Principle Component Analysis (PCA) is used to reduce the to the model-based algorithm. Some combining algorithms are
data dimension, and then weighted Fuzzy C-Means (FCM) is also proposed to make up the weakness of single algorithm. A
adopted to cluster the big data. The optimal number of clusters hybrid clustering algorithms combining partitioning methods
is determined adaptively by integrating a clustering validity
function into the clustering algorithm. The simulation results and hierarchical methods is proposed in [14], which is suitable
show that the proposed algorithm could achieve considerable for large scale load data clustering.
improvement both in time complexity and clustering accuracy FCM is a soft partitioning method which is superior to the
through comparing four clustering validity indexes. traditional crisp partitioning method like K-Means, because
most objects have ambiguous attributes and may belong to
Index Terms—Load profiles, customer classification, PCA, more than one cluster [9]. However, there is a shortcoming of
weighted FCM, cluster validity
FCM that the number of clusters should be pre-defined. Many
I. I NTRODUCTION efforts have been made to determine the optimal number of
clusters, a popular way of determining the optimal number of
With the development of the smart grid, sensors and meter- clusters is to find the “knee” of the curve [9], [10], which is
ing devices such as smart meters are increasingly installed still a challenge.
into distribution networks [1]. The way to use and extract Due to the massive characteristic of the load data, many
valuable information from these big data is a serious challenge studies have attached great importance to reducing the data
faced by the electricity companies. The time-varying power dimension before clustering analysis, aiming to lower the
consumption data collected from smart meters form load computational complexity. There are three different dimension
profiles, and different load profiles of customers show their reduction methods are discussed in [14], respectively Sammon
different consumption behaviors. Conducting load profile clus- Mapping, SOM and PCA. Results show that PCA is superior
tering is an important task in grouping electricity customers to other methods because of the minimum computation time.
into classes and obtaining the typical load profiles (TLPs), In order to assess the performance of different algorithms,
which is significant for many applications, such as load fore- a measure of adequacy should be used. Research in this area
casting [2], abnormal electricity consumption detection [3], has proposed many different indices [10], [14] according to a
load control [4], designing electricity tariff offers [5], real- metric that samples within one partition should be compacted
time demand side response policy [6] and developing market and samples within different clusters should be separated, such
strategies [7]. as the clustering dispersion index (CDI), the modified Dunn
Many methods or techniques for clustering load profiles index (MDI), the scatter index (SI), the Davies-Bouldin index
have been proposed in literatures. They investigate clustering (DBI), and the sum of squared error (SSE).
analysis by using the existed classical clustering methods Existing literatures of load clustering analysis mainly focus
including partitioning methods, hierarchical methods, density- on the selection of clustering algorithm, but lack of considera-
based methods, grid-based methods and model-based methods. tion on different importance of features. In this paper, in order
Considering the large scale characteristic of load data, cluster- to cover the issues above, we propose an adaptive weighted
ing methods applied to load profile clustering mainly focus on Fuzzy C-Means clustering algorithm based on PCA (PCA-
partitioning algorithm (such as K-means [8], Fuzzy C-Means WFCM) to cluster load profiles and determine TLPs of smart
grid customers. Based on the conventional FCM algorithm,
This work is supported by the National Natural Science Foundation of
China (No.61461029), and Fundamental Research Funds for the Central we add different weights to the reduced features by variance
Universities (2014ZD03-01). contribution rate obtained from the PCA processing to improve
clustering accuracy. Moreover, the clustering validity function Step 4) Cluster validity analysis and TLPs generate. Four
Davies-Bouldin (DB) is integrated into the clustering algo- cluster validity indexes are used to evaluate the
rithm to determine the optimal number of clusters adaptively. clustering results, which are PC, PE, SSE and DB
The data sets based on the real smart meter readings collected respectively, and the TLPs are generated by the center
in Ireland are applied to evaluate the proposed PCA-WFCM profile of every cluster after clustering.
algorithm.
This paper is organized as follows. The load profile clus-
   



tering procedure is described in Section II. The PCA-WFCM
clustering algorithm is presented in Section III. The simulation
results are given in Section IV, and then conclusions are
presented in Section V. 

II. C LUSTERING P ROCEDURE

Load clustering algorithms are used to group similar load
profiles. A load profile represents the power demand over a
specific period of time with different intervals. The data set
used in this paper was gathered by Commission for Energy 
Regulation (CER) and accessed via the Irish Social Science
Data Archive (ISSDA) [15]. In this data set, smart meter
records are given at half hourly intervals. Thus, there are 48
records per day and all the records are used as load profile 
 



features. Fig. 1. Two-stage clustering approach.
We describe clustering data as follows. Load data are
denoted as X = {x1 , x2 , · · · , xN }, where xj is a daily
load profile, and xj = {xj1 , xj2 , · · · , xjS } demonstates that III. PCA-WFCM A LGORITHM
each load profile has S features, where S ≤ 48. Clustering
analysis is to divide X into K classes, 2 ≤ K ≤ N , and In this paper, an adaptive weighted FCM clustering algorith-
V = {v1 , v2 , · · · , vK } is the cluster center vector. m based on PCA is proposed. The PCA is adopted to reduce
The classification of customers and generation of their TLPs the feature dimension of the original load data and far less
are achieved by applying clustering techniques, which include than 48 features are obtained. Then, the variance contribution
the following basic steps. of each principal components is used as the feature weight
and the weighted FCM clustering is performed. The cluster
Step 1) Data cleaning. Abnormal load data may occur due to number is adaptively selected based on the clustering validity
the network failure or meter error, and the values that function DB combined with the clustering algorithm.
are obviously wrong should be modified or deleted
before clustering analysis. For example, we remove A. PCA
those daily load profiles with 0 values or less than 48 The PCA [16] is a popular linear reduction method that
records. projects a data set embedded in an S-dimension space into an
Step 2) Data preprocessing. In this paper, we cluster the L-dimension subspace, with L  S. Mathematically, the pro-
load profiles based on the shape of a load profile cedure requires computing the eigenvector of the covariance
instead of by absolute values, therefore the data matrix of the data. The zero-mean data points X = {x i } =
should be normalized, that is, scaled to a specific {xi − xi }, i = 1, 2, · · · , N , and the covariance matrix C is
range such as [0.0, 1.0]. Here we apply the min-max defined as
normalization method, which is defined as normij = 1   T
N
xij −min xij C= x ix i , (1)
1 ≤ i ≤ K, 1 ≤ j ≤ N .
j
max xij −min xij , N i=1
j j
Step 3) Load clustering. As illustrated in Fig.1, A two-stage and the eigenvector of the covariance matrix is computed as
load pattern clustering method is used in the clustering
procedure. In the first stage, clustering analysis is 
S

applied on each customer’s daily load profiles in par- CU = UΛ ⇒ C = UΛUT = λi u i u i T , (2)

i=1
allel, and then typical load profiles of each customer
are obtained. In the second stage, all the typical load where U = [u1 , u2 , · · · , uS ] is eigenvector set, and Λ =
profiles in first stage are clustered and the final user diag(λ1 , λ2 , · · · , λS ) is eigenvalue. The data can be projected
classification results are obtained. In each stage, a from S-dimension into L-dimension as Y = X UL , where
proposed adaptive weighted FCM algorithm based on UL = [u1 , u2 , · · · , uL ] are the first L eigenvectors of U.
PCA is applied as the clustering algorithm. Supposing the eigenvalues of the covariance matrix C satisfy
 
S
clusters cannot be pre determined. To overcome this challenge,
λ1 ≥ λ2 ≥ · · · ≥ λS ≥ 0, then λl / λi is defined as
i=1 an adaptive approach to determine the number of clusters is
the variance contribution rate of lth principal components. proposed where no assumptions on K are made a priori. We
L 
S
λi / λi is the cumulative contribution rate of the first conduct the proposed PCA-WFCM algorithm by varying K
i=1 i=1 from 2 to Kmax , and choose the optimal value of K based
L principal components. To avoid the information loss, the
on the minimum value of cluster validity function DB which
cumulative contribution rate is generally larger than 95%.
will be presented in Part C. Here, Kmax is predefined value
B. PCA-WFCM that represents the maximal number of clusters, which should
be larger than the optimal number. Algorithm 1 presents the
The PCA-WFCM algorithm proposed in this paper is based
on conventional FCM [17] algorithm, which is a fuzzy clus- procedure of PCA-WFCM in detail.
tering algorithm. Each sample point has a membership degree
Algorithm 1 PCA-WFCM
with respect to each cluster. Since uij is defined as the
membership degree of jth sample to ith cluster, the fuzzy 1: Input the maximum number of clusters Kmax , the fuzzifier
partition matrix is U = {uij } and cluster center vector is m, iterative threshold ε, cumulative contribution rate σ and
V = {vi }. data X to be clustered.

S1 
S
In this algorithm, different weights are assigned to the 2: PCA processing. According to λi / λi ≥ σ, choose
dimension reduced features after the PCA dimension reduc- i=1 i=1
the first S1 features as the clustering features. Each feature
tion. The weights are determined according to the variance S1
contribution rate. The lth feature weight is defined as ωl = weight is wj = λj / λi , j = 1, 2, · · · , S1 . The feature
S
 i=1
λl λi , l = 1, 2 · · · , L. weight vector is W = {wj }, and the dimension-reduced
i=1 data is Xnew .
The proposed algorithm aims to determine cluster centers
3: K ← Kmax , DBmin ← ∞.
V and fuzzy partition matrix U by minimizing the objective
4: while K > 1 do
function J defined as
5: Initialize

the fuzzy partition

matrix U using Eq. (4).

K 
N
6: while
J (t) − J (t−1)
> ε do
(uij ) dij 2 ,
m
J(U, V ) = (3) 7: Calculate cluster centers V by using Eq. (7).
i=1 j=1
8: Calculate U by using Eq. (6).
subject to 9: end while

K 10: Calculate DB by using Eq. (11)-(12).
0 ≤ uij ≤ 1, uij = 1, (4) 11: if DB < DBmin then
i=1 12: DBmin ← DB,
where dij is weighted euclidean distance from sample xj to 13: U∗ ← U, V∗ ← V, where U∗ and V∗ are the
cluster center vi as optimal clustering results.
 end if
 L 14:
 K ←K −1
dij = 
2 15:
ωl (xjl − vil ) . (5)
16: end while
l=1
17: Output U∗ and V∗ .
The exponent m in (3) is a control parameter that specifies
the level of fuzziness. Larger m will attach more emphasis
on the fuzziness of the function (3). The value of m is often C. Clustering Validity Functions
set to 2. Applying derivative to (3) and (4), we can derive the Cluster validity functions are often used to evaluate the
computational formulation of uij and vi as performance of different clustering methods and deal with the
1 number of clusters. There are two important types for FCM,
uij = K  2/m−1 (m = 1), (6) one is based on the fuzzy partition of sample set and the other
 dij
dlj is on the geometric structure of sample set.
i=1
According to the main idea of validity functions based on

N
m fuzzy partition, the less fuzziness means the better perfor-
(uij ) xj mance. The representative functions for this type are Partition
j=1
vi = , (7) Coefficient (PC) and Partition Entropy (PE), which is defined

N
m
(uij ) by Bezdek [18].
j=1 PC measures the amount of “overlapping” between clusters.
where xj represents the jth sample. It is defined as
The main disadvantage of the FCM algorithm is the re-
1 
K N
quirement for the pre-determined K which is the number of 2
IPC = (uij ) , (8)
clusters. In the real load pattern clustering, the number of N i=1 j=1
where uij is the membership of data point j in cluster i. IV. S IMULATION R ESULTS
The disadvantage of PC is lack of direct connection to some
In this simulation, the data set used contains smart meter
properties of the data themselves. The optimal number of
records of more than 6000 customers in Ireland. The smart
clusters is at the maximum value.
meter readings have been recorded from 14 July, 2009 to
PE singely measures the fuzzyness of the cluster partition,
31 December, 2010. There are two groups of customers in
which is similar to the Partition Coefficient.
this data set, residential customers and small to medium
enterprizes. Note that only the data of residential customers
1 
K N
are used.
IPE = − uij log(uij ). (9)
N i=1 j=1 1000 residential customers are left to do clustering analysis
after data cleaning, and each consumer has 531 daily load
The optimal number of clusters is at the minimum value. records. In the first stage, clustering analysis is applied on
The main idea of the validity function based on the geo- each customer’s 531 daily load profiles separately. On each
metric sample structure is that samples within one partition customer’s daily load profile clustering, the number of clusters
should be compacted and samples within different clusters is determined by the clustering validity function DB, and at
should be separated, e.g., the Sum of Squares Error (SSE) last 2046 typical load profiles are obtained. In the second stage,
[19] and Davies-Bouldin (DB)[20]. the total 2046 typical load profiles are clustered again. All the
ISSE is defined by the euclidean distance of a sub class to simulation results below are based on the data of 2046 typical
the cluster center, which is given by load profiles obtained from the first stage.
Four clustering validity functions mentioned in section III

K  are applied to make performance comparison among the three
ISSE = d2 (vi , x), (10) algorithms, namely, FCM, FCM based on PCA (PCA-FCM),
i=1 x∈Xi and the proposed PCA-WFCM algorithm. We set the fuzzifier
m = 2, cumulative contribution rate σ = 0.95, iterative
1
 vi is the cluster center which is defined as vi =
where
threshold ε = 1 × 10−5 , and the maximal number of clusters
x , and d(vi , x) is the euclidean distance between
mi
x∈Xi Kmax = 20.
vectors. The smaller ISSE means the better clustering effect.
ISSE decreases with the increase of the number of clusters,
1.4
and the “knee” of the ISSE curve can be determined as the FCM
PCA−FCM
optimal cluster numbers. 1.2 PCA−WFCM

IDB aims to quantify the ratio of the total variation within 1

clusters and the separation of clusters, which is defined as
0.8
time (s)

1 
K
0.6
IDB = Ri , (11)
K i=1 0.4

0.2
d(xi ) + d(xj )
Ri = max , (12)
i=j d(ci , cj ) 0
0 200 400 600 800 1000 1200 1400 1600 1800 2000
data set size

where d(xi ) is the intra-cluster distance of cluster i, and Fig. 2. Computing time for each clustering algorithm.
d(ci , cj ) is the distance between cluster center i and j. The
optimal number of clusters should minimize the value of the
index. We conduct the three clustering algorithms by varying the
We conclude a brief summary of 4 selected cluster validity data set size from 100 to 2000, and the time of each algorithm
functions in Table I. consumed is recorded. The time consumption of three different
clustering algorithms is illustrated in Fig. 2. It shows that PCA-
TABLE I: Cluster Validity Functions FCM and PCA-WFCM obviously outperform FCM with the
Validity function Functional description Optimal growth of the data size due to the PCA dimension-reduction
1

K 
N
2
processing. It can also be found that the time consumed
Partition Coefficient IPC = N (uij ) Max(IPC )
i=1 j=1 by the proposed algorithm PCA-WFCM is very close to

K N
Partition Entropy 1
IPE = − N uij log(uij ) Min(IPE ) PCA-FCM. Hence, the weighted processing in PCA-WFCM
i=1 j=1

K  does not consume too much time but can improve clustering
Sum of Squares Error ISSE = d2 (vi , x) Min(ISSE ) performance much more in the following figure.
i=1 x∈Xi

1

K d(xi )+d(xj ) Fig. 3 shows the performance of three different clustering
Davies-Bouldin IDB = max d(c Min(IDB )
K
i=1 i=j i ,cj )
algorithms. We use four clustering validity index to assess the
performance between the three clustering algorithm, namely
 1 2.5 3.5
FCM FCM PCA−WFCM
0.8 PCA−FCM 2 PCA−FCM
PCA−WFCM PCA−WFCM
0.6 1.5
PC

PE
0.4 1

0.2 0.5 3

0 0
5 10 15 20 5 10 15 20
number of clusters number of clusters

DB
4
x 10
6 120
FCM FCM
5 PCA−FCM 100 PCA−FCM 2.5
PCA−WFCM PCA−WFCM
80
4
SSE

DB
60
3
40
2 20

1 0 2
5 10 15 20 5 10 15 20 2 4 6 8 10 12 14 16 18 20
number of clusters number of clusters number of clusters

Fig. 3. Validity index for each clustering algorithm. Fig. 5. DB index.

1 2.5 that the PCA dimension reduction processing can not only
0.8 2 lower the time complexity, but also can improve clustering
0.6
1.5 performance by removing noises. The weighted processing can
PC

PE

0.4
1 improve clustering performance further but without too much
0.2
0.5
time consumption.
0
FCM PCA−FCM PCA−WFCM FCM PCA−FCM PCA−WFCM In Fig. 3, we can see the optimal number of clusters should
4
be 8, which can be chosen by the “knee” of SSE curve.
x 10
100
The optimal number of clusters determined by PCA-WFCM
5
80 algorithm adaptively is also 8, which is chosen at the minimum
4 60 value of DB index, as we can see in Fig. 5.
SSE

DB

40
3
20
The total 2046 load profiles are clustered into 8 clusters
2 0 by the proposed PCA-WFCM clustering algorithm in terms
FCM PCA−FCM PCA−WFCM FCM PCA−FCM PCA−WFCM of the optimal number of clusters. Each center of the clusters
represents the typical load profile of that cluster of customers.
Fig. 4. Validity index for each clustering algorithm on boxplot.
The final 8 typical load profiles are presented in Fig. 6. The
horizontal axis means daily time, and the vertical axis is power
consumption, whose unit is kWh. Fig. 6 shows that the 8
typical load profiles are in big difference. Some have only
PC, PE, SSE and DB, which are described in section III. one peak, and some have two or more. Some consume more
Repeated executions of the clustering algorithms have been power at night, and some use power all the day. In a word, the
performed by varying the number of clusters from 2 to 20 load profiles encode the behaviors of the customers and can be
and computing the clustering validity index for all algorithms. utilized to provide the demand-oriented services for different
The best clustering is achieved at the minimum value of types of users.
PE, SSE, DB and maximum value of PC. In Fig. 3, we
can see the proposed PCA-WFCM algorithm always exhibits    

better performance with respect to the FCM and PCA-FCM    

algorithms, with lower values of PE, SSE, DB and higher value
of PC. In addition to the better validity index value of PCA-    

WFCM, the curve is also more smooth than the others.    

                           
The fourth subgraph in Fig. 3 shows that the DB value of
the FCM algorithm fluctuates considerably, and the magnitude    
is much larger than the other two algorithms. It is difficult to   

show a clear contrast of absolute magnitude between them
  
in curvilinear figure. Thus we use boxplot to show a further 
  
clustering performance contrast in Fig. 4. The proposed PCA-
 
 
WFCM algorithm outperform the other two algorithms with                            
best clustering effect and least volatility, and note also that Fig. 6. Daily load profiles.
the PCA-FCM algorithm is better than FCM with superi-
or clustering validity index value. Hence, we can conclude
 V. C ONCLUSION [7] R. F. Chang, C. N. Lu, “Load profiling and its applications in power
market,” Power Engineering Society General Meeting, vol. 2, 2003.
Clustering load profiles in terms of customers’ time-varying [8] G. J. Tsekouras, N. D. Hatziargyriou, and E. N. Dialynas, “Two-
power consuming behaviors is an important task in load fore- stage pattern recognition of load curves for classification of electricity
casting, abnormal electricity consumption detection, demand customers,” IEEE Transactions on Power Systems, vol. 22, no. 3, pp.
1120-1128, 2007.
side response policy and so on. In this paper, three clustering [9] Z. Zakaria, K. L. Lo, and M. H. Sohod, “Application of fuzzy clustering
algorithms, respectively FCM, PCA-FCM and the proposed to determine electricity consumers’ load profiles,” Power and Energy
PCA-WFCM algorithm, have been presented for performing Conference, 2006. PECon’06, pp. 99-103, 2006.
[10] G. Chicco, R. Napoli, and F. Piglione, “Comparisons among clustering
the load profile clustering of the smart grid customers. Then techniques for electricity customer classification,” IEEE Transactions on
four clustering validity function (PC, PE, SSE, DB) are used Power Systems PWRS, vol. 21, no. 2, pp.933-940, 2006.
to assess the performance of different clustering algorithms. [11] S. Haben, C. Singleton, and P. Grindrod, “Analysis and clustering of
residential customers energy behavioral demand using smart meter data,”
The simulation results demonstrated that the proposed PCA- IEEE Transactions on Smart Grid, 2015.
WFCM algorithm could achieve considerable improvement [12] G. Chicco, R. Napoli, and F. Piglione, “Application of clustering
both in time complexity and clustering accuracy, and the pro- algorithms and self organising maps to classify electricity customers,”
Power Tech Conference Proceedings, vol. 1, pp. 1-7, 2003.
posed algorithm can adaptively determine the optimal number [13] G. Chicco, I. S. Ilie, “Support vector clustering of electrical load pattern
of clusters without the need of pre-definition. data,” IEEE Transactions on Power Systems, vol. 24, no. 3, pp. 1619-1628,
2009.
R EFERENCES [14] Y. Zhou, C. Zheng, and X. Jian, “Ensemble clustering algorithm
combined with dimension reduction techniques for power load profiles,”
[1] A. Vojdani, “Smart Integration,” IEEE Power and Energy Magazine, vol. Proceedings of the Chinese Society of Electrical Engineering, vol. 35,
6, no. 6, pp. 71-79, 2008. no. 15, pp. 3741-3749, 2015.
[2] M. Chaouch, “Clustering-Based Improvement of Nonparametric Func- [15] http://www.ucd.ie/issda/data/commissionforenergyregulationcer/.
tional Time Series Forecasting: Application to Intra-Day Household-Level [16] J. E. Jackson, “A user’s guide to principal components,” John Wiley &
Load Curves,” IEEE Transactions on Smart Grid, vol. 5, no. 5, pp. 411- Sons, 2005.
419, 2014. [17] J. C. Bezdek, R. Ehrlich, W. Full, “FCM: The fuzzy c-means clustering
[3] A. H. Nizar, Z. Y. Dong, and Y. Wang, “Power Utility nontechnical loss algorithm,” Computers & Geosciences, vol. 10, no. 2, pp. 191-203, 1984.
analysis with extreme learning machine method,” IEEE Transactions on [18] J. C. Bezdek, “Pattern recognition with fuzzy objective function algo-
Power Systems, vol. 23, no. 3, pp. 946-955, 2008. rithms,” Advanced Applications in Pattern Recognition, 1981.
[4] H. T. Yang, S. C. Chen, and P. C. Peng, “Genetic k-means-algorithm- [19] I. P. Panapakidis, M. C. Alexiadis, G. K. Papagiannis, “Enhancing the
based classification of direct load-control curves,” Proc. Inst. Elect. Eng., clustering process in the category model load profiling,” Generation,
Gen., Transm., Distrib., vol. 152, no. 4, pp. 489-495, 2005. Transmission & Distribution, IET, vol. 9, no. 7, pp. 655-665, 2015.
[5] G. Chicco, R. Napoli, and P. Postolache, “Customer Characterization [20] D. L. Davies, D. W. Bouldin, “A Cluster Separation Measure,” IEEE
Options for Improving the Tariff Offer,” IEEE Power Engineering Review, Transactions on Pattern Analysis and Machine Intelligence, vol. PAMI-
vol. 18, no. 1, pp. 381-387, 2003. 1, no. 2, pp. 224-227, 1979.
[6] A. J. Conejo, J. M. Morales, and L. Baringo, “Real-Time Demand
Response Model,” IEEE Transactions on Smart Grid, vol. 1, no. 3, pp.
236-242, 2010.