DRAFT 1

Pricing Residential Electricity Based on Individual

Consumption Behaviors
Siddharth Patel∗ , Raffi Sevlian†, Baosen Zhang‡, and Ram Rajagopal∗
∗ Department of Civil and Environmental Engineering, Stanford University, Stanford, CA, 94305, USA
† Department of Electrical Engineering, Stanford University, Stanford, CA, 94305, USA
‡ Department of Electrical Engineering, University of Washington, Seattle, WA, 98195, USA
arXiv:1312.1243v4 [math.OC] 5 Aug 2017

plans: all customers face a single rate, and each customer’s

Abstract—The conventional practice of retail electric utilities
is to aggregate customers geographically. The utility purchases bill is computed by multiplying that rate by the customer’s
electricity for its customers via bulk transactions on the wholesale
total usage. However, most utilities are moving away from
market, and it passes these costs along to its customers, the
this simple scheme because of cost sharing among customers.
end consumers, through their rate plan. Typically, all residential
consumers are offered the same per unit rate plan, which leads The wholesale market price varies greatly based on time of day
to cost sharing. Some consumers use their electricity at peak and geographical location.1 If all consumers face a single rate,
hours, when it is more expensive on the wholesale market, and then those that use electricity at cheaper times or locations
others consume mostly at off peak hours, when it is cheaper, but are subsidizing those that use electricity at more expensive
they all enjoy the same per unit rate through their utility. This
paper proposes a method for the utility to segment a population times or locations. Enabled by advances in customer metering
of consumers on the basis of their individual consumption (e.g. advanced metering infrastructure), utilities have started
patterns. An optimal recruitment algorithm is developed to designing rates more representative of usage.
aggregate consumers into groups with a relatively low per unit New rates mainly take two forms: geography-based and
cost of electricity on the wholesale market. Enough consumers
time-based. For example, Pacific Gas and Electric Company
are grouped to ensure reduced forecast error and consequently
diminish wholesale electricity costs for the aggregator. The (PG&E) divides northern California into zones and charges
resulting optimal rate groups are stable in that no one consumer a different rate for each zone. Customers also face time of
can unilaterally improve her outcome. use (TOU) prices which charge different prices for electricity
Index Terms—Smart meter, electricity consumption, utility, consumed at different times of the day. However, both methods
aggregation, load-serving entity still do not take individual consumer behaviors into account.
It is known that consumers from the same geographical area
may exhibit drastically different behaviors [4], as illustrated in
I. I NTRODUCTION
Figure 1. Also, time of use prices may lead to large changes

E LECTRICAL retail utilities function as intermediaries be- in customer bills since small shifts in temporal behaviors can
tween the wholesale market and end consumers. Utilities result in significant cost differences. Furthermore, prior work
purchase in bulk for the consumers they represent and then in [5] established that residential consumers are sensitive to the
factor the cost of these bulk purchases into the rate plans average price of electricity, more so than to the marginal price.
offered to the consumers. This middleman service provides Therefore a constant average price is also more beneficial in
advantages for the system operator and the consumers. The influencing customer behaviors. We investigate how an agent
system operator can deal with a mass of customers through can provide differing average prices for its customers.
a single agent instead of undertaking many thousands of
transactions with end consumers. The consumers are spared
the complexities of entering the wholesale market. A more
1.5

fundamental benefit of aggregating consumers is the reduction

kW

of uncertainty in load forecasts. While the day-ahead consump-

tion of individual consumers is difficult to forecast to within
0.0

50% accuracy, the aggregate consumption of a large group

of consumers can be accurately forecasted with errors smaller 5 10 15 20
than 2% [1]–[3]. Thus, by aggregating a group of customers, Hour of day
utilities can provide the system operator with higher-accuracy
forecasts of load. As a consequence, utilities face less risk in Fig. 1. The two curves are the mean daily usage patterns for two different
consumers in Bakersfield, California, over a period of one year. The solid
their transactions on the wholesale market. green curve is for a consumer who uses electricity mostly at times when the
Traditionally, the cost of the bulk electricity is allocated price of electricty is high, whereas the dashed red curve is for a consumer
who uses electricity mostly at off-peak times. This demonstrates that two
to the residential consumers using simple proportional rate consumers nearby each other can have very different consumption behaviors.
This research was supported in part by the TomKat Center for Sustainable
Energy, the Precourt Institute for Energy Efficiency, and the Thomas V. Jones 1 For California, prices in the afternoon at the inland areas tends to be much
Stanford Graduate Fellowship in Science & Engineering. higher than prices at night for coastal areas.
DRAFT 2

In this paper, we design an optimal aggregation and pricing the trade-off between rate and uncertainty. In Section V, we
scheme by explicitly considering the behaviors of individual show how to segment the entire population using our method.
consumers. We aggregate consumers into groups where a We close with some concluding remarks in Section VI.
single rate is charged for all consumers within a group. This
design balances the two competing desires of rate design:
limiting cost sharing and reducing uncertainty. We propose a II. P ROBLEM S TATEMENT
method to optimally segment consumers into groups for given
levels of uncertainty tolerated by the groups. Our scheme can We consider the scenario of an LSE who purchases electric-
be used by any agent that purchases electricity for consumers ity on behalf of a group of residential consumers in a two-stage
on the wholesale market, such as a utility or other load-serving wholesale market. We begin by defining the costs this LSE in-
entity (LSE). For the remainder of this paper, we will use LSE curs for purchasing electricity. We introduce a selection vector
N
to denote any such agent. u ǫ {0, 1} , where N is the size of the entire population of
We model the electricity market as a simple two-stage mar- consumers, and ui = 1 if the ith consumer is part of the group
ket. At the day ahead stage, the LSE forecasts its customers’ of consumers for which the LSE purchases electricity. Let d(i)
consumption and purchases some amount of electricity [6], [7]. denote the electricity consumption of the ith consumer on a
Any discrepancy between the amount purchased in advance given day. We assume that this consumption is nonnegative
and the customers’ actual consumption is settled at the real (i.e., no reverse power flow). The total consumption of the
time price. Thus, the LSE faces risks in its wholesale market group of consumers on the given day is
transactions due to forecast errors and to uncertainty in the d = ΣN (i)
i=1 ui d (1)
real time price, whose value is unknown at the day ahead
stage and can spike sharply [8]. Aggregation enables the LSE The two stages of the wholesale electricity market are the
to mitigate the risk of high forecast errors. day ahead market and the real time market. At the day ahead
An important application of our method is designing rate stage, the LSE forecasts the aggregate consumption of the
plans in deregulated retail markets. The deregulation of the consumers for the next day, d̂. Based on that forecast, the
retail electricity market has opened the possibility for LSEs LSE purchases an amount of electricity for them, d̃, at the
to offer a variety of rate plans for residential consumers. In day ahead price, p. The cost incurred at the day ahead stage
ERCOT’s geography, for example, there are over 200 plans is pT d̃. The next day, the consumers’ actual consumption, d,
with a range of prices and schemes. These plans have expe- is realized. The difference between what they consume and
rienced significant turnover of their customer base, which is what the LSE previously purchased is settled at the real time
undesirable from the standpoint of an individual LSE [9], [10]. price, q. We assume that p and q are nonnegative.
We show that our design method induces a stable partitioning For the kth day, the total cost ck that the LSE pays is:
of the consumers, in the sense that no consumer can reduce
his cost by unilaterally moving from one group to another. ck = pTk d̃k + qTk (dk − d̃k ). (2)
Over a period of K days, the LSE will pay an average per
A. Our approach unit cost of electricity rK , given by:
We develop a fractional integer program for aggregating ΣK 1 K
k=1 ck K Σk=1 ck
groups of consumers with the lowest per unit cost of electricity, rK = = . (3)
ΣKk=1 1T dk K Σk=1 1T dk
1 K
and we present an optimal solution to this nonconvex problem.
The LSE then uses the group’s average per unit cost as the (Note that the rates for regulated utilities are defined in a
basis for its rate. As the group size gets larger, this average similar way as the average per unit cost over a period of time.)
per unit cost increases, but the forecast error decreases [3]. The amount of electricity purchased at day-ahead d̃k may
We quantify this trade-off in our dataset of hourly smart meter be different than the forecasted consumption d̂k . For example,
readings for over 100,000 residential consumers for one year. the LSE may regularly decide to purchase slightly more than
We propose a method for the LSE to determine its preferred the forecasted amount in order to avoid the risk of paying
group size and composition. The LSE can extend this method a very high real time penalty. On the other hand, the real
to segment an entire population of consumers into different time price and the day ahead price are typically nearly equal
groups based on their average cost to serve and a common in expectation [8], so the LSE needs to carefully balance
acceptable level of forecast error. This segmentation scheme conservatism and cost.
creates groups of consumers that are stable in the sense that The LSE seeks to minimize rK , which is a random vari-
no one consumer can improve her situation unilaterally. able. For mathematical simplicity, instead of working with
the expectation of the entire ratio, we focus on the numer-
B. Outline ator.Suppose that the LSE enters into a contract with the
In Section II of this paper, we formalize the aggregation consumers it is representing for a period of K days. For large
problem facing the LSE. In Section III, we present an optimal K, we can replace the numerator in the right-most fraction in
algorithm for constructing groups of consumers with a low equation (3) with the expected value of ck :
per unit cost of electricity. In Section IV, we calculate the K Edk [ck ]
rK = . (4)
uncertainty faced by a group of consumers and demonstrate ΣK T
k=1 1 dk
DRAFT 3

We can expand the expectation in the numerator of (4): simplifications in place, the rate paid by the LSE per unit of
electricity is:
Edk [ck ] = Edk pTk d̃k + qTk (dk − d̃k ) .
 
(5)
K Edk pTk dk
 
We can write d̂k = dk + ǫk ; in other words, the forecasted rK = T
= λK (9)
ΣK
k=1 1 dk
consumption is the actual consumption plus an error term, ǫk .
As discussed previously, the LSE can choose at the day ahead When recruiting consumers, the LSE seeks to minimize the
stage to buy a different amount of electricity than its forecast. rate that it will pay for their electricity. The LSE does not
Let the difference between the two be δk . Then d̃k = d̂k + δk . know how they will consume electricity in the future, but it
We now rewrite the right hand side of (5) as: can use historical data to compute an estimate of the λK it
 T  T would pay if it were serving the ith consumer:
Edk pk (dk + ǫk + δk ) + Edk qk (−ǫk − δk ) . (6)
 
(i)
ΣH pT dh
We apply the fact that E [xy] = Cov(x, y) + E [x] E [y] to λ̂(i) = h=1 Th (i) , (10)
expand (6): ΣH
h=1 1 dh

= Edk pTk dk
 
+tr (Cov (pk , ǫk )) + Edk [pk ]T Edk [ǫk ] where H is the number of days of historical data available.
We assume that the consumers’ history is a good predictor of
+tr (Cov (pk , δk )) + Edk [pk ]T Edk [δk ] their future behavior in terms of how aligned their electrical
T
−tr (Cov (qk , ǫk )) − Edk [qk ] Edk [ǫk ] consumption is with the day-ahead price.
T
−tr (Cov (qk , δk )) − Edk [qk ] Edk [δk ] To achieve the lowest possible rate, the LSE would choose
to recruit and serve the single consumer with the lowest value
We assume that the LSE uses an unbiased forecaster, so of λ̂. This would be the consumer whose consumption is most
E [ǫk ] = 0. Furthermore, we assume that the electricity market orthogonal to the day-ahead price vector, given the assumed
is an efficient market, meaning that E [pk ] = E [qk ]. Under nonnegativity of consumption and price. In other words, this
these conditions, we can simplify the above sum to: consumer uses electricity mostly at off-peak hours.
This may not be a feasible recruitment plan for the LSE
 T 
Edk pk dk + tr (Cov (pk − qk , ǫk )) + tr (Cov (pk − qk , δk )) .
(8) for two main reasons. First, a single consumer likely will not
The following two sections examine how the LSE can be a viably large basis for the LSE’s operation. Second, the
use this model to evaluate the per unit cost and uncertainty electrical consumption of an individual residential consumer is
involved in purchasing electricity for its customers. highly variable day-to-day and subject to forecasting errors of
50% [4]. This may lead to large fluctuations in the day-to-day
per unit cost of electricity paid by the LSE.
Therefore, the LSE needs a method to recruit and aggregate
III. C OST- BASED AGGREGATION
multiple consumers into a group that has a low cost to serve.
For a group specified by the selection vector u, we define the
We begin by outlining some simplifcations that allow us cost to serve metric λ̂ as follows,
u
to focus on the cost of serving an individual consumer and
groups of consumers. In this section we will assume that the ΣH pTh dh
forecasting errors are independent of the difference between λ̂u = h=1 , (11)
ΣH
h=1 1T dh
the day ahead and real time price. This means that the
second term of (8), tr (Cov (pk − qk , ǫk )), equals zero. This where dh is the consumption of the group on the hth day.
assumption is standard in the literature [11], [12].2
The first term in (8), Edk pTk dk , represents how aligned
 

the group’s consumption is with the day ahead price. If the

group uses most of its electricity during times when the price A. Optimal Recruitment Algorithm
is high, then that term will be large. The last term in (8),
tr (Cov (pk − qk , δk )), captures whether, on average, the LSE Suppose the LSE wishes to recruit the M consumers who
saves money or loses money by purchasing something different will have the lowest cost to serve among any group of that
than what the forecaster predicts for the group’s consumption. size. Let PH denote the concatenation of ph vectors for h =
(i) (i)
We will assume that the LSE is unable to make money in 1 to H. Similarly, DH is the concatenation of dh vectors.
this way. In this case, the LSE’s best strategy is to set δk = Choosing the M best consumers to minimize λ̂u is given by
0, and we are left with Edk [ck ] = Edk pTk dk . With these the following optimization problem:


PN (i)
2 The independence assumption may not hold generally. For example, ui PTH DH
suppose the LSE’s forecast errors are correlated positively to the forecast minimize Pi=1
N T (i)
(12a)
u
errors of all buyers on the market. Suppose on a given day that the LSE’s i=1 ui 1 DH
forecast error is negative. We would then expect that the total amount of subject to 1T u = M, ui ∈ {0, 1}. (12b)
electricity purchased in the market at the day ahead stage will be below the
total demand. This would drive up the real time price due to a shortage of (i) (i)
generation scheduled at the day ahead stage. Let ti = PTH DH and wi = 1T DH . We rewrite the
DRAFT 4

optimization as: First define the set of feasible values of λ as Λ = {λ : ∃ u ∈

T
T {0, 1}N , 1T u = M, (t − λw) u ≤ 0}. This defines a set
u t
minimize T (13a) of attainable servicing costs for M consumers. Therefore, the
u u w
T solution to (14) is λ⋆ = inf{λ ∈ Λ}. Bisection will find λ⋆ if
subject to 1 u = M, ui ∈ {0, 1}. (13b) the following condition holds for ǫ > 0: λ ∈ Λ =⇒ λ+ǫ ∈ Λ
Finally, we introduce a slack variable λ and obtain: and λ ∈ / Λ =⇒ λ − ǫ ∈ / Λ. That is, if there is a unique
transition point, then the bisection method is guaranteed to
minimize λ (14a) find it.
u,λ
We first prove that λ ∈ Λ =⇒ λ + ǫ ∈ Λ. If λ ∈ Λ,
subject to (t − λw)T u ≤ 0 (14b) T
T
then there exists uλ satisfying (t − λw) uλ ≤ 0. The vector
1 u = M, ui ∈ {0, 1}. (14c) uλ is feasible for λ + ǫ since (t − (λ + ǫ)w)T uλ = (t −
T T
In general, the above combinatorial optimization problems, λw) uλ − ǫw uλ ≤ 0. The first term is nonpositive because
called linear fractional programs, are difficult to solve. Because uλ is feasible for λ. The second term is always positive since
all parameters in our problem are positive, there is an efficient wi > 0 and ǫ > 0.
bisection algorithm for a feasibility problem which tries to find Now we prove that λ ∈ / Λ =⇒ λ − ǫ ∈ / Λ. If λ ∈ / Λ
T
u and λ to satisfy (14b) and (14c). This can be performed in a then ∄ u λ such that (t − λw) u λ ≤ 0. However, ∃ u ′
such
T T
greedy fashion per Algorithm 1. For a given λ, we rank each that (t − λw) u′ ≤ (t − λw) u, ∀u : u ∈ {0, 1}N and
T ′
element of the vector (t − λw) and choose the smallest M 1 u = M . That is, u is a selection T
vector which produces
T
elements with the selection vector u. If (t − λw) u ≤ 0, then the smallest value of (t − λw) u. We call this vector u′ a
we have found a feasible solution for the given value of λ; candidate for λ. If λ is infeasible, T ′
it must be the case that the
otherwise, no solution exists for this value of λ. candidate satisfies (t − λw) u > 0; otherwise, λ would be
feasible. Now consider a candidate for λ − ǫ, and call it u′′ .
Algorithm 1 The LSE can use this alorithm to select the group For u′′ we can state the following:
of M customers who had the lowest cost to serve λ̂u over a
(t − (λ − ǫ)w)T u′′ = (t − λw)T u′′ + ǫwT u′′ (15)
period of historical data.
T ′ T ′′
1: Initialize bisection method:
≥ (t − λw) u + ǫw u (16)
2: λ ← max{t1 /w1 , . . . , tN /wN } >0 (17)
3: λ ← min{t1 /w1 , . . . , tN /wN }
The inequality in (15) - (16) holds since u′ is a candidate for λ
4: Set γ ⊲ Convergence threshold
and must therefore satisfy (t − λw)T u′ ≤ (t − λw)T u for all
5: Bisection method:
u, including u′′ . Since (t − (λ − ǫ)w)T u′′ > 0, we conclude
6: while λ − λ > γ do
that λ − ǫ is infeasible.
7: λ = (λ + λ)/2 ⊲ Update current λ
8: Compute (t − λw) Now the LSE has a method for putting together a group
9: Sort (t − λw) in ascending order to obtain{i1 . . . iN } of consumers of size M who will be relatively cheap to
10: Construct uλ,i = 1 if i ∈ {i1 , . . . , iM } service, assuming all of their electricity can be purchased at
11: if (t − λw)T uλ ≤ 0 then ⊲ Feasibility test the day-ahead price. The LSE must decide how to set the
12: uλ feasible appropriate size M . A larger group will have a higher cost to
13: else serve because it will have to include more consumers whose
14: uλ ← ∅ consumption aligns more closely with peaks in electricity
15: end if prices. On the other hand, a larger group offers advantages
16: if uλ = ∅ then of lower forecasting errors and easier administration.
17: λ←λ ⊲ Infeasible, increase lower bound
18: else IV. AGGREGATING TO M ITIGATE U NCERTAINTY
19: λ←λ ⊲ Feasible, decrease upper bound
In order to analyze the effects of uncertainty on the total
20: end if
cost faced by the LSE, we consider a somewhat different
21: end while
market design. Suppose the LSE is unable to sell back surplus
22: Result:
electricity at the real time settlement. Such an arrangement
23: Minimum λ
T is typically captured in deviation charges and is used by
24: Associated selection vector uλ ⊲ 1 uλ = M
the system operator to discourage LSEs from purchasing
excessive amounts of electricity on the day ahead market
Theorem 1. Algorithm 1 returns the minimum feasible value [11]. Alternatively, the operator may want LSEs to commit
of λ (within a tolerance of γ). to specific load profiles at the day ahead stage with a large
penalty for consuming less than they committed to. For this
Proof. To show the bisection method applied to the feasibility market design, the total cost paid by the LSE in (2) becomes:
problem is optimal, we must show that a unique minimum λ
ck = pTk d̃k + qTk dk − d̃k + ,
 
exists and that bisection will always find that λ. (18)
where [x] + denotes max(x, 0).
DRAFT 5

For a given group of consumers, the LSE chooses d̃k to of the effect of forecast uncertainty on prices.
⋆
minimize Edk [ck ]. At the optimal choice d̃k , the following Thus, the LSE faces a tradeoff when assembling a group
first order condition must hold: of consumers to service. After showing some initial empirical
relationships in the data, we will propose a heuristic algorithm
∇d̃k Edk [ck ] =0 (19) for the LSE to segment the population into stable groups that
⋆
d̃k =d̃k
can be offered different rates based on how they consume
The day ahead price is known to the LSE when it purchases electricity.
electricity on the day ahead market, so pk is a realized random
variable at this stage. The gradient can be applied as shown: V. S EGMENTING THE POPULATION
 i 
h ⋆ A. Data description
0 = pk + Edk ∇d̃k qTk dk − d̃k (20a)
h
+ The data are hourly smart meter readings over a period of
⋆
i
= pk + Edk −qTk diag(~1{dk > d̃k }) , (20b) one year (summer 2010 to summer 2011) for 110,000 residen-
tial customers of PG&E. In addition, we use day ahead and real
⋆
where ~1{dk > d̃k } is a vector whose ith element is one if the time prices published by the California Independent System
⋆ Operator (CAISO) over the same period. One limitation of
ith element of dk is greater than that of d̃k . Next, we assume
⋆ our data is that it only spans one year. We use part of the
that the difference between dk and d̃k is independent of the
real time price. data - the first nine months - to evaluate and segment the
h consumer population, and the rest of the data to validate the
⋆
i
0 = pk − Edk [qk ] diag(Edk ~1{dk > d̃k } )
T
(21a) segmentation scheme. Therefore, we may be introducing a
⋆ seasonal bias into the segmentation in our simulation, but that
= pk − Edk [qk ] diag(~P{dk > d̃k }),
T
(21b) would readily be fixed by using data spanning an entire year
⋆ for the segmentation algorithm presented below.
where ~P{dk > d̃k }) is a vector whose ith element is the
⋆
probability that the ith element of dk is greater than that of d̃k .
⋆
Equation (21b) gives an easy way to compute the optimal d̃k , B. Forecasting method
assuming the distribution of dk is known. This is a variation For the purposes of this paper, we use a relatively simple
on the classic newsvendor model [13], [14]. forecaster. We use an ARMA model, with temperature as an
⋆
Recall that d̃k = dk + ǫk + δk⋆ . Therefore, we can write external regressor, to predict the daily total electricity usage by
⋆
~P{dk > d̃ } as ~P{−ǫk > δ ⋆ }, so the optimal day ahead the consumers, ŷ. We use a vector ARMA model, again using
k k
purchase quantities are based on the distribution of the fore- temperature as an external regressor, to predict the normalized
cast errors. If we assume that the forecast errors are jointly load shape, ŝ. This is a vector whose elements are the fraction
Gaussian with zero mean and covariance Σ, we can write of the daily total electricity consumed in each hour. In other
T
δk⋆ = fk (Σ). Substituting the optimal electricity purchase into words, 1 ŝ = 1. Finally, we multiply the predicted daily total
(18), and taking the expectation, we arrive at: by the predicted normalized load shape to obtain the predicted
 T  consumption for the next day: d̂ = ŷŝ.
Edk [ck ] = Edk pk dk + gk (Σ) (22a)
gk (Σ) = Edk [fk (Σ)] + Edk [qk ]T Edk [−ǫk − fk (Σ)] + .
 
C. Range of λ̂(i)
(22b)
To begin with, we evaluate the range of λ̂(i) values in the
Recall that (19) relates to a component of (4), so we can entire population. Recall that the LSE seeks to minimize the
substitute (22a) back into (4) to obtain: per unit cost of electricity that it incurs on the wholesale
market. One natural way for it to do this is to choose
K Edk pk dk
 T 
K
Σk=1 gk (Σ)
rK = + (23a) consumers who use electricity at off-peak times with respect
1 T DK 1 T DK to the day ahead price, and λ̂(i) captures this alignment for
ΣK g k (Σ) the historical consumption of the ith consumer.
= λK + k=1T . (23b)
1 DK Figure 2 shows how λ̂(i) increases as we go from the
In the next section of this paper, we will show how the first cheapest consumer to the (i)
most expensive consumer in our
and second terms of (23) vary with the composition and size dataset, ranked by their λ̂ metric computed over the first nine
of the group of consumers selected by the LSE, in other words months of the year. The most expensive consumer is almost
how they vary with u and kuk 1 . For now we will discuss these twice as expensive as the cheapest consumer, a significant
relationships in general terms. difference. This suggests that the LSE may be able to group
together cheaper consumers and offer them a lower rate.
The LSE varies u by controlling whom it recruits into its
group. If the LSE recruits only consumers who use most of
their electricty at peak hours, its per unit cost of electricity will D. Application of optimal recruitment algorithm
be higher, and this is captured in λK . Furthermore, if the LSE As mentioned before, the LSE would likely need to aggre-
recruits only a very small number of consumers, it will face gate consumers together to service them practically. We apply
larger forecast errors, which may lead to higher variation in Algorithm 1 using the first nine months of data as the historical
rK , partially through the gk (Σ) term. See [15] for a treatment period. Let λ̂M denote the minimum value of λ achieved using
 DRAFT 6

same period. Then

4.0
λ (cent/kWh)
q
1 ˆ 2


T ΣTt=1 d(t) − d(t)
CV = 100 1 (%). (24)
ΣTt=1 d(t)
3.0

Figure 4 illustrates the effect of group size on forecast error.

^ (i)

For both the random groups and optimal groups, the coefficient
2.0

of variation of the forecast error decreases with group size. The

0 20000 60000 100000 decrease for random groups is smooth and monotone, while
the curve for the optimal groups exhibits more complicated
User ranking
behavior. If the error curve for optimal groups coincided with
Fig. 2. A plot of λ̂(i) vs. consumer rank, in order of increasingly expensive that of the random groups, we could assert that CV , and by
consumers. The difference between the cheapest and the costliest consumer
is about 2.0 cent/kWh. The steep slopes at either end of the curve mean that
extension Σ, is a function solely of group size - but that is
there are relatively few very cheap consumers or very expensive consumers. not the case. It is clearly a function of both u and kuk 1 , both
The horizontal orange line is the average per unit cost for the population. composition and size.
The CV is related to the variability in electricity costs faced
by the LSE [15]. If the CV is high, and if the day ahead and
the algorithm given a group size M . Figure 3 illustrates how
real time prices tend to be different, the LSE can expect a
λ̂M varies with M . Note that at M = 39, 800, approximately
higher variation in its per unit cost of electricity from one day
37% of the population, λ̂M is within 5% of the population
to the next because the higher forecast errors will require it to
average. On the other hand, λ̂500 is 2.75 cents/kWh, which is
purchase (or sell) more electricity at the real time settlement.
17% lower than the population average, so the optimal group
of size 500 could be offered a meaningfully cheaper rate plan.
100
Optimal Groups
3.4

Random groups, mean

λM(cent/kWh)

Random groups, 95% conf.

CV
20
2.8

5
2.2
^

10 100 1000 10000

10 100 1000 10000 M
M Fig. 4. CV is decreasing in M because the consumption of larger groups can
be forecasted more accurately. The black line is the mean CV for randomly
Fig. 3. λ̂M is increasing in M because as the group size increases, the constructed groups, and the dashed lines are the 95% confidence interval. The
algorithm includes consumers who are more and more aligned with the day blue line is for groups constructed using the optimal recruitment algorithm.
ahead price vector. The horizontal dashed line is λ̂M when M is the size of the Until about M = 1000, the optimal groups have higher forecasting errors
entire population. In other words, it is the average per unit cost of electricity than the random groups.
for servicing the entire population at the day ahead price. Randomly selected
groups of consumers have a λ̂ close to this population average.
Therefore, if the LSE wants to control the variation in its
The LSE now knows how its per unit cost of electricity will per unit cost of electricity, it must aggregate a certain number
change based on how large of a group of low-cost consumers it of consumers together. The LSE faces a trade-off - a smaller
recruits. Practical operational considerations may set a lower group size allows for a lower per unit cost, while a larger group
limit on group size, but the LSE needs a way to determine size allows for a lower variation in per unit cost. Suppose the
how big of a group it should aggregate, in other words, how LSE sets an upper limit CV on the forecast error as a way of
to select M . We propose that the LSE take into account how controlling the variation in its costs. Then it should determine
group size affects its forecasting error. the minimum group size M such that the CV for optimal
The larger the group, the more accurately its aggregate groups of that size is less than or equal to CV . Finally, it
consumption can be forecasted. To demonstrate this result in should run the optimal recruitment algorithm to choose the
the data, we use the forecaster described previously to predict lowest-cost group of consumers of size M . Now the LSE
hourly group consumption for the last three months of the knows which consumers to recruit to get a group that will
year. We do this for two types of groups - groups constructed have a low per unit cost of electricity and an adequately small
by randomly assembling individuals, and groups constructed variation in that cost. For example, if the LSE set a limit of
using the optimal recruitment algorithm. The error metric we CV = 10%, it would find on Figure 4 that it needs to recruit
use is the coefficient of variation, CV , which is commonly a group of size M = 209. It would then run Algorithm 1 with
used in forecasting literature as a measure of performance. M = 209 and obtain the selection vector u that would identify
Let d(t) be the actual consumption of the group of consumers the consumers it should recruit to the group. The LSE would
ˆ be the forecasted consumption for the
at time t, and let d(t) then refer to Figure 3 to determine that the long run average
DRAFT 7

per unit cost for this group will be 2.70 cents per kWh, and Proof. Suppose that a consumer has been assigned to group ui
it can offer a rate plan to this group based on that cost. and offered that group’s average rate, λ̂M i . The consumer has
no incentive to join the group ui+1 because that group pays a
E. Segmenting the entire population higher rate. The consumer would like to join the group ui−1
to enjoy their lower rate. However, the existing consumers in
We now turn to how the LSE could use the methods we’ve
group ui−1 will not agree to this because they already have
presented to serve an entire population. The LSE is interested
in designing stable rate schemes, in which customers do not enough consumers in their group to reduce their risk level to
CV . Adding this new consumer will only lead to an increase in
have an incentive to jump frequently from one to another.
This consideration is relevant to current practice - load serving their rate because we know that the optimal selection algorithm
entities in ERCOT have reported substantial customer turnover, did not choose this consumer to be part of ui−1 .
which is undesirable [9], [10]. In our method, the LSE will
segment the population into groups that have different long
run rates but similar levels of variation in daily cost.
The LSE starts by considering the entire population and

20
establishing an accepted level of variation, CV . It generates a

CV
forecast error curve similar to that for the optimal groups in

10
Figure 4, and it selects the smallest group size M 1 such that
the CV for that group is less than or equal to CV . There is

5
an optimal group of low-cost consumers u1 corresponding to
this group size M 1 . The LSE takes this group of consumers
10 100 1000 10000
and serves it as an aggregate, offering them a rate plan based

3.4
λM (cent/kWh)
on their group cost λ̂M 1 .
After removing that first set of consumers from the pop-
ulation, the LSE repeats this process on the remaining con- 2.8
sumers, with the same CV limit, until the entire population
is segmented into groups. Say that the process results in P
total groups. These groups, u1 , u2 , ..., uP , will be of various
2.2

sizes. Each subsequent group will be offered a higher rate

^

than the preceding group because the lowest-cost consumers 10 100 1000 10000
have already been assigned to prior groups. In other words,
λ̂M i ≤ λ̂M i+1 . However, by construction, every group will M
have the same level of risk, CV . Fig. 5. These two figures illustrate the next step in the segmentation process
when using CV = 10 as the limit. The LSE faces a new curve for CV
Returning to our previous example, if the LSE sets CV = vs. group size M . The LSE applies the CV = 10 threshold to the forecast
10, the first group of consumers it recruits will be of size error curve, determines the required group size, and then reads off their group
M 1 = 209, and it will offer that group a rate plan based per unit cost from the bottom figure. The red arrows trace out that process.
on an average per unit cost of λ̂M 1 = 2.70 cents per kWh. than Note that the starting point for the optimal group λ̂M is noticeably higher
it was for the entire population in Fig. 3. This makes sense given that
The LSE then removes this group of consumers from the the cheapest consumers were already recruited into the first group, so the
population. Figure 5 illustrates the next iteration of the LSE’s remaining consumers will have higher per unit costs. The orange dashed line
segmentation process. The LSE again applies the CV = 10 on the λ̂M plot is the per unit cost for servicing the entire population, the
same value as in Fig. 3.
threshold and chooses to go with a group of size M 2 = 145,
as seen in the top panel. The LSE recruits these consumers We simulate this pricing design scheme on the entire popu-
into a group and offers them a rate plan based on its average lation of consumers in our dataset, using CV = 6, 5.5, and 5.
per unit cost of λ̂M 2 = 2.81 cents per kWh, as illustrated in We evaluate the daily per unit cost of electricity that the LSE
the bottom panel. The LSE would repeat this process until would incur for servicing each group. Figure 6 illustrates the
all consumers had been assigned into a group. Note that results of our simulation. As expected, when using the higher
it is possible that on the last iteration, the CV curve for forecast error threshold, the LSE segments the population into
the remaining consumers will be entirely above CV . In that smaller groups. For all three cases, the groups segmented out
case, the LSE must aggregate together all of the remaining earlier incur costs noticeably lower than the social average
consumers, and this last group will have a forecast error higher cost, whereas the groups formed later incur higher costs.
than the other groups. Alternatively, the LSE can decide not For CV = 6, the LSE segments the population into 70
to serve the remaining consumers, or to segment them on the groups. The first 69 all satisfy the forecast error threshold
basis of a new, higher CV threshold. and have per unit costs ranging from λ̂M 1 = 2.87 to
Proposition 1. This method produces stable pricing plans in λ̂M 69 = 3.29 cents per kWh. The average size of these groups
the following sense: assuming that all consumers agree to the is 738 consumers. All of these groups, covering almost 51,000
variation limit CV , and given the initial segmentation of the consumers, have a per unit cost lower than the population
population by the LSE, no single consumer can unilaterally average. The 70th group lumps together the remaining 57,000
take an action that would improve her outcome. consumers, with a per unit cost of λ̂M 70 = 3.40, which is
DRAFT 8

Figure 7 shows the average daily aggregate load shapes for

x x x three groups (i = 2, 22, 64) from the segmentation performed
Mi (users)
5000 with CV = 6. The groups have distinct consumption patterns,
x reflected in their differing per unit costs. This demonstrates
x x
x
x xx x xx
the effectiveness of the segmentation algorithm.
xx x x
xxx x x xx xxx xx xxx xxxx xx x
xx x xxxx x x xx xx xx x x xx x x xx xxxxxx
xxx x x x
xx xx xxxxxxx xx xx x xxxx x x x xx x
x xx
x x xxx xx x xx x VI. C ONCLUSION
100

We developed and proposed a method for an LSE to

0 10 20 30 40 50 60 70 segment a population of electricity consumers on the basis of
their average per unit cost. Using their historical consumption
λMi (cent/kWh)

x x
x patterns, the LSE identifies and aggregates consumers who
3.3

xxx
x xxxxx xxxxxxxxxxxxxxxxx are cheaper to serve. The LSE aggregates enough consumers
xx xxxxx xxxxxxxxxxxxxxxxxx to reduce the forecast error to an acceptable level, which is
xxxxxxxxxxxxxxxxxxxxxxxx
3.1

x xxxxxx
xx xxx xxxxx common across all consumer groups. We quantified the trade-
xxxxxxxxxx x CV=6
x
xxx
xxxx x CV=5.5 offs involved in this process when segmenting a population
x
2.9

xx
x x CV=5 of over 100,000 PG&E residential consumers. Our simulation
^

demonstrated that the LSE can offer each group a different

0 10 20 30 40 50 60 70 average per unit rate plan that is based on the extent to which
i they consume their electricity at peak times.
Fig. 6. These figures illustrate the results of segmenting the entire population
of consumers using three different values of the CV threshold. The first figure ACKNOWLEDGMENT
shows how the optimal group size M i varies as the segmentation proceeds. We would like to thank Pacific Gas and Electric Company
The CV = 6 segmentation proceeds with smaller group sizes because
it doesn’t require aggregating as many consumers together to achieve that
for providing us with the smart meter data used in this study.
forecast error threshold. In each of the three cases, the last and largest group
is well over 50,000 people. These last groups did not achieve the required CV R EFERENCES
threshold. The second figure shows the per unit costs for servicing each of the
[1] W. R. Christiaanse, “Short-term load forecasting using general exponen-
groups. The orange dashed line is the per unit cost for the entire population,
tial smoothing,” Power Apparatus and Systems, IEEE Transactions on,
a reference point for this segmentation. As expected, in each case, the first
groups have per unit costs below the population average, and the later groups vol. PAS-90, no. 2, pp. 900–911, 1971.
approach and then exceed that average. [2] M. Hagan and S. M. Behr, “The time series approach to short term load
forecasting,” Power Systems, IEEE Transactions on, vol. 2, no. 3, pp.
785–791, 1987.
[3] R. Sevlian and R. Rajagopal, “A model for the effect of aggregation
0.07

on short term load forecasting,” in PES General Meeting — Conference

Load fraction

Exposition, 2014 IEEE, July 2014, pp. 1–5.

[4] S. Patel, R. Sevlian, B. Zhang, and R. Rajagopal, “Aggregation for load
servicing,” in PES General Meeting — Conference Exposition, 2014
0.04

IEEE, July 2014, pp. 1–5.

[5] K. Ito, “Do consumers respond to marginal or average price? evidence
from nonlinear electricity pricing,” National Bureau of Economic Re-
search, Working Paper, 2012.
0.01

[6] H. Allcott, “Real time pricing and electricity markets,” Harvard Univer-
sity, 2009.
5 10 15 20 [7] M. Roozbehani, M. Dahleh, and S. Mitter, “Volatility of power grids
under real-time pricing,” Power Systems, IEEE Transactions on, vol. 27,
Hour of day no. 4, pp. 1926–1940, 2012.
Fig. 7. The three curves are the average daily aggregate load shapes for three [8] D. Kirschen and G. Strbac, Fundamentals of Power System Economics.
groups created by the segmentation algorithm using CV = 6. The dashed John Wiley and Sons, 2004.
red curve is for the second group segmented out. Its consumption is mostly [9] Advanced Energy Economy. (2013) 21st Century Electricity System
off peak, and its per unit cost to serve is 2.90 cents/kWh. The dotted blue CEO Forum Summary. [Online]. Available: https://goo.gl/4RNfq7
curve is for the 22nd group segmented out, with per unit cost to serve of [10] P. Carson. (2012) Texas retail: analyzing
3.10 cents/kWh. The solid green curve is for the 64th group segmented out. data to retain customers. [Online]. Available:
Its consumption is mostly on peak, and its per unit cost to serve is 3.25 http://www.intelligentutility.com/article/12/09/texas-retail-analyzing-data-retain-custom
cents/kWh. [11] E. Bitar, R. Rajagopal, P. Khargonekar, K. Poolla, and P. Varaiya,
“Bringing wind energy to market,” Power Systems, IEEE Transactions
on, vol. 27, no. 3, pp. 1225–1235, Aug 2012.
above the population average. This last group has a forecast [12] Y. Zhao, J. Qin, R. Rajagopal, A. Goldsmith, and H. V. Poor, “Risky
power forward contracts for wind aggregation,” in 2013 51st Annual
error of CV = 7.9, exceeding the CV = 6 threshold. Allerton Conference on Communication, Control, and Computing (Aller-
For comparison, when using CV = 5, the first 13 groups ton), 2013.
satisfy the forecast error threshold and have costs ranging from [13] N. C. Petruzzi and M. Dada, “Pricing and the newsvendor problem: A
review with extensions,” Operations Research, vol. 47, no. 2, pp. 183–
2.90 to 3.20 cents per kWh. The average size of these groups 194, 1999.
is 1892 consumers, and in total, they include about 25,000 [14] M. Khouja, “The single-period (news-vendor) problem: literature review
consumers. The remaining 83,000 consumers are placed into and suggestions for future research,” Omega, vol. 27, no. 5, pp. 537 –
553, 1999.
the last group, with a per unit cost of 3.36 cents per kWh, [15] B. Zhang, R. Rajagopal, and D. Tse, “Network risk limiting dispatch:
which is above the population average, and a forecast error of Optimal control and price of uncertainty,” Automatic Control, IEEE
CV = 7.1, well above the threshold. Transactions on, vol. 59, no. 9, pp. 2442–2456, Sept 2014.