IEEE TRANSACTIONS ON POWER SYSTEMS, VOL. 37, NO.

3, MAY 2022 1695

Fast Newton-Raphson Power Flow Analysis Based

on Sparse Techniques and Parallel Processing
Afshin Ahmadi , Member, IEEE, Melissa C. Smith , Senior Member, IEEE,
Edward R. Collins, Senior Member, IEEE, Vahid Dargahi , Member, IEEE,
and Shuangshuang Jin, Senior Member, IEEE

Abstract—Power flow (PF) calculation provides the basis for more frequently and in a shorter time scale to handle the system
the steady-state power system analysis and is the backbone of dynamics. As a result, it is necessary to develop faster mathemat-
many power system applications ranging from operations to plan-
ical methods and utilize more computational power to ensure the
ning. The calculated voltage and power values by PF are essential
to determining the system condition and ensuring the security adequacy and security of energy delivery. Fortunately, advance-
and stability of the grid. The emergence of multicore processors ments in computer technology and the widespread availability
provides an opportunity to accelerate the speed of PF computa- of multicore processors open up new possibilities to address this
tion and, consequently, improve the performance of applications problem.
that run PF within their processes. This paper introduces a fast
Parallel computing has proven as a viable solution to ac-
Newton-Raphson power flow implementation on multicore CPUs
by combining sparse matrix techniques, mathematical methods, celerate several computationally intensive applications in the
and parallel processing. Experimental results validate the effec- power system [1]–[3]. These applications typically belong to
tiveness of our approach by finding the power flow solution of a real-time control and simulation, optimization, and probabilistic
synthetic U.S. grid test case with 82,000 buses in just 1.8 seconds. assessment. Many of these studies rely on the solution of power
Index Terms—Parallel, multicore, sparse, power flow, newton- flow (PF) within their process. PF analysis aims to obtain voltage
raphson, OpenMP, SIMD. magnitudes and angles at load buses, real and reactive power
flow through the transmission lines, and voltage angles and
reactive power injection at generator buses. This information is
I. INTRODUCTION essential to determine the steady-state condition of the network
OWER systems throughout the world are undergoing a and ensure the security and stability of the grid. However, PF
P significant transformation, mainly driven by the increase
in penetration of renewable energy, integration of distributed
is a non-linear, computationally demanding problem, where the
solution is only meaningful for a short time since the state of the
energy resources, and advances in digital technologies. Future power system continuously changes.
grids are powered by clean energy, enable the bidirectional Over the years, researchers have developed various numerical
flow of electricity, and have a centralized-decentralized con- and analytical methods to solve power flow, such as Newton-
trol scheme. The resulting benefits are improved reliability Raphson (N-R), Gauss-Seidel (G-S), Fast Decouple (FD), Holo-
and resiliency, more efficient supply and delivery of energy, morphic Embedding, Continuation, and several others. To date,
reduced environmental impact, and cost-effective energy gen- Newton-Raphson is still the most commonly used method be-
eration. However, this transformation will make the planning, cause of its quadratic convergence property. On the other hand,
operation, and control of the power grid more complicated and this method requires more computational resources compared
computationally demanding. The variability and uncertainty of with others [4]. Researchers around the world have investi-
decision variables result in complex models that must be solved gated the possibility of using parallel processing to boost the
Newton-Raphson Power Flow (NRPF) performance. These at-
tempts can be classified based on the parallel programming
Manuscript received August 19, 2020; revised February 10, 2021, April 20,
2021, and August 12, 2021; accepted September 19, 2021. Date of publication model and processor architecture: shared memory, distributed
September 28, 2021; date of current version April 19, 2022. This work was memory, graphic processing unit (GPU), configurable chips,
supported in part by the NSF-MRI Award 1725573. Paper no. TPWRS-01413- and hybrid approaches. Only a few studies have employed
2020. (Corresponding author: Afshin Ahmadi.)
Afshin Ahmadi, Melissa C. Smith, and Edward R. Collins are with the shared and distributed memory models to accelerate the
the Holcombe Department of Electrical, and Computer Engineering, Clem- solution of the NRPF algorithm on multicore processors. In [5], a
son University, Clemson, SC 29634 USA (e-mail: [email protected]; maximum speedup of 5.6× was achieved for a 1,944-bus system
[email protected]; [email protected]).
Shuangshuang Jin is with the School of Computing, Clemson University, by employing OpenMP and eight CPU cores. Reference [6]
North Charleston, SC 29405 USA (e-mail: [email protected]). reported a speedup of 3.3× using four threads and OpenMP for a
Vahid Dargahi is with the School of Engineering and Technology, University power system with 1,354 buses. Guera and Martinez-Velasco [7]
of Washington, Tacoma, WA 98402 USA (e-mail: [email protected]).
Color versions of one or more figures in this article are available at integrated the high-performance Intel MKL PARDISO solver
https://doi.org/10.1109/TPWRS.2021.3116182. into the source code of the MATPOWER [8] library and the
Digital Object Identifier 10.1109/TPWRS.2021.3116182 solution of a 9,241-bus test system was found in 560 ms.

0885-8950 © 2021 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See https://www.ieee.org/publications/rights/index.html for more information.

Authorized licensed use limited to: UNIV OF ENGINEERING AND TECHNOLOGY LAHORE. Downloaded on November 28,2022 at 05:43:53 UTC from IEEE Xplore. Restrictions apply.
 1696 IEEE TRANSACTIONS ON POWER SYSTEMS, VOL. 37, NO. 3, MAY 2022

Distributed computing for the solution of NRPF was investi- scope of this study since our focus is on balanced transmission
gated in [9] and [10], with an average speedup of 2× in both networks.
studies. The communication time delay between nodes limits Review of the literature shows that recent studies for ac-
the performance of the distributed model. Additionally, authors celerating PF calculations are mostly designed to exploit the
in [11] examined the possibility of solving PF equations on computational power of GPUs. Although the reported speedups
field-programmable gate arrays (FPGAs) and achieved about 6x are encouraging, there are some disadvantages to this approach.
higher performance compared with the CPU implementation. First, the overhead time for data transfer between the host
However, using FPGA devices for accelerating power system computer and GPU poses an obstacle for the performance of
applications has not received much attention by engineers and these implementations. Second, only a limited number of GPU-
scientists since it requires significant programming effort and based sparse direct solver libraries are currently available and
expertise. they typically utilize the entire device to find the solution of
Over the past decade, there have been enormous advance- a linear system, which poses a significant limitation when it
ments in GPU hardware and programming tools, which has comes to applications that are reliant on concurrent execution of
made these devices more powerful, popular, and accessible. As many PF scenarios. Finally, GPUs are generally equipped with
a result, a significant number of researchers have focused on less memory compared with shared-memory computers, which
GPU-based implementations of both PF algorithms and appli- makes them undesirable for memory-intensive applications such
cations that are reliant on PF calculation such as contingency as contingency analysis.
analysis [12]–[15]. A hybrid GPU-CPU NRPF implementation
based on vectorization parallelization and sparse techniques was II. CONTRIBUTIONS
proposed in [16], where the solution of a 3,012-bus system
This paper aims to maximize the performance of Newton-
was calculated in 206 ms. Although the results are promising,
Raphson power flow on multicore CPUs by combining software
unfortunately, the scalability and effectiveness of the proposed
techniques, mathematical methods, and parallel processing. The
approach is unknown since test systems are limited to 3,012-bus
main contributions of this study are summarized as follows:
and less. A GPU version of three different PF algorithms was r Both SIMD (Single Instruction, Multiple Data) vectoriza-
developed in [17] using the CUDA library and dense matrices.
tion and multicore processing are targeted to accelerate
For a system with 974 buses, the run time was 19.6, 10.8, and 5.5
power flow calculations.
seconds for G-S, N-R, and P-Q decoupled power flow methods, r The implementation employs the Compressed Sparse Row
respectively. Moreover, the performance of Gauss-Jacobi and
(CSR) storage format to address two major constraints in
N-R power flow algorithms on GPUs was studied in [18]. Results
computing PF solution - memory and time.
for a 4,766 bus system shows an execution time of 3.98 sec- r A parallel approach for forming and updating the sparse
onds for N-R and 3.06 seconds for Gauss-Jacobi. Furthermore,
Jacobian arrays is introduced that significantly improves
authors in [19] have compared the parallel implementation of
the execution time.
N-R and G-S algorithms on both CPUs and GPUs using sparse r The nested dissection algorithm is applied to reduce the
techniques with the objective of accelerating the concurrent PF
fill-in and improve the computation time and memory usage
calculation of many cases of a single network. Reported speedup
in solving the system Ax = b.
for a 2,383-bus system ranges from 6× to 13× depending on the r Various combinations of scenarios are benchmarked on
target architecture and number of simultaneous runs. In the case
power systems ranging from 1,354 to 82,000 buses.
of the G-S power flow method, although the reported speedups r The proposed NRPF implementation is combined with
in [17]–[21] are better than the sequential G-S algorithm, this
contingency analysis, and results are presented to further
PF method cannot compete with the NRPF method since it is
demonstrate the significance of this research.
prone to divergence and requires many iterations to obtain a
solution. Moreover, the nonlinear devices in the network cannot
be modeled in the G-S method. III. POWER FLOW PROBLEM
Additionally, references [22] and [23] sought to execute the The PF study aims to calculate the bus voltages and power
fast decoupled power flow (FDPF) on GPUs. In [22], exploiting flow in the network given the nodal admittance matrix (Ybus ),
both a GPU-based preconditioned conjugate gradient solver and known complex power at load buses (PQ), known voltage mag-
an inexact Newton method improved the performance up to 2.86 nitude and angle for the slack generator bus, and known voltage
times for test systems upto 13,173 buses. Researchers in [23] magnitude and injected real power for the remaining generator
compared the performance of running FDPF on GPUs with two buses (PV). Network equations in a power system are commonly
different fill-in reduction algorithms, namely, reverse Cuthill- formulated by the node-voltage method, which results in a
Mckee (RCM) and approximate minimum degree (AMD). By complex linear system of equations in terms of injected bus
utilizing CUDA libraries, sparse matrices, and the AMD algo- currents,
rithm, a 4.19x speedup was achieved for a 13,659 bus power
system. Parallel implementation of the PF algorithms for an I = Ybus V (1)
unbalanced distribution network has been investigated in several However, the complex power values are generally known in a
studies as well [25]–[28]. However, this topic is beyond the power system rather than the bus currents. Thus, Eq. (1) should

Authorized licensed use limited to: UNIV OF ENGINEERING AND TECHNOLOGY LAHORE. Downloaded on November 28,2022 at 05:43:53 UTC from IEEE Xplore. Restrictions apply.
 AHMADI et al.: FAST NEWTON-RAPHSON POWER FLOW ANALYSIS BASED ON SPARSE TECHNIQUES AND PARALLEL PROCESSING 1697

TABLE I
PSEUDO-CODE OF IMPLEMENTED NEWTON-RAPHSON POWER FLOW

Fig. 1. A Sparse Matrix and its Corresponding CSR Arrays.

following matrix equation,

   
Δδ −1 ΔP
=J (6)
Δ|V | ΔQ
where ΔP and ΔQ vectors are the mismatch between the
scheduled and calculated power values (i.e., power mismatch),
and the entries of the Jacobian matrix, J, are partial derivatives
of Eqs. (4) and (5). As shown below, the Jacobian matrix has
four varying dimension submatrices depending on the number
of PQ (npq ) and PV (npv ) buses.
be modified to integrate complex power equations, which results
in a set of nonlinear power flow equations. The injected complex
power into the ith bus is,
 n 

∗ ∗ ∗
Si = Pi + jQi = Vi Ii = Vi Yik Vk (2)
k=1
Generally, the NRPF algorithm is comprised of three main
where Pi and Qi are the injected active and reactive powers into functions: a routine to build the nodal admittance matrix, an
the bus i, respectively, Vi is the voltage of the bus i, Yik is the iteration loop to minimize the power mismatch, and a function to
entry i, k of the admittance matrix, and n is the number of buses. calculate network information such as line losses, branch flows,
Substituting the complex voltage and admittance variables with etc. The entire process is presented in Table I. Typically, control
their exponential form yields, parameters are adjusted in an outer loop to correct violations
n in bus voltages, reactive power at generator buses, and so on.

Pi + jQi = |Vi | |Vk ||Yik |ej(δi −δk −θik ) (3) This loop adjusts system parameters, such as transformer tap
k=1 and shunt control, and reruns the iteration loop to remove the
violations. There is no difference in the pseudo-code of the
where δ and θ are the voltage and load angles, respectively. NRPF algorithm for sparse and dense matrix implementations.
Moreover, the entry Yik in Eq. (3) can be decomposed into Gik + However, programs that employ sparse matrix techniques are
jBik , where G is the conductance and B is the susceptance more complicated to code. For instance, one of the difficulties
value. Expanding Eq. (2) and separating the real and imaginary that we encountered was forming/updating the sparse Jacobian
parts results in the following power flow equations, matrix (line 7) with the calculated values of partial derivatives
n (line 5). This paper presents a novel parallel solution in Sec-

Pi = |Vi ||Vk |[Gik cos(δi − δk ) + Bik sin(δi − δk )] (4) tion IV-D to address this problem.
k=1
n IV. DESIGN AND IMPLEMENTATION

Qi = |Vi ||Vk |[Gik sin(δi − δk ) − Bik cos(δi − δk )] (5) Generally, improving the execution time of a program is not a
k=1 straightforward task. It involves examining computational bot-
tlenecks, redesigning algorithms, selecting high-performance
Generally, there are four quantities associated with each bus,
libraries, and converting code to utilize parallel processing. All
namely, Pi , Qi , |Vi |, and δi . Depending on the bus type, two
of the building blocks and design schemes used in this study
of these variables are known, and the remaining are unknown.
to maximize the peak performance of the NRPF algorithm are
Iterative methods such as Gauss-Seidel and Newton-Raphson
discussed in this section.
are commonly employed to solve the above nonlinear power
flow equations. In the N-R method, the nonlinear simultaneous
A. Sparse Matrix Storage
equations are linearized based on Taylor’s series around an initial
guess, and all of the higher-order terms are ignored. Using N-R A sparse matrix is defined as a matrix with few non-zero ele-
to linearize the power flow equations in (4) and (5) yields the ments. Storing only the non-zero values and their respective row

Authorized licensed use limited to: UNIV OF ENGINEERING AND TECHNOLOGY LAHORE. Downloaded on November 28,2022 at 05:43:53 UTC from IEEE Xplore. Restrictions apply.
 1698 IEEE TRANSACTIONS ON POWER SYSTEMS, VOL. 37, NO. 3, MAY 2022

Fig. 2. An Illustration of Nested Dissection Algorithm.

TABLE II
Fig. 3. Levels of Hardware Parallelism.
IMPACT OF MATRIX REORDERING ON THE NUMBER OF NONZEROS ELEMENTS

and it creates a pattern of zeros similar to the top-right figure.

Additionally, because the elimination process in V1 and V2 are
independent, it can be performed in parallel. It should be noted
that V1 and V2 can themselves be reordered by a separator to
create a recursive structure in the matrix.
Because nested dissection has a parallel nature and typically
and column indices will significantly reduce memory usage and results in less fill-in for large problems compared to other meth-
computation time, because operations on zero elements are elim- ods, we selected this algorithm for fill-reducing of the sparse
inated. The power flow problems have a sparse nature because Jacobian matrix. Table II shows the number of nonzero elements
power system nodes are only connected to a few adjacent buses, in the LU factor matrices of the sparse Jacobian matrix before
which makes the nodal admittance matrix highly sparse [16]. and after applying the nested dissection algorithm. The details
Therefore, this sparse property can be effectively utilized to of these power system cases are provided in the next section.
improve computational performance and reduce memory usage. There are several high-performance sparse direct solvers for
There are several standard formats for storing a sparse matrix. In multicore architectures such as Intel Math Kernel Library (aka
this study, compressed sparse row (CSR) is selected for storing Intel MKL), PETSc, MUMPS, Eigen, Pardiso, and SuiteSparse,
the sparse arrays since it is widely supported by existing libraries. to name a few. This study employed Intel MKL PARDISO [29]
Fig. 1 shows the CSR representation of matrix A. It is important direct sparse solver because it is robust, memory-efficient, tar-
to mention that external libraries were not used to produce or gets both SIMD and multicore parallelization, and outperformed
modify the sparse CSR arrays due to the unsatisfactory perfor- other CPU-based libraries in most cases [7], [16], [19]. This
mance of those libraries. library also supports various sparse storage formats, matrix
types, and provides many numerical options. Additionally, the
B. Fill-Reducing Ordering of Sparse Jacobian Matrix nested dissection algorithm is supported by this solver, which
makes it easier to integrate this method into the code.
The N-R method for solving power flow entails finding the
solution of Ax = b in each iteration. The computational work
and memory for solving this sparse linear system of equations C. Multicore Processing and SIMD Vectorization
depends on the fill-in of the factorization process. fill-in happens The operating frequency (i.e., clock) of computer processors
when zero values turn into non-zeros during the factorization. was continuously increasing for many years until heat generation
A standard technique to reduce the fill-in and improve the became an obstacle and forced the clock speed to remain in
computational performance is to reorder the rows and columns of the 1-5 GHz range for about a decade now. However, chip
matrix A and solve the equivalent system (P AQ)(QT x) = P b, designers overcame this problem and addressed the increasing
where P and Q are permutation matrices. The rearrangement demand for higher computational power by designing single
process depends only on the pattern of non-zero elements in processors, with multiple low-frequency low-voltage comput-
matrix A and not the values. The reordering algorithms to reduce ing cores inside. These multicore processors can achieve high
the fill-in are broadly categorized into minimum degree and computational throughput by dividing the work among multiple
nested dissection methods. The first is naturally sequential but resources while the power consumption and heat generation
fast to compute. The latter uses a divide and conquer strategy are significantly reduced. Generally, hardware parallelism can
on the graph to detect and eliminate reordering and is suited for take place at different levels: Single Instruction, Multiple Data
parallel implementation. (SIMD) unit, multicore, multi-socket, many-cores, and clusters,
As illustrated in Fig. 2, in the nested dissection method, we as depicted in Fig. 3. Each of these technologies is powerful on
partition vertices V into V1 , V2 , and VS , where VS is called the its own, but only when combined can the highest performance
separator (i.e., highlighted points in red). If we first reorder V1 be achieved. This study targets both SIMD and multicore level
and V2 , and then VS , no fill-in will occur between V1 and V2 , parallelism for solving power flow.

Authorized licensed use limited to: UNIV OF ENGINEERING AND TECHNOLOGY LAHORE. Downloaded on November 28,2022 at 05:43:53 UTC from IEEE Xplore. Restrictions apply.
 AHMADI et al.: FAST NEWTON-RAPHSON POWER FLOW ANALYSIS BASED ON SPARSE TECHNIQUES AND PARALLEL PROCESSING 1699

Fig. 4. SIMD Vectorization. Fig. 5. Parallel OpenMP Loop.

1) Exploiting SIMD Vectorization: SIMD performs same in- stands for Open Multi-processing, is an Application Program
struction on multiple data elements in a clock cycle. Today’s Interface (API) that provides a powerful directive-based pro-
modern processors are equipped with up to 512 KB vector gramming model for developing parallel applications on plat-
registers that offer higher data-level parallelism while using lessforms ranging from multicore and shared-memory systems, to
power. For instance, a CPU with an AVX-512 instruction set coprocessors and embedded systems. OpenMP API supports
can hold 8 × 64-bit double precision floats and execute a single parallel programming in C, C++, and Fortran languages. There
instruction on them in parallel, which is 8 times faster than per- are many advantages associated with OpenMP: the developed
forming a single instruction at a time. Although modern CPUs code is portable between standard compilers and operating
provide direct access for SIMD level vectorization, exploiting systems, and existing sequential programs can conveniently be
this feature is not automatic and requires advanced programming converted to parallel implementations. In OpenMP, the com-
skills and software changes. Modern compilers allow access to piler handles the low-level implementation details for gen-
SIMD features by using intrinsic functions similar to the C style erating and terminating the threads. Most compilers support
function calls instead of writing the code in assembly language. OpenMP.
Fig. 4 describes the difference between scalar and vectorized The NRPF code is comprised of many for-loops that can
SIMD operation of a Fused Multiply Add (FMA) instruction for effectively be parallelized with OpenMP directives. Without
calculating combined addition and multiplication. Data types OpenMP, the for-loop iterations in Fig. 5 are processed in
have the following naming convention, serial and by only one thread. On the other hand, the omp
__m<register_width_in_bits><data_type> parallel for pragma provides a way to utilize multiple CPU
where register_width_in_bits is architecture dependent (e.g., cores and perform the calculations faster and more efficiently.
64 bits in MMX, 128 bits in SSE, 256 bits in AVX, etc.) and This directive allows the user to specify several configuration
parameters: the number of threads to use, shared and private
data_type is either d, i, or no letter for double precision, integer,
or single precision floating point, respectively. For example, variables, workload scheduling mechanism, and reduction op-
__m256 d vector contains eight 64-bit doubles. Intrinsic func- eration. Workload scheduling is helpful when the amount of
tions have the following format, work varies across iterations. For example, if no scheduling
_mm<register_width_in_bits>_<operation>_<suffix> mechanism is specified, all large jobs may be assigned to thread
where register_width_in_bits is similar to data type, op- 0, and all small jobs may be processed by thread 1. Hence, thread
eration is arithmetic, logical or data operation to perform on 1 will finish faster, and will stay idle until thread 0 completes
the data stream, and suffix is one or two letters that indi- its task. As a result, the total execution time will significantly
cate the data type the instruction operates upon. For example, increase. OpenMP provides three scheduling options to balance
_mm512_fmadd_pd performs fused multiply add on three given the workload: static, dynamic, and guided. Interested readers
vectors as demonstrated in Fig. 4. Notice that single underscore are referred to the OpenMP manual to understand the difference
prefix is used to call intrinsic functions and double underscore between each work scheduling mechanism. However, based on
is used to declare data types. In this study, the high-performance extensive testing and evaluation, guided scheduling was deter-
Intel MKL functions are thoroughly used to maximize the mined to be the best choice in terms of performance and accuracy
peak performance of sparse matrix operations since they are of the results for this study.
optimized to take advantage of both SIMD and multicore Another OpenMP for-loop option that was widely used in our
level parallelism (i.e., mkl_sparse_spmm, mkl_sparse_z_add, implementation of NRPF is the parallel reduction primitives.
mkl_sparse_z_mm, mkl_sparse_z_mv). Reductions are used in loops with dependencies, where a series
2) Shared-Memory Parallel Programming: there are a hand- of variables must be processed to produce an output (e.g., sum,
ful of extensions for developing shared-memory parallel pro- max, min) at the end of the parallel region. Unlike atomic
grams, including pthreads [30], OpenMP [31], OpenACC [32], operations, thread synchronization is not needed in the case
Intel TBB [33], etc. Among these extensions, OpenMP, which of parallel reductions. Thus, the overall performance is much

Authorized licensed use limited to: UNIV OF ENGINEERING AND TECHNOLOGY LAHORE. Downloaded on November 28,2022 at 05:43:53 UTC from IEEE Xplore. Restrictions apply.
 1700 IEEE TRANSACTIONS ON POWER SYSTEMS, VOL. 37, NO. 3, MAY 2022

Fig. 6. Proposed Algorithm for Forming the Sparse Jacobian Arrays.

higher. A parallel reduction can reduce the operations from N the program only uses these two mapping arrays to locate and
steps to log2 N steps [19]. update the values of the Jacobian matrix. The proposed strategy
is shown in Fig. 6. Examples are provided in this figure to better
illustrate the output of each step in the process. These examples
D. Initializing and Updating the Sparse Jacobian Matrix relate to the matrix A in Fig. 1 and are only for illustrative
Populating the Jacobian matrix in the dense implementation purposes. The key steps are as follows:
of NRPF is straightforward because the matrix dimension is 1) The function receives the sparse sub-matrices J1 to J4
(npv + npq × 2)2 , and each entry can bed directly accessed that contain the partial derivative values, together with the
given the row and column index. Thus, memory allocation at list of PV and PQ buses.
the beginning and updating the values at each iteration can be 2) The number of nonzero elements in each row of the
performed quickly. However, this is not the case for the sparse Jacobian matrix are calculated by processing the J1 to
implementation because the total number of nonzero elements J4 arrays. This step is parallelized by using the OpenMP
in the matrix and the number of nonzero entries in each row for-loop pragma and sum reduction. The number of
are needed to allocate memory for the CSR arrays. Therefore, nonzero elements is necessary to allocate memory and
two problems must be addressed: forming the CSR arrays and create the row pointer array Jrow . The binary search is
updating the values at each iteration. used to quickly find the bus types.
We propose a parallel workflow for generating the CSR ar- 3) Next, the sparse column Jcolumn and value Jvalues arrays
rays using the OpenMP lock mechanism and the binary search are constructed by processing the input data. This step is
algorithm. The OpenMP lock mechanism is necessary to ensure parallelized with OpenMP locks so that multiple threads
that only one thread can modify a data element at a time. We can work on values in each matrix row simultaneously.
also need to determine whether a bus number belongs to a PV We also create the mapping arrays during this process.
or PQ bus during the process. Binary search is beneficial for this 4) Because of the parallel approach used in the previous
purpose. It is a fast algorithm to find the index of a value within step, the column indices in the Jcolumn array are not in
a sorted array by repeatedly splitting the search space in half and incremental order. We use a parallel sorting algorithm to
comparing the target value with the middle element of the list sort the column indices in increasing order. The mapping
until the value is found or the list is empty. arrays are also sorted accordingly.
Creating sparse Jacobian arrays is time-consuming, and re- To further explain the parallel approach for constructing the
peating the same procedure at every iteration will undesirably sparse Jcolumn array, let us assume that the second row of the
affect the execution time. To address this issue, we develop two matrix A in Fig. 1 is being processed by two threads in parallel.
mapping arrays while populating the Jacobian CSR arrays in the There are two nonzero columns in this row. Suppose that the
first iteration and use them in successive iterations to update the thread that is processing the column number of value “9” has
Jacobian values. We take advantage of the fact that the Jacobian finished first, and the column number is to be saved in the
matrix’s sparse row and column arrays remain the same, and Jcolumn array. This thread is aware that positions one and two in
only the values change. The first mapping array (Jtype ) refers to the column indices array are dedicated to the second row, but it
one of the four partial derivative arrays J1 to J4, and the second is unaware of which position belongs to the column index of the
mapping array (Jposition ) refers to the position of that value in “9” entry. Therefore, this thread begins by checking the status
the respective partial derivative array. So, after the first iteration, of position one in the Jcolumn array and locks the access to it

Authorized licensed use limited to: UNIV OF ENGINEERING AND TECHNOLOGY LAHORE. Downloaded on November 28,2022 at 05:43:53 UTC from IEEE Xplore. Restrictions apply.
 AHMADI et al.: FAST NEWTON-RAPHSON POWER FLOW ANALYSIS BASED ON SPARSE TECHNIQUES AND PARALLEL PROCESSING 1701

TABLE III TABLE IV

SUMMARY OF TEST POWER SYSTEMS COMPARISON OF THE SOLUTION ACCURACY BETWEEN MATPOWER AND
PROPOSED IMPLEMENTATION (TOLERANCE 1E-8)

so that other threads will not be able to make any changes. If

the value in this position is -1 (i.e., this array is initialized with VI. EXPERIMENTAL RESULTS
-1), the respective column index will be written in this position,
and the lock will be released. Otherwise, the thread checks the A. Accuracy of the Program
status of the next position until a vacant position is found. Due The accuracy of the developed NRPF code in this study is
to the nature of this parallel process, the order of index values in compared with MATPOWER 7.1 as the reference. MATPOWER
the Jcolumn array may not be incremental. Therefore, a sorting is a well-known, high-performance, and open-source power
algorithm is utilized at the end to address this matter. system simulation package based on MATLAB. Both accuracy
and speed tests in MATPOWER have been performed on the
V. EXPERIMENTAL SETUP above-mentioned computer using MATLAB 2021a. It is impor-
tant to mention that there is no difference in the convergence
A. Software and Hardware Configuration property of the proposed NRPF implementation and the original
The proposed NRPF code is developed in C++ and compiled NRPF method. Results in Table IV show that the number of iter-
as a 64-bit application, using the Intel compiler with /Q2 switch. ations to converge to the solution is the same as MATPOWER.
We used the Intel MKL library that comes with Intel OneAPI Additionally, the difference in the value of error norm between
Base Toolkit 2021.3. The dsecnd function of MKL library MATPOWER and our program for three stages of the iterative
was employed to measure the time duration throughout this process is presented. Here, f0 refers to the error norm difference
study. Also, the peak memory usage was measured by using at the beginning of the process, fn−1 is the difference before the
the mkl_peak_mem_usage function. The operating system of last iteration, and fn is the difference value in the final iteration.
the test machine was CentOS Linux 8.2 Core. The experiments The insignificant difference in the final values of error norm is
in this study are conducted on a two-socket machine with Intel due to the conversion of input data to double-precision format
Xeon Gold 6148 processors (total of 40 cores with a base clock and rounding in floating point. In fact, error values are exactly
speed of 2.4 GHz and 376 GB of DDR4 DRAM). Each CPU the same prior to the last iteration.
core is equipped with AVX-512 instruction set and two FMA
instructions. Hyperthreading was disabled during the experi-
mental evaluation; therefore, the number of threads is equal to B. Performance Analysis
the number of actual CPU cores. This section provides a comprehensive insight into the per-
formance of the proposed implementation. We evaluate two
scenarios and each is discussed as follows:
B. Benchmark Systems 1) Case #1: Table V presents the average runtime obtained
The benchmark power systems in this study are taken from for essential steps of the NRPF program when both SIMD and
the MATPOWER 7.1 [34] package. These power systems are OpenMP are utilized (i.e., the impact of matrix reordering and
used to test the performance of our NRPF program and provide use of sparse arrays is not presented separately, but without
a comparison with other studies. Table III shows the summary applying these methods, the execution time and memory usage
information of each test case. The 118 and 300 bus systems are are many times higher than the obtained results). Note that
standard IEEE test cases. The power systems containing 1354, parallelization at the SIMD level is still active when only one
2869, 9241, and 13659 buses represent different parts of the CPU core is utilized. The bold number in front of the function
European high voltage transmission network. The power flow name shows the total computation time for that function. This
data for Polish and French networks are embedded in the 2,383 value may be slightly higher than the sum of time values for
and 6,470 bus cases, respectively. Lastly, the synthetic 25,000 the listed steps because some unimportant steps are omitted
and 82,000-bus transmission systems are developed to mimic due to the space limit. Execution time is recorded for utilizing
the U.S. North East and the entire continental U.S. power grid one and multiple CPU cores. In the case of multiple cores,
models [35]. Selected swing bus numbers in this study are similar the indicated core number is when the peak performance was
to those in MATPOWER. achieved. Important observations are summarized as follows:

Authorized licensed use limited to: UNIV OF ENGINEERING AND TECHNOLOGY LAHORE. Downloaded on November 28,2022 at 05:43:53 UTC from IEEE Xplore. Restrictions apply.
 1702 IEEE TRANSACTIONS ON POWER SYSTEMS, VOL. 37, NO. 3, MAY 2022

TABLE V
AVERAGE EXECUTION TIME OF PROPOSED NEWTON-RAPHSON POWER FLOW IMPLEMENTATION WITH SIMD AND MULTICORE LEVEL PARALLELISM (SECOND,
100 TRIALS, TOLERANCE 1E-8)

1) Results show that combining sparse techniques, mathe-

matical methods, parallel processing, and existing high-
performance libraries can significantly reduce the com-
putation time of the NRPF method. Even when only one
CPU core is used, SIMD level parallelism provides very
satisfactory performance. It is also observed that utilizing
multiple CPU cores further improved the performance.
However, performance enhancement in smaller power
systems is less tangible because of their smaller workload.
2) On the other hand, there is a trade-off between the number
of CPU cores and peak performance achievement. As
shown in Fig. 7, execution time starts to degrade after
utilizing a certain number of cores due to the synchro- Fig. 7. Average Iteration Time vs. Number of Cores.
nization overhead. Our analysis shows that all parts of the
algorithm begin to slow down once the saturation point
is reached, but the performance degradation of computa- usage belongs to the SyntheticUSA case with 548 MB,
tionally expensive routines is more significant than other which means running ten concurrent power flow studies
parts. would require more than 5.5 GB of memory.
3) Solving the linear system of equations is the most time- 5) Enforcing the reactive power limit of generators (i.e.,
consuming part of the NRPF algorithm. With the re- Qlim) results in more iterations, which ultimately leads
ordering strategy, solver time is 5 to 20 times higher. to slower execution time. This feature is typically im-
On the other hand, forming the sparse Jacobian arrays plemented as an outer loop around the core power flow
is the second most time-consuming routine in the PF function to correct the violations in a brute force fashion
loop. Comparing the runtime between one and multi- by converting the generator’s bus type to PQ, turning the
ple CPU cores shows that our proposed parallel ap- unit off, and re-running the power flow. Comparing one
proach improved the speed up to 1.75×. More importantly, and multiple core results further confirms the importance
we were able to eliminate this operation in consecutive and effectiveness of using OpenMP to thread the NRPF
iterations by utilizing the mapping arrays that were code on top of the SIMD parallelism, especially when
developed when forming the sparse Jacobian arrays. For equipment limits are enforced.
instance, the SynthicUSA case would require one addi- 2) Case #2: We have so far investigated the performance of
tional second to form the sparse Jacobian matrix without running a single power flow study with SIMD and OpenMP
this strategy. parallelization. In this scenario, we aim to study the impact of
4) Prior knowledge of memory usage is beneficial when it our proposed NRPF on the performance of applications that rely
comes to simultaneous execution of multiple power flow on many PF calculations. For this purpose, we have developed
studies or when the target processor is a GPU. Please note a contingency analysis program with the proposed NRPF as its
that in many of the Intel MKL routines, memory buffers for core algorithm. Similar to Case #1, solution tolerance and the
each thread are created. Thus, peak memory usage can be maximum number of iterations is 1E-8 and 10, respectively. The
affected by the number of threads. The highest memory number of CPU cores is the same as those used previously to

Authorized licensed use limited to: UNIV OF ENGINEERING AND TECHNOLOGY LAHORE. Downloaded on November 28,2022 at 05:43:53 UTC from IEEE Xplore. Restrictions apply.
 AHMADI et al.: FAST NEWTON-RAPHSON POWER FLOW ANALYSIS BASED ON SPARSE TECHNIQUES AND PARALLEL PROCESSING 1703

TABLE VI TABLE VII

EXECUTION TIME OF CONTINGENCIES (SECOND) PERFORMANCE COMPARISON OF IMPLEMENTED NEWTON-RAPHSON POWER
FLOW WITH OTHER REFERENCES

reach the maximum peak performance. The N-1 contingency

analysis in this study considers the loss of one generator or
one transmission line in the system. However, we limit the line
contingencies to a maximum of 2,000 to restrict the excessive
calculation time for large power systems. Additionally, the test
power systems are first solved to create a base case. Afterward,
each contingency scenario uses a copy of this base case to apply
the outage and solve the power flow.
Four combinations are simulated to tabulate the data in Ta-
ble VI. When OpenMP is not active, multicore processing of
for-loops plus the parallel strategy for forming Jacobian arrays
is disabled. However, the SIMD level parallelization is always
active regardless of the OpenMP status. Results show that using
OpenMP improves the performance anywhere between 5% to
35% depending on the power system size. Typically, the perfor-
mance gain is higher for larger power systems. This observation
is consistent with the results in the previous scenario. It is
also possible to solve multiple contingency cases concurrently.
However, this study is left for future research.

C. Comparison With Other Studies

Conducting a fair performance comparison between our pro-
posed implementation of NRPF with other studies was very
challenging. Studies are different in terms of the implementation
platform (i.e., CPU, GPU, Hybrid), matrix storage format (i.e.,
dense, sparse), functions that are included in the total execution
time (i.e., sometimes only the power flow loop has been studied),
and test power systems (i.e., different sizes). There are also some
recent studies with a focus on concurrently running many power
flows on CPUs or GPUs, but without providing the execution
time for only one instance of the NRPF. To the extent possi-
ble, we attempted to provide a complete comparison between
the proposed implementation and other references. Only the
fastest reported runtime is selected from studies with multiple
implementations. The same number of CPU cores as in Table V
was used to benchmark the MATPOWER performance (i.e., two
cores for 118 and 300 buses systems). MATPOWER-NR-Qlim
is the runtime of MATPOWER when generator’s reactive power
limit was enforced.
As shown in Table VII, the only instance where GPU was able advance caching techniques to improve the performance in the
to compete with the multicore CPUs is the 118-bus system; even subsequent executions of a program. Because we run MAT-
though the difference is less than 3 ms. For medium and large POWER several times to obtain the average execution time, these
power systems, the runtime of our developed NRPF is 1.5-2.5× techniques influence the reported value, and the actual runtime
better than MATPOWER. For instance, the execution time of is expected to be slower. As expected, fast decoupled power
the proposed algorithm is 2.1× better than MATPOWER for the flow is typically 2-3 times faster than the N-R method because
82,000-bus system, which is also the fastest reported runtime in it is the simplified version of N-R with less computational cost
the literature for a system of this size at the time of publication. per iteration. However, it requires more iterations to converge
It is worth mentioning that MATLAB uses memoization and than NRPF, which is the reason for the 12 s computation time

Authorized licensed use limited to: UNIV OF ENGINEERING AND TECHNOLOGY LAHORE. Downloaded on November 28,2022 at 05:43:53 UTC from IEEE Xplore. Restrictions apply.
 1704 IEEE TRANSACTIONS ON POWER SYSTEMS, VOL. 37, NO. 3, MAY 2022

of the 82,000-bus power system. It is also known that FD [14] S. Huang and V. Dinavahi, “Real-time contingency analysis on massively
power flow may diverge for cases that are solvable by the parallel architectures with compensation method,” IEEE Access, vol. 6,
pp. 44 519–44530, 2018, doi: 10.1109/ACCESS.2018.2864757.
N-R method. [15] G. Zhou et al., “A novel GPU-accelerated strategy for contingency screen-
ing of static security analysis,” Int. J. Elect. Power Energy Syst., vol. 83,
pp. 33–39, 2016.
VII. CONCLUSION [16] X. Su, C. He, T. Liu, and L. Wu, “Full parallel power flow solution: A
GPU-CPU-based vectorization parallelization and sparse techniques for
In this paper, we presented the development of a fast Newton- Newton-Raphson implementation,” IEEE Trans. Smart Grid, vol. 11, no. 3,
pp. 1833–1844, May 2020.
Raphson power flow on multicore CPUs by combining software [17] C. Guo, B. Jiang, H. Yuan, Z. Yang, L. Wang, and S. Ren, “Performance
techniques, mathematical methods, and parallel processing. We comparisons of parallel power flow solvers on GPU system,” in Proc. IEEE
proposed a parallel approach for developing the Jacobian CSR Int. Conf. Embedded Real-Time Comput. Syst. Appl., 2012, pp. 232–239.
[18] J. Singh and I. Aruni, “Accelerating power flow studies on graphics
arrays and also a set of mapping arrays that eliminate the need processing unit,” in Proc. Annu. IEEE India Conf., 2010, pp. 1–5.
for creating the Jacobian arrays in successive iterations. The [19] V. Roberge, M. Tarbouchi, and F. Okou, “Parallel power flow on graphics
accuracy of the proposed algorithm was verified by comparing processing units for concurrent evaluation of many networks,” IEEE Trans.
Smart Grid, vol. 8, no. 4, pp. 1639–1648, Jul. 2017.
the error norm with MATPOWER program. The code was [20] C. Vilacha, J. Moreira, E. Miguez, and A. F. Otero, “Massive Jacobi power
precisely profiled to show the computation time for each step of flow based on SIMD-processor,” in Proc. 10th Int. Conf. Environ. Elect.
the algorithm and provide a reference for future work in this area. Eng., 2011, pp. 1–4.
[21] A. Ahmadi, F. Manganiello, A. Khademi, and M. C. Smith, “A parallel
Based on the results obtained, the performance of the proposed Jacobi-embedded Gauss-Seidel method,” IEEE Trans. Parallel Distrib.
approach is better than GPU-based accelerated power flows. To Syst., vol. 32, no. 6, pp. 1452–1464, Jun. 2021.
the best of our knowledge, this is the first time in the literature [22] X. Li, F. Li, H. Yuan, H. Cui, and Q. Hu, “GPU-based fast decoupled power
flow with preconditioned iterative solver and inexact Newton method,”
that the performance of the NRPF algorithm on an 82,000-bus IEEE Trans. Power Syst., vol. 32, no. 4, pp. 2695–2703, Jul. 2017.
system was measured, and the proposed implementation was [23] S. Huang and V. Dinavahi, “Performance analysis of GPU-accelerated
able to solve the system in 1.87 seconds, which is 2.1 times fast decoupled power flow using direct linear solver,” in Proc. IEEE Elect.
Power Energy Conf., 2017, pp. 1–6.
faster than the high-performance MATPOWER library. This [24] M. M. A. Abdelaziz, “OpenCL-accelerated probabilistic power flow for
study opens the opportunity to improve the performance of other active distribution networks,” IEEE Trans. Sustain. Energy, vol. 9, no. 3,
applications that are reliant on PF computation. pp. 1255–1264, Jul. 2018.
[25] D. Ablakovic, I. Dzafic, and S. Kecici, “Parallelization of radial three-
phase distribution power flow using GPU,” in Proc. 3rd IEEE PES Innov.
Smart Grid Technol. Europe, 2012, pp. 1–7.
REFERENCES [26] M. Abdelaziz, “GPU-OpenCL accelerated probabilistic power flow anal-
ysis using Monte-Carlo simulation,” Electric Power Syst. Res., vol. 147,
[1] R. C. Green, L. Wang, and M. Alam, “Applications and trends of high
pp. 70–72, 2017.
performance computing for electric power systems: Focusing on smart
[27] T. Cui and F. Franchetti, “A multi-core high performance computing
grid,” IEEE Trans. Smart Grid, vol. 4, no. 2, pp. 922–931, Jun. 2013.
framework for distribution power flow,” in Proc. North Amer. Power Symp.,
[2] R. C. Green, L. Wang, and M. Alam, “High performance computing for
2011, pp. 1–5.
electric power systems: Applications and trends,” in Proc. IEEE Power
[28] Intel Math Kernel Library. [Online]. Available: https://software.intel.com/
Energy Soc. Gen. Meeting, 2011, pp. 1–8.
mkl
[3] D. M. Falcão, “High performance computing in power system applica-
[29] B. Nichols, D. Buttlar, and J. P. Farrell, Pthreads Programming. Se-
tions,” in Proc. Int. Conf. Vector Parallel Process., 1996, pp. 1–23.
bastopol, CA, USA: O’Reilly Associates, 1996, pp. 1–29.
[4] M. Čepin, Assessment of Power System Reliability: Methods and Applica-
[30] OpenMP. [Online]. Available: https://www.openmp.org/
tions. Berlin, Germany: Springer, 2011, pp. 147–154.
[31] OpenACC. [Online]. Available: https://www.openacc.org/
[5] H. Dağ and G. Soykan, “Power flow using thread programming,” in Proc.
[32] Intel Threading Building Blocks. [Online]. Available: https://software.
IEEE Trondheim PowerTech, 2011, pp. 1–5.
intel.com/tbb
[6] A. Ahmadi, S. Jin, M. C. Smith, E. R. Collins, and A. Goudarzi, “Parallel
[33] Matpower 7.0. [Online]. Available: https://matpower.org/download/
power flow based on OpenMP,” in Proc. North Amer. Power Symp., 2018,
[34] A. B. Birchfield, T. Xu, K. M. Gegner, K. S. Shetye, and T. J. Overbye,
pp. 1–6.
“Grid structural characteristics as validation criteria for synthetic net-
[7] G. Guerra and J. A. Martinez-Velasco, “Evaluation of MATPOWER and
works,” IEEE Trans. Power Syst., vol. 32, no. 4, pp. 3258–3265, Jul. 2017.
OpenDSS load flow calculations in power systems using parallel comput-
[35] H. Jiang, D. Chen, Y. Li, and R. Zheng, “A fine-grained parallel power
ing,” J. Eng., vol. 2017, no. 6, pp. 195–204, 2017.
flow method for large scale grid based on lightweight GPU threads,” in
[8] R. D. Zimmerman, C. E. Murillo-Sánchez, and R. J. Thomas, “MAT-
Proc. IEEE 22nd Int. Conf. Parallel Distrib. Syst., 2016, pp. 785–790.
POWER: Steady-state operations, planning, and analysis tools for power
systems research and education,” IEEE Trans. Power Syst., vol. 26, no. 1,
pp. 12–19, Feb. 2011.
[9] R. S. Kumar and E. Chandrasekharan, “A parallel distributed computing
framework for Newton-Raphson load flow analysis of large interconnected
power systems,” Int. J. Elect. Power Energy Syst., vol. 73, pp. 1–6,
2015.
[10] L. Ao, B. Cheng, and F. Li, “Research of power flow parallel computing
based on MPI and P-Q decomposition method,” in Proc. Int. Conf. Elect. Afshin Ahmadi (Member, IEEE) received the M.S.
Control Eng., 2010, pp. 2925–2928. degree in electrical engineering from the Univer-
[11] X. Wang, S. G. Ziavras, C. Nwankpa, J. Johnson, and P. Nagvajara, sity of the Philippines, Philippines, in 2012, and the
“Parallel solution of Newton’s power flow equations on configurable Ph.D. degree in computer engineering from Clemson
chips,” Int. J. Elect. Power Energy Syst., vol. 29, no. 5, pp. 422–431, 2007. University, SC, United States, in 2020. He is cur-
[12] G. Zhou et al., “GPU-accelerated batch-ACPF solution for N-1 static rently a Power System Application Developer with
security analysis,” IEEE Trans. Smart Grid, vol. 8, no. 3, pp. 1406–1416, the Electric Reliability Council of Texas (ERCOT).
May 2017. His main research interests include high-performance
[13] I. Araújo, V. Tadaiesky, D. Cardoso, Y. Fukuyama, and Á. Santana, computing in power and energy systems, smart grid,
“Simultaneous parallel power flow calculations using hybrid CPU-GPU and power system planning.
approach,” Int. J. Elect. Power Energy Syst., vol. 105, pp. 229–236, 2019.

Authorized licensed use limited to: UNIV OF ENGINEERING AND TECHNOLOGY LAHORE. Downloaded on November 28,2022 at 05:43:53 UTC from IEEE Xplore. Restrictions apply.
 AHMADI et al.: FAST NEWTON-RAPHSON POWER FLOW ANALYSIS BASED ON SPARSE TECHNIQUES AND PARALLEL PROCESSING 1705

Melissa Crawley Smith (Senior Member, IEEE) re- Vahid Dargahi (Member, IEEE) received the Ph.D.
ceived the B.S. and M.S. degrees in electrical engi- degree in electrical engineering, with an emphasis in
neering from Florida State University, Tallahassee, power electronics and power systems, from Clemson
Florida, in 1993 and 1994, respectively, and the Ph.D. University, Clemson, SC, USA, in 2017. From 2016 to
degree in electrical engineering from the University 2017, he was a Graduate Research Assistant with the
of Tennessee, TN, United States, in 2003. She is eGRID Center, CURI, North Charleston, SC, USA.
currently an Associate Professor of electrical and From 2018 to 2019, he was a Postdoctoral Research
computer engineering with Clemson University. She Fellow with the Electrical and Computer Engineer-
has more than 25 years of experience developing ing Department, UC Santa Cruz, CA, USA. He is
and implementing scientific workloads and machine currently an Assistant Professor with the School of
learning applications across multiple domains, in- Engineering and Technology, University of Washing-
cluding 12 years as a research associate at Oak Ridge National Laboratory. Her ton, Tacoma, WA, USA. His current research interests include power systems,
current research interests include the performance analysis and optimization of parallel processing, power electronics circuits, novel converter topologies, wide-
emerging heterogeneous computing architectures (GPGPU- and FPGA-based bandgap semiconductor devices, multilevel inverters, control of power electronic
systems) for various application domains including machine learning, high- systems, grid-tied inverters, and active rectifiers.
performance or real-time embedded applications, and image processing.

Shuangshuang Jin (Senior Member, IEEE) received

Edward Randolph Collins (Senior Member, IEEE) the Ph.D. degree in computer science from Wash-
received the B.S.E.E. degree from North Carolina ington State University in 2007. She is an Asso-
State University, Raleigh, NC, USA, and the Ph.D. ciate Professor with the School of Computing with a
degree in electrical engineering from the Georgia joint appointment in the Department of Electrical and
Institute of Technology, Atlanta, GA, USA. In 1989, Computer Engineering at Clemson University. Her re-
he joined the Department of Electrical and Computer search interests include high-performance computing
Engineering, Clemson University, North Charleston, (HPC), big data analysis, and scientific computation
SC, where he is currently a Professor. He directs the and visualization. She has over 15 years of inter-
research activities in the Power Quality and Industrial disciplinary research experience applying advanced
Applications Laboratory at Clemson University and computing technology to pressing scientific and engi-
has focused his research on power quality and grid neering domain areas. Prior to joining Clemson University, Dr. Jin was a senior
integration, in addition to power electronics, machines and controls. He was in research scientist at Pacific Northwest National Laboratory, and contributed
several leadership positions at Clemson related to electric power and energy, to the development of a variety of HPC-based power system applications
including executive Director of the Energy Innovation Center. such as look-ahead dynamic simulation, distributed state estimation, massive
contingency analysis, GridLAB-D, and GridPACK, etc.

Authorized licensed use limited to: UNIV OF ENGINEERING AND TECHNOLOGY LAHORE. Downloaded on November 28,2022 at 05:43:53 UTC from IEEE Xplore. Restrictions apply.