NATIONAL ELECTRIC POWER REGULATORY AUTHORITY (NEPRA) INQUIRY REPORT TOTAL POWER BLACKOUT IN THE COUNTRY ON January 23, 2023INQUIRY INTO THE MATTER OF TOTAL POWER BLACKOUT IN THE COUNTRY OCCURRED ON JANUARY 23, 2023 DUE TO SYSTEM DISTURBANCE IN NTDC & K-ELECTRIC TRANSMISSION NETWORK ‘TABLE OF CONTENTS CONTENTS PAGE.NO. Part-I - 1 | Background 2 | Brief Introduction of the Power System of Pakistan: 3__ | National Power Control Center (NPCC) ~ System Operator (SO) ————-~ —_J,_-———_f 2 4 Generation | 5. | Transmission System 3 | 6 | Distribution of Electricity 4 Part-2 pees) anon a | Proceedings 5 Part-3 18 20 10 | Analysis 23 "| Implementation status of the findings/recommendations of inqui 1 5 ee qury 24 | reports furnished thereof for such break downs in the past 30 Conelusion 12 | indings | l 32Part 1 Background2. 21 22 23 BACKGROUND: Total power collapse/blackout in the country occurred on 23° January 2023 at 07:34:43:800 Hrs. National Electric Power Regulatory Authority (NEPRA), being an exclusive Regulator of electric power sector in Pakistan, took serious notice of the referred blackout and decided to probe into the ‘matter through an Inquiry Committee (IC) comprising the NEPRA as well as the market experts. 1.1 Constitution of Committee: NEPRA vide office Orders No. MON-1/2023/110 dated 10" February 2023 and No. MON- 01/202/153 dated 23° February 2023 constituted an Inquiry Committee (IC) to probe in the ‘matter of total power collapse/blackout in the country on 23" January 2023, Annex. 1 & 2 1. [ Mr. Imran Kazi Senior Advisor, M&E, NEPRA | Convene? 2. | Mr. Nadir Ali Khoso ‘Market Expert Member 3. | Mr. Ghulam Abbas Memon | Market Expert Member 4. | Syed Safeer Hussain Market Expert Member 5. | Mr. Manu Ram Market Expert Co-Opt Member 6. | Syed Agib Ali Shah Dy. Director, M&E, NEPRA Secretary 1.2 Terms of Reference (TORS) i) To determine, the causes and the faults which resulted in the cascade trippings and system collapse; ii) To fix the responsibility for the country wide blackout occurred on 23.01.2023; iii) To review, whether the restoration efforts were adequate and the power supply was restored within the reasonable time; iv) Tocheck/update the implementation status of the findings/recommendations of inquiry reports furnished thereof for such break downs in the past; ¥) _ Torecommend, the remedial measures to be taken in order to avoid recurrence of such events in future; BRIEF INTRODUCTION OF THE POWER SYSTEM OF PAKISTAN: The main segments of electric power sector are Generation, Transmission and Distribution. In Karachi and its suburb the Generation, Transmission and Distribution of electric power are being ‘maintained and operated by a privatized vertically integrated utility namely the K-Electric (KE). KE, besides having its own power generation plants, is importing power from NTDC through Central Power Purchasing Agency Guarantee Limited (CPPAG) and other private power generation companies. The details of KE’s generation plants and transmission network along with other power plants supplying electric power to KE is attached as Annex. 3 & Annex. 4. The distribution and supply of electric power in its area is being done by KE itself. In the larger area of the country, the generation, transmission, distribution and supply of the electric. power is being done by different companies. The electric power generation, is being done through different private (Independent Power Producers-IPPs) as well as public sector entities like WAPDA, PEDO, GENCOs etc. Transmission of electric power is being done by National ‘Transmission and Despatch Company (NTDC), a company wholly owned by the Government of Pakistan. The recently developed, + 660kV High Voltage Direct Current (HVDC) transmission WA ragetorse wu ® \\system is operated by Pak Matiari Lahore Transmission Company (PMLTC). The Central Power Purchasing Agency (CPPA-G) works as power agency on behalf of Ex-WAPDA DISCOs. 2.4 Generally, the high voltage transmission system to transfer bulk electric power generation consists of two regions i.e. North and South; At 500 KV voltage level these two regions are connected through five HVAC circuits while a + 660kV HVDC transmission system is also transmitting electric power from Matiari in South to Lahore in North. A map showing the NTDC network is attached as Annex §. The Distribution and Supply of electric power in this area is being done through ten (10) Ex-WAPDA distribution companies (WDISCOs), wholly owned by the Government of Pakistan. A map showing the territories of the distribution companies is attached as Annex 6. 3. NATIONAL POWER CONTROL CENTRE (NPCC) - SYSTEM OPERATOR (SO): NPCCINTDC is discharging the duties under the Grid Code approved by the NEPRA. The NPCCi/System Operator (SO) is responsible to operate the system in an economic manner without ‘compromising the system stability, reliability and safety. 4, GENERATION: . The brief regarding electric power generation capacity (dependable) as on 30.06.2022 and goneration there from during fiscal year 2021-22 as provided in NEPRA’s State of Industry Report 2022 is reproduce here below: ‘Dependable Capacity (MW) I WAPDA Hydel 9443, 2 IPPs Hydel 1009 3 Total Hydel 10452 4 ‘GENCOs 3900 s IPPs 17258 6 SPPS/CPPS 257 7 Nuclear 3345 8 Total Thermal 24760 9 Wind 1838 10 Solar 530 1 ‘Bagasse/Biomass 278 ‘Total CPPA G System 37858 iz KW Own 2135 iB IPPs 354 14 ‘SPPs/CPPs 106 15 KANUPP 0 16 Solar 79 17, Total KE System 2674 18 Grand Total 40532 4.1 The share of public sector power plants in power sector of Pakistan is around 21521 MW; out of which 11893 MW from thermal, 9524 MW from Hydel and 104 MW is procured from Iran. A list of power plants in NTDC as well as in KE system is attached as Annex ~ 7 and Annex-8. Www aw wr5. TRANSMISSION SYSTEM: 5.1 NTDC is the only National Grid Company (NGC) in Pakistan and its transmission network is spread all over the country except the area served by K-Electrie. NTDC is responsible to maintain and operate long transmission network of SO0KV and 220K; details of NTDC system is given here below: 17 x Grid Station of 500 KV with capacity of 32700 MVA; 67 circuits of 500 kV lines with total length of 8388 kn; ‘50 x Grid stations of 220 kV with capacity of 28160 MVA; 163 circuits of 220 kV lines with total length of 11611 km. 5.2 Diagram of NTDC system is attached as Annex-9. + 660kV HVDC Transmission Line (T/Line) is ‘maintained by NTDC, while +: 660 kV Converter Stations at Matiari and Lahore are maintained by Pak Matiari Lahore Transmission Company (PMLTC). HVDC T/Line is capable to carry 4000 MW. Due to non-availability of the adequate AC network at Lahore side, the HVDC system is underutilized, 5.3 Detail of South and North regions of NTDC is as under: Northern Region: Region | _500kV Grid Stations 220kV Grid Stations No. No. Islamabad | Rawat, Sheikh | 02. | Sangjani, University, Shahi Bagh, Mardan, | 11 ‘Munda Daud Khail, Burhan, Mansehar, Banu, Chakdara, Nowshera and DI Khan Lahore | Gatti, Sheikhupura, | 06 | NKLP Lahore, Ravi Lahore, Kala Shah Kaku, | 20 ‘New Lahore, Nokhar, Sarfaraz Nagar, Bund Road, Ghakkar, Gujjrat, Yousufwala and Sialkot, Wapda Town, —_Nishatabad, Faisalabad West Sammandri_ Road, Ludewala, Jaranwala, Shalamar, Ghazi road, Bandala, Toba Tek Singh, Okara, Kasowal and Lalian Multan | Multan, — Muzaffar | 04 | Lal Sohanra, Bahawalpur, Vehari, | 07 garh, DG Khan and Muzaffargarh, Chishtian, NGPS Piran Ghaib Rahim Yar Khan and TPS Muzaffargarh ah neSouthern Region: Region 500kV Grid Stations [ No. 220KV Grid Stations ‘No. Hyderabad | NKI, Jamshoro, | 06 | Hala Road Hyderabad, TM Khan, Jhimpir- | 07 Shikarpur, Dadu, 1, Jhimpir-Il, Daharki, Rohri, 220/132 kV Guddu, Guddu 747 switchyard Guddu ‘Quetta - 00 | Quetta, Loralai, ‘Sibbi, Khuzdar, DM | 05 Jamali 5.4 Transm n System of K-Electric: KE is operating and maintaining transmission network of 220kV and 132KV; brief regarding KE transmission system is given here below: © 11x Grid Station of 220 kV with capacity of 4580 MVA; ‘© Total length of 220 kV transmission line is 364 km; ‘* 69x Grid stations of 132 KV with capacity of 7465 MVA; ‘© Total length of 132 kV transmission line is 838 km; Besides its own transmission system, KE has following fou KV arid system: {erconnections with the NTDC’s 220 220 KV NKL-KDA; 220 KV NKI-Baldia; iii, 220 KV Jhimpir I!— KDA 1; iv, 220KV Jhimpir Il -KDA 2. Diagram of KE Transmission System is attached as Annex- 10, 6. Distribution of Electricit Following companies, under the licenses granted by NEPRA, are engaged in distributing and supplying of electric power in their designated areas: i, Peshawar Electric Supply Company Limited (PESCO) ii, Tribal Area Electricity Supply Company (TESCO) iii, Islamabad Electric Supply Company Limited (ESCO) iv. Lahore Electric Supply Company Limited (LESCO) Faisalabad Electric Supply Company Limited (FESCO) Gujranwala Electric Power Company Limited (GEPCO) Multan Electric Power Company Limited (MEPCO) ‘Sukkur Electric Power Company Limited (SEPCO) Hyderabad Electric Supply Company Limited (HESCO) X. Quetta Electric Supply Company Limited (QESCO) ‘Maps showing the teritories of above distribution and supply companies are at Annex = 11. wilh 3 ey Page 4 of 34Part 2 Proceedings7. PROCEEDINGS: 7.1 To probe in the matter it was decided to approach the relevant entities through field visits and/or by requiring relevant information and documents. IC visited the following entities to discuss the issues and obtained the information/ documents and briefs relating to total power blackout in the ‘country on 23rd January 2023 and restoration of the system thereafter: National Power Control Centre (NPC), Islamabad; K-Blectric, Karachi; NTDC 500 / 220 kV NKI grid station; Port Qasim Power Plant, Karachi; Lucky Power Plant, Karachi; K-2 & K-3 Nuclear power plants, Karachi; + 660kV HVDC Converter Station, Matiar; 220/132 KV Jhimpir-1 and Jhimpir-II Grid Stations; Tarbela Power Station; NTDC Head Office, Lahore; = 660kV HVDC Converter Station, Lahore; « 7.2. The summary of those presentations/briefings is given below; 7.3 National Power Control Centre (NPCC), Islamabad (Annex-12) ‘* System was operating normally and within permissible limits, no intimation was given by any entity i.e, NTDC Grid Stations, K-Electric, Power Plants and DISCOs. At 07:34 Hrs., ‘the system collapsed with following sequence of events; 07:33:36 Oscillations noticed 07:34:02 to 07:34:14 Frequency hunting 07:34:14 System collapsed ‘S0OkV Guddu 747-RY Khan circuit tripped on Power Swing and SOOkV supply failed 07:34:14:895 Commutation Failure occurred at HVDC :34:14:908 Sudden power loss (2306 MW to 2154 MW) 07:34:14:908 following SOOKV Cet. tripped; Time ‘Names of S00kV Cet, ‘Indication Guddu-DG~ Khan Cot | DG Khan End: Power Swing Block 908 | tripped from both ends | Guddu End: Over Voltage Guddu-Muzafargarh Cet | Muzafargarh End: Power Swing Block 07:34:14:908 | tripped from Muzatfargarh end Guddu 747-RY Khan Cot | Guddu 747 End: 07:34:14:908 | tripped from Guddu 747 ends “ower Swing Block 07:34:14:908 System split into Northern & Southern regions * 07:34:15:417 following SOOKV Cet. tripped; \/ ye 3 & \ vasa nyTime ‘Names of S00KV Cet. Indication K2-K3 - NKI Cot. tripped | K2-K3 End: DTT received from NKT 07:34:15:417 | from both ends ‘NKI End: Over Voltage (640kV) NKI- =~ Jamshoro Cet | NKTEnd: Over Voltage (640kV) 07:34:15:417 | tripped from both ends Jamshoro End: DTT Received from NKI Guddu 747-RY Khan Cot | Guddu 747 End: Power Swing Block 07:34:15:417 | tripped from Guddu 747 end = 07:34:15:568 Frequency Limit Controller (FLC) at HVDC automatically activated due to difference in frequencies of North and South zones. = 07:34:29:987 HVDC blocked Annex-13: * S0OKV Guddu ~ Muzafargarh Cot, was opened at 11:56 Hrs on 18/01/23 due to foggy weather. . * 500kV Moro RYK Cet was opened at 23:22 Hrs on 21/01/23 for voltage control 7.3.1 Power Plants Trippings Time Names of Power Plant Indication (07:34:15:900 (K2) | K2-K3 P/H (900+1040 MW) ‘Over Speed 07:34:15:935 (K3) 073416 Port Qasim P7H (620 MW) Run-back 073423 Lucky P/H (606 MW) ‘Under Frequency 07:34:25 Engro Thar (150 MW) Under Frequency 07:34:25 | ‘Thar Energy Lid (151 MW) ‘Under Frequency 07:34:25 ‘Shanghai Electric (1230 MW) Under Frequency + Low Frequency Oscillations are evident from all the power plants which may be attributed to complexities of the power system, weak transmission network, weak damping system and harmonies induced due to inverter based generation, * Commutation Failure resulted in sudden power loss in HVDC link, 350 commutation failure in HVDC have been witnessed. Commutation failure is frequent in winters. + The Failure of Over-Frequency Tripping at Port Qasim: After the splitting event, frequency of southern region went beyond 51.5 Hz, Port Qasim did not trip at it’s designed over frequency trip setting i.e, 51.4 Hz [Inst. trip], instead it run backed slowly. This failure of 620 MW tripping of Port Qasim, resulted in sustained over- frequency in south region [max recorded =51.568 Hz], which consequently tripped K- 2 and K-3 power plants (1940 MW) (Trip setting=51.5 Hz for 250msec]. Tripping of YNL BR a — u1940 MW in southern island, acted as a major disturbance leading to cascaded tripping of other generators in south. 7.3.2 Restoration SeNo. | Restoration | Power House “tempt | Warsak | Tarbels—] Mangia] Was p a OBTSHi=s | O8TORIn- | TOTOF | TOTOHs- rozstics | opioris | 1140Hke | Onwards z Second WORST OIG 100g | TaIOTRAS | = 7h TROETO2 | TOES | Wase-Ie24 | = 7 Treen BS [nerae | 5 Fit 5 Tass] = E 3 Soak 5 Tae] > 7 Seventh 5 ~[isaa-i8as—f- 3 = Trea [EY 7am 5 "With the help of Black Start Facility at UCH-1 Power Plant, the power supply of some Grid Stations in SEPCO, QESCO and HESCO was restored including ‘main cities ie. Sukur, Jacobabad, Quetta and Khuzdar. 714 K-Bleetric (Annex 14) 7.A.1 Bricfing/Presentation of K-Electric is summarized below; "On January 23, 2023 prior to power breakdown, KE and NTDC networks were running synchronized. KE network was being operated in split mode ie., major portion of KE network including generation was connected at S00kV NKI and another island was connected to 220kV Jhimpir for the purpose of network load flow conditions. * Plant wise generation and demand position @0730 brs in KE network (Annex 15) "Sequence of Tripping Events; Time Sequence of Event Source 07:34:14:906 | Disturbance in 500KV network & Over-frequency in | DFR 07:34:15:205 Taal rao om ‘BQPS3 units from 239 to 79 MW | BQ3 DFR O7:34:15:350 ‘NKJamsoro and NKI-K2KS tripped on over= | Calelaed voltage15:457 | CTS Operation in the system initiated — (Baldiay _] SCADA 7:34:15:588 | CTS Operation in the system (KDA) SCADA 73416149 | CTS Operation in the system (Mauripur) SCADA 73416224 | UFR initiated in the system SCADA7DFR 07:34:16:860 | UVLS initiated in the system SCADA7DFR 07:34:17:706 | BOPS-3, Unit-10 tripped on thermal load variation | BQ3 (07:34:18:323 | BQPS-3, Unit-20 tripped on under frequency BOS protection (07:34:18:323 | Breakdown in KE island connected with SOOKV | DFR 07:34:43:800 Beinn in KE island connected with 220kV | DFR impir 624 MW load was rejected through cross trip scheme and under frequency/voltage protection System Restoration through Tapal & Gul Ahmad (08:32 Hrs — 16:22 Hrs) ‘System Restoration through BQPS-2 and CCPP (16:25 Hrs — 1900 Hrs) 4101/2023 * 03:22 Hts , KE received power from National Grid Jhimpir © 05:28 Firs, BQ2— CCP island synchronized with NTDC. * 05:34 & 05:35 Hrs, 220kV NKI - KDA & 220 KV NKI - Baldia circuits also normalized respectively. ‘+ Compliance of Recommendation of Breakdown dated January 09, 2021 S.No ‘Recommendations by NEPRA. Actions by KE The islanding facility available at KCCPP and _ | With the commissioning of HSD at BQPS-II could not be utilized as KCCPP was | CPP, black start facility is also off due to low gas pressure whereas only one | available at KCCP. Also, with ‘machine out of the two machines of BQPS-I | integration of PLL gas, BQ2 1 | came on island for short interval but could not | complex now also possess Black survive. start facility. Also, BQPS III will have black start option post commissioning on secondary fuel (HSD). The quantum of load to be rejected through Detailed study was conducted by 2. | under frequency relays along with time and | consultant and under frequency and setting coordination between those of under voltage schemes were revised ‘transmission lines and machines needs to be __| which responded accurately in the Woh Rv me18 16 1 18 reviewed for making those efficient and recent breakdown. Details of UFLS effective. The feeders which were supposed to | and UVLS are annexed, but did not trip on under frequency shall be checked and kept health NTDC 500/220kV NKI Grid Station, Karachi The information by the NKI G/Station is placed at Annex 16 Port Qasim Power Plant, Karachi (Annex-17) Essence of the Port Qasim Power Plant presentation is given below; (660 MW Unit 1 was under shutdown, Unit 2 was operating at 656 MW load. 07:34:16 His, the system voltage increased from SOIKV to S46kV and dropped to 463KV after 1 second. "07:34:16 His, frequency increased from 50.5hz to 51.458Hz of Unit 2 and the speed of Unit 2 steam turbine increased to 3084 rpm. Unit 2 ST tripped resulting in decrease in frequency and speed, = 07:34: activated. = 07:34:30 Unit 2 tripped. Unit 2 restored on 04:44 Hrs. dated January 24, 2023. * Connected to National Grid at 12:53 Hrs. January 25, 2023 after 50.32 His "As per PPA agreement, the maximum restoration time is 72 hours after restoration of SO0kV Grid System (Annex-18). However, the generator was synchronized with busbar after 50.32 hours ‘Hts, Unit 2 boiler tripped but due to grid failure, reverse power protection was not Lucky Power Plant, Karachi (Annex-19) From the data provided by Lucky Power Plant, following has been observed; Generation before the breakdown was 606 MW + 07:34:24:12 the power plant tripped ‘Generator tripped immediately after tripping of Lucky-Matiari Cet, due to disturbance in Grid frequency and Lucky ~ Port Qasim Cet. voltage became zero resulting in power loss at the complex = Frequency at the time of tripping was 47.0 Hz K2, K3 Karachi Brief of the K2, K3 presentation is given below (Annex-20); Capacity of K2 Power Plant is 1100 MW and K3 Power Plant is 1100 MW Before the power breakdown, K2 was generating 1040 MW and K3 was generating 900 MW. ‘At 07:34:15:216 DTT trip signal received from NKI Grid at K2/K3 Grid NKI Line Breakers Opening Time: 9ETTI20JA (07:34:15:248) & SETT130JA (07:34:15:250) 07:34:15,719, K3 tripped on Over Frequency Split Tripping setting. The K2 trip command issue time was 07:34:15:670, whereas breakers opening time was 07:34:15.719 & 07:34:15,721 and the Grid Freq, @ Breaker opening was 51.642 Hz, V=25.73kV Wwe Bev » Page 9 of 34= 07:34:15.722, K2 Power Plant tripped on Over Frequency Split Tripping setting. The K2 trip command issue time was 07:34:15:669, whereas breaker opening time was 07:34:15,722 (another breaker earlier opened at DTT) and the Grid Freq. @ Breaker opening was 51.642 Hz, ‘V= 25.73kV 7.9 + 660kV HVDC Converter Station, Matiari (Annex-21) ‘Summary of the briefing given by HVDC, Matiari Converter Station is as under; + 660kV HVDC Project from Matiati to Lahore starts from Matiari Converter Station in Sindh and ends at Lahore Converter Station in Punjab. The project includes two 660kV DC converter stations, 886km HVDC Bi-pole transmission lines. "The rated capacity is 4000MW. The rated voltage is + 660kV, whereas the rated maximum ‘current of each pole 3030A. 07:34:13:222, AC voltage started decreasing i.e. 478kV 07:34:15:568, FLC activation 16:130 first instant when FLC achieve 1000MW 07:34:29:987, Pole | block 07:34:30:001, Pole II block Power before Tripping was 2400MW : AC voltage at the time of tripping was 46.6KV DC voltage at the time of tripping was 43KV 7.10 +660kV HVDC Converter Station, Lahore (Annex-22) Summary of the presentation given by HVDC, Lahore Converter Station is as under; 07:34:30:258, Pole 1 block 07:34:30:270, Pole II block 07:34:14:895, Pole | Commutation Failure 07:34:14:895, Pole II Commutation Failure 13KV (L-N) RMS 26kV 07:34:30:241, DC voltage at ¥ Bloc! Commutation Failure occurred at low voltage indication Power before Tripping was 2400MW 7.10.1 Out of Step Splitting Devices: Out of step protection devices have been installed at the different locations to solve the problem of system out of step between North and South power grid under extremely serious fault (Annex-23); 7.11 220/132kV Jhimpir - II Grid Stations (Annex-24) Brief of RE Jhimpir-II submissions is as under; Capacity = 3 x 250 MVA 07:00:00 Load = 3 MW 07:00:00 Voltage = 234kV 07 Frequency = 49.8 Hz 07:35 Hrs Main Supply Fail Page 10 of 34 AL7A2 220/132kV Shimpir - I Grid Station Briefing given by the 220kV/132kV RE Jhimpir-I is as under (Annex-25); Capacity = 4x250 MVA 07:00 His. Load = 7 MW on each transformer 07:00 Hrs. Voltage = 230V 07:00 Hrs, Frequency = 49.7 Hz 7:35 Hts. Main Supply Fail, 7.13 Tarbela Power Station (Annex-26) Brief of the Tarbela Power Station is as under; = Total installed capa Pre Fault Conditions: ‘+ Loading of Tarbela + 4 Extension = 686 MW + All 220kV/SOOKV T/Lines were energized except 500kV Ghazi Barotha T/Line 2 Bus Bar Voltage: © 220kV Bus Bar No 1&2 D 232/235kV © S00KV Bus Bar No I & 2 529/539 kV ‘+ 07:34:30 sudden dip in the system frequency from (50Hz to 0 Hz) caused the tripping of generating units & transmission lines of Tarbela Power Station. 4888 MW. 7.13.1 Restoration events submitted by Tarbela Power Station 23/01/2023 07:35 His, DG Set ~2 started on Auto 07:38 Hrs. DG Set — I started on Manual 07:56 Hrs. Unit No. 2 started and Busbar 2 energized 08:07 Hrs, Unit No. | taken off the Busbar 08:52 Hrs. Unit No. 2 started and synchronized with the system 08:55 Hrs. 220kV Burhan 3 energized (09:00 His. 220kV Cet. Burhan 2 energized 09:10 Hrs, severe frequency swing (45Hz— 55 Hz) observed in the system and the energized circuits i.e. Burhan 2-3, ISPR tripped on loss of voltage 09:16 Hrs, Unit No 3 started and 220kV busbar 1 & 2 energized 09:40 Hrs. 220/500kV Auto T/F 2 energized 09:55 His. 220/500kV Auto T/F 1 energized 10:24 Hrs. serious frequency swing observed (48Hz — 55 Hz) in the system and resultantly tripping of units occurred, whereas the tripping of 220kV Cets, occurred on loss of voltage 10:36 Hrs, Busbar 1 & 2 energized by Unit No. 3 11:38 His. frequency swing observed (45Hz — 51 Hz) in the system and resultantly tripping of units occurred, whereas the tripping of 220kV Cets. occurred on loss of voltage 11:49 Hrs. 220KV Busbar 1 energized and later 220kV Cets. & auto T/Fs energized 12:54 His, frequency swing observed (46H1z ~ 54 Hz) in the system and resultantly tripping of units occurred, whereas the tripping of 220kV Cets. occurred on loss of voltage "12:59 Hrs. Unit No, 6 started and auto T/Fs and 220kV/SOOKV Busbar energized. Later-on, other Units and 220kV Tilines energized 7 Ye 3 ra oat" 14:26 Hrs. frequency swing observed (47Hz— 54 Hz) in the system and resultantly tripping of units occurred, whereas the tripping of 220KV Ccts. occurred on loss of voltage 14:38 Hrs. Busbar 2 energized and subsequently 220kV Cets also energized 15:22 His. frequency swing observed (45Hz — $5 Hz) in the system and resultantly tripping of units occurred, whereas the tripping of 220kV Cots. occurred on loss of voltage Hrs, Busbar energized and subsequently 220kV Cets also energized 15:45 Hrs. frequency swing observed (45Hz — 55 Hz) in the system and resultantly tripping. of units occurred, whereas the tripping of 220kV Cets. occurred on loss of voltage . ‘Hrs. Busbar energized and subsequently 220kV Cets also energized Hrs. frequency swing observed (48Hz — 55 Hz) in the system and resultantly tripping of units occurred, whereas the tripping of 220kV Ccts. occurred on loss of volt ela Uni opera mode ‘18:54 Hrs. Busbar | was energized after receiving voltage from far end via 220kV ISPR ‘T/Line. Subsequently, other units of Tarbela, 220/S00kV Auto T/Fs, and 220/S00KV T/Line were energized. "22:33 His, Busbar 2 was also energized forward by energization of remaining 2 units 24/01/2023: 00:05 Hrs. Unit No, 14 started & synchronized with system 00:16 Hrs. 220/S00kV Auto T/F 3 energized 00:23 Hrs, remaining ‘T/Lines energized 7.14 Mangla Power Station (Annex-27) Brief of the Mangla Power Station data is as under; ‘Installed capacity = 1100 MW. + Pre Fault Conditions: Zero Irrigation Indent, No power generation as units were standby/shutdown and on maintenance etc. 220kV Busbar 1 was under shutdown for shifting of its protections 23/01/2023: * 07:35 system blackout occurred * 10:10 Hrs. Unit No. | started and switched over to 132kV Busbar 1 + 10:30 Hrs. Unit No. 2 started and kept running on SNL/FSNL. Unit stopped due to oil leakage in inlet valve pressure system 11:27 Hrs. after receiving voltages from far end 220kV Mangla New Rawat 2 energized 11:55 Unit 2 started on FSNL 12:45 Unit No. 2 switched on SNL mode 12:56 220kV Cet. Mangla, New Rawat 1 & 2 tripped due to hunting in voltages and frequency 12:58 Hrs. Unit No. 2 connected to Busbar 2 ‘+ 13:17 Hrs. severe hunting/ fluctuation observed in voltages (1 60KV to 2S0kV) and frequency (47hz to $3h2) causing abnormal sound from the P.T of Unit No. 2 13:51 Hrs. severe hunting/ fluctuation observed in frequency (46hz to 54 hz) 14:25 Hrs. severe hunting/fluctuation observed in frequency (47hz to SShz) causing abnormal sound from the P.T of Unit No. 10"15:47 Hrs. due to severe fluctuation in frequency (45.Shz to 62hz) causing abnormal sound from the P.T of Unit No. 9810 and both units tripped along with tripping of 220kV Cet ‘Mangla New Rawat 2 * 16:24 Hrs. due to severe hunting/fluctuation in frequency (46hz:to 58hz) causing abnormal. sound from the P.T of Unit No. 2 and Unit tripped along with tripping of 132kV Cot Rajjar and 220kV Cet Mangla Gujrat 1 ‘17:30 Hrs. Severe hunting / fluctuation observed in frequency (49.04Hz to 53.37Hz) causing abnormal sound from the P.T of Unit No. 2,9 & 10. = 22:47 His, Unit No. | taken off the bar from Island Mode. Afterwards, 132kV CB closed and resultantly station supply shifted to the National Grid from radial mode 7.18 Ghazi Barotha Power Station Brief of the Ghazi Barotha Power Station data is as under (Annex-28); Installed capacity = 1400 MW. Pre Fault Conditions at 07:33 Hrs: Generation (Unit 2) = 250 MW. All 500/220kV energized except S00KV Barotha — Tarbela 2. Voltage Busbar 1 & 2 = 526kV 23/01/2023, "07:34 Hrs. Fault Occurrence power system collapse / de-energization of HV & MV system at Ghazi Barotha Power stations and tripping of the Unit 2 + Note: An abrupt system frequency dip and failure of system voltages caused tripping and no fault at Ghazi Barotha 7.15.1 Restoration by Ghazi Barotha Power Station "07:44 Hrs. January 23, 2023 Restoration Started "06:14 Hrs. January 24, 2023 : Complete Restoration yu 2 Vy \ Page 13 of 347.16 —NTDC Head Office, Lahore (Annex-29) Event description provided by NTDC Head Office, Lahore is reproduced below; ‘Sr.No. Time Event Description 1. | Priori 07301, | HVDC Flow =2400 MW HVAC Flow = 1730 MW 2 [0730 Tin, Curtailment of wind generation lifted. Congestion on HVAC network HVDC Flow = 2400 MW ee eel HVAC Flow = 2252 MW 3. [0733 Tis, System Frequency = 50.73 Hz. Generator at Ghaai Barotha Power Plant ordered to shutdown o bring the system frequency within nominal range. &__[ 073230 His, | Oscillations started inthe network parameters 5. | 073406 Tis, | Oscillation recorded in the Winity oF 500 RV Guddu GS. Voltage on 500 KV system swinging between 276 kV (Ph-Ph) to S10KV Phy } Curent on 500 kV T/Line swinging between 495 A to 1594 A 6. (07:34:14,895 Hirs., | Commutation falure at Lahore Converter Station. Voltages on 00 KV AC Bus ~ 410 kV (value provided by M/s, PMLTC), T_[ 73414 908i, [a S00KV Gaddis 747 — Rabi Yar Khan tipped. GISEVLTSEAY b. 500 kV DG Khan ~ Gud tripped, (99.6 kV / 1.74 kA) ©. 500 kV Gueddu ~ Muzafforgach tripped. (62 kV /2.02 kA) O7341S His, | a Vollage and Frequency overshoot in southem system b, 07:34:15.417 Hrs., 500 kV NKI~K2/K3 & NKI—Jamshoro T/Lines tripped on over voltage (V > 600 kV). © © 07:34:15,500 Hirs,, Frequency controller of HVDC activated. Power flow on HVDC T/Line momentarily increased to 3400 MW 4d. 07:34:15,722 His, K2/K3 tripped on over frequency (f> 51.5 Hz) Rest of operating power plants in southern region tripped on under frequency. 073429987 is., ‘Loss of AC voltage at Matiari AC busbar causing HVDC bi-pole block leading to system collapse. rH Bed Page 14 of 34Sr] Name ofCkt./ | Grid Relay No.| Power Plant | Station/ena | TYPE°fRelay | indications a Zone I Trip, | Voltage recorded at time of Tripping: ~ Phase A,B,C | s35ey ne 500 kV Guddu ; GE D60 Power Swing | 1. | 747—Rahim Yar | Guddu 747 ‘ ‘urrent recorded at oe Distance Relay | detected | sen of Tripping: = LI5KA Voltage recorded at Schneider | 706179 | cine of Thpping: = 2, | 500KV DG Khan | 6 i¢hon | Electric P43 | Phase A,B,C | 99.6KV | © —Guddu O1d ox Distance Relay | Power Swing cect et | 174 ka, Voltage recorded at . time of Tripping: = 500 kV ArevaP443 | Zone1 Trip | 62kV 3. | M — | Muzatfargarh Guddu O1€ Distance Relay | Phase.A,B,C | Current recorded at time of Tripping: = 2.02 kA S00 RV Gadd * | 747 -Guddu O14 oe) 300K Muzaffargarh — No Trip DG Khan WoL ae Page 15 of 34TAT Pre-Event Power System Conditions 7.17.1 During the morning time, power system demand starts increasing owing to offices, school and other business activities as per usual trend. DISCOs started load management from 07:00 Hrs in groups with difference of 15 minutes. In this way, load management in FESCO and MEPCO was started at 07:30 hours. Usually, the wind generation remains restricted during the night due to less downwards regulation margin on thermal plants because of minimum system demand during night hours, which is witnessed since many years back. Resulting in higher SOOKV system voltage at various locations within the transmission network due to which some Tilines are switched offopened to manage the system stability and equipment safety. 7.17.2, Before four minutes of the collapse, the system generation was approximately 11,683 MW at 07:30 hours, As the cheaper generation resources of indigenous coal and nuclear are located in the most Southern part of the country therefore as per the Economic Merit Order (EMO) the generation in Southern region was evacuated to Northern region through HVAC. and HVDC parallel T/Line corridors. The status of power generation and its flow on 23° January 2023 @ 0725 Hrs., & 07:34 Hs. region wise, before event, as provided by System ‘Operator (NPCC) is given here below: ‘ ‘© Generation Status at 23" January 2023 @ 0725 Hrs. is as under; 0 ‘* Generation Status at 23% January 2023 @ 0734 Hrs. is as under; South a Cesta [es] ‘SalasPower | el! = | 7.17.3 The consolidated status of region wise generation along with plants names on 23" January 2023 is as under: yy UL zea \ Page 16 of 34South Region (MW) North Region (MW) @ @ 0734 @ @ 0734 0730Hrs | Hrs 7308s | Hrs Gadd 5-13 145 145 | Tarbela 754 745 UCH-I 372 372 | Ghazi Barotha 380. 250 Engro 145 145 | Karot 95 95. Foundation 178 178 | Neelum Jhelum 0 0 UCH-I 188 188 | Other Hycel PP BI 231 Port Qasim 21 (621 [AGL 94 ia Engro Thar 150 150 | Atlas 0 19 Lucky 606 (606 | Liberty Tech 160 177 ‘Thar Ei 150 150 | Sahiwal Coal 621 621 Shangai 72301230 [HBS 1190 [1190 KANUPP-IL 1040 [~ 1040 | Balloki 1183 [1183 KANUPP-IIT 870 ‘900 | Chashnupp-T 310 310 Total Bagasse 136 136 | Chashnupp-It 316 316 Total Renewable 0 591 | Chashnupp-IT 318 318 (Chashnupp-IV 0 0 Total 3831 | 6452 3852 [5569 ‘Total (North + South) @ 07:30 | 11,683 | Total (North + South) @07:34 | 12,021 7A7A The status of 500KV transmission circuits, connecting South and North zones, on 23% January 2023 @ 0700 Hrs, as ise “rego er (ors 0 Trippi ing Sequence =) OC lcd & Pats Tipped (0734298675) provided by System Operator (NPC), is given as below: =F (ree et Page 17 of 34Part 3 Event Occurrence Findings Conclusion8, Event Occurrence: 8.1 Wind generation curtailment lifted at 07:30 Hrs. and as a result about 500 MW generation injected in the system, which caused the overloading of Transmission Lines as well as low voltage under steady state conditions in view of long AC Corridor of about 1100 KM, from Thar in South to Haveli Bahadur Shah in North with insufficient VAR generation, as the Guddu 747 Plant was under forced shutdown, This reduced the T/Line capacity in light of Surge Impedance Level (SIL). 8.2 Consequently, apart from rising trend in system frequency due to increase in wind generation. the oscillations on the system also started, At the same time the Generation in Northern region was reduced by shutdown of the one unit and ramp down the generation of other unit of Ghazi Brotha at about 07:32:22,187 Hrs. This action aggravated the situation as system was already weak and under stress due to high AC flow and low voltage/less VAR compensation in middle. 8.3 Thus, the ation of the event started with the depression of voltages as shown in ‘graph of Annex- 30. This increased the oscillations which can also been seen from the graphs of Multan, Lahore & Muzaffargarh attached as Annex-31. Due to these oscillations in voltage and current, the Commutation Failures occurred at 07:34:14:895 Hrs. at voltage 391 kV recorded at Lahore Converter Station, graph attached as Anmex-32, 84 After 13 mili-seconds of activation of the Commutation Failure, the DC power of about 304 MW, was also shifted on AC corridor. Consequently the power swing was generated on the system. However out of step protection scheme as per attached Annex-33 for splitting the system in different Islands, did not operate, Resultantly all three HVAC circuits as mentioned in Annex 33 (i) tripped ‘on Power Swing (unstable) and HVAC system split into two regions at 07:34:14:908 Hrs. but both regions still remained connected through HVDC system, 8.5 In view of the load flow, where about 4130-4652 MW was being exported from South to North, the splitting of the HVAC network into North and South sub-systems caused frequency rise in the Southern region up-to 51.525 Hz having excess generation and depressed frequency/voltage in Northern region having excess load. 8.6 Meanwhile the Frequency Limit Controller (FLC) at Converter Grid Station initiated but before completion of its cycle, Over Voltage protection at NKI operated and tripped SOOKV NKI-Jamshoro and NKI-K2/K3 circuits from both ends at 07:34:15:250 hours The tripping of above-mentioned cireuits isolated NKI from 500 KV National Grid. Thus the export of $21 MW to K. Electric became zero and K.E went into the Island status. 8.7 Although the consequences would not have been avoided, the rejection of load of $21 MW pushed the frequency rise trend in the South region. Looking into further, it was noted that the time delay setting of over voltage low set element at NKI end for Jamshoro and NKI-K2/K3 circuits was very low i.e; 100 mili-second. After pointing out the same by committee, the above setting was revised to 8 seconds and 11 seconds on Jamshoro and K2/K3 circuits respectively and for High set elements of both the circuits from 0.00 to 0.10 see on 24-02-2023 (Annex-34). \w/ We RM Nm8.8 At 07:34:15:408, FLC of HVDC activated and provided a support of additional 1000 MW to North to balance the frequency in both regions. Prior to that, another catastrophic factor was that Port Qasim Generating Unit #2, in operation, started Run Back instead of tripping/isolation from National grid at over frequency setting, implemented as 51.4 Hz (instantaneous) as per the agreed setting, attached as Annex-35, provided by the plant Management, This resulted in sustained over frequeney which caused tripping of K-2 (1040 MW) and K-3 (900 MW) at 07:34:15:935 at agreed Trip setting of 50.5Hz with time delay of 0.25 sec. With the tripping of 1940 MW of K2. K3 plants. system experienced severe dip in frequency. ‘8&9 Although about 426MW was rejected through Under Frequency Load Shedding scheme in South region, but being insufficient, caused HVDC system to block at 07:34:29:987 Hrs. Resultantly cascaded outages of power plants started in south i.e.; Lucky Power (606 MW), Engro Thar (150 MW), Shanghai Electric (1230 MW), and Thar Energy Limited (151 MW) on under-frequency protection, and thus the collapse of NTDC Southern Island. With this the export to KE from NTDC 220KV network (Jhampir Grid) became zero at 07:34:43:800 Hrs. 8.10 In mean time the North region also collapsed inspite of load rejection of 3834 MW through Automatic Load Shedding Schemes i.e, Under Frequency, Under Voltage and Rate of Change of Frequency (ROCOF). 8.11 With regards to KE, prior to the event, it was running synchronized with NTDC. Major portion ofits, network, including all its generation, was connected with NKI and the other portion with 220K V NTDC Jhampir-ll grid station. KE total load was 1246MW, out of which 708 MW import from ‘NTDC (NKI-521MW and Jhimpir II-187MW), and the remaining 538 MW from its own generation (BQPS I11-498MW and SNPC-40MW). 8.12 Upon isolation from NKI at 07:34:15:250, it faced deficiency of 521MW, however, the NKI-KE Cross Trip Scheme operated which rejected the load of 283MW from KE network. Since the Power deficit was still there, the under frequency scheme operated through which 341MW was rejected ‘Thus a total of 624 MW was rejected against shor fall of S21 MW so KE system should have sustained but Unit #10 (249MW) of BQPS-IM tripped on ‘Combustion Chamber Acceleration’, which does not seem to be justified’. This caused the other Unit #20 (239MW) of BQPS III and SNPC (40MW) to trip on over loading, 8.13 At 07:34:43:800, upon collapse of NTDC network, export from 220K V Jhimpir-II became zero. Thus KE system completely collapsed. Wy ye OR A Page 19 of 349, Restoration of Power System 9.1 Timely and reliable Restoration of Power is an essential task after the blackout. The prompt initiation of the restoration process was critical to minimize the impact of the power outage. In this regard availability of Black Start facility at multiple geographical locations (load centers) of the network is essential for speedy restoration of the power system. As per the details provided by the System Operator NPC, first unit at Tarbela was synchronized at 08:10 Hrs. on 23.01.2023, However, record from NPCC and WAPDA shows that restoration attempts failed multiple times from Tarbela & Mangla Power houses contributing towards delay in system restoration of North Region, whereas Uch-1 Power Plant, being with Black Start facility, was started at 09:39 Hrs. on 23.01.2023 through which partial areas of QESCO, SEPCO, HESCO and MEPCO were restored. Moreover, auxiliary supply to most of the power plants in southern region was also provided. 9.1.1 In the aftermath of the event, restoration activities were started. Power Plants having black start facility were instructed for starting of generating units for restoration of system. 9.1.2. The Black-Start initiated at the following power houses: + Tarbela Power Station + Mangla Power Station + Warsak Power Station + UCH-1 Power Plant 9.1.3 KAPCO was also instructed to initiate Black Start Process on direction of Honorable Minister of Power (MoE), despite expiry of its PPA, since 24-10-2022, but it was not successful due to technical fault, 9.2, Restoration in Northern Region: 9.2.1 Black Start from Tarbela: 9.2.1.1 Black start facility from Tarbela power station was availed and generating unit # 3 synchronized at 08:10 Hrs. and power supply to some of the grid stations in the Jurisdiction of ESCO i.e., Islamabad and Rawalpindi was restored and generation was increased gradually. However, Tarbela generating units remained unstable as reported by NPCC, resulting in large frequency swings ranging from 45-55 Hz leading to tripping of generators at 0910 Hrs. 9.2.1.2 After that, nine (09) similar startups were initiated from Tarbela with different combinations of transmission lines and different area loads, but all attempts failed and the island could not expand and could not synch with Mangla. As per NPCC the DG sets of AGL could not synch due to unstable behavior of high fluctuating Tarbela generating units i We Bw Page 20 of 39.2.2 9.2.3 9.2.4 9.2.1.3 Tarbela power house stance however was that the variation in load causes much stress and variatior/fluctuation in frequency and eventually caused tripping of machines. Black Start from WARSAK Power Station: Black start facility was initiated from Warsak Power Station as well and supply was restored to some of the grid stations in the PESCO area. Black Start from Mangla Power Station: Mangla power station was also instructed for the black start, however, the power station ‘could not start the units due to unavailability of IRSA indent (Indent = 0 cusees) as per the report of Mangla power station. After getting clearance from IRSA for water discharge, the power station initiated the black start activities and synchronized Generating Unit # | on 132 KV Bus Bar and restored its auxiliary supply with 132 KV Mirpur circuit at 10:10 Hrs. ‘Mangla-Tarbela Coupling attempts: 9.2.4.1 Supply extended from Tarbela to Mangla power stations at 1123 Hrs. and instructed to close interconnector Transformer breaker and synch generating units to couple it with Tarbela, Mangla Unit #2 was on FSNL (Full Speed No Load), however, Mangla was unable to synchronize with the system, Meanwhile Tarbela again tripped at 1138 Hrs, 9.2.4.2. Mangla synchronized its unit #2 with 132kV Busbar at 1210 Hirs. The 220/132kV Interconnector Transformer was energized by Mangla at 1219 Hrs, Thereon, Black Start facility was again availed from Tarbela power station and supply was extended to 220kV Sangjani (ISPR). Supply from Mangla also extended to 220kV Snagjani (ISPR) and attempted to couple both power houses at 220kV ISPR. 220kV breaker controlling 220kV Rawat-Mangla circuit tripped from Mangla end at 1340 Hrs, 9.2.4.3 Supply extended from Tarbela to Mangla power station. However, while attempting to cover Mangla, Unit #9, 10 along with 220kV, Mangla-Rawat circuit- I tripped at 1547 Hrs, 9.2.4.4 Supply extended from Mangla to Tarbela, Tarbela units got synch one by one and after the successful coupling at 1852 Hrs. on 23.01.2023 the restoration was expanded to the rest of network. Ve We zry\ Page 21 of 349.3, Restoration in Southern Region: 93.1 Black Start from UCH-I Power Pla In southern region, Black start capability of UCH-1 power complex was availed after the system collapse. The power plant started its first generating unit at 0939 Hrs. and supply t0 220 KV Sibbi Grid station was restored through 220 kV UCH ~ Sibbi circuit. The ‘generation gradually increased in the island by extending its supply to nearby power plants ie., UCH-I, Foundation, Engro Power and TPS Guddu. As the generation inreased, supply was extended to a number of grid stations in the jurisdiction of SEPCO, QESCO, HESCO and MEPCO, SUMMARY OF RESTORATION FROM DIFFERENT SOURCES Sr | Restoration | Power House No. | Attempt | Warsak (HRs) | Tarbela Mangia (HRs) | UCH-I (HRs) (HRs) T | First O813His- ‘o910ris | TO10Hrs- 0939Firs- 1024Hrs * | 1140Firs Onwards | 2 | Second | 1035-1252 | 1024 1210-1245 | = 3 | Third 1302-1502 [1138 1258-1624 4 _| Fourth 1533-Onwards [1254 1629-Onwards S| ith = 1426 = : 6 _| Sixth : 1519 5 5 7 | Seventh [= 1529 5 = I 8 | Bighth [= 1545 = oilney 9 [Ninth = 1846 = 9.4 Delay in Restoration ~ Challenges and Deficiencies 9.4.1 The restoration process took almost 20 hours which is more than expected. This was due 10 different factors which are discussed below: 9.4.1.1 One of the major causes of delay in restoration process was the unavailability of black start facility in the central region of the network which is crucial for restoration. However, in the northem region of the country, the black start fac ‘was available with Tarbela, Mangla, and Warsak Power Stations. In the southern region, the only black start facility at UCH-I power plant, was utilized. 9.4.1.2 The restoration process of the power system was also impacted by the lack of mock testing ofthe black start facility, where ever available, The mock testing isa crucial step for the preparation of NPCC and all relevant stakeholders to handle a blackout scenario, as it ensure the healthiness of Black Start facility. 9.4.1.3 As obvious from the summary, UCH-I power plant remained stable in island mode afier its starting but Tarbela did not, although hydel power stations have much higher technical capability to absorb jerks than a thermal power plant. Further SME BN Yengoing into details it has been learnt that UCH-I power plant operated on speed control mode while running in island mode whereas Mangla and Tarbela were operating in power control mode. 9.4.1.4 The lack of SCADA and remote operations has significant impact on the restoration process of the power system. SCADA is an essential tool that allows the system operator to monitor all system parameter which helps system operator in expediting the action. Equally important is the sequence of events, specially related to equipment, of complete network, which presently are not available, and can be recorded at NPC only through SCADA. 9.5. Restoration in K Electri 9.5.1 K-Blectric started its restoration at 0832 Hrs through Tapal, BQPS II and KCCP simultaneously through Black Start in island on 23.01.2023. At 1000 Hrs restoration from Gul Ahmed Power Plant was also started. However, all Power Plants with Black Start facility could not sustain in island modes and tripped multiple times. 9.5.2 Last successful restoration was started at 2107 Hrs from Tapal and BQPS II GT-4 at 2153 Hrs in island modes. K Electric received power from NTDC at 0322 Hrs dated 24.01.2013 through Jhimpir Grid, Restoration completed at 05:08 Hrs. on 24.01.2023 Hrs. after synchronization of all Power sources, 10, Analysis: 10.1 In light of the events narrated above, Inquiry Committee is of the opinion that following faciors contributed towards black out. While issuing the operational instructions by NPCC to Generating, plants due consideration should have been paid to system parameters, such as Out of Service and length of SOOKV lines, required VARs and locations of Generating Plants vs. Load, elaborated as under. 10.2 Although the addition of Wind generation was not required, however, if added then prior to it firing angle at HVDC should have been adjusted for additional transmission to avoid any effect on HVAC transmission lines. 10,3 De-synchronization of Ghazi Barotha Unit # 5, which is not only located in load area, supports the system VARs requirement also, should have been avoided as the water reservoir level. being 334.2 meters at that time, was very much appropriate to keep the plant running. 10.4 Guddu 747 power plant, contribution of which towards balancing the system parameters plays an important role in view of its geographical location, was under forced outage on account of non- insurance despite its availability. 10.5 Ramping down of the only running Unit #2 of Port Qasim instead of tripping on Over Frequency \“/ NL ao i eee10.6 Out of Step Devices installed in the system did not operate, which could have split the network in different stable Islands. 10.7 Although not so much contributing, the operation of Over Voltage Protection at NK, due to which 521MW load of KE was cut-off, pushed up the over frequency trend, 10.8 KE should have survived after isolation from NK, as proportionate load was rejected through Cross Trip and Under Frequency schemes but due to inappropriate behavior and tripping of one of the two running machines at BQPS-III, KE system also collapsed, 10.9 Delay in restoration of the NTDC system was mainly due to failed attempts of black start facilities at WAPDA hydel stations and KAPCO, Other factors include delay in synchronization of various power plants and non-availability of SCADA system in NPCC. Whereas KE system was also delayed due to repeated failed attempts on black start. 10.10 The only successful restoration on first attempt was through UCH I Power Plant black start facility 11, Implementation status of the findings/recommendaticns of inquiry reports furnished thereof for such break downs in the past. M4 Implementation status of Recommendation of Inquiry Committee of NTDC i.r.0 Partial Blackout of October 13, 2022 ‘Sr# Actions proposed Implementation Status | 8.1 | During planning and design phases, condition on aged | __ Under implementation transmission lines shall be assessed while | interconnecting with new power plants %2 | During execution stage, quality of _the| _ Underimplementation | material/workmanship shall be. ensured as per | International Standards 83 [O&M activities of transmission lines including Implemented thermography needs to be carried out as per SOP with availble resources, 84 | Ongoing projects of permanent netowrk for evacuation | Under implementation of power in general and in particular from K2/K3 shall be 85 __| Implementaion of SCADAAII shall be ensured as per | __ Under implemented timelines to enhance the live visi parameters to System Operator 86 | Phasor Measurement Units is required to be installed to | Under implemented timely indicate the oscillations in the system 8&7 | Stability Control System shall be activated at the| Partially Implemented earliest to enhanvce the system stablility BB_| For more effective maintenance of T/Line, a yardstick | Partially Implemented for creation of new T/Line Divisions/ Sub-Divisions be developed so that with the addition of new T/Line, the requisite posts be fot sanctioned aceordin 1 yy WR ONS senad lity of system89 ‘TiLine maintenan staff shall be equipped with proper PPEs, T&P, spare material/hardware and vehicles to ensure healthiness of NTDC’s T/Lines as per NTDC ‘mainteancne SOPs Partially Implemented B10 Condition assessment of the section of old 500 KV ‘TiLines near coastal/polluted areas should be carried ‘out on war footing basis and remedial measures may be taken accordinlgy Under implemented Br Based on the visial inspection and documentary evidences, the Enquiry Committee is of the opinion that if deemed necessary, disciplinary action under the relevant rules may be intiated against the technical teams from project dielivery south involded in execution of interim armagment and asst management south for lack of O&M activities as per SOP keeing in view the crticitcality of said arrangment, Partially Implemented 2 Implementation status of Recommendation of Inquiry Committee of NEPRA i.t.0 Total Power System Collapse dated January 09, 2021 (Pertaining to NTDC/NPCC) Sr. ‘Actions proposed Implementation Status 64.1 ‘As regard to Power Swing phenomenon, a study should be arranged from a tional firm to suggest Out Of Step (Power Swing Blocking (PSB) and Out of Step Tripping (OST) in order to avoid major power breakdowns and blackouts Partially Implemented 642 NTDC, KE and Power Plants shall prepare contingency plans and establish procedures for restoration of supply in case of emergency, partial and complete power failure Implemented 643 ‘A Centralized Event Recorder having resolution of milli seconds (ms), is essentially required for proper analysis of events and suggestion of remedial measures ‘Not Implemented 644 ‘No operation of Power Houses and Grid stations be carried out without the knowledge and directions of NPCC Implemented 645 ‘A function should be developed at the power houses with capacity of 200 MW and above so that instead of tripping of all machines, at least one of that shall remain available in Island Mode during under frequency and shall be in a position to energize the bus Not Implemented \/ me OO wobar to facilitate the startup of other machines and speedy restoration of the network 646 The black start facility shall be made available at the | Partially Implemented power plants suggested/selected by NPC especially | for those located in South and Central zones 647 Tn case of under frequency situation, at least one Not implemented machine in power plants shall be made available in island mode in order to facilitate for early restoration of supply. In this regard, necessary amendments in any applicable document i.e, PPA, Tariff Determination, License or Grid Code etc, be made if required 648 Tripping of generating machine on over Implemented frequency/over voltage/over speed shall be managed through settings for their sequential tripping instead of their simultaneous outage 649 The quantum of load to be rejected throught under | Partially Implemented frequency relays along with time and setting coordination between those of transmission lines and machines needs to be reviewed for making those | efficient and effective. The feeders which were supposed to but did not trip on under frequency shall | be checked and kept healthy 64.10 The power plants which were not available for Implemented synchronization despite the notice of NPC shall be dealt with under PPA yo ®t Page 26 of 343 Implementation status of Recommendation of Inquiry Committee of NEPRA i.t.o Total Power System Collapse dated January 09, 2021 (Pertaining to Guddu) Sr. ‘Actions proposed Implementation Status 64.11 Grid Station Equipment and Protection/Control System of 220kV Switchyard Guddu shall be maintained in accordance with SOPs issued by TSG NIDC/manufacture of the equipment Partially Implemented | 64.12 | Single Line Diagram of 220kV switchyard Guddu and Implemented Mimic diagram of its 220kV control room be updated and made available at site as well as with NTDC. 64.13 | Interlocking scheme of equipment installed at 220KV | Partially Implemented switchyard Guddu shall be checked and ensured for proper working 64.14 Indiscriminate tripping GT 14 at Guddu on Unit Differential Relay checked and set night Implemented 64.15 Routine testing of HV equipment at Guddu switchyard including protective relays shall be carried out regularly : Implemented 64.16 “Technical trainings be arranged for staff and engineers at TPS Guddu for their skill improvement and capacity building Implemented 64.17 The departmental enquiry initiated by Guddu Power Plant management against the delinquent staff regarding the instant breakdown be concluded Implemented | 64.18 Management reforms are required at TPS Guddu for systematic working Implemented vie Ss \ Page 27 of 3414 Implementation status of Recommendation of Previous Inquiry Committees of NTDC/NPCC/Ministry ete i..o Partial / Full Blackouts N Sr. | Title & Brief of Inquiry Report ‘Ketions proposed | Implementation # along with date and year Status 1 ‘System Disturbance in North | During canal closure and foggy | Not Region seasons i.e. December 15 to | Implemented February 15%, Hydel ‘A hearvy jetk was observed on the | Generation is recommended to system duye to damage of isolator | be maintained at 1500 MW | controlting 220kV Muzaffargarh — | (minimum) at all times and at Multan Circuit 2000MW during peak hours 09:21 Hrs on 15.01.2016 2 [System Disturbance in North | Achieving of load shedding Notaaaael Region through under frequency relays | Implemented at 132kV level can be better System disturbance occurred due | managed and controlled in | to tripping of auto transformer T3 | NTDC grid station on radial | along with S00kV Guddu Old — | feeders Guddu 747 circuit, | 09:21 Hrs on 21.01.2016 Report by NTDC 2016 Certain number of 220kV CBs Not installed at Guddu Power | Implemented House was reported last year by NTDC as very old which have completed their life time and need to be replaced 2018 The power plant management Not shall ensure enough supply of | Implemented | MVARs to the network. ‘Adequate supply of MVARS | maintains the system voltage | and hence decreases the | possibilities of voltage collapse | breakdowns. The power plants | ‘operator must follow the NPCC instructions regarding voltage regulation and incase of failure to follow the same, penalty be imposed on the power plant i, Delay in system | restoration has also been caused due to non-availability of | yi ze ¥ ee aes2018 ii. black Start Facility in Power Plants located in | central region. The | Power Purchase | Agreement for the | future plants needs to include the black start | facility for quick | restoration of power | supply to load centers. | Fully functional SCADA system helps in better monitoring & contol of power system and analyzing of realtime disturbances. Unavailability of SCADA system at major stations and power plants has also caused delay in system ‘operation / restoration High voltages in SOOKV network is a) major problem in system restoration and the same can be mitigated by installation of Shunt Reactors on appropriate stations. Study of _islanding schemes needs to be carried out by International Consultants to avoid the system — going towards Out of Step / ut of Synchronism | Not Implemented Page 29 of 3412. FINDINGS: Based on the site visits, examination of the documents and discussion with the concerned, following, are the findings of the Inquiry Committe. 12.1 Complete power blackout occurred at 07:34:43:800 Hrs. on 23.01.2023 and full. restoration of NTDC system was achieved at 0322 Hrs. on 24.01.2023. 12.2 Failure of supply in the KE system happened at 07:34:43:800 Hrs. on 23.01.2023 and complete restoration was done at 0508 Hrs. on 24.01.2023. 12.3 The sequence of events which took place on during the occurrence of blackout are described below: ii, iii, iv. vi vill xi Generation of power from Wind Power Plants in South was increased whereas hyde! generation was decreased in the North Power supply in HVAC was above the capacity of lines which resulted in Oscillations. Reduced line voltage at North caused Commutation Failure which temporarily decreased power flow from HVDC Lahore converter station. Some power of HVDC was diverted to HVAC which generated Power Swing, Out of step protection schemes did not operate, hence North and South zones split on HVAC system. However, both zones remained connected through HVDC. In South zone, over frequency was observed while in North zone, under frequency was experienced. Due to difference in frequencies of the HVAC at Matiari and Lahore Converter Stations Frequency Limitation Controller (FLC) was activated. Prior to completion of FLC process, 500kV NKI- Jamshoro and 500kV NKI-K2/K3 tripped on over voltage which ceased export of 521 MW from National Grid to KE Grid. This resulted under frequency in KE Grid and over frequency in the National Grid. Instead of tripping, Port Qasim Power Plant went into Run Back mode at the set over frequency settings which maintained the system over frequency and resultantly K2/K3 Power Plants generating 194© MW tripped. Abrupt loss of power caused severe dip in frequency which caused the blocking of both HVDC poles. Resultantly cascaded outages of other Power Plants in Southern region ive, Lucky, Engro Shanghai & Thar on under frequency occurred. Power Supply to South zone including KE was suspended from Grid Network resulting in complete blackout in the South Region. In the meanwhile the North region also collapsed inspite of load rejection of 3834 MW through Automatic Load Shedding Schemes i.e. Under Frequency, Under Voltage anni i Vy of mw Page 30 of 34xiii, xiv. xv. Power Supply from NTDC network Grid Stations to Karachi network was completely suspended due to tripping of Jhimpir Il - KDA circuits. Despite suspension of Power Supply from National Grid, KE could have avoided blackout, had BOPS Power Station survived as sufficient load was rejected via Cross Trip and under frequency schemes. Due to non-availability of central event recorder great difficulty was faced in assessment of accurate sequence of events. 12.4 Following are contributing factors for cascaded tripping: Injection of wind power generation in South and curtailment of hydel power generation in Ghazi Barotha Power Station at North by NPC. Running back of Port Qasim Power Plant instead of shutdown at set over-frequenc) setting. ‘ 12.5 Restoration: 12 2 2 5.1 National Grid system was completely restored at 0322 Hrs. on 24.01.2023 5.2 KE Grid was restored on at 0508 Hrs. on 24.01.2023. 5.3. Factors for delayed restoration are enumerated below: i, Frequent tripping of Tarbela units while energization/loading of circuits, fi, Delayed synchronization of Mangla Power Station with bus bar due to internal fault. iii, Late synchronization of Tarbela and Mangla Power Stations by System Operator. iv, Delayed availability of following Power Plants despite Notice to Sync (NTS) by System Operator: Nandipur, Sahiwal Coal, Guddu, China Hub, HUBCO, Halmore. Rousch Power Plants took long time to syne inspite of NTS by NPCC. Detaily are attached as Annex 36, yb %*® Page 31 of 3413. Conelusion: 13.1 Causes and Faults which resulted in cascade tripping and system collapse Induction of power generation at South and reduetion of generation at North caused oscillations in the system which gave rise to power swing. Power swing could not be controlled due to non- ‘operation of out of step protection devices. Hence in the South initially over frequency and later under frequency, whereas only under frequency occurrence in the North caused in the cascaded ‘rippings of lines and power houses which resulted in total power blackout/breakdown in the ‘country. 13.2 Respor lity for the country wide blackout Following entities are jointly responsible for the event: 13.2.1 NTDC: 13.2.1.1 For issuing dispatch instructions without due consideration of prevailing network conditions. 13.2.1.2 For inconsistent protection settings at NKI Grid. 13.2.1.3 For non-activation of Out of Step Splitting System 13.2.2. Central Power Generation Limited Company (CPGCL) GENCO-lI, TPS Guddu for non-availability of its 747 Guddu in the system on account of non-insurance. 13.2.3 Port-Qasim Electric Power Company Pvt. Ltd. Power plant for failure to trip at the set over-frequency values. 13.2.4 K-Blectric for failure to remain in island mode due to tripping of Bin Qasim Power Station - IIT (BQPS-II]) 13.3. Restoration of Power Supply within reasonable time Delay in restoration of the NTDC system was mainly due to failed attempts of black start facilities at WAPDA Hydel Power Stations and KAPCO. Other factors include delay in synchronization of various power plants and non-availability of SCADA system in NPC. Whereas, KE system was also delayed due to repeated failed attempts on their black start facilities. 13.4 Status of previous inquiry report findings/recommendations Recommendation given in the Inquiry Report of NEPRA regarding Total Power Collapse occurred on January 09, 2021 & NTDC Inquiry Report dated October 13, 2022 regarding Partial Shutdown have been partially implemented. 13.5 Remedial measures/recommendations: 13.5.1 VAR compensations study shall be carried out and required measures in light of study shall be taken to avoid Power Swing. wil B® yes13.5.2. The stability of HVDC system needs to be ensured through proper study to avoid frequent signals of ‘Commutation Failure’. 13.5.3, NTDC staff needs to be trained for thorough understanding of HVDC transmission system, 13.5.4. Fully functional SCADA facility for complete system related to NTDC, GENCOS, IPPs and DISCOs is essentially required for system operator in order to ensure the ‘on line monitoring of system parameters in all respect for sake of system security, stability and analysis of events through GPS synched time event recorder. 13.5.8. Implementation of protection settings issued by NTDC for power plants, mutually agreed, shall be ensured under witness of NTDC through CPPA-G, 13.5.6 NTDC shall ensure the healthiness and operation of recently installed Out Of Step devices, as the same did not operate during the event. 13.8.7 Quantum of load to be rejected through under frequency schemes need to be increased, especially in South region, where itis considerably less, 13.5.8 Simultaneous tripping of K2/K3 Units on Over Frequency shall be made stage wise, 13.5.9 SECL, Thal Nova and Thar Energy Power Plants are not included in the Over Frequency tripping scheme, hence shall be considered. 13.5.10 SOP to be developed between all stake holders to maintain required water indent for relevant Hydel Power Station under acute emergency situations. 13.5.11 KE shall set right the cause of undesired tripping of Unit # 10 at BQPS-IIL 13.8.12 KE shall also set right their Black Start facilities as multiple attempts were made during restoration and as such not satisfactory. 13.8.13 Periodical testing of Black Start facilities shall be carried out to ensure their satisfactory performance at the time of need. 13.5.14 Black start facilities are required at some additional locations, better study based near Lahore and Hyderabad to help early restoration during Black Out. 13.5.15 Repeated tripping of Machines at Tarbela and Mangla during Black Start need to be checked and set right, as successful restorations have been carried out through these stations in the past. \ Page 33 of 3413.5.16 To ensure the availability of synchronic relay at major SOOKV & 220kV Grid Stations to couple/synchronize the islands 13.8.17 To ensure the healthiness of Power System Stabilizers (PSS) at power plants to damp intra-area oscillations 13.5.18 To ensure the installation of modern technologies device such as Wide Area Management (WAM) including Phasor Management Units (PMU) to detect pe mitigated by Remedial Action Scheme (RAS) 4 Te Welw Nadir Ali Khoso Gh Syed Safeer Hussain Market Expert (Member) Mark Market Expert (Member) wy aN Za A eZ ; & ‘Manu Ram Syed Aqib Ali Shah . Market Expert (Co-Opt/Member) Dy. Dir. (M&E) NEPRA (Secretary) Imran Kazi Senior Advisor (M&E) NEPRA (Convener) Date: March 14, 2023 Page 34 of 34