SPARK USE CASE- Distributed Reinforcement Learning for Electricity Market Bidding with Spark - SPARK SUMMIT SESSION
6 likes2,346 views
SPARK SUMMIT SESSION - A majority of the electricity in the U.S. is traded in independent system operator (ISO) based wholesale markets. ISO-based markets typically function in a two-step settlement process with day-ahead (DA) financial settlements followed by physical real-time (spot) market settlements for electricity. In this work, we focus on obtaining equilibrium bidding strategies for electricity generators in DA markets. Electricity prices in DA markets are determined by the ISO, which matches competing supply offers from power generators with demand bids from load serving entities. Since there are multiple generators competing with one another to supply power, this can be modeled as a competitive Markov decision problem, which we solve using a reinforcement learning approach. For power networks of realistic sizes, the state-action space could explode, making the RL procedure computationally intensive. This has motivated us to solve the above problem over Spark. The talk provides the following takeaways: 1. Modeling the day-ahead market as a Markov decision process 2. Code sketches to show the markov decision process solution over Spark and Mahout over Apache Tez 3. Performance results comparing Mahout over Apache Tez and Spark.
Ad
More Related Content
What's hot (20)
Viewers also liked (10)
Ad
Similar to SPARK USE CASE- Distributed Reinforcement Learning for Electricity Market Bidding with Spark - SPARK SUMMIT SESSION (20)
Ad
More from Impetus Technologies (20)
Recently uploaded (20)
SPARK USE CASE- Distributed Reinforcement Learning for Electricity Market Bidding with Spark - SPARK SUMMIT SESSION
- 1. © 2014 Impetus Technologies1 Impetus Technologies Inc. Distributed Reinforcement Learning for Electricity Market Bidding with Spark Vishnu Nanduri, Ph.D. Vijay Srinivas Agneeswaran, Ph.D. 2014 Spark Summit (June 30th-July 2nd), San Francisco, CA
- 2. © 2014 Impetus Technologies2 Speaker Bios Vishnu is a Senior Data Scientist with experience in Reinforcement Learning, Supervised Machine Learning, Stochastic Optimization, and Statistical Analysis. He was a faculty member at the University of Wisconsin- Milwaukee before joining Impetus. Vijay is the Director for BigData Labs at Impetus, with experience in cloud, grid, peer-to-peer computing, and machine learning for Big- Data. He is the author of the recently published book “Big Data Analytics Beyond Hadoop.”
- 3. © 2014 Impetus Technologies3 Agenda What is an electricity market? Why does it need to be modeled? Are there other markets that need/or could use such modeling? Why reinforcement learning? • Background and basics Why Spark? What other options exist? Solution strategy over spark Ongoing and future work 1 2 3 4 5 6 7
- 4. © 2014 Impetus Technologies4 Electricity Markets in North America Source: http://www.caiso.com/about/Pages/OurBusiness/UnderstandingtheISO/Opening-access.aspx
- 5. © 2014 Impetus Technologies5 Prior to Market Restructuring Generators Transmission Distribution Consumers Vertically Integrated Old Structure
- 6. © 2014 Impetus Technologies6 Field Force Use Case – The Challenges ISO balances supply and demand and decides the price- quantity allocations Independent System Operator (ISO) Multiple Competing Suppliers submit Price-Quantity bids Load Serving Entities (LSEs) submit demand bids
- 7. © 2014 Impetus Technologies7 Why does it need to be modeled? To understand long-term behavior of market Equilibrium behavior Assessment of market power Investigate policy prescriptions
- 8. © 2014 Impetus Technologies8 Modeling Approach • Bid each day (each hour) for supplying power the following day • Stochastic processes underlying the day ahead energy market operations are modeled as a Markov chain • Decisions embedded on a Markov chain are modeled as Markov decision processes • Multiple generators competing in a market is modeled as a competitive Markov decision process (a.k.a., stochastic game)
- 9. © 2014 Impetus Technologies9 CMDP Model • Let B = Set of Supply/Load Buses in the Network M = # of loads N = # of suppliers/generators • Xt = (qt, pt), is the system state at any day ‘t’ where, • qt = (q1t, q2t, , q3t, ,……… qt|B|)- load vector • pt = (p1t, p2t, , p3t, ,……… pt|B|)- price vector • Dt = ( Dlt : l Є { 1,2,3,…….N}) where, • Dlt = lth generator’s bid
- 10. © 2014 Impetus Technologies10 CMDP Model • X = {Xt : t Є N}- System State process • D = {Dt : t Є N}- Decision process • The joint process (X, D ) is a Competitive Markov Decision Process • Average reward criterion and non-zero sum nature of the game make solution more challenging • Hence, simulation based optimization called Reinforcement Learning (RL)
- 11. © 2014 Impetus Technologies11 RL Basics • The theory of RL is founded on two important principles: • Bellman’s equation and the theory of stochastic approximation. • Any reinforcement learning model contains four basic elements: System environment (simulation model) Learning agents (market participants) Set of actions for each agent (action spaces) System response (participant rewards)
- 12. © 2014 Impetus Technologies12 Are there other applications? Generation capacity expansion planning Carbon allowance markets
- 13. © 2014 Impetus Technologies13 Reinforcement Learning in Electricity Markets: A Schematic Agent 1 Agent 2 Agent ‘n’ Independent System Operator System State System State System State action action action reward reward reward Update State Update State Update State Learn & Update Knowledge Base Learn & Update Knowledge Base Learn & Update Knowledge Base
- 14. © 2014 Impetus Technologies14 Why Spark? • Hadoop Map-Reduce is not well suited for iterative machine learning algorithms • Twister/HaLoop/Apache Hama Fault-tolerance is questionable, Enterprise readiness is not clear • GraphLab Can only take distributed snapshots – no automatic recovery. • Possible application to the spot market – real-time electricity bidding.
- 15. © 2014 Impetus Technologies15 RL Implementation over Spark
- 16. © 2014 Impetus Technologies16 RL Spark Code
- 17. © 2014 Impetus Technologies17 RL Code xxx xxx xxx
- 18. © 2014 Impetus Technologies18 Work-in-progress • Giraph like APIs for GraphX Allow user to specify master code VS slave code Explicit barrier synchronization • Reinforcement learning over Spark Can be used by independent system operator or by individual generators Can be used in other competitive bidding-based markets such as carbon allowance markets, which is a multi-million dollar industry in U.S. and Europe. • RL over Hadoop/Stratosphere Comparison with Spark/GraphX
- 19. © 2014 Impetus Technologies19 Thank you. Questions??
Editor's Notes
- #3: Just a brief bio about the two of us. I have a research and academic background and so does Vijay.
- #4: In this talk I will briefly cover the following topics. This talk probably has a very different flavor when compared to the others. I will begin talking about what electricity markets are and then discuss the need for them to be modeled. I will briefly highlight Other markets that behave in a similar fashion. I will discuss why reinforcement learning (unsupervised machine learning) Is a viable approach for this challenging problem. Then Vijay will discuss how we intend to exploit the inherent parallelisms within the algorithm and Share a potential solution strategy over spark. WE will also briefly highlight the current state of work and discuss future work.
- #5: Ok..so let’s talk about electricity markets in North America. In north America, in the early 1990s some serious structural reforms were introduced, which forced divestment of centrally owned power generation and encouraged the investment in the electricity industry by individual investors. Currently there exists competition in the wholesale side and in some areas in the retail side as well,. Even though wholesale power trades account for a significant percentage/As you can see here several regions in the US come under several what we call as ISOs or RTOS. I will go over how those work in just a second,
- #7: So let’s talk about what an electricity market is… So we have multiple generators now competing to sell power and multiple large distribution companies or load serving entities competin to buy power. This market is what we are looking to model and understand. And since this decision making happens each day, we have a very real need for it to be modeled and understood especially since there have been some market design flaws. We all remember about the California market crisis, where the prices went through the roof.
- #12: Show AA example with animation
- #13: Show AA example with animation