Scribd
Upload
85 views

IIM7064 Dynamic Programming

This document outlines the syllabus for a course on dynamic programming taught by Professor Ya-Tang Chuang. The course will cover modeling sequential decision problems using dynamic programming, solving algorithms like value iteration and policy iteration, and structural results for optimal policies. Students will work in teams to present research papers, and individually on a term project applying dynamic programming to a real-world problem. The goal is for students to learn to formulate and analyze problems with dynamic programming.

Uploaded by

Armee Justitia
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
85 views

IIM7064 Dynamic Programming

This document outlines the syllabus for a course on dynamic programming taught by Professor Ya-Tang Chuang. The course will cover modeling sequential decision problems using dynamic programming, solving algorithms like value iteration and policy iteration, and structural results for optimal policies. Students will work in teams to present research papers, and individually on a term project applying dynamic programming to a real-world problem. The goal is for students to learn to formulate and analyze problems with dynamic programming.

Uploaded by

Armee Justitia
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 18

IIM7064 Dynamic Programming

Ya-Tang Chuang

Department of Industrial and Information Management


National Cheng Kung University

1 / 18 Ya-Tang Chuang Dynamic Programming


TODAY ’ S OUTLINE

About this class

Introduction of dynamic programming

Some examples

2 / 18 Ya-Tang Chuang Dynamic Programming


C LASS OVERVIEW

Office hours: By appointment; [email protected]


TA: Allen Lin, [email protected]
No required textbook, but you can refer the following books
Sheldon Ross, Introduction to Stochastic Dynamic Programming,
Academic Press, 1983
Martin L. Puterman, Markov Decision Processes - Discrete
Stochastic Dynamic Programming, Wiley, 1994. (electronic book)
Important Dates
10/06: Team formation
11/03: Select one research paper for your representation

3 / 18 Ya-Tang Chuang Dynamic Programming


C OURSE TOPICS

Modeling: Modeling problems as a dynamic program


Backward induction
Finite horizon
Infinite horizon

Solving algorithms
Value iteration
Policy iteration

Structural results of the optimal policies


Modular functions
Monotone policies

Time permitting: Other selected topics

4 / 18 Ya-Tang Chuang Dynamic Programming


C OURSE SCHEDULE

Week Progress Description


1 Introduction, Course Orientation, and Q&A
2-4 Deterministic Problems
5 Stochastic Problems
6-7 Finite-Horizon Problems
8 Backward Induction
9-10 Structural Results of Optimal Policy
11 Infinite-Horizon Problems
12 Solving Algorithms
13 Midterm Exam
14 Partially Observable Markov Decision Processes
15-16 Paper Presentation
17-18 Project Presentation

5 / 18 Ya-Tang Chuang Dynamic Programming


PAPER PRESENTATION

Team of 2-3 members

Select one paper from the list provided

Each team will be given 20 minutes

Each member should contribute equally during presentation

Each team should ask at least two questions for other’s


presentation

Your presentation should include


Problem set up
Main results
Main contributions
Criticism

Scores will be based on peer evaluation

6 / 18 Ya-Tang Chuang Dynamic Programming


T ERM PROJECT

Modeling one sequential decision making problem as a Markov


decision process

Solving this problem through dynamic programming

Providing numerical or theoretical results

Giving a presentation

Scores will be based on peer evaluation

You also need to write a report; your report should include


Problem set up
Main results
Potential improvements

7 / 18 Ya-Tang Chuang Dynamic Programming


C OURSE OBJECTIVES

Understand how dynamic programming (DP) is used to model a


variety of situations, and what properties are desirable in the DP

Be able to analyze DP equations that may be helpful to “solve” a


given problem

Know the basic concepts of complexity theory, in particular,


understand the situations under which the optimal policy may be
numerically intractable

Learn basic techniques for deriving properties of value function


and optimal policies

8 / 18 Ya-Tang Chuang Dynamic Programming


G OAL OF THE COURSE

Primary course objective: After taking this course, you should be


able to formulate a given problem using dynamic programming
and derive solutions

We will not spend time teaching coding, but you should have the
ability to complete your project

9 / 18 Ya-Tang Chuang Dynamic Programming


G RADING

I don’t view grades in graduate courses as very important. You


should be here because you want to learn. Nevertheless, need
to assign grades:
Assignments 20%
Midterm Exam 30%
Paper Presentation 10%
Term Project Presentation 10%
Term Project Report 30%

Assignments: You can discuss with other people, but assignment


solutions should be your own work

In any assignment, you must properly give credit to any outside


resources you use (such as books, papers, web sites, other
people, etc.)

10 / 18 Ya-Tang Chuang Dynamic Programming


Q UESTIONS ABOUT THE COURSE ?

11 / 18 Ya-Tang Chuang Dynamic Programming


W HAT IS DYNAMIC PROGRAMMING ?

Dynamic Programming is a collection of mathematical tools used


to analyze sequential decision processes.
Such processes entail a sequence of actions taken toward some
goal, often in the face of uncertainty.

During each discrete time period, the process goes through a


sequence of steps that involve (1) state observation, (2) selection
of an action or control, (3) incurrence of a cost, and (4) state and
time update.

If the sequential decision processes terminate after a finite


number of stages, they are called finite horizon processes.
Otherwise, they are called infinite horizon processes.

12 / 18 Ya-Tang Chuang Dynamic Programming


T HE BIG PICTURE OF MODEL

At a given state, the decision maker can take an action/control


which takes the system to another state and the process is
repeated
In the process, the decision maker may receive a reward or incur
some cost, which depends on the current state and action
State transitions are usually stochastic and are a function of the
actions
The end goal is to optimize some function of the sequence of
cost/reward

13 / 18 Ya-Tang Chuang Dynamic Programming


A PPLICATIONS

PChome/Amazon online shopping

How to select add-on product to increase success probability

Figure: Rice cooker with a cooking pot (from Amazon)

14 / 18 Ya-Tang Chuang Dynamic Programming


A PPLICATIONS

PChome/Amazon online shopping

How to select add-on product to increase success probability

Figure: PS4 with a hard disk drive (from PChome)

15 / 18 Ya-Tang Chuang Dynamic Programming


A PPLICATIONS

How to determine the price of tickets to maximize total reward

16 / 18 Ya-Tang Chuang Dynamic Programming


A PPLICATIONS

Preventive maintenance

17 / 18 Ya-Tang Chuang Dynamic Programming


A PPLICATIONS

Discharge or continue treatment

18 / 18 Ya-Tang Chuang Dynamic Programming

You might also like