Reinforcement Learning @FAU2022

Reinforcement Learning (RL) is an area of Machine Learning that has recently made large advances and has been publicly visible by reaching and surpassing human skill levels in games like Go and Starcraft. These successes show that RL has the potential to transform many areas of research and industry by automatizing the development of processes that once needed to be engineered explicitly.

In contrast to other machine learning paradigms, which require the presence of (labeled or unlabeled) data, RL considers an agent that takes actions in an environment and learns from resulting feedback. The agent maximizes a reward signal that it receives for desirable outcomes, while at the same time trying to explore the world in which it operates to find yet unknown, potentially more rewarding action sequences–a dilemma known as the exploration-exploitation tradeoff. Recent advances in machine learning based on deep learning have made RL methods particularly powerful since they allow for agents with particularly well performing models of the world.

Lecturers & Course Instructors

Christopher Mutschler
(course instructor)
Sebastian Rietsch
(teaching assistant / exercises)

The lectures will be recorded and provided on YouTube before the lecture slot. The lecture slot can be used as a consultation hour and if you give me a short ping we can organise an ad-hoc zoom meeting. The Q&A session for the lecture content will take place at the beginning of the exercise session.

Course Material

We compiled a YouTube playlist for this summer term here. However, you can also access the videos directly below, or if you are curious, you can have a look on the previous year’s edition as we will only occasionally record new content. However, exercises will change a bit.

 

  Topic Video Duration Material
28.04.2022: Lecture 1: Introduction to Reinforcement Learning
   
1.00/01 Opening Remarks & Introduction to RL live/zoom slides (pdf)
1.02 Markov Decision Processes link 37:49 slides (pdf)
05.05.2022: Q&A on Lecture 2: Dynamic Programming (slides, video)
 1:47:05  
2.01 Introduction to Dynamic Programming link 24:48 slides (pdf)
2.02 Dynamic Programming: Value Iteration link 28:14 slides (pdf)
2.03 Dynamic Programming: Policy Iteration link 25:45 slides (pdf)
2.04 Summary link 12:03 slides (pdf)
2.05 Hands-On: DP on Frozen Lake link 16:15 code (zip)
12.05.2022: Q&A on Lecture 3: Model-free Prediction (slides, video)
 1:39:06  
3.01 Introduction to Model-free Reinforcement Learning link 7:45 slides (pdf)
3.02 Model-free Prediction with Monte Carlo link 31:19 slides (pdf)
3.03 Model-free Prediction with Time-Difference Learning link 47:49 slides (pdf)
3.04 Hands-On: TD Learning on Frozen Lake link 12:13 code (zip)
19.05.2022: Q&A on Lecture 4: Model-free Control (slides, video)
 1:51:17  
4.01 Model-free Control link 58:33 slides (pdf)
4.02 Hands-On: OpenAI Gym Intro link  14:37 code (zip)
4.03 Hands-On: Q-Learning on Frozen lake link  15:15 code (zip)
4.04 Summary on Model-free Reinforcement Learning link 22:52 slides (pdf)
26.05.2021: Holiday/Vacation
02.06.2022: Q&A on Lecture 5: Value Function Approximation (slides, video)
2:04:39  
5.01 Value Function Approximation link 14:11 slides (pdf)
5.02 Linear Value Function Approximation link 34:26 slides (pdf)
5.03 Deep Q-Networks (error on slide #15 corrected in pdf) link 44:13 slides (pdf)
5.04 Hands-On: DQN on Cartpole link  22:51 code (ipynb)
5.05 Summary link 8:58 slides (pdf)
09.06.2022: Q&A on Lecture 6: Policy-based RL – Part 1 (slides, video)
2:03:41  
6.01 Introduction to Policy-based RL link 54:44 slides (pdf)
6.02 Policy Gradients  link  54:49 slides (pdf)
6.03 Hands-On: Monte-Carlo Policy Gradient link 14:08 code (zip)
16.06.2021: Holiday/Vacation
23.06.2022: Q&A on Lecture 7: Policy-based RL – Part 2 (slides, video)
2:01:05  
7.01 Actor-Critics link 21:48 slides (pdf)
7.02 Trust-Region Policy Optimization (TRPO) link  1:06:09 slides (pdf)
7.03 Proximal Policy Optimization (PPO) link  17:50 slides (pdf)
7.04 Deep Deterministic Policy Gradient (DDPG) link  15:18 slides (pdf)
30.06.2022: Q&A on Model-based RL – Part 1 (slides, video)  1:38:05  
8.01  Introduction to Model-based RL  link 33:01 slides (pdf)
8.02 Background Planning link 19:22 slides (pdf)
8.03 Online Planning with Discrete Actions link 45:42 slides (pdf)
07.07.2022: Q&A on Model-based RL – Part 2 1:08:05  
9.01 Online Planning with Continuous Actions link 22:09 slides (pdf)
9.02 Real-World Application: Uncertainty in Model-based RL link  39:25 slides (pdf)
9.03 Summary on Model-based RL link 6:31 slides (pdf)
14.07.2022: Q&A on Exploration Strategies 1:32:20  
10.01 Motivation, Multi-Armed Bandits, Regret link 33:45 slides (pdf)
10.02  Classic Exploration Strategies link 58:35 slides (pdf)
21.07.2022: Q&A on Exploration in Deep RL 2:07:43  
11.01 Count-based Exploration link 32:32 slides (pdf)
11.02 Prediction-based Exploration link 56:25 slides (pdf)
11.03 Memory-based Exploration link 38:46 slides (pdf)
28.07.2022: Course Wrap-Up   slides (pdf)
  Summary on Content      
  Discussion of Course Evaluation Results      
  Q&A on the Exam      

 

Exercises

 

  # Topic Material Due Date
28.04.2022 1 Markov Decision Processes ex1.pdf 05.05.2022
05.05.2022 2 Dynamic Programming ex2.pdf | ex2_skeleton.zip 12.05.2022
12.05.2022 3 OpenAI Gym & TD-Learning ex3.pdf | ex3_skeleton.zip 19.05.2022
19.05.2022 4 TD Control ex4.pdf | ex4_skeleton.zip 02.06.2022
19.05.2022 5 PyTorch ex5.pdf | ex5_skeleton.zip 02.06.2022
26.05.2022 no exercises / holiday
02.06.2022 6 DQNs v0.2 on Cartpole ex6.pdf | ex6_skeleton.zip 09.06.2022
09.06.2022 7 DDQNs on Atari ex7.pdf (Update) | ex7_skeleton.zip 23.06.2022
09.06.2022 8 VPG ex8.pdf | ex8_skeleton.zip 23.06.2022
16.06.2022 no exercises / holiday
23.06.2022 9 A2C ex9.pdf | ex9_skeleton.zip 07.07.2022
30.06.2022 10 MCTS ex10.pdf | ex10_skeleton.zip 14.07.2022
07.07.2022        
14.07.2022 11 Bandits   21.07.2022
21.07.2022 12 RND on MountainCar   28.07.2022
28.07.2022        

 

Course Evaluation

The evaluation of the lecture and the exercises will be made available here.

Literature

  • Richard S. Sutton and Andrew G. Barto. 2018. Reinforcement Learning: An Introduction. A Bradford Book, Cambridge, MA, USA. [link]
  • Bellman, R.E. 1957. Dynamic Programming. Princeton University Press, Princeton, NJ. Republished 2003: Dover, ISBN 0-486-42809-5.
  • UC Berkeley CS188: Intro to AI [link]
  • University College London Course on RL [link]
  • Advanced Deep Learning and Reinforcement Learning (UCL + DeepMind) [link]
  • https://cs.stanford.edu/people/karpathy/reinforcejs/gridworld_dp.html
  • https://cs.stanford.edu/people/karpathy/reinforcejs/gridworld_td.html

Interesting talks, articles, and blog-posts:

  • Joelle Pineau: Reproducible, Reusable, and Robust Reinforcement Learning [youtube]
  • David Silver: AlphaGo, AlphaZero, and Deep Reinforcement Learning | AI Podcast #86 with Lex Fridman [youtube]
  • Technion Research: Why does reinforcement learning not work (for you)? [link]
  • RL algorithms quick overview [link]

Code examples and exercises:

  • GitHub Repo of Denny Britz: https://github.com/dennybritz/reinforcement-learning/tree/master/DP