Reinforcement Learning @FAU2025

Reinforcement Learning (RL) is an area of Machine Learning that has recently made large advances and has been publicly visible by reaching and surpassing human skill levels in games like Go and Starcraft. These successes show that RL has the potential to transform many areas of research and industry by automatizing the development of processes that once needed to be engineered explicitly.

In contrast to other machine learning paradigms, which require the presence of (labeled or unlabeled) data, RL considers an agent that takes actions in an environment and learns from resulting feedback. The agent maximizes a reward signal that it receives for desirable outcomes, while at the same time trying to explore the world in which it operates to find yet unknown, potentially more rewarding action sequences–a dilemma known as the exploration-exploitation tradeoff. Recent advances in machine learning based on deep learning have made RL methods particularly powerful since they allow for agents with particularly well performing models of the world.

Lecturers & Course Instructors

Christopher Mutschler
(course instructor)
Alexander Mattick
(teaching assistant / exercises)
Nico Meyer
(teaching assistant / exercises)

Course Material

The lecture will take place on Thursdays from 16:15 to 17:45 in H9 (Werner-von-Siemens-Hörsaal).

 

#Week Date Topic Material/Information
1 24.04. Introduction to RL, Markov Decision Processes 01 Intro RL, MDPs.pdf
2 01.05. public holiday  
3 08.05. Dynamic Programming 02 Dynamic Programming.pdf
4 15.05. Model-free Prediction 03 Model-free Prediction.pdf
5 22.05. Model-free Control 04 Model-free Control.pdf
6 29.05. public holiday  
7 05.06. Value Function Approximation, DQNs 05 Value Function Approximation.pdf
8 12.06. Policy-based RL #1 06 Policy-based RL1.pdf
9 19.06. public holiday  
10 26.06. Policy-based RL #2 07 Policy-based RL2.pdf
11 03.07. Exploration-Exploitation, Regret, Bandits 08 Exploration-Exploitation.pdf
12 10.07. Exploration in Deep RL, Intrinsic Motivation 09 Exploration in DeepRL.pdf 
13 17.07. Model-based RL #1 (Discrete Actions) 10 Model-based RL.pdf
14 24.07. TBD: “Reinforcement Learning for and with Foundation Models” (Guest Lecture by Dr. Georgios Kontes, Fraunhofer IIS)
Course Wrap-Up, Discussion of Evaluation Results

11 LLM_RL_2025.pdf
11 WrapUp.pdf

 

Exercises

The exercises will take place on Tuesdays from 14:15 to 15:45 in H9 (Werner-von-Siemens-Hörsaal). Exercise sheets will be available after the lecture in the week before. We will give a wrap up of the contents and discuss the solutions of the exercises in the exercise sessions. 

Week Date Topic Material Who?
1 22.04. no exercises
2 29.04. MDPs (slides) ex1.pdf Nico Alex
3 06.05. no exercises 
4 13.05. Dynamic Programming (slides) ex2.pdf, ex2_skeleton.zip Alex
5 20.05. OpenAI Gym, PyTorch-Intro (slides)
TD-Learning (slides)
ex3.pdf, ex3_skeleton.zip
ex4.pdf, ex4_skeleton.zip
Nico
6 27.05. TD-Control (slides) ex5.pdf, ex5_skeleton.zip Nico
7 03.06. Intermediate exam    
8 10.06. no exercises
9 17.06. DQN (slides) ex6.pdf, ex6_skeleton.zip Nico
10 24.06. VPG (slides) ex7.pdf, ex7_skeleton.zip Alex
11 01.07. A2C (slides) ex8.pdf, ex8_skeleton.zip Nico
12 08.07. Multi-armed Bandits (slides) ex9.pdf, ex9_skeleton.zip Alex
13 15.07. RND/ICM (slides) ex10.pdf, ex10_skeleton.zip Alex
14 22.07. MCTS (slides) ex11.pdf, ex11_skeleton.zip Alex

 

Course Evaluation

The evaluation of the lecture and the exercises will be made available here.

Literature

  • Richard S. Sutton and Andrew G. Barto. 2018. Reinforcement Learning: An Introduction. A Bradford Book, Cambridge, MA, USA. [link]
  • Bellman, R.E. 1957. Dynamic Programming. Princeton University Press, Princeton, NJ. Republished 2003: Dover, ISBN 0-486-42809-5.
  • UC Berkeley CS188: Intro to AI [link]
  • University College London Course on RL [link]
  • Advanced Deep Learning and Reinforcement Learning (UCL + DeepMind) [link]
  • https://cs.stanford.edu/people/karpathy/reinforcejs/gridworld_dp.html
  • https://cs.stanford.edu/people/karpathy/reinforcejs/gridworld_td.html

Interesting talks, articles, and blog-posts:

  • Joelle Pineau: Reproducible, Reusable, and Robust Reinforcement Learning [youtube]
  • David Silver: AlphaGo, AlphaZero, and Deep Reinforcement Learning | AI Podcast #86 with Lex Fridman [youtube]
  • Technion Research: Why does reinforcement learning not work (for you)? [link]
  • RL algorithms quick overview [link]

Code examples and exercises:

  • GitHub Repo of Denny Britz: https://github.com/dennybritz/reinforcement-learning/tree/master/DP