CS 277: Control and Reinforcement Learning

Winter 2021

Note: This course was previously offered as CS 295.

Course logistics

• When: Tuesdays and Thursdays at 5–6:20pm
  • Lectures will be recorded and added to this playlist with access for uci.edu accounts.
• Where: zoom
• Announcements and forum: piazza
  • Important course announcements will be made on the forum.
  • Please post on the forum, publicly or privately, all course-related questions (no emails please).
• Assignments: gradscope
  • Published on this page biweekly.
• Instructor: Prof. Roy Fox
  • Office hours: calendly
  • Enrolled students are welcome to:
    • schedule 15-minute slots (more than once if needed);
    • give at least 4-hour notice;
    • attend individually or with friends.

Grading policy

• Assignments: 88%
  • 4 best of 5 assignments count for 22% each.
  • No late submission.
• Participation: 5%
• Bonus: 7%

Schedule

(subject to changes)

Assignments

• Assignment 1; due Friday, January 22, 2021 (Pacific Time). Solution
• Assignment 2; due Friday, January 29, 2021 (Pacific Time). Solution
• Assignment 3; due Tuesday, February 16, 2021 (Pacific Time). Solution
• Assignment 4; due Friday, February 26, 2021 (Pacific Time). Solution
• Assignment 5; due Friday, March 12, 2021 (Pacific Time).

Resources

Courses

• Sergey Levine (Berkeley)
• Pieter Abbeel (Berkeley)
• Dimitri Bertsekas (MIT; also available in book form; also see 2017 book)
• David Silver (UCL)

Books

• Vincent François-Lavet et al., An Introduction to Deep Reinforcement Learning
• Richard Sutton & Andrew Barto, Reinforcement Learning: An Introduction (2nd edition)
• Csaba Szepesvári, Algorithms for Reinforcement Learning

RL libraries

• RLlib
• Acme
• Coach
• Coax
• MushroomRL
• Spinning Up

More resources

• Awesome Deep RL

Further reading

Imitation Learning

• Behavior Cloning with Deep Learning Bojarski et al., CVPR 2016
• DAgger Ross et al., AISTATS 2011
• Goal-conditioned imitation learning Ding et al., NeurIPS 2019
• DART Laskey et al., CoRL 2017
• HVIL Fox et al., Infer2Control @ NeurIPS 2018

Temporal-difference methods

• DQN Mnih et al., Nature 2015
• Double Q-learning van Hasselt et al., NeurIPS 2010
• Double DQN van Hasselt et al., AAAI 2016
• Clipped double Q-learning Fujimoto et al., ICML 2018
• Dueling networks Wang et al., ICML 2016
• NAF Gu et al., ICML 2016

Policy-gradient methods

• REINFORCE Williams, ML 1992
• Policy-Gradient Theorem Sutton et al., NeurIPS 2000
• DPG Silver et al., ICML 2014
• DDPG Lillicrap et al., ICLR 2016
• Off-policy policy evaluation Precup et al., ICML 2000
• Off-policy policy gradient Liu et al., UAI 2018
• TRPO Schulman et al., ICML 2015
• PPO Schulman et al., arXiv 2017

Actor–critic methods

• A3C Mnih et al., ICML 2016
• GAE Schulman et al., ICLR 2015

Model-based methods

• MCTS Kocsis and Szepesvári, ECML 2006
• ILQR Li and Todorov, ICINCO 2004
• DDP Mayne, IJC 1966
• Dyna Sutton, ACM SIGART Bulletin, 1991
• E³ Kearns and Singh, ML 2002
• R-max Brafman and Tennenholtz, JMLR 2002
• Local models Levine and Abbeel, NeurIPS 2014
• PBVI Pineau et al., IJCAI 2003

Exploration

• UCB in MAB Auer et al., ML 2002
• Thompson sampling in MAB Agrawal and Goyal, COLT 2012
• Count-based exploration Bellemare et al., NeurIPS 2016
• Thompson sampling in RL Gopalan and Mannor, COLT 2015

Inverse Reinforcement Learning

• Feature matching Abbeel and Ng, ICML 2004
• MaxEnt IRL Ziebart et al., AAAI 2008
• GAIL Ho and Ermon, NeurIPS 2016

Control as Inference

• LMDP Todorov, NeurIPS 2007
• Full-controllability duality Todorov, CDC 2008
• Information–value tradeoff Rubin et al., Decision Making with Imperfect Decision Makers 2012
• VI duality Levine, arXiv 2018
• Soft Q-learning Fox et al., UAI 2016
• Deep SQL Haarnoja et al., ICML 2017
• SAC Haarnoja et al., ICML 2018

Structured Control

• Options framework Precup et al., AI 1999
• Option–critic method Bacon et al., AAAI 2017
• Skill trees Konidaris et al., NeurIPS 2010
• Spectral method Mahadevan and Maggioni, JMLR 2007
• Exact inference Fox et al., arXiv 2017
• PHP Fox et al., ICLR 2018
• FuN Vezhnevets et al., ICML 2017

Multi-Task Learning

• Fine-tuning SQL Haarnoja et al., ICML 2017
• Learning with pre-trained perceptual features Levine et al., JMLR 2016
• Learning with model transfer Fu et al., IROS 2016
• Domain randomization Peng et al., ICRA 2018
• Domain adaptation Bousmalis et al., ICRA 2018
• Goal generation for curriculum learning Florensa et al., ICML 2018
• Multi-task policy distillation Teh et al., NeurIPS 2017
• Multi-task hierarchical imitation learning Fox et al., CASE 2019

Multi-Agent RL

• MADDPG Lowe et al., NeurIPS 2017
• NFSP Heinrich and Silver, DRL @ NeurIPS 2016
• Double Oracle McMahan et al., ICML 2003
• PSRO Lanctot et al., NeurIPS 2017

Academic honesty

Don’t cheat. Academic honesty is a requirement for passing this class. Compromising the academic integrity of this course is subject to a failing grade. The work you submit must be your own. Academic dishonesty includes, among other things, copying answers from other students or online resources, allowing other students to copy your answers, communicating exam answers to other students during an exam, or attempting to use notes or other aids during an exam. If you do so, you will be in violation of the UCI Policy on Academic Honesty and the ICS Policy on Academic Honesty. It is your responsibility to read and understand these policies, in light of UCI’s definitions and examples of academic misconduct. Note that any instance of academic dishonesty will be reported to the Academic Integrity Administrative Office for disciplinary action, and may fail the course.