285-LEC12

13 flashcards · Computer Science · by John Viljoen

1. explain the pseudo code of model-based RL v1.5

2. what shortcomings do open-loop methods have?

   they are suboptimal, as not taking into account future information will result in suboptimality there is no room for adaptation once a trajectory is formed

3. what is the distributional shift problem in standard model-based RL?

   we can move away from the region in which the model was trained. the model that has been trained is itself imperfect to begin with

4. what is the problem with backpropping into policy directly like this:?

   actions at the end of a trajectory matter far less than actions at the start.

5. How to solve the vanishing/exploding gradient problem in a learned policy + learned dynamics system?

6. What is the equation for the policy gradient? (for stochastic policies and transitions?)

7. What is the backprop gradient? (for deterministic policies and transitions)

   (massive product of jacobians)

8. why might the policy gradient be more stable than the backprop gradient?

9. why are long model-based rollouts a problem?

   for the same reason as long rollouts in imitation learning are a problem, small errors in the imperfect model that has been learned will compound over time.

10. how quickly does error accumulate in long model-based rollouts?

11. pros and cons of long and short model-based rollouts?

12. solution to long model-based rollout problem

13. explain model-based RL with short rollouts pseudo code (v3.0)