Taming the Noise in Reinforcement Learning via Soft Updates

Roy Fox*, Ari Pakman*, and Naftali Tishby

32nd Conference on Uncertainty in Artificial Intelligence (UAI), 2016

BibTeX

@conference{ Fox2016Glearning,
  Title = "Taming the Noise in Reinforcement Learning via Soft Updates",
  Author = "Roy Fox and Ari Pakman and Naftali Tishby",
  Booktitle = "32nd Conference on Uncertainty in Artificial Intelligence (UAI)",
  Year = "2016"
}

Model-free reinforcement learning algorithms, such as Q-learning, perform poorly in the early stages of learning in noisy environments, because much effort is spent unlearning biased estimates of the state-action value function. The bias results from selecting, among several noisy estimates, the apparent optimum, which may actually be suboptimal. We propose G-learning, a new off-policy learning algorithm that regularizes the value estimates by penalizing deterministic policies in the beginning of the learning process. We show that this method reduces the bias of the value-function estimation, leading to faster convergence to the optimal value and the optimal policy. Moreover, G-learning enables the natural incorporation of prior domain knowledge, when available. The stochastic nature of G-learning also makes it avoid some exploration costs, a property usually attributed only to on-policy algorithms. We illustrate these ideas in several examples, where G-learning results in significant improvements of the convergence rate and the cost of the learning process.