Skip to main content

Postponed Updates for Temporal−Difference Reinforcement Learning

Harm van Seijen and Shimon Whiteson

Abstract

This paper presents postponed updates, a new strategy for TD methods that can improve sample efficiency without incurring the computational and space requirements of model-based RL. By recording the agent's last-visit experience, the agent can delay its update until the given state is revisited, thereby improving the quality of the update. Experimental results demonstrate that postponed updates outperforms several competitors, most notably eligibility traces, a traditional way to improve the sample efficiency of TD methods. It achieves this without the need to tune an extra parameter as is needed for eligibility traces.

Book Title
ISDA 2009: Proceedings of the Ninth International Conference on Intelligent Systems Design and Applications
Month
November
Pages
665−672
Year
2009