Dopamine neurons learn to encode the long-term value of multiple future rewards

Kazuki Enomoto, Naoyuki Matsumoto, Sadamu Nakai, Takemasa Satoh, Tatsuo K. Sato, Yasumasa Ueda, Hitoshi Inokawa, Masahiko Haruno, Minoru Kimura

Research output: Contribution to journalArticleResearchpeer-review

53 Citations (Scopus)

Abstract

Midbrain dopamine neurons signal reward value, their prediction error, and the salience of events. If they play a critical role in achieving specific distant goals, long-term future rewards should also be encoded as suggested in reinforcement learning theories. Here, we address this experimentally untested issue. We recorded 185 dopamine neurons in three monkeys that performed a multi-step choice task in which they explored a reward target among alternatives and then exploited that knowledge to receive one or two additional rewards by choosing the same target in a set of subsequent trials. An analysis of anticipatory licking for reward water indicated that the monkeys did not anticipate an immediately expected reward in individual trials; rather, they anticipated the sum of immediate and multiple future rewards. In accordance with this behavioral observation, the dopamine responses to the start cues and reinforcer beeps reflected the expected values of the multiple future rewards and their errors, respectively. More specifically, when monkeys learned the multistep choice task over the course of several weeks, the responses of dopamine neurons encoded the sum of the immediate and expected multiple future rewards. The dopamine responses were quantitatively predicted by theoretical descriptions of the value function with time discounting in reinforcement learning. These findings demonstrate that dopamine neurons learn to encode the long-term value of multiple future rewards with distant rewards discounted.

Original languageEnglish
Pages (from-to)15462-15467
Number of pages6
JournalProceedings of the National Academy of Sciences of the United States of America
Volume108
Issue number37
DOIs
Publication statusPublished - 13 Sep 2011
Externally publishedYes

Keywords

  • Basal ganglia
  • Decision making
  • Primate
  • Temporal difference learning

Cite this