PRINCETON UNIVERSITY
Animal Learning and Decision Making: Psychological, Computational and Neural Perspectives
Spring 2009, Mondays 1:30-4:20, Green Hall 0-N-2



Yael Niv
Email: yael at princeton dot edu


Format and content:

The purpose of this seminar is to provide a modern, integrative view of classic animal learning phenomena from experimental psychology, through the lens of contemporary learning theory, computational models of learning and decision making, and current neuroscientific knowledge. Each 3-hour lecture will focus on one topic in experimental psychology and will be divided into roughly three parts. First, we will discuss the major behavioral findings and the learning theory that pertains to them. Following this, in the second and third hours, we will discuss computational theories pertaining to the topic and what is currently known about the neural basis of these behaviors or computations.

Course requirements: Active class participation (20%), reading of assigned literature, 4 out of 5 problem sets (40%), final Exam (40%; open book/notebook; no laptops by unversity rules).

Syllabus topics and presentations*:
Note: handouts are put up before class and are a tentative idea of what we will talk about. They also have intentional gaps where I will ask you questions and prefer to not reveal the answers yet. Slide presentations are put up after the class and account for what we actually covered, icluding corrections of errors in the handouts which we noticed in class.

  1. Introduction - motivation, structure, requirements of the course; defining learning and decision making; MarrÕs three levels of inquiry; conditioning; what we know about the brain handout keynote PDF
  2. Classical conditioning I - basic phenomena, error driven learning and the Resorla-Wagner model handout slides
  3. Classical conditioning II - second order conditioning and the Temporal Difference model, dopamine, opponent processes: appetitive vs. aversive, inhibitory vs. excitatory, fMRI of aversive conditioning handout slides
  4. Recap of TD learning & Instrumental conditioning I - recap of TD learning, opponent processes model, fMRI of temporal difference errors, Thorndike, basic instrumental procedures, Skinner & free operant behavior handout slides
  5. Instrumental conditioning I continued - Behaviorism, free operant schedules, learning Q values, Actor/Critic framework, neural substrates inform about models handout slides
  6. Instrumental conditioning II - actions and habits, S-R versus R-O, devaluation, dual neural pathways, uncertainty based arbitration handout slides
  7. Instrumental conditioning III - modeling action selection, comparing models using neural data handout slides
  8. Modeling free operant behavior - a reinforcement learning model of response vigor and motivation, tonic dopamine handout slides
  9. Classical vs. instrumental conditioning & extinction - comparison of instrumental and Pavlvovian procedures and processes, omission, Pavlovian-instrumental transfer, extinction in young and old rats, models of extinction, handout slides & notes
  10. Learning as a model of disorders - latent inhibition and schizophrenia, drug addiction handout slides & notes
  11. Generalization and discrimination - configural vs. elemental theories, learning latent causes and Bayesian inference handout slides & notes
  12. Summary & a brief touch on topics we did not cover - hippocampal maps, spatial learning and representations, attention and associability, timing, other neuromodulators, amygdala, summary - the big picture handout slides
* Many thanks to all my colleagues and fellow teachers who have (mostly unknowingly) contributed much of the material that is included in the above slides


Readings: (for PDFs go to blackboard)

  • Textbook: Learning and Behavior: A Contemporary Synthesis by Mark E. Bouton. (See also the companion website for chapter outlines, summaries and online quizzes.)

  • For 9 Feb: Chapters 1 (historical perspective on learning) and 3 (classical conditioning) in the textbook
  • For 16 Feb: Rescorla (1988) - Pavlovian conditioning: its not what you think it is, Americal Psychologist 43(3):151-160. (PDF on BlackBoard)
  • For 23 Feb: Textbook pages 103-118 (Rescorla-Wagner model), 360-368 (opponent process model) and Tobler, Dickinson & Schultz (2003) - Coding of predicted reward omission by dopamine neurons in a conditioned inhibition paradigm, The Journal of Neuroscience 23:10402-10410. (PDF on Blackboard)
  • For 2 Mar: Textbook reading
    • Chapter 2 , pages 46-50 - introduction to instrumental conditioning (review)
    • Chapter 7 - Behavior and its consequences - instrumental conditioning in detail (you are responsible for the section on "theories of reinforcement" which we will not go over in class)
  • For 9 Mar: Dickinson (1985) - Actions and Habits: The development of behavioural autonomy, Philosophical Transactions of the Royal Society of London, Series B, Biological Sciences 308:67-78. (PDF on Blackboard)
  • For 23 Mar:
    • Tolman (1948) - Cognitive maps in rats and men, Psychological Review 55:189-208. (PDF on Blackboard)
    • Chapter 9 , pages 330-358 - Motivation and instrumental behavior - in case you did not read this yet (I assigned it two weeks ago by mistakey)
    • Niv, Daw, Joel & Dayan (2006) - Tonic dopamine: Opportunity costs and the control of response vigor, Psychopharmacology 191(3):507-520. (PDF on Blackboard)
  • For 30 March: Niv (in press) - Reinforcement Learning in the Brain, Journal of Mathematical Psychology (PDF on Blackboard)
  • For 6 April: Kim & Richardson (2008) - The effect of temporary amygdala inactivation on extinction and reextinction of fear in the developing rat: unlearning as a potential mechanism for extinction early in development, Journal of Neuroscience 28:1282-1290. (PDF on Blackboard)
    • Write down (and bring to class) a 2-3 point summary of the paper (2-3 short sentences)
    • Write down (and bring to class) 3 (or more) questions about extinction that arose from reading this paper
  • For 13 April: (note, one of the papers has changed since we discussed the reading in class)
    • Redish (2004) - Addiction as a computational process gone awry, Science 306:1944-1947 (PDF & questions on Blackboard)
    • Yogev et al. (2004) - Latent inhibition and overswitching in schizophrenia, Schizophrenia Bulletin 30(4):713-726 (PDF & questions on Blackboard)
    • Here is an (optional) list of Focus Questions for the Yogev et al. paper. The idea is to give you some guidelines as to how to get the most out of reading an experimental paper. This might be superfluous as you may have already figured this out for yourself, but I find that this is one of the harder things in science, so I thought an example might help.
  • For 20 April: Pearce (1994) - Similarity and discrimination: a selective review and a connectionist model Psychological Review, 101:587-607 (PDF on Blackboard)


Assignments:

Note: Your 4 highest graded assignment will form 40% of your final grade (the Matlab assignment is compulsory, but you can choose to not hand in one other assignment, or to hand in all and the worst won't count). Assignments handed in late will lose half a grade for every day late.


Some books I recommend:

  • Sutton, R. and Barto, A. - Reinforcement Learning (1998) - A very good and accessible book explaining the computational field of reinforcement learning (also available online here.
  • Mackintosh, N.J. - The Psychology of Animal Learning (1974) - Great book, although hard to find (out of print).
  • Mackintosh, N.J. - Conditioning and Associative Learning (1983) - Shorter than the above and covers mostly theoretical aspects, not for complete beginners.
  • Dickinson, A. - Contemporary Animal Learning Theory (1980) - Although this was contemporary a long time ago, it is still very good and easy to read.


Extra Material:


Interesting (and useful) Links:

Last modified: Yael Niv - April 2009