Why Are We
Conscious?
Consciousness is a Simulator with a Memory for Solving the
Temporal Credit Assignment Problem in Reinforcement Learning
Bill Hibbard
University of Wisconsin
This poster available from:
Contact Bill Hibbard at:
Reinforcement Learning
Temporal Credit Assignment Problem
- Time delays between actions and rewards
- Complex causal relations between actions and rewards
- Multiple actions precede a reward
This is the central problem for intelligent organisms and robots, which must adapt their behavior to the complexities of the world by learning.
The Temporal Difference (TD) Solution
Time traces:
Stimulus
Action
Reward
Reward Prediction
before training
Reward Prediction
after training
The TD solution uses a set of time delays triggered by Stimulus.
Before training the Reward Prediction mirrors the Reward, increasing the weight of the Stimulus Working Memory so the Reward Prediction will mirror the Stimulus after training and strengthen the weight of the Stimulus triggering the Action.
Before training the Reward Prediction also increases the weight of the appropriate Delays, which will negate the Reward in the Reward Prediction after training.
The Reward Prediction signal strongly resembles the observed output of dopaminergic neurons in the substantia nigra pars compacta (SNc) in monkeys during reinforcement learning.
Dopamine has the effect of strengthening weights of neural connections, such as between Stimulus and Action.
The Delay signals resemble the behavior of striosomal neurons, and other midbrain structures have been identified with other elements of the TD solution.
However, observed TD solution behavior in brains only applies to sensorimotor learning with short and predictable delays between Stimuli and Rewards, and is inadequate for learning complex behaviors.
The TD solution can learn a sequence of Actions, by working backward through the sequence and using the Reward Prediction for one Action as the Reward signal for the previous Action.
Suri cites indirect evidence that dopamine also plays a role in more sophisticated TD solution learning involving an internal model that simulates Stimuli, Actions and Rewards.
Suri says internal model represents events on a compressed time scale, and speculates about neural mechanisms for this.
Brown, J., Bullock, D., and Grossberg, S. How the basal ganglia use parallel excitatory and inhibitory learning pathways to selectively respond to unexpected reward cues. Journal of Neuroscience 19(23), 10502-10511. 1999.
Suri, R. E. TD models of reward predictive responses in dopamine neurons. Neural Networks, special issue on Neuromodulation, invited contribution, 2002, (submitted).
Consciousness is a Simulator
Consciousness is a simulator that enables the brain to process experiences that are not actually occurring.
Consciousness is a simulator of experience that can attach to or detach from perception and physical action as necessary - the decision to attach is under control of the simulator, but may be overruled by perceptions (e.g., hard to ignore pain).
During attachment, details of perception are filled in by simulation, causing various illusions and false memories.
The simulator detaches during dreaming, remembering and imagining - detached simulation of an event causes the same cortical neurons to fire that actually experiencing the event does.
Brains Need Simulators for Solving the Temporal Credit Assignment Problem
Many complex stimuli, actions and rewards occur on long and unpredictable time scales, whereas simulations of those events can occur on short and predictable time scales suitable for the brain's known TD solution to the temporal credit assignment problem.
Simulation allows complex situations to be explored many times for more efficient learning (e.g., thinking about future lines of play in a chess game, rather than actually playing each line).
The Extended Reticular-Thalamic Activation System (ERTAS) Theory of Consciousness
Feedback loops exist between all areas of the cortex and the thalamus, gated by the nucleus reticularis thalami (nRt); this gate is thought to implement conscious attention.
40 Hz oscillations in these feedback loops bind together areas of the cortex representing diverse content to form a constantly evolving conscious focus.
The basal ganglia, which control plans, choices and motor skills, also send signals to the nRt, suggesting that control of attention is similar to control of motion.
Perceptual input to the cortex flows through the thalamus and its gating function.
Crick, F. 1984. Function of the thalamic reticular complex: the searchlight hypothesis. Proceedings of the National Academy of Science USA, 81, 4586-4590.
Baars, B. J. 1988. A Cognitive Theory of Consciousness.
Newman, J. Thalamocortical foindations of conscious experience.
Edelman, G. M., Tononi, G., A Universe of Consciousness. New York: Perseus Books Group, 2000.
The ERTAS Theory and Simulation
Consciousness binds diverse content in order to simulate a wide variety of situations in detail.
The constantly evolving sequence of conscious content simulates cause and effect in a sequence of world events …
… or conscious content may evolve due to an interruption of attention to simulate a different sequence of events.
Consciousness and Emotion
The positive and negative valences of emotion exist to motivate behavior - in humans that primarily means to reinforce learning of behavior.
Douglas Watt presents evidence from brain architecture, diseases of consciousness and neurodevelopment that emotion plays a central role in consciousness.
This suggests that consciousness serves reinforcement learning.
Watt, D. F. Emotion and consciousness: implications of affective neuroscience for extended reticular thalamic activating system theories of consciousness.
Consciousness and Learning
Jeffrey Gray presents evidence that mechanisms involving the hippocampus constantly compare perceptions to predictions, and trigger conscious attention of mismatches via connections to the thalamus.
That is, consciousness is directed to situations where learning is needed.
Unfamiliar behaviors are the subject of conscious attention, but once learned they are often performed without conscious
attention.
Gray, J. A. 1995. The contents of consciousness: a neuro-psychological conjecture. Behavior and Brain Sciences, 18(4).
Consciousness and Natural Selection
1. A solution of the temporal credit assignment problem for reinforcement learning is essential for any complex organism.
2. The TD solution for sensorimotor learning only solves the problem for short and predictable delays between stimuli and rewards.
3. Consciousness is a simulator for solving the problem in more general situations and more efficiently.
4. This is why consciousness gives organisms a survival and reproduction advantage.
Simulation and Duality
A sufficiently complex simulation will include models of the simulations in other brains.
Human simulations of other human's simulations are inferred from perceptions of other human's behaviors, especially via language.
My simulation of red is based on visual perception, whereas my simulation of your simulation of red is based on perception of your language. There is no basis to compare them.
Human brains lack any means for direct neural connection with each other, but if future technology provides one then humans may get better comparisons of their models of red.
If our consciousness is a simulator of the world, shaped by but not identical with our perceptions, then it is inevitably separated from the world.
But ultimately everyone must make their own judgement about the relation of their subjective experience to the world.