Have We Missed Half of What the Neocortex Does? by Jeff Hawkins (12/15/2017)
Download as PPTX, PDF•3 likes•559 views
Jeff Hawkins explores the functionalities of the neocortex, emphasizing the need to reverse engineer its operations to develop biologically accurate theories for machine intelligence. He discusses how cortical columns process allocentric location and learn predictive models for sensory and motor sequences, highlighting their complex architecture and integration of input over time. The document outlines ongoing research and proposals for understanding cortical column functions, including their role in predicting sensory input and learning object structures.
Have We Missed Half of What the Neocortex Does? by Jeff Hawkins (12/15/2017)
- 1. MIT December 15, 2017 Jeff Hawkins [email protected] Have We Missed Half of What the Neocortex Does? Allocentric Location as the Basis of Perception
- 2. 1) Reverse engineer the neocortex - an ambitious but realizable goal - seek biologically accurate theories - test empirically and via simulation 2) Enable technology based on cortical theory - active open source community - basis for Machine Intelligence
- 4. L2 L3a L3b L4 L6a L6b L6 ip L6 mp L6 bp L5 tt L5 cc L5 cc-ns L2/3 L4 L6 L5 Input The Cortical Column 1) Cortical columns are complex - Twelve or more excitatory cellular layers - Two parallel FF pathways - Parallel FB pathways (not shown) - Numerous intra- and inter-column connections (not shown) - Inhibitory neurons/circuits are equally complex 2) The function of a cortical column must also be complex. 3) Whatever a column does applies to everything the cortex does. L5: Calloway et. al, 2015 L6: Zhang and Deschenes, 1997 Simple Output, via thalamus 50%10% Cortex Thalamus Output, direct L5 CTC: Guillery, 1995 Constantinople and Bruno, 2013 A Couple of Thoughts Output
- 5. Observation: The neocortex is constantly predicting its inputs. How do networks of neurons, as seen in the neocortex, learn predictive models of the world? Research:
- 6. 1) How does the cortex learn predictive models of extrinsic sequences? 2) How does the cortex learn predictive models of sensorimotor sequences? Current research: How do columns compute allocentric location? - Grid cells in entorhinal cortex solve a similar problem - Big Idea: cortical columns contain analogs of grid cells and head direction cells - Starting to understand the function of numerous layers and connections “Why Neurons Have Thousands of Synapses, a Theory of Sequence Memory in the Neocortex” Hawkins and Ahmad, Frontiers in Neural Circuits, 2016/03/30 - Big Idea: Pyramidal neuron model for prediction - A single layer network model for sequence memory - Properties of sparse activations “A Theory of How Columns in the Neocortex Learn the Structure of the World” Hawkins, Ahmad, and Cui, Frontiers in Neural Circuits, 2017/10/25 - Extension of sequence memory model - Big Idea: Columns compute “allocentric” location of input - By moving sensor, columns learn models of complete objects
- 7. Proximal synapses: Cause somatic spikes Define classic receptive field of neuron Distal synapses: Cause dendritic spikes Put the cell into a depolarized, or “predictive” state Depolarized neurons fire sooner, inhibiting nearby neurons. A neuron can predict its activity in hundreds of unique contexts. 5K to 30K excitatory synapses - 10% proximal - 90% distal Distal dendrites are pattern detectors - 8-15 co-active, co-located synapses generate dendritic spike - sustained depolarization of soma HTM Neuron Model Prediction Starts in the Neuron Pyramidal Neuron Major, Larkum and Schiller 2013
- 8. Properties of Sparse Activations L2 L3a L3b L4 L6a L6b L6 ip L6 mp L6 bp L5 tt L5 cc L5 cc-ns Example: One layer of cells, 5,000 neurons, 2% (100) active 1) Representational capacity is virtually unlimited (5,000 choose 100) = 3x10211 2) Randomly chosen representations have minimal overlap 3) A neuron can robustly recognize an activation pattern by forming 10 to 20 synapses 4) Unions of patterns do not cause errors in recognition Hypothesis: Cellular layers use unions to represent uncertainty Hawkins, Ahmad, 2016 Ahmad, Hawkins, 2015 Pattern 1 (100 active cells) Cell robustly recognizes pattern1 by forming synapses to small sub- sample of active cells Union Patterns 1-10 (1,000 active cells) Cell still robustly recognizes pattern 1
- 9. A Single Layer Network Model for Sequence Memory - Neurons in a mini-column learn same FF receptive field. - Neurons forms distal connections to nearby cells. No prediction Predicted input (Hawkins & Ahmad, 2016) (Cui et al, 2016) - High capacity (learns up to 1M transitions) - Learns high-order sequences: “ABCD” vs “XBCY” - Makes simultaneous predictions: “BC…” predicts “D” and “Y” - Extremely robust (tolerant to 40% noise and faults) - Learning is unsupervised, continuous, and local - Satisfies many biological constraints - Multiple open source implementations (some commercial) t=0 t=1 Predicted cells fire first and inhibit neighbors Next prediction t=2 t=0 t=1
- 10. 1) How does the cortex learn predictive models of extrinsic sequences? 2) How does the cortex learn predictive models of sensorimotor sequences? Current research: How do columns compute allocentric location? - Grid cells in entorhinal cortex solve a similar problem - Hypothesis: cortical columns contain analogs of grid cells and head direction cells - Starting to understand the function of numerous layers and connections “Why Neurons Have Thousands of Synapses, a Theory of Sequence Memory in the Neocortex” Hawkins and Ahmad, Frontiers in Neural Circuits, 2016/03/30 - Pyramidal neuron model - A single layer network model for sequence memory - Properties of sparse activations “A Theory of How Columns in the Neocortex Learn the Structure of the World” Hawkins, Ahmad, and Cui, Frontiers in Neural Circuits, 2017/10/25 - Extension of sequence memory model - Big Idea: Columns compute “allocentric” location of input - By moving sensor, columns learn models of complete objects
- 11. How Could a Layer of Neurons Learn a Predictive Model of Sensorimotor Sequences? Sequence memory Sensorimotor sequences SensorMotor-related context Hypothesis: By adding motor-related context, a cellular layer can predict its input as the sensor moves. What is the correct motor-related context? L2 L3a L3b L4 L6a L6b L6 ip L6 mp L6 bp L5 tt L5 cc L5 cc-ns 50% Sensory feature
- 13. Two Layer Model of Sensorimotor Sequence Memory Feature @ location Object Stable over movement of sensor With allocentric location input, a column can learn models of complete objects by sensing different locations on object over time. Sensor Feature Allocentric Location Pooling Seq Mem Changes with each movement
- 14. Object Feature @ Location Location on object Column 1 Column 2 Column 3 Sensor feature Sensorimotor Inference With Multiple Columns Each column has partial knowledge of object. Long range connections in object layer allow columns to vote. Inference is much faster with multiple columns.
- 15. FeatureFeatureFeatureLocationLocationLocation Output Input Objects Recognized By Integrating Inputs Over Time
- 16. FeatureLocationFeatureLocationFeatureLocation Column 1 Column 2 Column 3 Output Input Recognition is Faster with Multiple Columns
- 17. Yale-CMU-Berkeley (YCB) Object Benchmark (Calli et al, 2017) - 80 objects designed for robotics grasping tasks - Includes high-resolution 3D CAD files YCB Object Benchmark We created a virtual hand using the Unity game engine Curvature based sensor on each fingertip 4096 neurons per layer per column 98.7% recall accuracy (77/78 uniquely classified) Convergence time depends on object, sequence of sensations, number of fingers. Simulation using YCB Object Benchmark
- 18. Pairwise confusion between objects after 1 touch Convergence 1 finger 1 touch
- 19. Pairwise confusion between objects after 2 touches Convergence 1 finger 2 touches
- 20. Pairwise confusion between objects after 6 touches Convergence 1 finger 6 touches
- 21. Pairwise confusion between objects after 10 touches Convergence 1 finger 10 touches
- 22. Convergence Time vs. Number of Columns This is why we can infer complex objects in a single grasp or single visual fixation.
- 23. 1) How does the cortex learn predictive models of extrinsic sequences? 2) How does the cortex learn predictive models of sensorimotor sequences? Current research: How do columns compute allocentric location? - Hypothesis: cortical columns contain analogs of grid cells and head direction cells - Starting to understand the function of numerous layers and connections “Why Neurons Have Thousands of Synapses, a Theory of Sequence Memory in the Neocortex” Hawkins and Ahmad, Frontiers in Neural Circuits, 2016/03/30 - Pyramidal neuron model - A single layer network model for sequence memory - Properties of sparse activations “A Theory of How Columns in the Neocortex Learn the Structure of the World” Hawkins, Ahmad, and Cui, Frontiers in Neural Circuits, 2017/10/25 - Extension of sequence memory model - Big Idea: Columns compute “allocentric” location of input - By moving sensor, columns learn models of complete objects
- 24. Entorhinal Cortex environments A B C X Y Z R S T Room 3 Room 2Room 1 Location - Encoded by Grid Cells - Unique to location in room AND room - Location is updated by movement Orientation (of head to room) - Encoded by Head Direction Cells - Anchored to room - Orientation is updated by movement Location - Unique to location on object AND object - Location is updated by movement Orientation (of sensor patch to object) - Anchored to object - Orientation is updated by movement Cortical Column objects Hypothesis: Cortical columns contain analogs of grid cells and head direction cells A C B X Y Z Stensola, Solstad, Frøland, Moser, Moser: 2012 Location and Orientation are both necessary to learn the structure of rooms and predict sensory input. Location and Orientation are both necessary to learn the structure of objects and predict sensory input.
- 25. L3 L4 L6a L6b L5a L5b Mapping Orientation and Location to a Cortical Column (most complex slide) Sensation Orientation 1) A column is a two-stage sensorimotor model for learning and inferring structure. 2) A column usually cannot infer a Feature or Object in one sensation. - Integrate over time (sense, move, sense, move, sense..) - Vote with neighboring columns 3) This system is most obvious for touch, but it applies to vision and other sensory modalities. Because this architecture exists throughout the neocortex, it suggests we learn, infer, and manipulate abstract concepts the same way we manipulate objects in the world. Location Sensation @ Orientation Feature Feature @ Location Object Motor updated (HD cell-like) Motor updated (grid cell-like) Seq mem Pooling Seq mem Pooling Meaning Operation
- 26. Rethinking Hierarchy Every column learns complete models of objects. They operate in parallel. Inputs project to multiple levels at once. Columns operate at different scales of input. Sense Simple features Complex features Objects Classic Objects Objects Objects Sensor array Proposed Region 3 Region 2 Region 1
- 27. Rethinking Hierarchy Every column learns complete models of objects. They operate in parallel. Inputs project to multiple levels at once. Columns operate at different scales of input. Non-hierarchical connections allow columns to vote on shared elements such as “object” and “feature”. Sense Simple features Complex features Objects Classic Sensor array Objects Objects Objects Sensor array vision touch Proposed Region 3 Region 2 Region 1
- 28. Summary Goal: Understand the function and operation of the laminar circuits in the neocortex. Method: Study how cortical columns make predictions of their inputs. Proposals 1) Pyramidal neurons are the substrate of prediction. Each neuron predicts its activity in hundreds of contexts. 2) A single layer of neurons forms a predictive memory of high-order sequences. (sparse activations, mini-columns, fast inhibition, and lateral connections) 3) A two-layer network forms a predictive memory of sensorimotor sequences. (add motor-derived context and a pooling layer) 4) Columns need motor-derived representations of location and orientation, of the sensor relative to the object. These are analogous to grid and head direction cells. 5) A framework for the cortical column. - Columns learn complete models of objects as “features at locations”, using two sensorimotor inference stages. 6) The neocortex contains thousands of parallel models, that resolve uncertainty by associative linking and/or movement of the sensors.
- 29. Open Issues Behaviors: how are they learned, encoded, and applied to objects? Detailed model of hierarchy including thalamus How can the model be applied to “Where” pathways, and how do “What” and “Where” pathways work together Collaborations There are many testable predictions in this model, a “green field”. We welcome collaborations and discussions. We are always interested in hosting visiting scholars and interns.
- 30. Numenta Team Subutai Ahmad VP Research Marcus Lewis Thank You