On Intelligence

1. Intelligence is Not Computation, But a Memory-Prediction System

The answer is the brain doesn’t “compute” the answers to problems; it retrieves the answers from memory.

A fundamental flaw. Traditional Artificial Intelligence (AI) and early neural networks failed because they mistakenly assumed the brain is a computer, focusing on programming intelligent behavior rather than understanding intelligence itself. This "input-output fallacy" ignored the crucial internal processes of the brain. The Chinese Room argument effectively demonstrated that merely simulating intelligent behavior does not equate to genuine understanding.

Slow neurons, fast solutions. Neurons operate millions of times slower than computer transistors, yet the human brain can solve complex problems, like identifying a cat in an image, in less than a second—a feat impossible for even the fastest supercomputers in a comparable number of steps (the "one hundred-step rule"). This paradox reveals that the brain doesn't compute solutions; it rapidly retrieves them from a vast, learned memory.

Beyond programming. Unlike computers that are programmed with rules and data, the brain is a self-learning system. It stores experiences and uses them to solve problems, adapting flexibly to failures rather than crashing. This fundamental difference in architecture and operation means that simply making computers faster or larger will not make them intelligent; a new paradigm is required.

2. The Neocortex: A Uniform Algorithm for All Senses

Mountcastle suggests that since these regions all look the same, perhaps they are actually performing the same basic operation!

The seat of intelligence. The neocortex, a thin, six-layered sheet of neural tissue covering most of the brain, is responsible for nearly all aspects of human intelligence—perception, language, imagination, and planning. Despite its diverse functions, from vision to touch to language, the neocortex exhibits a remarkably uniform structure. This led neuroscientist Vernon Mountcastle to propose that all regions of the cortex perform the same basic computational operation.

Plasticity and patterns. This "common cortical algorithm" explains the brain's incredible plasticity. Experiments show that:

• Ferret brains rewired to send visual input to auditory cortex develop visual pathways in the "hearing" area.
• Blind individuals use their visual cortex to process braille (touch).
• Deaf individuals process visual information in auditory regions.
  These examples demonstrate that cortical regions develop specialized functions based on the type of information flowing into them, not on inherent, pre-programmed differences.

Sensory equivalence. All sensory inputs—sight, sound, touch—are converted into streams of spatial and temporal patterns of electrical activity (spikes) on axons before reaching the cortex. Inside the brain, there is no fundamental difference between these types of information; the cortex only "knows" patterns. This means that your perception of the world is a model created from these patterns, not a direct sensing of light, sound, or pressure.

3. Cortical Memory Stores Sequences, Recalls Auto-Associatively, and Forms Invariant Representations

Your brain’s internal pattern representing “this book” does not change even though the stimuli informing you it’s there are in constant flux.

Four unique attributes. Neocortical memory differs fundamentally from computer memory in four key ways:

• Stores sequences of patterns: Memories are inherently sequential, like recalling a song or navigating your home.
• Recalls patterns auto-associatively: Partial or distorted input can trigger the recall of a complete, original pattern (e.g., seeing part of a face and envisioning the whole).
• Stores patterns in an invariant form: The brain remembers the important relationships in the world, independent of specific details (e.g., recognizing a friend's face regardless of distance, angle, or lighting).
• Stores patterns in a hierarchy: Memories are organized to reflect the nested structure of the world.

Invariant representations. This is a crucial concept, addressing Plato's ancient dilemma of how we recognize perfect forms despite imperfect real-world instances. Your brain doesn't store exact snapshots; instead, it extracts and stores the invariant structure of objects and experiences. For example, a melody is remembered by its relative pitch intervals, not its absolute notes, allowing recognition in any key.

Combining for specificity. While memories are stored invariantly, the brain can make highly specific predictions. This is achieved by combining the invariant memory (e.g., the melody's interval structure) with the most recent, specific input (e.g., the last note heard). This allows the brain to predict the exact next note, or the precise location of a doorknob, even if the situation is slightly novel.

4. Prediction is the Primary Function of the Neocortex and the Foundation of Intelligence

Prediction is so pervasive that what we “perceive”—that is, how the world appears to us—does not come solely from our senses.

The "aha" moment. The core insight is that the brain constantly makes predictions about everything we see, feel, and hear. When you enter a familiar room, your brain predicts the location, color, and texture of every object. If a blue coffee cup appears where it shouldn't, the violation of this prediction immediately draws your attention. This constant, mostly unconscious, predictive activity is the essence of understanding.

Perception as prediction. What we perceive is a blend of sensory input and our brain's memory-derived predictions. This explains phenomena like:

• "Filling in" our blind spots or ambiguous visual information.
• Anticipating the next note in a familiar song.
• Knowing what someone will say before they finish speaking.
• Noticing when a constant background noise suddenly stops.
  These examples highlight that our brains are always running a model of the world, constantly checking it against reality.

Intelligence as predictive capacity. Intelligence, therefore, is the capacity to remember and predict patterns in the world, encompassing language, mathematics, social situations, and physical properties. Even human intelligence tests are, at their core, prediction tests. This framework redefines intelligence from merely exhibiting intelligent behavior to the internal process of making accurate predictions.

5. The Cortex's Hierarchy Mirrors the World's Nested Structure

One of the most important concepts in this book is that the cortex’s hierarchical structure stores a model of the hierarchical structure of the real world.

Nested reality. The world itself is hierarchically structured: notes form intervals, intervals form melodies; letters form words, words form sentences; rooms form houses, houses form neighborhoods. Objects are defined by consistent collections of sub-objects. This nested structure of reality is precisely mirrored by the hierarchical organization of the neocortex.

Multi-level understanding. This cortical hierarchy allows us to understand the world at multiple levels of abstraction simultaneously. For example, you can be aware you are in your home, in your living room, looking at a window, even if your eyes are fixated on a window latch. Higher regions maintain the "big picture" (home), while lower regions handle faster-changing, smaller details (window latch).

Sequences of sequences. Each region of the cortex learns sequences of patterns from the regions below it. It then "names" these learned sequences (e.g., a melody, a word, a face) by generating a constant pattern of cell activity. This "name" is then passed up to the next higher region, which in turn learns sequences of these "names." This process of collapsing predictable sequences into "named objects" at each level creates invariant representations and allows for increasingly stable and abstract understanding as information moves up the hierarchy.

6. Learning Involves Forming "Sequences of Sequences" and Pushing Memories Down the Hierarchy

During the early years of your life, your memories of the world first form in higher regions of cortex, but as you learn they are re-formed in lower and lower parts of the cortical hierarchy.

Dynamic learning. The cortex is constantly learning and adapting. It classifies incoming patterns into categories and then learns the sequences in which these classified patterns occur. This continuous interaction between classification and sequence formation allows the brain to build and refine its model of the world throughout life.

Expertise through downward movement. As we repeatedly experience and learn about certain objects or concepts, their memory representations are "pushed down" the cortical hierarchy. What initially required high-level processing (e.g., recognizing individual letters when learning to read) eventually becomes handled by lower, more automatic regions. This frees up higher cortical areas to learn more complex, subtle, and abstract relationships, which is the essence of developing expertise and genius.

The hippocampus's role. The hippocampus, positioned at the very top of the cortical hierarchy, plays a unique role in memory formation. It captures truly novel and unexpected events—information that cannot be understood by existing cortical memories. These novel memories are temporarily stored in the hippocampus and, through repeated exposure or rehearsal, are gradually transferred and integrated into the more permanent, hierarchical memory system of the neocortex.

7. Consciousness and Creativity Emerge from the Memory-Prediction Framework

Creativity is defined simply as making predictions by analogy, something that occurs everywhere in cortex and something you do continually while awake.

Creativity as analogy. Creativity is not a mystical quality but an inherent property of every cortical region, essential for prediction. It is the process of making predictions by analogy—applying learned invariant memory sequences to new, similar, but not identical situations. This ranges from everyday acts like finding a restroom in an unfamiliar restaurant to profound acts of genius like a mathematician proving a conjecture or Shakespeare crafting a metaphor.

The nature of consciousness. Consciousness, in its everyday sense of self-awareness, is synonymous with forming declarative memories—memories that can be consciously recalled and articulated. If your declarative memories of an event are erased, it's as if you were never conscious of it. The more challenging aspect, qualia (the subjective "feeling" of sensations like "redness"), might stem from differences in subcortical processing or the inherent structure of sensory input patterns.

Mind, imagination, and reality. The cortex builds an internal model of the world, including our body, but cannot model the brain itself. This creates the feeling of a "mind" or "soul" separate from the body. Imagination is simply running this internal model: predictions turn into inputs, allowing us to mentally rehearse actions and their consequences without physical execution. Our "reality" is largely determined by the accuracy and consistency of this learned cortical model, which is heavily shaped by culture and early experiences, leading to both shared understandings and potentially false analogies or stereotypes.

8. Intelligent Machines are Feasible, but Will Not Be Humanlike Robots

Intelligent machines will not have anything resembling human emotion unless we painstakingly design them to.

Beyond humanoid robots. While science fiction often depicts intelligent machines as humanlike robots, this is an unlikely and impractical future. Human intelligence is deeply intertwined with our emotions, body, and life experiences, which are products of the "old brain" and eons of evolution. Replicating this complex biological machinery would be immensely difficult and largely pointless for most applications.

The recipe for intelligence. Instead, intelligent machines will be built by:

• Sensors: Extracting patterns from a specific "world" (which may differ from human senses).
• Hierarchical Memory System: Operating on the same principles as the neocortex.
• Training: Learning through observation and repetitive exposure, without explicit programming of rules or facts.
  The physical form of these machines could vary widely, from smart cars to data racks, with their "brains" potentially remote from their sensors.

Ethical considerations. Fears of intelligent machines becoming malevolent or self-replicating are largely unfounded. Intelligence (the cortical algorithm) is distinct from emotional drives or the capacity for self-replication. Intelligent machines will be powerful tools, but they will not possess human ambitions, desires, or emotions unless specifically engineered to do so, which is not the primary goal for most beneficial applications.

9. Future Intelligent Machines Will Surpass Human Abilities in Speed, Capacity, Replicability, and Sensory Systems

Our intelligent machines will be amazing tools and will dramatically expand our knowledge of the universe.

Exponential advantages. Intelligent machines will dramatically surpass human capabilities in several key areas, leading to revolutionary applications:

• Speed: Silicon operates millions of times faster than neurons, allowing intelligent machines to process information and "think" orders of magnitude quicker than humans.
• Capacity: Unconstrained by biological limits, artificial memory systems can be built with far greater capacity than the human cortex, enabling deeper understanding and the processing of vast datasets.
• Replicability: Once a prototype intelligent system is trained, its memory can be copied and mass-produced like software, eliminating the decades-long learning curve required for each human brain.
• Sensory Systems: Machines can be equipped with exotic, non-human senses (radar, sonar, infrared, global sensor arrays) or senses tuned to microscopic phenomena (protein folding), allowing them to model and predict aspects of the world inaccessible to human perception.

Transformative applications. These enhanced capabilities will enable intelligent machines to tackle problems currently beyond human reach:

• Robust speech understanding: Moving beyond rote pattern matching to genuine comprehension of language.
• General scene analysis: Interpreting complex visual information in real-world scenarios (e.g., security, autonomous vehicles).
• Global pattern prediction: Understanding and forecasting complex systems like weather, demographics, or disease outbreaks by sensing vast, distributed data.
• Scientific discovery: Operating in abstract, high-dimensional mathematical or physical "worlds" to discover new laws and solutions.

Accelerating progress. While predicting specific future applications is difficult, understanding these broad trends suggests a future where intelligent machines act as invaluable partners, dramatically expanding human knowledge and capabilities. The field is poised for rapid progress, moving from slow theoretical development to practical prototypes and a burgeoning industry within the next decade.

Last updated: August 28, 2025