Using Mechanistic Interpretability for Cognitive Science

Based on a guest lecture for CS 7651: Human and Machine Learning at Georgia Tech, October 12, 2026.

Neural networks can recognize objects, use language, play games, and sometimes solve problems that seem to require planning or reasoning. We have become remarkably good at measuring what they can do. We are much less certain about how they do it.

That difference is the starting point for mechanistic interpretability (MI). It is also the source of a question I have been thinking about for some time: Can studying the mechanisms of artificial neural networks teach us anything fundamental about cognition?

This post follows three questions from my lecture:

1. Why study the internal mechanisms of neural networks rather than their behavior alone?
2. What can we learn from neurons, features, circuits, representational geometry, and dynamics?
3. How can an observation about a network become a mechanistic explanation—and, potentially, a scientific insight about cognition?

I will use a few examples, from a network that learns to play Othello to a language model that plans the ending of a poem. Toward the end, I will return to the harder question: what, if anything, do discoveries about artificial computation imply about human cognition?

1. Why look inside a neural network?

Imagine two people answering the same arithmetic problem:

36 + 59 = 95.

One computes the units and tens separately, carrying the extra ten. Another notices that 59 is almost 60, computes 36 + 60, and subtracts 1. Both give the same answer. If we observed only their inputs and outputs, we could not tell which procedure they used.

The same ambiguity appears in neural networks. Good performance on a task establishes that a system can perform the task, but not which representation or algorithm it uses. Two models might achieve similar accuracy by relying on very different information and computations. Conversely, similar-looking internal components might play different roles in different systems.

This does not make behavioral testing unimportant. Behavior defines the phenomenon we want to explain, and carefully designed behavioral experiments can distinguish many hypotheses. But input–output agreement rarely identifies a unique internal mechanism. If we want to understand how a learned system computes, internal evidence gives us another set of constraints.

One motivation for studying mechanisms is practical: perhaps understanding a model will help us diagnose failures, predict generalization, or intervene more reliably. A second motivation is scientific: learned networks may provide clues about which computations can support memory, abstraction, or reasoning.

I also think there is a simpler motivation. Artificial neural networks are fascinating computational systems. We have constructed the learning procedures that produce them, but we often do not understand the resulting organization. It is reasonable to want to study them simply because they are there.

A broad view of mechanistic interpretability

When people hear mechanistic interpretability, they often think of attention-head circuits, sparse autoencoders, or the LLM-focused research tradition around reverse-engineering Transformers. Those are important examples, but I prefer a broader definition:

Mechanistic interpretability studies how neural networks implement their behavior through internal representations, transformations, and interactions.

Under this view, neuron analysis, causal interventions, matrix factorization, population geometry, and dynamical-systems analysis can all contribute to mechanistic understanding. There is no reason for MI to be restricted to language models, or to a particular class of interpretability tools.

However, there is an equally important boundary: not every interpretable observation is already a mechanism. Finding a feature that correlates with a concept, or a component whose removal hurts performance, is evidence. What that evidence establishes requires a separate argument.

This distinction motivates much of the rest of the post.

2. Five ways to look inside

An artificial neural network has many possible explanatory levels. Choosing one is not merely a matter of convenience: it is already a hypothesis about where the relevant computation is organized.

Neurons: What does a unit respond to?

A natural starting point is the individual neuron. In computer vision, Network Dissection (Bau et al., 2017) quantified whether individual convolutional units responded to recognizable visual concepts such as objects, parts, materials, or textures.

The intuition is simple. For a given image, a convolutional unit produces a spatial activation map. We threshold that map to identify where the unit responds strongly, compare it with a human-labeled segmentation mask, and quantify their overlap—for example, using intersection over union (IoU). If a unit repeatedly activates over windows, trees, or other meaningful regions, we can describe its selectivity.

A complementary technique is activation maximization. Instead of presenting the network with an ordinary image, we optimize an input image to make a selected neuron respond strongly. The interactive Feature Visualization article in Distill (Olah, Mordvintsev, & Schubert, 2017) contains some of the clearest demonstrations: moving through a visual network, one encounters patterns resembling edges, textures, parts, and increasingly complex objects.

These are useful observations. Yet a neuron that responds to trees need not be the mechanism for recognizing trees. Selectivity alone does not establish what downstream computations do with its activity.

Features: Information may not live in one neuron

Neural representations can be distributed. A concept may correspond to a direction, subspace, or pattern of activation across many neurons, rather than to one dedicated unit.

This motivates dictionary learning and sparse autoencoders (SAEs). Roughly, an SAE tries to express a high-dimensional activation vector as a sparse combination of learned feature directions:

The hope is that the learned coordinates are more interpretable than the original neuron axes. One striking example in Anthropic’s Mapping the Mind of a Large Language Model (2024) is a feature associated with the Golden Gate Bridge, activated by references to the bridge across languages and modalities.

This suggests that a meaningful concept need not coincide with an individual neuron. But it raises another question: Is the recovered feature basis uniquely privileged, or simply a useful decomposition produced by our measurement method? A readable dictionary can help us investigate a network without automatically becoming the network’s true set of atomic computational units.

Circuits: How do components work together?

Even if we know what individual components represent, we may not know how their interactions produce an answer.

A classic example is Interpretability in the Wild: A Circuit for Indirect Object Identification in GPT-2 Small (Wang et al., 2023). Consider a prompt such as:

When Mary and John went to the store, John gave a drink to …

The expected next name is Mary, not John. A high-level description might say that the model tracks the names, recognizes that John has already appeared in a relevant role, and promotes Mary as the answer. But which internal components implement these operations?

The researchers used interventions, including activation patching and path patching, to investigate contributions from particular attention heads and the interactions among them. In activation patching, we replace an internal activation from one run with the corresponding activation from another run and observe how the output changes. By targeting paths rather than just components, we can ask whether information travels through a proposed connection.

The resulting account involved multiple classes of attention heads, rather than a single “Mary neuron.” It is also an instructive reminder of the difficulty of circuit analysis: even a relatively localized behavior can involve a surprisingly elaborate internal organization.

The goal is not simply to draw a more detailed wiring diagram. It is to reconstruct a computation performed by interacting parts, and to determine how much of the model’s behavior the proposed circuit really explains.

Population geometry: How is knowledge organized?

Another approach is to treat neural activity as a population-level representation. Each input corresponds to a point in a high-dimensional activation space. Instead of asking which individual neuron detects a concept, we ask how relations among many inputs are organized geometrically.

This is especially relevant for concepts. A concept must be sufficiently stable to remain recognizable, yet flexible enough to change with context. The word apple might evoke a fruit, a company, or something very different depending on the sentence in which it appears.

In our work, Language Models Represent and Transform Concepts with Shared Geometry (Hu, Niu, & Varma, 2026), we studied contextual changes in conceptual representations as a field of displacements over a vocabulary. We found that contexts do not simply shift every concept by one identical vector: the directions and magnitudes of change differ across concepts, with semantically structured variation. Some of this organization is also reproducible across model families.

This is a different explanatory lens from individual-neuron analysis. It allows us to ask whether flexible conceptual behavior is related to systematic transformations of an underlying representational structure.

But geometry creates a familiar inferential problem. A geometric regularity is not, by itself, evidence that the network uses that geometry to produce its behavior. To establish a functional mechanism, we need to connect the structure to downstream computation, generalization, or intervention.

Dynamics: Computation can be a trajectory

Finally, some computations are best described not by a static representation but by how activity evolves over time.

In Opening the Black Box: Low-Dimensional Dynamics in High-Dimensional Recurrent Neural Networks (Sussillo & Barak, 2013), the authors studied recurrent networks trained on tasks including a three-bit flip-flop memory problem.

A three-bit memory has eight possible states. After training, the recurrent network’s high-dimensional dynamics could be interpreted using a lower-dimensional structure with stable memory states, or attractors, and unstable structures associated with transitions between them. The authors examined fixed points and the local Jacobian of the recurrent dynamics to understand where trajectories remain stable and where small perturbations can grow.

This provides a mechanistic account of memory that does not depend on finding one neuron for each bit. Memory corresponds to a stable state of a dynamical system, and changing memory corresponds to moving between those states.

The lesson from these five approaches is not that one is better than the others. They illuminate different aspects of a learned computation. The first scientific decision is to ask what exactly needs to be explained, and which level of analysis is appropriate.

3. From seeing structure to explaining a mechanism

We have methods for detecting selective units, extracting features, reconstructing circuits, measuring geometry, and analyzing dynamics. But a microscope does not supply the logic of scientific inference.

A finding inside a model might be interesting without settling the mechanism behind a behavior. To see the difference, let us follow three discoveries that ask increasingly demanding questions.

Case study I: Does a language model construct a world model?

An elegant starting point is Emergent World Representations (Li et al., ICLR 2023). The authors trained a GPT-like sequence model to predict moves in the game Othello. The model receives a sequence of moves; it is not directly given an image or symbolic encoding of the current board state.

What might the network learn? One possibility is a collection of statistical regularities about legal move sequences. Another is an internal representation of the evolving board.

The first test was to probe the hidden activations. A probe takes the model’s internal state and tries to reconstruct the state of each square on the board. Successful decoding reveals that board information is accessible in the hidden representation.

But consider the inferential gap. Imagine a computer program that stores a detailed log of its input history but never uses that log when choosing an output. We might be able to reconstruct the current board from the log, yet that would not show that the program uses a board representation in its decision-making.

Decodable does not necessarily mean used.

The next step, therefore, was an intervention. The researchers changed internal activations in a way intended to alter the represented status of a board square, while leaving the preceding move sequence unchanged. The model’s subsequent move predictions shifted in ways consistent with the modified board state.

That is a much stronger result than successful probing alone. It supports the claim that board-state information is functionally implicated in the model’s computation. It does not, however, provide a complete account of how every move is generated.

This is the central distinction between finding a representation and understanding its computational use.

For an accessible walkthrough, see Kenneth Li’s Othello-GPT explanation in The Gradient. The official Othello World repository includes training, probing, and intervention notebooks.

Case study II: Can a next-token predictor plan ahead?

Autoregressive language models generate text one token at a time. It is tempting to interpret that process as purely reactive: produce one word, inspect the partial sentence, and then produce the next.

But one-token-at-a-time output does not imply one-token-at-a-time planning.

Consider a model asked to write a rhyming poem. Two explanations are possible:

• Improvisation: The model generates most of the line and only then finds a suitable rhyme.
• Planning: The model selects a likely ending in advance, then constructs earlier text to reach that ending.

Both strategies might produce equally fluent poetry. How could we tell them apart?

In On the Biology of a Large Language Model (Lindsey et al., 2025), researchers used attribution-graph methods to investigate Claude 3.5 Haiku. In poetry examples, they identified internal features associated with possible future rhyming words before the model generated the corresponding line.

An early activation is suggestive, but not decisive. The stronger evidence came from interventions: manipulating candidate rhyme-related features at particular points could shift the later ending and influence the intervening words. This is what a planning hypothesis predicts. A feature that contributes to a future target should affect not just the final word but the construction of the line leading up to it.

Notice the difference between two experimental questions:

• Does removing this feature change the answer?
• If this feature specifies a planned ending, does changing it at the planning stage reorganize the earlier sequence toward a different ending?

The second question tests a computational role, rather than merely causal relevance.

The evidence does not establish that all language generation works this way. It supports an account of particular poetry-generation behavior in a particular studied model. The researchers also emphasize that their attribution graphs are incomplete reconstructions, not literal and exhaustive renderings of the underlying computation.

That caution matters: an intervention effect is not equivalent to a full algorithmic explanation. The value of an intervention depends on the specific prediction it distinguishes.

Case study III: How does a language model keep track of beliefs?

Now consider a simple story:

Bob pours beer into a bottle. Carla pours coffee into a cup. What is in Carla’s cup?

A model can answer “coffee.” But to do so reliably across longer stories, changing objects, and multiple characters, it must preserve relationships between who, what object, and what state. When characters see different events, the problem becomes harder: the model must also distinguish reality from what each character believes.

The key question is not merely whether a model can answer these questions. It is how it organizes and retrieves the relevant information.

Prakash et al., Language Models Use Lookbacks to Track Beliefs (ICLR 2026) studied this using controlled stories, causal analysis, and abstraction-based interventions. They identified a recurring lookback mechanism: information linking characters, objects, and states is encoded through reference-like signals, and subsequent operations retrieve the relevant state and answer when queried. Stories involving who could observe whom required additional processing related to visibility.

The researchers’ account matters because the explanation is relational. It is not enough to find a neuron associated with “Carla,” another with “cup,” and another with “coffee.” We need to know how the model binds the right pieces together and how one operation supplies information to the next.

To illustrate the intuition, imagine a deliberately simplified algorithm:

def answer(story, question):
    state_pointer = find_relevant_state(story, question)
    return retrieve_value(story, state_pointer)

The example is conceptual pseudocode, not a literal implementation recovered in the paper. Its purpose is to separate two computational roles: selecting the right state and retrieving the value stored there. Interventions that selectively disrupt these roles can test whether a proposed organization explains the model’s outputs.

This moves us beyond a collection of causally important components toward a model of how those components work together. It also illustrates a critical boundary: identifying a belief-tracking algorithm inside an LLM is not the same as demonstrating that the system possesses human-like Theory of Mind, or that humans use the same algorithm.

The project website provides an accessible overview, and the research code contains the authors’ experiments.

4. A logic of mechanistic discovery

What connects the examples above? They use different models and methods, but they all face the same problem: How do we turn evidence about internal activity into a justified claim about computation?

This question motivated a perspective I developed with collaborators, Mechanistic Interpretability Needs a Logic of Inference (Hu, Jiang, Xie, Ratan Murty, & Varma, 2026). We proposed five recurring principles, underpinned by a general principle about measurement.

P0 — Measurement: At every step, ask how the result depends on the probe, metric, intervention, corruption distribution, or other operational choices. Measurement is part of the evidence, not a transparent window onto a preexisting answer.

These are not five boxes to tick in order. They are a recursive research logic. A circuit analysis may reveal that the natural explanatory object is actually a lower-dimensional algorithm. A geometric result may suggest a representation, but intervention may reveal that the relevant information is not used. A mechanism that appears stable on one class of inputs may fail on another.

The point is not to demand a complete reverse-engineering of every neural network. It is to make claims proportionate to evidence—and to design the next experiment around what remains uncertain.

5. What do ANN mechanisms tell us about cognition?

So far, most examples establish facts about artificial systems. These can be fascinating and scientifically valuable in their own right. But the title of this lecture asks a more demanding question: How can interpreting ANNs contribute to cognitive science?

I find it useful to distinguish three contributions.

5.1 Computational existence proofs

A neural network can demonstrate that a cognitive ability is computationally possible without a mechanism we had assumed was necessary.

For example, a network that learns to maintain an internal game state from sequences alone challenges the intuition that a world representation must be explicitly supplied in advance. A next-token generator that sometimes plans future wording suggests that sequential output need not correspond to purely myopic internal computation.

These results enlarge the set of plausible computational explanations. They do not establish which explanation humans use. An existence proof shows that a mechanism can support a capability, not that it must or does support it in the brain.

5.2 Constraints on cognitive theories

A stronger contribution may come from understanding which representations and algorithms naturally emerge under particular learning conditions.

Suppose a cognitive theory proposes that flexible conceptual behavior requires an explicit controller that selects a new representation for every context. But suppose networks repeatedly acquire structured, context-dependent transformations over a shared conceptual space, and we can demonstrate that this organization is sufficient for flexible generalization. That finding would not prove that human concepts work the same way. It would, however, challenge a claim about the computational requirements of conceptual flexibility.

To make this argument convincing, we need more than a resemblance between representational plots. We need to identify the conditions that generate the organization, show how it supports a function, and test when the proposed explanation ceases to work.

In this sense, the most interesting question may not be simply what mechanism does this model use?, but rather: Why do learning systems discover this sort of mechanism, and what capabilities does it make possible?

5.3 Brain–model alignment is not mechanistic understanding

There is also a more direct route through NeuroAI. We can compare neural-network representations with brain activity, use models to predict neural responses, or train models under constraints derived from neural data. Such work can provide useful in silico models of aspects of brain computation.

Yet good brain–model alignment does not settle the computational mechanism. Two models might predict similar neural measurements while implementing different internal algorithms. A representation that aligns well with a brain region may tell us what information is shared without explaining how that information is transformed or used.

Mechanistic analysis can help formulate more precise hypotheses about these predictive models. But explaining an aligned ANN is still not, by itself, proof that the same mechanism is implemented in the brain. That stronger inference requires evidence that discriminates between possible biological mechanisms.

This leaves an important complementary pair of questions:

• Correspondence: Does the artificial system capture relevant properties of neural or behavioral data?
• Mechanism: What computation explains those properties, and does that account transfer across systems?

Neither question replaces the other.

6. A particularly instructive example: The mathematics of semantic development

One of my favorite papers that connects neural-network analysis with cognitive theory is A Mathematical Theory of Semantic Development in Deep Neural Networks (Saxe, McClelland, & Ganguli, PNAS 2019).

The authors begin with familiar phenomena of semantic development: broad conceptual distinctions often emerge before fine ones; children can temporarily make systematic semantic errors; category typicality affects judgments; and inductive generalization tends to become more specific with development.

Could a common computational principle account for several of these findings?

From experience to statistical structure

Imagine a learning environment containing objects and their properties. A canary can fly; a salmon can swim; an oak has bark. A network receives an item as input and learns to predict its properties.

The structure of experience can be summarized by an input–output cross-correlation matrix, which the authors decompose using singular value decomposition (SVD):

Different singular modes correspond to statistical distinctions within the domain. In a hierarchically structured environment, a strong mode might distinguish animals from plants, while weaker modes distinguish birds from fish or trees from flowers. The singular vectors characterize which items and properties participate in each distinction; the singular values characterize their statistical strengths.

At this point, we have a description of the data. We have not yet explained learning or cognition.

Why learning dynamics matter

The crucial step is to analyze a deep linear network trained on those associations. A deep linear network has no more input–output expressive power than a single linear layer. Yet its parameterized learning dynamics are different.

Under the assumptions analyzed in the paper, the learned map can be written as

where the diagonal entries of $A(t)$ describe how much each statistical mode has been learned at time $t$. In the relevant regime, stronger modes are acquired earlier and their learning trajectories can show relatively sharp, stage-like transitions.

The distinction is important: SVD alone reveals the available structure; the learning dynamics predicts when that structure becomes part of the model’s computation. Under idealized conditions, the authors can derive these trajectories analytically rather than relying only on numerical training.

That connection between data statistics and learning dynamics generates several cognitive phenomena:

• Hierarchical differentiation. Broad distinctions emerge before finer ones because the corresponding modes are learned at different rates.
• Semantic illusions. An early-learned mode may temporarily support a mistaken property inference before later modes correct it. The network can make a systematic error even though the training environment does not contain that false fact.
• Typicality and category coherence. The geometry of learned semantic modes provides mathematical characterizations of representative category members and of which category structures are easiest to learn.
• Changing inductive generalization. Early representations are less differentiated, so a newly learned property can generalize broadly. Later representations distinguish items more finely, leading to narrower generalization.

This is what makes the paper instructive. It does not simply report that a network contains a semantic hierarchy. It shows how environmental statistics, a particular learning mechanism, the geometry of internal representations, and observable behavior can be connected within one theoretical account.

A question worth asking: Why train a network if SVD explains so much?

There is an apparent objection. If many semantic distinctions and measures of typicality already exist in the SVD of the item–property matrix, why bother with the network?

For static structure, this objection has force: much of what the network ultimately represents is inherited from the training data’s statistics.

The additional contribution of the learning analysis is time and mechanism. The static decomposition does not, by itself, predict a sequence of developmental transitions or the temporary rise and correction of erroneous beliefs. The learning dynamics explains how modes are acquired at different rates and how their changing contributions alter behavior.

This is also why the theory has informative limits. It relies on simplified networks and specific assumptions about learning and data. It offers an elegant sufficient computational explanation of several phenomena, not a demonstration that human brains literally perform this SVD or implement the same learning equations.

Moreover, this is a mathematical cognitive-modeling study, not a case in which researchers first reverse-engineered an unrelated pretrained AI system and then discovered a new fact about human cognition. I include it because it shows a standard of explanatory integration that I would like mechanistic studies of modern ANNs to approach: one coherent mechanism connecting multiple phenomena, rather than an ever-expanding catalog of interesting internal patterns.

7. Where does this leave mechanistic interpretability?

The field has developed increasingly sophisticated ways to observe and manipulate neural networks. In some settings, we can identify meaningful features, map partial computational circuits, and intervene on internal variables with impressive specificity. Those achievements are real.

But a science of mechanisms should aspire to more than describing the contents of trained networks. Three distinctions seem particularly important to me:

Description versus explanation. A geometric regularity or interpretable feature is a finding. Explaining why it arises, which computation it supports, and what follows from it is a further step.

Local mechanisms versus general principles. A component-level explanation for one model and prompt may be correct yet highly implementation-specific. More general science asks what remains stable across tasks, learning conditions, architectures, and populations—and why.

Artificial cognition versus human cognition. A mechanism found in an ANN can establish computational possibility, suggest an alternative cognitive theory, or motivate a new experiment. It cannot automatically become a claim about the human brain.

My current view is that MI’s deepest scientific opportunity lies in moving toward the principles of learned computation: under what circumstances do learning systems develop particular representations and algorithms, what functions do those structures support, and which of these principles can help explain cognition beyond a single model?

That is a more demanding agenda than identifying a neuron, feature, or circuit. It is also, to me, a more compelling one.

We may eventually find that some aspects of intelligence are realized through stable, general computational principles, while others are highly contingent on architecture, data, and training. Either outcome would be scientifically interesting—provided we can formulate predictions that distinguish them.

The original question therefore remains open:

Can we use the mechanisms discovered inside artificial neural networks to build explanations of cognition that are more precise, predictive, and general than the theories we have today?

I do not think we have a complete answer yet. But studying the internal computations of learned systems gives us an unusually direct way to investigate it.

Papers, visual explanations, and code

The following resources are especially useful for readers who want to explore the examples or reproduce parts of the analysis. Figures in this post were adapted from the original lecture slides; credit for each published research figure appears in its caption.

For hands-on experiments with activation caching, patching, and interventions in language models, TransformerLens is a useful starting point. For sparse-feature analysis, see SAELens. These are research tools, not substitutes for deciding what scientific question an experiment is meant to answer.