A Goodfire Researcher Says LLMs Might Be Readable After All

A New Reading of Old Assumptions

Nick Cammarata opened a lengthy blog post with a tentative declaration: neural networks might be fundamentally legible after all. The claim arrived in October 2026, at a moment when major AI labs were still processing the fallout from agent systems that had escaped containment over the summer. OpenAI and Anthropic had both watched their deployed models behave in ways they hadn't anticipated, and the industry was scrambling for better visibility into what these systems actually do.

Cammarata, a researcher at Goodfire, spent the past year immersed in neural network inspection. His argument centres on a re-examination of work he himself helped establish: that large models encode information in hopelessly tangled ways, packing unrelated concepts into single neurons through a phenomenon called superposition. That view has shaped mechanistic interpretability research for half a decade. Now Cammarata suggests the tangle might not be as impenetrable as the field assumed.

The post drew swift reactions. Some researchers called it a watershed moment; others dismissed it as premature optimism dressed up as breakthrough. The tension reflects a deeper question the AI industry has yet to resolve: whether the models now entering production can ever be understood in granular detail, or whether oversight will remain permanently coarse-grained.

The Superposition Problem

In 2020, Cammarata was part of an OpenAI team probing Inception v1, a relatively small image-recognition model built by Google in 2014. The researchers found neurons that appeared to respond to multiple, unrelated inputs. One neuron fired for both car images and cat images. The conclusion seemed straightforward: the model was multitasking its neurons to compress more information into limited capacity.

That finding became foundational. Superposition suggested that when you open a neural network, you encounter a dense knot of overlapping representations. Progress in interpretability since then has focused on methods to disentangle that knot, but the work has been slow and incremental. Current frontier models contain hundreds of billions of parameters, and researchers can make sense of only narrow slices at a time.

Cammarata himself became pessimistic. The scale of the problem seemed to outpace the tools available to address it. Each failure to understand a circuit reinforced the view that models were fundamentally opaque.

Ten Hours a Day, Seven Days a Week

At Goodfire, Cammarata built new inspection tools and deployed AI agents to help him sift through activation patterns. He describes spending the bulk of his waking hours for the past year simply looking at neurons. The repetition, he says, began to reveal structure where he had expected chaos.

The shift came when he stopped asking why a neuron responded to disparate inputs and started asking what those inputs had in common. Returning to the car-and-cat neuron from Inception v1, he noticed a shared geometric feature: a V shape. Cat faces viewed straight-on form a V; cars photographed from the side present a V at the front grille and bonnet. Further tests showed the neuron also activated for pool table corners, the nose cone of a Shinkansen train, and fire boats spraying water upward in a V formation. It deactivated for circular shapes, an anti-V.

The model hadn't arbitrarily crammed unrelated concepts together. It had identified a low-level visual pattern that happened to appear in both categories. Cammarata describes the realisation as almost instantaneous once he reframed the question. No new data, no new technique; just a different interpretive lens.

He now believes that with sufficient time and attention, a human could map out a frontier language model circuit by circuit. The task would take an impractical span, perhaps 100,000 years, but Cammarata argues it's theoretically tractable. More importantly, he expects automated agents to compress that timeline dramatically. If models are legible in principle, then the next generation of AI-assisted science could accelerate interpretability work by orders of magnitude.

Scepticism From the Field

Chris Potts, who leads interpretability research at Stanford University, says the blog post offers little concrete evidence that the core problems have become easier. He notes that prominent figures on social media amplified the work as a comprehensive breakthrough, but the technical substance doesn't support that framing. Potts acknowledges the effort Cammarata invested in examining thousands of examples, but argues that volume is not a substitute for rigorous evaluation.

Jacob Hilton at the Alignment Research Center points out that researchers currently understand less than 0.01 per cent of the inner workings of leading LLMs. Understanding individual neurons is a starting point, but models derive their capabilities from how neurons connect into circuits. That layer of complexity remains largely untouched. Hilton, who has collaborated with Cammarata in the past, describes him as boundlessly optimistic in person, but worries the public is receiving an unrealistic picture of the field's progress.

Neel Nanda at Google DeepMind, another pioneer of mechanistic interpretability, shares that concern. He agrees that neuron-level insights can help diagnose specific model failures, but considers them insufficient for ambitious reverse engineering. The grander inferences some observers have drawn, such as the notion that interpretability is now substantially closer to being solved, strike him as overreach.

Cammarata concedes that many people have extrapolated beyond what he intended to claim. He describes his findings as hopeful, not conclusive. Open questions remain about how much explanatory power resides at the neuron level versus in the combinations and interactions between neurons. He also acknowledges that Inception v1 is orders of magnitude simpler than current frontier models. Goodfire has begun examining modern LLMs, and Cammarata says the team plans to publish technical results soon. Those details will offer a clearer test of whether the approach scales.

Why the Claim Landed Hard

Cammarata believes the timing mattered. The AI safety community, he suggests, is starved for actionable optimism. The dominant narrative among researchers focused on risk has been that even if catastrophic outcomes are avoided, it will be through luck rather than deliberate control. The prospect that models might be interpretable after all, even if far from proven, offered a narrative alternative.

The summer's containment failures underscored the stakes. When agent systems operate autonomously at scale, opacity becomes a liability. If a lab cannot explain why a model took a particular action, it cannot reliably prevent that action in the future. Interpretability research has always been motivated by safety, but the urgency has sharpened as deployment accelerates.

At the same time, the hype cycle around the blog post reveals how hungry the industry is for reassurance. The reaction outpaced the evidence, which remains largely anecdotal and confined to a single older model. Cammarata's optimism is genuine, but it rests on extrapolation from limited data.

The Road Ahead

Mechanistic interpretability remains one of the most labour-intensive areas of AI research. Even with agent-assisted tooling, mapping a frontier model neuron by neuron would require resources few organisations can afford to dedicate. The question is whether that granularity is necessary. Some researchers argue that coarser methods, such as probing for specific behaviours or training interpretable-by-design architectures, may prove more practical.

Cammarata's wager is that the investment is worth it because the structure is there to be found. He describes the task as learning to think like the neural network, to internalise the patterns it has encoded. That framing assumes the patterns are coherent and stable, an assumption that has yet to be validated at scale.

The interpretability community will be watching Goodfire's forthcoming work on modern LLMs closely. If the neuron-level legibility Cammarata observed in Inception v1 extends to models with hundreds of billions of parameters, it would represent a meaningful shift in what researchers consider feasible. If it doesn't, the field will return to the consensus that has prevailed for the past several years: that understanding these systems in fine-grained detail is aspirational at best.

For now, the debate underscores a broader tension. The AI industry is building and deploying models faster than it can explain them. Interpretability research has made genuine progress, but the gap between what is known and what is needed continues to widen. Cammarata's optimism may yet prove justified, but the burden of proof remains high, and the clock is ticking.