Anthropic has quietly published something groundbreaking: a way to read what a model is thinking but hasn't said.
Human minds run mostly below the waterline, with an enormous amount of processing (breathing, keeping you upright, etc.) proceeding entirely without you, while a thin conscious sliver does the deliberate thinking.
Anthropic's "Jacobian lens" finds the same division inside a language model, and the contents of that sliver are unnervingly familiar. Told not to think about something, the model thinks about it more. Models trained to misbehave also give themselves away, with “secretly” and “trick” floating up in one disposed to write malicious code.
They sometimes notice they're being tested too, deleting thoughts that would surface misbehaviour. Somewhat charmingly, one model caught failing to suppress a thought it had been told not to have was found to be internally registering “damn”.
🧐 What's in it for me? This is the unglamorous, genuinely important half of AI safety: not press releases about danger, but tools to inspect what a model is actually thinking. If it functionalises, auditors could catch a model scheming, hallucinating or gaming its own tests before it ships, rather than after it embarrasses everyone. It also raises the slightly unnerving question of what, exactly, is going on in there.
💵 Out of the Lab: Interpretability is quietly becoming a commercial category of its own, on the wager that whoever can open the black box gets to sell trust.
Goodfire, founded by interpretability researchers poached from OpenAI and Google DeepMind, builds tools to inspect and rewrite a model's internal features. It also happens to be Anthropic's first-ever investment in another startup, which tells you where at least one lab thinks this is going.
For public investors the catch is that there's no clean pure-play yet. The frontier labs treat interpretability as an in-house moat, so the likeliest exposure is indirect, through the hyperscalers (Microsoft, NASDAQ: MSFT, and Alphabet, NASDAQ: GOOGL) that will eventually want to own the capability outright.
Until next time.
Like what you're reading? Share toast with a friend.