What it is
Open-source local web app that visualizes Jacobian-lens (J-space) readouts across layers during live chat with an open model (default: Qwen3.5-4B), with exportable replay files.
Gabriel’s notes
Quick take: Subtext is a “watch the model think” instrument that pipes Jacobian-lens readouts into a live chat UI. I haven’t personally used it yet, but it looks like the kind of tool that turns interpretability from a blog post into something you can actually poke at.
Subtext is an open-source project that runs a local Python server + browser UI to visualize “verbal workspace” readouts (the Jacobian lens / J-lens technique) while a model reads your prompt and generates its reply. The repo frames this in terms of Anthropic’s J-space / global-workspace findings and uses pre-fitted lens weights (published separately) to render per-token, multi-layer readouts in real time, plus replay exports you can open later without a GPU.
I have not personally used Subtext yet, but it’s on my “save for a rainy weekend” list. The practical appeal isn’t philosophical; it’s operational: if you’re building with LLMs, seeing what concepts are “active” before they hit the output can be a useful debugging superpower (and occasionally a reality check).
I saved this under Research because it’s a rare example of mechanistic interpretability that’s (a) interactive, (b) conversational, and (c) not locked behind a closed frontier model.
Good fit if you want to:
- Sanity-check what a local model appears to be tracking internally while it’s still “reading” (prefill) vs. “speaking” (generation).
- Debug multi-step prompts by watching intermediate concepts surface before they become text.
- Record and share interpretability artifacts (session exports) with teammates or clients without handing them a GPU bill.
- Learn the Jacobian-lens idea by playing instead of squinting at diagrams.
Pricing snapshot (auto-enriched):
Subtext is free to use in the “software cost” sense: the repo is licensed under Apache License 2.0. Your real costs are compute + time: the README calls out a GPU (or Apple Silicon) as the intended path, and notes the first run downloads the model and lens files (roughly multi‑GB). If you run this on a cloud GPU, that’s on you.
Work-use / compliance snapshot (auto-enriched):
Subtext runs locally (served on localhost per the setup instructions), which is good news for sensitive prompts—as long as you keep it local. It also exports sessions as JSON replays, which means your chat content can become a portable file that should be treated like any other sensitive log (store carefully; don’t email it casually). Subtext itself is Apache‑2.0, but you’ll also be pulling in separate components with their own licenses (for example, the referenced Qwen3.5‑4B model is Apache‑2.0 on Hugging Face, and Neuronpedia’s pre-fitted jacobian-lens weights are labeled MIT on Hugging Face). Telemetry / data collection behavior is Unknown / not confirmed; if that matters for your environment, review the code before running it on client data.
Alternatives (auto-enriched):
- Neuronpedia (Jacobian Lens demos) — great for quick, hosted exploration and broader interpretability tooling; less about your own continuous local chat session.
- anthropics/jacobian-lens (reference implementation) — closer to the core research code for fitting/applying lenses; more “library + experiments” than a polished real-time conversational UI.
Before you adopt it:
- Confirm you have the right hardware path (NVIDIA GPU vs. Apple Silicon vs. CPU fallback) and that performance is acceptable for your workflow.
- Decide where exported replay JSON files will live (and who gets access) before you generate a pile of them.
- Be disciplined about interpretation: the Subtext README explicitly notes that workspace readouts are not claims of subjective experience (and the lens has known limitations around multi-token concepts).