Mechanistic Interpretability in LLMs: Deconstructing Circuits in Transformer Weights

Reverse-engineer transformer weights. Explore circuit discovery, Sparse Autoencoders, and activation patching with practical Python code on Llama 3.

Hook & Executive Summary

For years, large language models (LLMs) were treated as inscrutable, high-dimensional black boxes. We optimized loss functions, observed emergent capabilities, and quantified performance via holistic downstream benchmarks, all without understanding the explicit algorithmic graphs executing across billions of weights.

Mechanistic interpretability rejects this black-box paradigm. Treating artificial neural networks not as statistical mysteries, but as compiled assembly code, it isolates the microscopic circuits responsible for reasoning, factual recall, and syntactic parsing.

TL;DR: Modern mechanistic interpretability shifts the field from passive observation to surgical reverse-engineering. By decomposing transformer weights into $W_{QK}$ and $W_{OV}$ circuit matrices, deploying Top-K Sparse Autoencoders (SAEs) to unpack superposition, and executing causal activation patching on open-weight models like Llama 3, researchers can isolate and steer exact subgraphs governing model cognition.


The Mathematical Anatomy of Transformer Circuits

To reverse-engineer a transformer, we must first discard the layer-by-layer sequential mental model and view the architecture through the lens of the residual stream as a communication bus, originally formalized by Elhage et al. (Anthropic).

Token Input -> Embeddings -> [Residual Stream x_0]
                                |      ^
                                v      |
                            [Layer 1 Head / MLP]
                                |      ^
                                v      |
                             [Residual Stream x_1]
                                ...
                             [Residual Stream x_L] -> Unembed -> Logits

The residual stream vector $x_l \in \mathbb{R}^{d_{model}}$ at layer $l$ is the sum of the input embedding plus the linear contributions of every preceding attention head ($a_i$) and multi-layer perceptron ($m_i$):

$$x_l = x_0 + \sum_{i=1}^{l} (a_i + m_i)$$

Because attention heads and MLP layers read from and write back into this shared vector space additively, the network operates through independent, parallel computational read-write cycles.

The $W_{QK}$ and $W_{OV}$ Decompositions

In standard Multi-Head Attention (MHA) or Grouped-Query Attention (GQA), each attention head $h$ in layer $l$ performs two distinct matrix transformations:

  1. The Attention Pattern Circuit ($W_{QK}$): Determines where information flows.
    $$A^h = \text{softmax}\left(\frac{(x W_Q^h)(x W_K^h)^T}{\sqrt{d_{head}}}\right) = \text{softmax}\left(\frac{x (W_Q^h (W_K^h)^T) x^T}{\sqrt{d_{head}}}\right)$$
    The bilateral product $W_{QK}^h = W_Q^h (W_K^h)^T \in \mathbb{R}^{d_{model} \times d_{model}}$ determines the bilinear form calculating which source token tokens attend to which destination tokens.

  2. The Output-Value Circuit ($W_{OV}$): Determines what information is moved.
    $$\text{HeadOutput}^h = A^h x W_V^h W_O^h$$
    Here, $W_{OV}^h = W_V^h W_O^h \in \mathbb{R}^{d_{model} \times d_{model}}$ is a low-rank matrix that linearly transforms the source token's representation before adding it directly back into the residual stream.

MLP Layers as Key-Value Memories

While attention heads route information across sequence positions, Multi-Layer Perceptrons (MLPs) perform localized memory lookups. In modern architectures like Llama 3, which utilize SwiGLU activations, the forward pass through an MLP layer is defined as:

$$\text{MLP}(x) = \left( \text{SiLU}(x W_{gate}) \odot (x W_{up}) \right) W_{down}$$

Mechanistically, the rows of $W_{gate}$ and $W_{up}$ act as associative memory keys detecting specific concepts (e.g., "the capital of France is"), while the columns of $W_{down}$ act as values that push corresponding factual feature vectors (" Paris") into the residual stream.


Circuit Typologies: From Induction Heads to Superposition

Reverse-engineering transformer weights has revealed that complex model capabilities are built from structured sub-circuits.

1. Induction Heads

Discovered across all transformers trained on sequential data, induction heads implement the general algorithm: "If token $[A]$ was previously followed by $[B]$, and token $[A]$ appears again, predict that $[B]$ will follow."

An induction circuit requires a minimum two-layer composition:

  • Layer $L$ (Previous Token Head): Writes the identity of token $i-1$ into position $i$.
  • Layer $L+1$ (Induction Head): Attends to position $j$ where token $j-1$ matches the current token, reading out token $j$ via its $W_{OV}$ circuit to boost its probability.

This two-layer sub-network is the foundational mechanism powering in-context learning and zero-shot pattern completion.

2. Polysemanticity and the Superposition Hypothesis

Historically, interpretability researchers attempted to interpret individual neurons in $W_{down}$. They hit a wall known as polysemanticity: an individual neuron might activate for academic citations, base64 strings, and discussions of medieval history.

Why does this occur? Modern mechanistic theory attributes this to the Superposition Hypothesis. Because reality contains far more features $M$ than the model has dimensional capacity $d_{model}$ ($M \gg d_{model}$), the network packs non-orthogonal, linearly independent feature directions into a lower-dimensional space, exploiting the Johnson-Lindenstrauss lemma and sparsity.

True Concept Features (Sparse, M-dimensional) 
      \      |      /  
       v     v     v   
 [Linear Bottleneck: d_model Dimensions] -> Neurons are Polysemantic
       |     |     |
       v     v     v
 [Top-K Sparse Autoencoder (SAE)] -> Recovers Monosemantic Features

3. Resolving Superposition via Sparse Autoencoders (SAEs)

To disentangle superposition, researchers train unsupervised Sparse Autoencoders on residual stream activations. An SAE projects the dense residual stream $x \in \mathbb{R}^{d}$ into a high-dimensional sparse latent space $z \in \mathbb{R}^{D}$ (where $D \gg d$, often $16\times$ to $64\times$ expansion):

$$z = f\left(W_{enc}(x – b_{dec}) + b_{enc}\right)$$
$$\hat{x} = W_{dec} z + b_{dec}$$

Where $f$ enforces sparsity—traditionally an $L_1$ penalty, or more recently, a parameterized Top-K activation function that directly zeroes out all but the $K$ largest activations (e.g., $K=32$ or $K=64$).

The learned columns of $W_{dec}$ represent true monosemantic features: distinct mathematical vectors corresponding purely to atomic concepts such as "Python syntax errors", "deception in strategic games", or "the Golden Gate Bridge".


Hands-On Walkthrough: Activation Patching on Llama 3 8B

Activation patching (also known as causal mediation analysis or interchange intervention) allows us to determine the exact functional importance of a specific attention head or MLP layer by substituting corrupted activations into a clean forward pass.

The Setup: Indirect Object Identification (IOI)

Consider the classic task:

  • Clean Prompt: "When John and Mary went to the store, Mary gave the bag to" $\to$ Target: " John"
  • Corrupted Prompt: "When Alice and Mary went to the store, Mary gave the bag to" $\to$ Target: " Alice"

We want to identify the exact attention head responsible for copying " John" to the output position.

import torch
from transformer_lens import HookedTransformer

# 1. Load open-weights model into a mechanistic interpretability hook framework
device = "cuda" if torch.cuda.is_available() else "cpu"
model = HookedTransformer.from_pretrained(
    "meta-llama/Meta-Llama-3-8B",
    device=device,
    dtype=torch.bfloat16
)

clean_prompt = "When John and Mary went to the store, Mary gave the bag to"
corrupted_prompt = "When Alice and Mary went to the store, Mary gave the bag to"

clean_tokens = model.to_tokens(clean_prompt)
corrupted_tokens = model.to_tokens(corrupted_prompt)

john_token_id = model.to_single_token(" John")
alice_token_id = model.to_single_token(" Alice")

# 2. Cache activations for clean and corrupted runs
clean_logits, clean_cache = model.run_with_cache(clean_tokens)
corrupted_logits, corrupted_cache = model.run_with_cache(corrupted_tokens)

def compute_logit_diff(logits):
    # Metric: Difference in logits between correct (John) and incorrect (Alice) token
    return logits[0, -1, john_token_id] - logits[0, -1, alice_token_id]

baseline_clean = compute_logit_diff(clean_logits).item()
baseline_corrupted = compute_logit_diff(corrupted_logits).item()

print(f"Clean Logit Difference: {baseline_clean:.4f}")
print(f"Corrupted Logit Difference: {baseline_corrupted:.4f}")

# 3. Patching a specific head: Hook into Layer 15, Head 4
target_layer = 15
target_head = 4
hook_name = f"blocks.{target_layer}.attn.hook_z"

def patch_head_activation(corrupted_z, hook):
    # Replace the activation of the target head with its activation from the clean cache
    corrupted_z[:, :, target_head, :] = clean_cache[hook.name][:, :, target_head, :]
    return corrupted_z

# 4. Run the corrupted forward pass with the patched clean head
patched_logits = model.run_with_hooks(
    corrupted_tokens,
    fwd_hooks=[(hook_name, patch_head_activation)]
)

patched_diff = compute_logit_diff(patched_logits).item()
normalized_recovery = (patched_diff - baseline_corrupted) / (baseline_clean - baseline_corrupted)

print(f"Normalized Logit Recovery by Patching L{target_layer}H{target_head}: {normalized_recovery * 100:.2f}%")

Interpreting the Results

If patching Layer 15, Head 4 recovers a large portion of the clean logit difference (e.g., $>60%$), this head is causally implicated as a Name Mover Head—responsible for extracting the non-repeated subject from early token positions and projecting its token identity into the final residual stream position.


Tooling & Interpretability Frameworks Benchmark

Researchers require specialized tooling capable of dynamic tensor interception, compute-graph editing, and distributed feature tracking. The table below compares the leading libraries in 2026.

Framework Primary Architecture Focus Multi-GPU / Distributed Support SAE Integration Level Execution Overhead Best For
TransformerLens (Neel Nanda et al.) Open weights (Llama, Mistral, GPT-2/J/NeoX) Limited (Single GPU / FSDP basic) Native via SAELens Medium (Hook management overhead) Fast hypothesis testing, academic exploration
NNSight (NDIF) Arbitrary PyTorch models (Llama 3 70B/405B) Native multi-node distributed tracing Modular (Remote activation extraction) Very Low (Direct torch trace hooks) Scaled inference, 70B+ cluster interventions
SAELens SAE Training & Inference across standard LLMs Native Hugging Face & TransformerLens Core framework Low to Medium Training dictionary learning models, feature steering
Baukit (David Bau lab) Vision & Language Models Basic External Low Causal tracing, ROME, direct weight edits

Computational Cost & Infrastructure Analysis (2026)

Mechanistic interpretability has evolved from inspecting toy single-layer models to training massive dictionary models across modern 8B–70B parameter foundations. This requires substantial computational investment.

Model Scale -> SAE Latent Expansion -> Minimum GPU Infrastructure -> Compute Cost (Est.)
---------------------------------------------------------------------------------------
Llama 3 8B  -> 16x (65k features)   -> 4x NVIDIA H100 80GB (100M tokens) -> ~$180 - $250
Llama 3 8B  -> 64x (262k features)  -> 8x NVIDIA H100 80GB (500M tokens) -> ~$1,200 - $1,600
Llama 3 70B -> 32x (262k features)  -> 32x NVIDIA H100 80GB (1B tokens)  -> ~$9,500 - $14,000
Llama 3 70B -> Full Layer Sweep    -> 64x NVIDIA H200 141GB Cluster     -> ~$65,000+

2026 Compute Economics Breakdown:

  • Cloud Spot H100 SXM5 Pricing: Averages between $2.30 to $2.85 per GPU/hour on standard cloud providers (Lambda Labs, RunPod, CoreWeave).
  • SAE Storage Footprint: A single 32x expansion Top-K SAE trained on Layer 24 of Llama 3 70B ($d_{model} = 8192$, latent dimension $D = 262,144$) occupies roughly 8.6 GB per checkpoint in bfloat16.
  • Full-Sweep Bottleneck: Extracting, buffering, and saving residual stream activations across all 80 layers of a 70B model requires hundreds of terabytes of high-throughput NVMe scratch arrays to prevent IO-throttling during SAE training runs.

Technical Limitations and Trade-offs

While mechanistic interpretability delivers precise insights, practitioners face severe real-world constraints:

  1. Reconstruction Loss vs. Feature Interpretability Trade-off:
    Training an SAE introduces reconstruction error. When substituting LLM activations with SAE outputs $(\hat{x})$, models suffer a degradation in downstream cross-entropy loss (often measured as a drop of 0.1 to 0.5 nats). Hyper-sparse representations (low $L_0$) yield clean, monosemantic latents but break downstream reasoning; dense latents preserve performance but reintroduce polysemanticity.

  2. Circuit Fragility and Context-Dependence:
    Circuits isolated under specific prompt templates (e.g., IOI) often fragment when prompt structure changes slightly. An attention head acting as an Induction Head under synthetic tokens may function as an acronym resolver under prose, suggesting dynamic multiplexing of circuits across contexts.

  3. Scaling Beyond 70B Parameters:
    Tracing circuits manually becomes computationally intractable on models like Llama 3 405B or DeepSeek-R1. Manual circuit annotation must transition toward automated interpretability, where secondary LLMs evaluate, name, and stress-test isolated SAE features via automated perturbation tests.


Implementation Roadmap for Research Teams

If your organization intends to stand up mechanistic interpretability pipelines for safety, auditing, or model steering, follow this four-phase operational blueprint:

[Phase 1: Instrumentation] -> [Phase 2: Causal Tracing] -> [Phase 3: Sparse Autoencoders] -> [Phase 4: Intervention & Steering]

Phase 1: Model Instrumentation and Hooking

  • Standardize on NNSight or TransformerLens.
  • Establish clean vs. corrupted evaluation datasets targeting specific behaviors (e.g., sycophancy, refusal mechanisms, multi-step calculation).
  • Verify zero execution overhead when hooks are dormant to avoid breaking production inference engines.

Phase 2: Coarse Causal Tracing

  • Run activation patching sweeps over entire residual stream blocks ($L_1 \dots L_{max}$).
  • Isolate the critical 2–5 layers where target decisions are committed.
  • Run head-level patching within those layers to isolate the attention heads driving the metric.

Phase 3: SAE Feature Extraction

  • Deploy pre-trained Top-K Sparse Autoencoders (e.g., from SAELens Hub) on the layers identified in Phase 2.
  • Project residual activations into latent space during problematic model outputs (e.g., generation of vulnerable code or hallucination).
  • Identify the exact sparse latents whose activations correlate ($r > 0.8$) with the target behavior.

Phase 4: Feature Steering and Ablation

  • Validate causality through activation clamping: dynamically add a constant scalar multiple of the target feature's decoder vector ($+ c \cdot W_{dec}^{(j)}$) during inference to induce the behavior, or subtract it to suppress the behavior.
  • Replace heavy RLHF/DPO runs with targeted activation clamps to suppress toxic or insecure outputs at run-time without retraining base model weights.

Frequently Asked Questions (FAQ)

Does mechanistic interpretability scale to frontier reasoning models?

Yes, but the locus of study shifts. In models exhibiting Chain-of-Thought (CoT) reasoning (such as OpenAI o1 or DeepSeek-R1), interpretability focuses on backtracking circuits, self-correction triggers in residual streams, and the emergence of planning features prior to tokens outputting verification steps.

What is the advantage of Top-K SAEs over traditional L1 regularized SAEs?

Traditional $L_1$ penalty autoencoders suffer from shrinkage bias, where the $L_1$ loss systematically underestimates the magnitude of true feature activations, harming model reconstruction fidelity. Top-K SAEs eliminate shrinkage by directly preserving the exact activations of the top $K$ largest latent features while zeroing the rest, achieving a strictly superior Pareto frontier between reconstruction fidelity and sparsity.

How does activation patching differ from Integrated Gradients or SHAP?

Integrated Gradients and SHAP provide input-feature attribution; they tell you which input tokens influenced an output. Activation patching provides internal mechanistic attribution; it tells you which weight matrices, layers, or hidden dimensions inside the model computed and transferred that information, exposing the model's internal causal DAG (Directed Acyclic Graph).