Activation patching¶

Activation patching asks which activations cause a behaviour. The model runs on a target prompt; one activation is overwritten with what it was on a source prompt (or with a value, or with noise); and the change in the output measures how much that activation mattered. Every row of the index is a location to patch, and gives one patched run.

Method	Method config	Fields	How it obtains each row’s effect
`exact`	`PatchingMethodConfig`	none	patches and runs the model, one run per row
`attribution`	`PatchingMethodConfig`	none	a first-order estimate from one forward and one backward pass for the whole index
`integrated_gradients`	`IntegratedGradientsConfig`	`interpolation_steps` (default `10`)	the same estimate, with the gradient averaged along the path from the source activation to the target one

A run¶

The classic sweep patches the residual stream of a clean prompt into a corrupted one, at every layer and position:

from transformer_lens.model_bridge import TransformerBridge

from entoscope import (
    MetricComparison,
    ModelAxis,
    ModelLayerLocation,
    Patcher,
    PatchingConfig,
    PatchingElementConfig,
    PatchingInput,
    PatchingMethodConfig,
    PatchingMetric,
    PatchingMetricConfig,
    PatchingType,
    SinglePatchingConfig,
)

model = TransformerBridge.boot_transformers("gpt2")
model.eval()

residual = PatchingElementConfig(
    type=PatchingType.REPLACE,
    activation=ModelLayerLocation.RESIDUAL_PRE,
    location=[ModelAxis.LAYER, ModelAxis.POSITION],
)
config = PatchingConfig(
    metric={
        "logit_diff": PatchingMetricConfig(
            type=PatchingMetric.LOGIT_DIFF,
            tokens=["correct", "distractor"],
            comparison=MetricComparison.NORMALIZED,
        )
    },
    method=PatchingMethodConfig(type="exact"),
    patch=[SinglePatchingConfig(source=residual, target=residual)],
)
patching_input = PatchingInput(
    config,
    "The capital of France is",  # source
    "The capital of Spain is",  # target
    tokens={"correct": " Paris", "distractor": " Madrid"},
)
Patcher(model).apply(patching_input, save_dir="runs/residual_sweep")

The config’s metric, which it names logit_diff, reads the tokens the input names correct and distractor. Each row of result.csv holds one patched run’s logit_diff_metric, reported normalized: 0 where the patch changed nothing, 1 where the target run now prefers “ Paris” over “ Madrid” as much as the source run does. For GPT-2 the grid shows the answer travelling. Up to layer 9, patching the " France" position (4) alone restores it almost fully; by layers 10 and 11 the last position carries it instead:

target_position    0    1    2    3     4     5
target_layer
              0.0  0.0  0.0  0.0  1.00  0.00
              0.0  0.0  0.0  0.0  1.00  0.00
              0.0  0.0  0.0  0.0  1.00 -0.00
              0.0  0.0  0.0  0.0  1.00  0.01
              0.0  0.0  0.0  0.0  1.00  0.01
              0.0  0.0  0.0  0.0  1.02  0.03
              0.0  0.0  0.0  0.0  1.02  0.04
              0.0  0.0  0.0  0.0  0.98  0.08
              0.0  0.0  0.0  0.0  0.98  0.09
              0.0  0.0  0.0  0.0  0.90  0.15
             0.0  0.0  0.0  0.0  0.30  0.65
             0.0  0.0  0.0  0.0  0.13  0.80

In a config file this is the residual_sweep entry of examples/patching.yaml, which has an entry for every variant on this page.

The two runs and the tokens¶

PatchingInput(config, source_prompt, target_prompt, tokens=...):

The target run is the one patched, and every metric is read off the patched target run.
The source run is what replace and value_scale read. A config whose sources only write a value or noise reads no source run, and source_prompt may then be None (see Without a source prompt).
tokens names the tokens the metrics read: each a string that tokenizes to a single token, or a token id. A string of several tokens is read as its first, with a warning. A config reads tokens by these names, so they are best chosen as roles, such as correct and distractor: the config then runs on any pair of prompts, each pair naming its own answers. Only the tokens a metric reads are looked at, so one mapping can serve several configs.

The metrics are read at the final position of each run, and the source and target prompts are each run once and cached before the patching starts. The tokens are resolved before either run, so plan() already refuses a metric that reads a token the input does not name, an empty string, or an id outside the vocabulary.

Metrics and comparisons¶

metric names every metric the run reads, each written to the result column <name>_metric. A metric gives its type, the tokens it reads by their names in the input’s tokens, and the comparison it is reported under. All of them come off the same patched run, so a second metric costs no extra forward pass:

metric:
  logit_diff:
    type: logit_diff
    tokens: [correct, distractor]
    comparison: normalized
  kl_divergence:
    type: kl_divergence
    comparison: indirect_effect

The metrics, read off a run’s final-position logits, where \(r\) and \(r'\) are the first and the second token a metric’s tokens lists:

Metric	Value	Tokens
`logit_diff`	\(\mathrm{logit}(r) - \mathrm{logit}(r')\)	2
`logprob_diff`	\(\log P(r) - \log P(r')\), numerically the same as `logit_diff`	2
`probability`	\(P(r)\)	1
`log_prob`	\(\log P(r)\)	1
`kl_divergence`	\(D_{KL}(P_{\text{source}} \,\Vert\, P_{\text{run}})\), how far the run’s whole distribution is from the source run’s, which it needs	0

A metric names exactly as many tokens as its type reads, or the input is refused as it is built.

The comparisons, where \(m\) is a metric and the subscripts name the run it is read off:

Comparison	Reported value	Needs a source run
`absolute`	\(m_{\text{patched}}\)	no
`indirect_effect`	\(m_{\text{patched}} - m_{\text{target}}\)	no
`normalized`	\((m_{\text{patched}} - m_{\text{target}}) / (m_{\text{source}} - m_{\text{target}})\)	yes

indirect_effect is activation patching stated as a causal quantity. normalized puts 0 at “the patch changed nothing” and 1 at “the patch reproduced the source run”. This is the ratio circuit work calls faithfulness, with the unpatched target playing the fully ablated model. A positive effect means the patch moved the model toward the source run for every metric: a divergence falls as the run nears the source, so kl_divergence’s indirect effect is its decrease.

The comparison is chosen per metric because the natural one differs: a divergence reads well as an effect, a logit difference as a fraction of the gap between the two runs. A metric wanted under two comparisons is named twice, once with each.

Several metrics of one type¶

Each metric reads tokens of its own, so one type can be read on several. The competing_answers entry of examples/patching.yaml sets the source run’s answer against the target run’s and against a third city neither prompt is about, and reads that city’s chance too:

metric:
  against_target:
    type: logit_diff
    tokens: [correct, distractor]
    comparison: normalized
  against_other:
    type: logit_diff
    tokens: [correct, other]
    comparison: indirect_effect
  other:
    type: probability
    tokens: [other]
    comparison: indirect_effect

With tokens={"correct": " Paris", "distractor": " Madrid", "other": " Rome"}, its columns are against_target_metric, against_other_metric and other_metric. Every metric column carries the suffix, so no name can take a column the run writes itself, generation, str_generation or one of the index’s.

Patch elements¶

patch lists the elements to patch. An element is a SinglePatchingConfig with a source and a target, each a PatchingElementConfig, which is a locator plus type and value. Row i of the source index is read and written at row i of the target index. A side with a single row pairs with every row of the other; otherwise the two need as many rows, and a pair that does not line up is rejected before anything runs. The two sides may name different locations, such as reading layer 10 and writing layer 11, as long as the slices fit.

The source’s type says what gets written:

Source `type`	Writes	Reads the source run
`replace`	the source run’s activation	yes
`value_scale`	the source run’s activation times `value`	yes
`value_replace`	the constant `value`; with the default `0.0` this is zero ablation	no
`gaussian_noise`	the target run’s own activation plus Gaussian noise, the corruption of causal tracing	no

The noise’s standard deviation is three times the target run’s own standard deviation over the whole activation, as in causal tracing (Meng et al., 2022). It is drawn from a fixed seed, so every row is corrupted by the same draw and a rerun repeats the sweep exactly. The target’s type is always replace. A type that cannot run, such as another target type or a source that reads a source run the input has no prompt for, is rejected as the PatchingInput is built.

An attn.pattern element patches attention weights alone: the head still mixes the target run’s values with them, and a query’s weights are not renormalized, so a patched row of the pattern need not sum to 1. Located by dest_pos and src_pos, it patches single entries. Those with src_pos past dest_pos are masked to 0 in both runs, so a replace there changes nothing and scores exactly 0, though exact still spends a run on it: a sweep of the whole pattern runs about half its rows for nothing, which a sweep of one query’s row, such as dest_pos: -1, does not. Every other source writes a weight there that no unpatched run can hold: a gaussian_noise or a nonzero value_replace makes the query attend to tokens after its own position.

Several elements at once¶

With more than one element, patched run i applies row i of every element together, so their indices must have the same number of rows, or one. An element with one row is applied on every run. The result is laid out along the element with the most rows, the first of them when several tie, so every patched run has a row of its own. On the result page, a one-row element is named in the subtitle and a varying one in every cell’s hover.

patch:
  - source: &late_heads  # varies: 4 layers x 12 heads
      activation: attn.z
      location: [layer, head]
      filter:
        - layer: {range: [8, 11]}
      type: replace
    target: *late_heads
  - source: &first_mlp  # one row, patched on every run
      activation: mlp.out
      location: [layer]
      filter:
        - layer: 0
      type: replace
    target: *first_mlp

Methods¶

`exact`¶

exact patches the model and runs it once per row. It assumes nothing about how the model responds, at the price of a forward pass per row. Patcher(model, batch_size=n) runs n rows per forward pass.

Backends can round differently at different batch shapes (MPS does, by up to half a logit in bfloat16). The source and target runs are therefore cached at the same batch width as the patched runs, and a ragged last batch is padded, so a patch that writes back what was already there reproduces the target run exactly.

exact also records what each patched run predicts next, as generation and str_generation.

`attribution`¶

Attribution patching estimates every row’s effect from the gradient of the metric at the target run’s activation: \(\nabla m \cdot (a_{\text{patch}} - a_{\text{target}})\). One forward pass and one backward pass per metric score the whole index, so the cost does not grow with the sweep. The estimate is only as good as the model’s local linearity, and a large intervention, such as zero-ablating a whole layer, leaves that behind. The replacement comes from the same source types as exact, and a row patching several elements sums their contributions.

`integrated_gradients`¶

integrated_gradients averages that gradient over interpolation_steps points on the straight path from the source run’s activation to the target run’s (Marks et al., 2024, Eq. 4), which recovers effects a single gradient at the clean point hides. As there, each transformer activation the index patches, such as each layer of a sweep, is interpolated on passes of its own, so a row’s score does not depend on what else the index holds. That costs one forward pass and one backward pass per metric for every step and transformer activation, except at the clean point, where one pass serves all of them. The cost grows with the transformer activations but not with the rows: every position, head and neuron of one comes from the same passes. The path runs between the two runs, so it needs a source prompt of the same token length as the target, which plan() checks too. It is meant for replace sources: a value or noise is written off that path, and attribution is the estimate to use there.

The gradient methods make no patched run, so their results have no generation. A common workflow is to estimate a large sweep with attribution, then confirm its strongest rows with exact.

Without a source prompt¶

Zero ablation writes a value, so it reads no source run. The source names only its type, a single row that pairs with every target row:

from entoscope.util.data import from_dict

zero_mlp = from_dict(
    PatchingConfig,
    {
        "metric": {
            "logit_diff": {"type": "logit_diff", "tokens": ["correct", "distractor"], "comparison": "indirect_effect"}
        },
        "method": "exact",
        "patch": [
            {
                "source": {"type": "value_replace"},
                "target": {"activation": "mlp.out", "location": ["layer"], "type": "replace"},
            }
        ],
    },
)
ablation = PatchingInput(zero_mlp, None, "The capital of Spain is", tokens={"correct": " Paris", "distractor": " Madrid"})
Patcher(model).apply(ablation, save_dir="runs/zero_mlp")

Reported as an indirect effect, each row is \(m(x \mid do(a = 0)) - m(x)\). Without a source run there is no normalized and no kl_divergence.

A source that reads no source run counts its positions in the target prompt: a value is written into the target run, and gaussian_noise corrupts the target run’s own activation, so position: -1 on such a source is the target’s last token, with or without a source prompt.

A normalized metric divides by the gap between the source and target runs. Two runs that score the same, such as one prompt on both sides, leave no gap: every row then reads inf, or nan where the patch changed nothing, and logs a warning.

Output¶

result.csv has one row per patched run:

the index of the element the result is laid out along, as source_* and target_* columns, a one-row source or target repeated on every row it pairs with;
a column per metric, <name>_metric, in the order the config names them;
generation and str_generation, the token the patched run predicts next, for exact.

Every element’s index is written to its own index-{i}.csv.

result.html draws one grid per result column, on a diverging scale centred on what an ineffective patch would score: the target run’s own metric for absolute, zero otherwise. Warm cells moved the model toward the source run. The subtitle names both prompts and what each predicts, then the plot’s metric type and the tokens it reads, by name and as spelled, since a metric’s name need not say. A cell’s hover gives its value, its patched run’s prediction and its full source and target coordinates. The collector’s figures hold the same grids, keyed by metric name, such as collector.figures["logit_diff"].

Recipes¶

All of these are entries of examples/patching.yaml:

Question	Entry
Where, layer by layer and token by token, is the answer carried?	`residual_sweep`
Against which rival does a patch move the answer?	`competing_answers`: two `logit_diff` metrics on different tokens
Which heads matter?	`head_sweep`: `attn.z` over `[layer, head]`
Which of a head’s attention weights matter?	`pattern_entries`: `attn.pattern` over `[layer, head, dest_pos, src_pos]`, at the last token’s row
Which MLP layers matter?	`zero_ablation`: `value_replace` into `mlp.out` over `[layer]`
Which tokens matter?	`noise_tracing`: `gaussian_noise` on `embed` over `[position]`
A whole sweep, cheaply	`attribution`, `integrated_gradients`
Specific points	`explicit_coordinates`
A sweep on top of a fixed patch	`heads_with_fixed_mlp`

References¶

Kevin Meng, David Bau, Alex Andonian and Yonatan Belinkov. Locating and Editing Factual Associations in GPT. 2022.
Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris and Jacob Steinhardt. Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small. 2022.
Fred Zhang and Neel Nanda. Towards Best Practices of Activation Patching in Language Models: Metrics and Methods. 2023.
Stefan Heimersheim and Neel Nanda. How to use and interpret activation patching. 2024.
Neel Nanda. Attribution Patching: Activation Patching At Industrial Scale. Blog post.
Aaquib Syed, Can Rager and Arthur Conmy. Attribution Patching Outperforms Automated Circuit Discovery. 2023.
Mukund Sundararajan, Ankur Taly and Qiqi Yan. Axiomatic Attribution for Deep Networks. 2017.
Samuel Marks, Can Rager, Eric J. Michaud, Yonatan Belinkov, David Bau and Aaron Mueller. Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models. 2024.