Configuration files

A config is a dataclass (LensConfig, ProbeConfig, PatchingConfig, CircuitConfig). You can build one in code, or read it from a YAML file and get the same object back. A file is the usual way to keep the configs of an experiment together and to run them from a script.

A config file

A file holds any number of configs, each under the id you load it by:

logit_lens:
  method: logit_lens
  top_k: 5
  source:
    activation: resid.post
    location: [layer, position]

tuned_lens:
  method: tuned_lens
  top_k: 5
  source:
    activation: resid.post
    location: [layer, position]

ConfigFactory reads one entry into the config class it is subscripted with:

from entoscope import ConfigFactory, LensConfig

config = ConfigFactory[LensConfig]("examples/lens.yaml").create("logit_lens")
print(config.config_id)  # logit_lens

The type argument is required, because it is the class the entry is read into. The entry’s id becomes the config’s config_id, which titles the run’s plan and result pages. A factory reads its file once, so creating several entries from one factory reads it a single time.

The keys are the config’s field names, and the values are their plain forms. Enums are written by value (resid.post, logit_diff, mean), locator keys as strings (layer, head), and filters and coordinates as the lists and mappings shown in Locating activations. Each operation page lists its config’s fields.

Reading is strict

An entry is checked as it is read, before any model is involved. A key the config does not have, a required key left out, or an enum value that does not exist raises a ValueError that names the class and the key:

ValueError: Unknown keys for LensConfig: ['top-k']

The checks go into method too: a method nothing registered raises and lists the methods that are registered, and a field the method does not take is an unknown key of its method config. A value its field cannot take is refused as well, such as a quoted "0.04" for a float or true for a count; an int written for a float field is read as that float.

A config built in code holds its fields’ types: enum members, a method config, a Filter or Coordinates. A raw shape held in code, such as a mapping assembled at runtime, is read through from_dict(PatchingConfig, {...}) from entoscope.util.data, which checks it exactly as strictly as a file.

The method field

method picks the operation’s method and holds its settings. In a file it is either the method’s name alone, which uses its defaults, or a mapping with type and the fields to set:

method: tuned_lens

method:
  type: tuned_lens
  translators: gpt2

In code it is an instance of the method’s config class, which always names its type. Methods with no settings of their own take their operation’s base class:

from entoscope import LensMethodConfig, TunedLensConfig

logit = LensMethodConfig(type="logit_lens")
tuned = TunedLensConfig(type="tuned_lens", translators="gpt2")

dataclasses.replace(config, method=tuned) switches a built config to another method. The operation pages list every built-in method’s fields, and Custom methods shows how a method names its config class.

Environment variables

A value may name environment variables as ${NAME}. Each is replaced by the variable’s value, or by NAME itself when the variable is unset. A value that is a single variable reads as the variable’s value would written in its place, so 0.5 is a float, -1 an int and true a bool:

my_data:
  type: jsonl
  path: ${DATA_DIR}/prompts.jsonl
  limit: ${LIMIT}

Reusing parts of a file

YAML anchors and aliases share a block between entries, which keeps the two sides of a patch, or a sweep repeated with several methods, written once:

residual_sweep:
  metric:
    logit_diff: &logit_diff
      type: logit_diff
      tokens: [correct, distractor]
      comparison: normalized
  method: exact
  patch:
    - source: &residual
        activation: resid.pre
        location: [layer, position]
        type: replace
      target: *residual

attribution:
  metric:
    logit_diff: *logit_diff
  method: attribution
  patch:
    - source: *residual
      target: *residual