Gemma 3 1B / recorded activations
Neural Observatory
A paper-styled view of one measured token as it passes through all 26 transformer blocks.
This compact Neural Observatory view accompanies the animal-neuron project. It shows Gemma 3 1B target-token activations for the sentence “They mentioned the word {TARGET}.”
Loading the recorded observatory…
Measured token
They mentioned the word pigeon.
Switching words changes the recorded target-token values shown below. It does not run inference.
Activation map
Whole model
One token, 26 blocks
Select any block to zoom in. Attention and residual paths stay neutral; measured gated-MLP activity carries the color.
Embedding → 26 transformer blocks → output. Layers 5, 11, 17, and 23 use global self-attention; the other layers use a local window.
Layer 12
Unit field
6,912 coordinates
A 96 × 72 field. Unit IDs increase from top-left to bottom-right. Spatial adjacency has no learned meaning.
How to read this observatory
The blue bar in each layer is the mean absolute standardized response of that layer’s strongest 69 units for the selected word. Its scale is fixed across all 320 words and all 26 layers. The orange diamond marks global self-attention and is an architecture marker, not an activation value.
Each field cell is one gated-MLP coordinate before proj_down. Cobalt marks values above the unit’s mean, rust marks values below it, and warm paper marks values near the mean. The numeric readout is the primary value; color is a second channel.
Category profiles are means of standardized activations across 40 words. For example, Animals +2.36 means the unit responds an average of 2.36 standard deviations above its own 320-word mean to animal words. It is not a probability, accuracy, or causal effect.
These are recorded target-token activations from They mentioned the word {TARGET}. They are not live inference, ablation, or causal evidence. Frozen-winner rank refers to units selected earlier on a separate animal/tool vocabulary; all other units are exploratory.
Download the manifest and provenance. Source model revision and file hashes are in the manifest.