Part 11 · Module 1: The Road to Attention

Incident: The causal flag that masked nothing

7 min read·13 Oct 2026

Module 1 · Incident file · About 30 minutes

What happened

On 16 March 2023 a PyTorch user, JahooYoung, opened issue #96941 on the PyTorch repository: "is_causal parameter in torch.nn.TransformerEncoderLayer.forward does not work" (github.com/pytorch/pytorch/issues/96941). The report was eleven lines of code on PyTorch 2.0.0. It built one encoder layer, ran the same input three ways (no mask, is_causal=True, and a real causal mask) and asserted that the flag should give the masked result, not the unmasked one. It did not.

The bug (red) was reported on 16 March 2023 and fixed (green) nine days later by requiring a mask; on torch 2.14.1 the flag alone now raises an error (measured here).

Five days later the PyTorch maintainer mikekgfb opened pull request #97214 (github.com/pytorch/pytorch/pull/97214). Its summary explains that PyTorch 2.0 had "added the is_causal parameter to legacy nn.Transformer* and nn.MHA APIs", but that on one internal code path "is_causal has no impact". It was approved on 24 March and landed on 25 March 2023 as commit c757647. On 20 April the same maintainer closed the issue with the rule that still holds: is_causal "is a hint", and "Undefined behavior occurs when is_causal is set and the mask is not a causal mask."

The issue records no damage figure, and none is claimed here. What a dropped causal flag puts at risk is plain from Lesson 1: a model meant to predict the next token that can see it.

Why it happened

On PyTorch 2.0.0 the flag worked on the fast path (green) but the slower path read only the mask, found none, and let every position see every other (red).

PyTorch 2.0 gave attention a fast new operator, scaled dot-product attention, which accepts either a mask or an is_causal flag. The PR explains that the older multi-head attention module also had a second, slower path, used when attention weights are requested or a padding mask is given. That path computes attention as matrix products with a mask, and, in the PR's words, "this code path does not have support for is_causal". A flag with no mask therefore reached a path that looked only at the mask, found none, and let every position attend to every other.

With the causal mask each position sees itself and earlier ones (blue) and the future is hidden (grey); without it the upper triangle (red) lets every position see later tokens.

A missing causal mask does not raise an error or change a shape. The output is (batch, time, width) either way. What changes is which cells of the attention matrix are allowed: with the mask, position 3 sees positions 0 to 3; without it, position 3 also sees 4 and beyond, including the token it is supposed to predict.

On torch 2.14.1 the flag alone fails loudly (green), and the flag with an empty mask silently gives the causal result (amber, uncertain): the hint overrides the mask.

The fix chose to require the mask. Today the issue's exact code raises an error instead of running, and when you pass a mask with the hint, the hint wins. Check both on your own PyTorch.

python

import torch

d_model = 3                                       # JahooYoung's repro, from issue #96941
layer = torch.nn.TransformerEncoderLayer(d_model, 1, 6, batch_first=True)
layer.eval()
x = torch.randn(1, 5, d_model)
try:
    layer(x, is_causal=True)
except RuntimeError as err:
    print("RuntimeError:", str(err).split(".")[0])

causal = torch.nn.Transformer.generate_square_subsequent_mask(5)
no_mask = torch.zeros(5, 5)                       # a mask that hides nothing
print(torch.__version__, "| hint + causal mask vs hint + empty mask, max difference:",
      (layer(x, src_mask=causal, is_causal=True) - layer(x, src_mask=no_mask, is_causal=True)).abs().max().item())

Code walkthrough

The first six lines are the issue's own setup: one layer, width 3, one head, a (1, 5, 3) input.
The try makes the call that silently did nothing on PyTorch 2.0.0. Measured here on torch 2.14.1 it raises RuntimeError: Need attn_mask if specifying the is_causal hint.
The last line passes the hint with an empty mask. The difference from the real causal mask is 0.0 (measured here): the hint overrode the mask. That is the "undefined behaviour" the closing comment warns about. The lab's Encoder1 passes both, and they agree.

The same failure at tinyformer-lab

Replay the 2.0.0 behaviour on the lab's one-layer encoder: the code says causal, but no mask reaches the layer. Run this in the lab's notebook, inside tinyformer-lab, after Step 8. If your runtime was reset, clone your repository first (git clone your GitHub copy, cd tinyformer-lab, pip install -e .).

The replay changes one argument: the flag-only encoder (red) calls the layer with no mask, the lab's encoder (green) passes the causal mask and the hint.

python

import time
import torch
from tinyformer.data import TASKS, set_seed
from experiments.m01_rnn_vs_attention import Encoder1, race

class FlagOnlyEncoder(Encoder1):
    """The 2.0.0 outcome: the model is meant to be causal, but no mask reaches the attention."""
    def forward(self, ids):
        T = ids.shape[1]
        x = self.embed(ids) + self.pos(torch.arange(T))
        return self.head(self.layer(x))           # what ran on 2.0.0 when only the flag was given

copy = TASKS["copy"]
MODELS = {}
for name, make in [("flag only (bug)", FlagOnlyEncoder), ("mask and hint (fix)", Encoder1)]:
    set_seed(11); model = make()
    t0 = time.perf_counter()
    r = race(model, copy, seed=11)
    MODELS[name] = model
    print(f"{name:20} scored accuracy {r['accuracy']:.3f}  unscored {r['unscored_accuracy']:.3f}"
          f"  ({time.perf_counter() - t0:.1f} s)")

Code walkthrough

FlagOnlyEncoder inherits everything from the lab's Encoder1 and changes one line: the layer is called with no mask. On PyTorch 2.0.0, self.layer(x, is_causal=True) computed exactly this, which is what the issue's assert caught.
Both models are trained by the lab's race on seed 11, so the only difference is the missing mask.

Measured here (author's Mac, one seed): both score 1.000 on the scored half. The leak is invisible in the number the lab reports. The only hint is the unscored half, 0.086 against 0.031 for the fixed model and 0.032 for chance (arithmetic). Each run took about 4 seconds.

Both models score 1.000 on the training measure; only when writing its own answers does the leaky model (red) fall to 0.816 while the fix (green) stays at 1.000.

The cost appears when the model has to write the answer itself, one token at a time, the way it will be used. Then there is no future token to peek at.

python

@torch.no_grad()
def write_copy(model, rows=512):
    """Give the 16 tokens and the separator, then let the model write all 16 answers itself."""
    model.eval()
    x = copy.inputs[-rows:]
    ids = x[:, :17]
    for _ in range(16):
        nxt = model(ids)[:, -1].argmax(-1, keepdim=True)    # the model's own best guess
        ids = torch.cat([ids, nxt], dim=1)                  # becomes its next input
    return (ids[:, 17:] == x[:, :16]).float().mean().item()

for name, model in MODELS.items():
    print(f"{name:20} writing its own answers: accuracy {write_copy(model):.3f}")

Code walkthrough

write_copy starts each held-out row from its first 17 tokens and appends the model's own prediction 16 times: generation, not the training set-up in which every true token is supplied.
[:, -1] takes the prediction at the last position. There, nothing lies to the right, so a model that learnt to read the next token has nothing to read.

Measured here: the flag-only model writes 0.816 of the answers correctly, the fixed model 1.000. Training said both were perfect.

The fix

Passing the mask moves the unscored half back to chance (0.031) and the written answers to 1.000 (green); the leaky version (red) reads tokens it should not.

The fix is the lab's own line: build the causal mask and pass it, with the hint if you like, never the hint alone. Encoder1 already does this (src_mask=causal, is_causal=True), which is why its numbers above are clean: 1.000 when trained, 1.000 when writing, 0.031 on the unscored half. On today's PyTorch the flag-only call raises an error, so the 2023 version of this bug cannot recur by that route. It can still recur in your own code, as FlagOnlyEncoder shows: a refactor that drops one argument trains just as well.

How you would have caught it

The causal check changes every token after position 20 (amber) and measures positions 0 to 20 (blue): the bug moved by 34.843 (red, fail), the fix by exactly 0.000 (green, pass).

The issue's author caught it with a behavioural test: compare outputs with and without the mask. The general form is Lesson 1's causal check. Change every token after position t; the outputs at positions up to t must not move at all.

python

@torch.no_grad()
def causal_check(model, t=20, rows=64):
    """Largest change at positions 0..t when every token after t is replaced. Causal means exactly 0."""
    model.eval()
    x = copy.inputs[:rows]
    changed = x.clone()
    changed[:, t + 1:] = 1 + changed[:, t + 1:] % 31        # a different content token everywhere after t
    return (model(changed) - model(x))[:, :t + 1].abs().max().item()

for name, model in MODELS.items():
    moved = causal_check(model)
    print(f"{name:20} earlier positions moved by {moved:.3f}  ->  {'PASS' if moved == 0 else 'FAIL'}")

Code walkthrough

changed[:, t + 1:] swaps every token after position 20 for a different one; 1 + token % 31 keeps it a content token but never the same.
The check compares logits at positions 0 to 20 before and after. Any non-zero change means an earlier position saw a later token.

Measured here: the flag-only model's earlier positions moved by 34.843 (FAIL); the fixed model's by exactly 0.000 (PASS). The check takes milliseconds, needs no training and does not depend on the task being hard. Module 5 makes it a permanent test in tests/.

Key points

A dropped causal mask changes no shape and raises no error. In PyTorch 2.0.0 the flag alone reached a path that ignored it (issue #96941, PR #97214, March 2023).
Today is_causal is a hint that needs a mask and may override it. Pass a real causal mask, every time.
Accuracy under teacher forcing could not see the leak here: 1.000 for both models. Writing its own answers showed 0.816.
Two cheap monitors catch it: accuracy on positions that should be at chance, and the behavioural causal check, which must print exactly zero.

Sources

JahooYoung, "is_causal parameter in torch.nn.TransformerEncoderLayer.forward does not work", PyTorch issue #96941, opened 16 March 2023, closed 20 April 2023. https://github.com/pytorch/pytorch/issues/96941
mikekgfb (PyTorch), "[Better Transformer] make is_causal a hint and force attn_mask to be set on is_causal=True in F.MHA", PyTorch pull request #97214, opened 21 March 2023, landed 25 March 2023 (commit c757647). https://github.com/pytorch/pytorch/pull/97214
PyTorch documentation, torch.nn.TransformerEncoder, version 2.14 (checked October 2026): "is_causal provides a hint that mask is the causal mask. Providing incorrect hints can result in incorrect execution". https://docs.pytorch.org/docs/2.14/generated/torch.nn.modules.transformer.TransformerEncoder.html