Part 12 · Module 1: Engineering Floor

Incident: Stolen CI secrets that never expired

9 min read·13 Oct 2026

Module 1 · Incident file · About 35 minutes

What happened

Seven days passed between the theft (red) and the first warning from a customer (amber), thirteen since the laptop was infected; access was cut and the breach disclosed on 4 January, and recovery followed (green).

On 4 January 2023 (US time), CircleCI, a hosted CI/CD service, disclosed a breach and told its customers to rotate the secrets they had stored with it. Eight days later its Chief Technology Officer, Rob Zuber, published the incident report (CircleCI, "CircleCI incident report for January 4, 2023 security incident", 12 January 2023). This section uses only what that report says.

On 16 December 2022, malware reached a CircleCI engineer's laptop; the report says antivirus did not detect it. The malware was used to "steal a valid, 2FA-backed SSO session": a signed-in session that had already passed two-factor checks. The attacker used it to impersonate the engineer and reach a subset of production systems, because "the targeted employee had privileges to generate production access tokens". Reconnaissance began on 19 December. On 22 December the attacker took customer environment variables, tokens and keys.

The data was encrypted at rest, but the attacker "extracted encryption keys from a running process". Nobody noticed until 29 December, when a customer reported suspicious GitHub OAuth activity. CircleCI shut off the engineer's access on 4 January and removed production access from nearly all employees.

The cost fell on customers too. CircleCI recommended "that all customers rotate their secrets, including OAuth tokens, Project API Tokens, SSH keys, and more". At publication, "fewer than 5 customers" had reported unauthorised access to their own systems. Among the report's recommendations: "Use OIDC tokens wherever possible to avoid storing long-lived credentials in CircleCI." Its own summary: "a security incident is a systems failure."

Why it happened

One infected laptop reached customer secrets through links that each trusted the last; red marks what the attacker took, blue what worked as designed.

Look at the chain, not the laptop. Each link was ordinary: a laptop, a single sign-on session, an engineer allowed to create production tokens, a store full of customer secrets. A session cookie that has already passed two-factor checks needs no second factor when it is replayed. Every link trusted the one before it, so one infected machine reached the most valuable data in the company.

Encryption at rest protects a copied disk (green); it does not stop an attacker inside production, who can read the key from the running service (red).

Encryption at rest protects data on a disk that is stolen or copied. A running service has to decrypt the data to use it, so the key sits in its memory. An attacker inside production can read the key from the running process, as the report says this one did. Encryption at rest is not the control for an attacker who is already inside.

A stored key stays useful to a thief until it is rotated (red); a credential issued for one job expires on its own (green).

The deeper reason is the secrets themselves. A long-lived credential works until someone rotates it. Copied on 22 December, a stored key stayed useful until each customer found and rotated it, after CircleCI's notice of 4 January, and CircleCI asked every customer to rotate them all. A short-lived credential, issued for one job and expiring by itself, is worth little to a thief an hour later. That is why the report's first recommendation is OIDC.

The same failure at Harbourline

The lab's CI pipeline holds no secrets. Suppose instead that, the week before kickoff, a first draft stored two in the CI service's settings: an AWS access key that lets integration tests fetch the TMS export from the harbourline-exports bucket, and the CRM token. Suppose the CI service's store is copied on Friday 13 March 2026 at 09:00. This replay is simulated; the access check is Lesson 4's simplified model of AWS's rules.

python

from datetime import datetime, timedelta
from fnmatch import fnmatch

def allowed(policy, action, resource):            # Lesson 4's simplified model, not the real evaluator
    decision = "deny (implicit)"
    for st in policy["Statement"]:
        if any(fnmatch(action, a) for a in st["Action"]) and any(fnmatch(resource, r) for r in st["Resource"]):
            if st["Effect"] == "Deny":
                return "deny (explicit)"
            decision = "allow"
    return decision

BUCKET = "arn:aws:s3:::harbourline-exports/"
CI_SECRETS = {                                    # simulated: the first draft's stored secrets
    "AWS_ACCESS_KEY_ID": {"created": datetime(2026, 3, 9, 10, 0), "expires": None,
        "policy": {"Statement": [{"Effect": "Allow", "Action": ["s3:*"], "Resource": [BUCKET + "*"]}]}},
    "CRM_TOKEN": {"created": datetime(2026, 3, 9, 10, 0), "expires": None, "policy": None},
}
STOLEN = datetime(2026, 3, 13, 9, 0)
print("copied at", STOLEN, ":", list(CI_SECRETS))

Code walkthrough

allowed is copied from Lesson 4: deny by default, allow when a statement matches, and an explicit deny wins.
CI_SECRETS is the simulated store. The AWS key has s3:* on the whole bucket, a typical "just make the test pass" policy, and expires is None: it works until someone rotates it.
STOLEN is the moment the store is copied.

python

def attacker_tries(cred, when, action, key):
    if cred["expires"] and when > cred["expires"]:
        return "expired"
    return allowed(cred["policy"], action, BUCKET + key)

ATTEMPTS = [(STOLEN + timedelta(minutes=5), "s3:GetObject", "tms/2026-03-13.csv"),
            (STOLEN + timedelta(minutes=6), "s3:PutObject", "tms/2026-03-13.csv"),
            (datetime(2026, 3, 16, 9, 0), "s3:PutObject", "tms/2026-03-16.csv"),
            (datetime(2026, 4, 17, 9, 0), "s3:GetObject", "tms/2026-04-17.csv"),
            (datetime(2026, 6, 5, 9, 0), "s3:DeleteObject", "tms/2026-06-05.csv")]

def replay(cred):
    for when, action, key in ATTEMPTS:
        print(f"{when:%a %d %b %H:%M}  {action:16} {key:20} {attacker_tries(cred, when, action, key)}")

replay(CI_SECRETS["AWS_ACCESS_KEY_ID"])

Code walkthrough

attacker_tries refuses an expired credential, then asks the policy.
ATTEMPTS are five simulated uses: a read and a write within minutes of the copy, a write on the first day of the engagement, a read a month later and a delete on the last day of the pilot.

You should see five lines, every one ending allow: the read and the overwrite on Friday 13 March, the overwrite of Monday's export on 16 March, a read on 17 April and a delete on 5 June.

When (simulated)	Action	Stored key
Fri 13 Mar 09:05	read export	allow
Fri 13 Mar 09:06	overwrite export	allow
Mon 16 Mar 09:00	overwrite export	allow
Fri 17 Apr 09:00	read export	allow
Fri 05 Jun 09:00	delete export	allow

Replayed on Harbourline's simulated CI store, the copied key reads, overwrites and deletes TMS exports for twelve weeks.

Every attempt succeeds, for twelve weeks. The write is the worst: whoever holds the key can replace the morning's TMS export before Mei's job loads it, and every claim downstream is computed from forged data. The policy was never the only problem. The key had no end date.

The fix

The same five attempts: the stored key allows all of them (red); the short-lived, read-only job credential allows one read and refuses the rest (green).

Two changes, in the order the report recommends. First, store nothing long-lived: the tests from the lab need no secrets, so the CRM token leaves CI completely. Second, when a job really must reach AWS, let it ask for a credential that expires and can only read. Below, issue_for_job plays the role of AWS issuing a short-lived credential to one CI job.

python

def issue_for_job(job_start, minutes=15):
    return {"created": job_start, "expires": job_start + timedelta(minutes=minutes),
            "policy": {"Statement": [{"Effect": "Allow", "Action": ["s3:GetObject"], "Resource": [BUCKET + "tms/*"]}]}}

CI_SECRETS_FIXED = {}                             # nothing stored: the job asks for what it needs
job_cred = issue_for_job(STOLEN - timedelta(minutes=2))   # a job that started at 08:58 that morning
print("stored secrets:", len(CI_SECRETS_FIXED), "| job credential expires:", f"{job_cred['expires']:%H:%M}")
replay(job_cred)

Code walkthrough

issue_for_job returns a credential that expires minutes after the job starts and allows only s3:GetObject under tms/.
CI_SECRETS_FIXED is empty: there is nothing in the store to copy.
replay(job_cred) runs the same five attempts against a credential copied from a job running at the time of the theft.

You should see stored secrets: 0 | job credential expires: 09:13. Then the read at 09:05 is allow, the write at 09:06 is deny (implicit), and the three later attempts are expired. The thief gets one morning's TMS file, read-only, for the 13 minutes left on the credential, instead of twelve weeks of write access.

In GitHub Actions, the same idea uses OpenID Connect. The job asks GitHub for a signed token, and AWS swaps it for temporary credentials for one role. GitHub's documentation says the provider "issues a short-lived access token that is only valid for a single job" and that it "then automatically expires". Not run here; verified against GitHub's "Configuring OpenID Connect in Amazon Web Services" and the configure-aws-credentials README (both checked October 2026):

yaml

permissions:
  id-token: write          # lets the job request the OIDC token
  contents: read
steps:
  - uses: aws-actions/configure-aws-credentials@v6
    with:
      role-to-assume: arn:aws:iam::111122223333:role/harbourline-ci-read-tms
      aws-region: eu-west-2
      role-duration-seconds: 900
  - run: aws s3 cp s3://harbourline-exports/tms/ ./exports/ --recursive

Code walkthrough

id-token: write lets the job request a token from GitHub; without it the step fails.
role-to-assume names an AWS role whose trust policy accepts tokens only from Harbourline's repository; the account number is a placeholder. eu-west-2 is AWS's London region.
role-duration-seconds: 900 asks for 15 minutes; the action's default is one hour.
The last line reads the TMS files. No access key appears anywhere in the repository or the CI settings.

How you would have caught it

An inventory checked in CI fails the first draft on three problems (red) and passes the fixed pipeline (green); on a bad day it is also the rotation list.

A review step finds this before anyone steals anything: list every credential the pipeline holds, with its age, its expiry and its permissions, and fail CI when one breaks the rules. The list also answers the first question on the day of an incident: what do we have to rotate?

python

def audit(secrets, today, max_age_days=90):
    problems = []
    for name, s in secrets.items():
        if s["expires"] is None:
            problems.append(f"{name}: never expires")
        if (today - s["created"]).days > max_age_days:
            problems.append(f"{name}: {(today - s['created']).days} days old")
        actions = [a for st in (s["policy"] or {"Statement": []})["Statement"] for a in st["Action"]]
        if any("*" in a for a in actions):
            problems.append(f"{name}: wildcard action {actions}")
    return problems

for label, store in (("first draft", CI_SECRETS), ("fixed", CI_SECRETS_FIXED)):
    found = audit(store, today=datetime(2026, 6, 5, 9, 0))
    print(f"{label}: {'FAIL' if found else 'PASS'}", *found, sep="\n  ")

Code walkthrough

audit flags three things: no expiry, older than 90 days, and a wildcard such as s3:* in a policy.
It runs on both stores on the last day of the pilot, 5 June 2026. *found prints one problem per line.

You should see first draft: FAIL with three problems: AWS_ACCESS_KEY_ID: never expires, AWS_ACCESS_KEY_ID: wildcard action ['s3:*'] and CRM_TOKEN: never expires. Then fixed: PASS. On 5 June the first draft's keys are 88 days old, just inside the 90-day limit, so age alone would not have caught them; the missing expiry does.

In CI, run the audit against an inventory file kept in the repository and stop the build when it fails, exactly as the lab's tests do. Review the inventory with Viktor Hale when access is approved: he needs it for his questionnaire anyway.

Key points

CircleCI's breach started on one laptop and ended in every customer's stored secrets; the report calls a security incident "a systems failure".
Encryption at rest did not help, because a running process holds the keys. Assume an attacker inside can read what your service can read.
A long-lived secret is worth as much to a thief as to you, until it is rotated. In the replay, one key gave twelve weeks of read, write and delete on Harbourline's exports.
Store nothing you do not need. Where a job needs cloud access, give it a short-lived, read-only credential through OIDC.
Keep an inventory of every credential, with age, expiry and permissions, and fail CI when it breaks the rules. It is also your rotation list on the worst day.

Sources

Rob Zuber, CircleCI, "CircleCI incident report for January 4, 2023 security incident", 12 January 2023. https://circleci.com/blog/jan-4-2023-incident-report/
GitHub Docs, "OpenID Connect", checked October 2026. https://docs.github.com/en/actions/concepts/security/openid-connect
GitHub Docs, "Configuring OpenID Connect in Amazon Web Services", checked October 2026. https://docs.github.com/en/actions/how-tos/secure-your-work/security-harden-deployments/oidc-in-aws
AWS, aws-actions/configure-aws-credentials README (inputs, role-duration-seconds default), checked October 2026. https://github.com/aws-actions/configure-aws-credentials