Lesson 4: Infrastructure: getting the claims tool off your laptop
Module 1 · Lesson 4 · About 50 minutes · Needs: Lesson 2 (a small web service); Docker on your own computer only to try section 1
The situation
The week before kickoff, Nadia Okafor forwards a note from Harbourline: the claims tool may run on one Linux virtual machine (VM) inside their network, and Viktor Hale, Head of IT and Security, wants three answers first. How is it packaged? Where do the credentials live? What does it need to reach? Mei Lin, who will own it after you leave, adds a fourth: when it breaks, where does she look?
This lesson follows one request. In a few weeks Joan Petrov will type HF-88301 into the tool, and that lookup will cross five pieces of infrastructure.
A real case, first
On Monday 4 January 2021, the first working day of the year, Slack went down. Laura Nolan's post-mortem on the Slack Engineering blog (1 February 2021) explains why. An AWS Transit Gateway, a managed hub linking Slack's private networks, "did not scale fast enough" when holiday traffic returned. Packet loss, on top of the usual hourly peak, saturated the web tier. At 6:57am PST, 99% of messages were being sent, against a usual rate above 99.999%.
Autoscaling did its job: "We attempted to add 1,200 servers to our web tier between 7:01am PST and 7:15am PST." Most never served. The service that prepares new servers "hit two separate resource bottlenecks": "the Linux open files limit" and "an AWS quota limit". The dashboards went dark too, because they reached their database across the same hub. AWS raised the hub's capacity and the network was normal by 10:40am.
Nothing exotic failed. You will set up the same pieces for Harbourline, at a smaller size.
What you will be able to do
- Package the service as an image and run it with its database using Compose.
- Load configuration that refuses to start without its secrets.
- Answer four questions on a Linux server.
- Find which network layer stopped a request.
- Write a least-privilege request and a CI pipeline that cannot lie.
1. Docker and Compose (Essential)
Viktor's VM has a different Python from your laptop and none of your libraries. Unless you ship the environment with the code, "it works on my machine" is all you can promise.
A container image is a package holding your code plus everything it needs: system files, Python, libraries. You describe it in a Dockerfile and build it once. A container is a running copy of an image, isolated from the rest of the machine. The same image runs on your laptop, in CI and on the VM, so they cannot drift apart.
# not run here, verified against https://docs.docker.com/reference/dockerfile/ and https://fastapi.tiangolo.com/deployment/docker/
FROM python:3.14-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY main.py .
RUN useradd --create-home appuser
USER appuser
EXPOSE 8000
CMD ["fastapi", "run", "main.py", "--port", "8000"]Code walkthrough
FROMpicks a small Linux with Python 3.14. Installing requirements before copyingmain.pylets Docker reuse that slow step when only code changes.- Copy named files:
COPY . .next to a.envfile bakes the secret into the image. USER appuseravoids running as root. Viktor will look for this.EXPOSE"doesn't actually publish the port" (Docker's reference).CMDstarts FastAPI's server.
The service needs a database, so you run two containers. Compose describes containers that belong together in one file, compose.yaml, and gives them a private network where each is reachable by its service name.
# not run here, verified against https://docs.docker.com/reference/compose-file/services/
services:
api:
build: .
ports: ["127.0.0.1:8000:8000"]
env_file: .env
depends_on:
db: {condition: service_healthy}
db:
image: postgres:17
environment:
POSTGRES_USER: claims
POSTGRES_PASSWORD: ${DB_PASSWORD:?set DB_PASSWORD in .env}
volumes: ["db_data:/var/lib/postgresql/data"]
healthcheck:
test: ["CMD-SHELL", "pg_isready -U claims"]
interval: 5s
volumes:
db_data:Code walkthrough
build: .uses the Dockerfile above;dbuses the official PostgreSQL image.portspublishes 8000 on the VM's own address only. Without an address, Docker's docs warn, it binds to all interfaces, "bypassing host firewall rules".env_filepasses your settings in (section 2).${DB_PASSWORD:?...}stops with that message if the value is missing.depends_onwaits for the database'shealthcheck. The volumedb_datakeeps the data when the container is replaced.
# not run here, verified against https://docs.docker.com/reference/cli/docker/compose/
docker compose up -d --build # build if needed, start both in the background
docker compose ps # what is running, and is the database healthy?
docker compose logs -f api # follow the API's output
docker compose down # stop and remove containers; the volume staysCode walkthrough: -d gives your terminal back. down keeps db_data unless you add -v, so never add it out of habit.
Break it: point the API at localhost
Set the database address to localhost:5432 and the API reports "connection refused" although the database is running. Inside a container, localhost is that container. The fix is the service name, db:5432. The error never mentions the cause.
2. Secrets and configuration (Essential)
The same image must run on your laptop, the test VM and production, with different addresses and credentials. Build three images and you no longer know which one you tested.
Configuration is everything that differs between those places. A secret is configuration that grants access, such as the CRM token. Neither belongs in the code or the image. The usual home is environment variables, set by whatever starts the service.
The token lives in a .env file listed in .gitignore; you commit a .env.example with names and no values. Compose loads the file into the container, and the code reads it once, at start-up.
import os
REQUIRED = ["CRM_BASE_URL", "CRM_TOKEN", "DATABASE_URL"]
SECRETS = {"CRM_TOKEN", "DATABASE_URL"}
class ConfigError(Exception):
pass
def load_config(env=os.environ):
missing = [name for name in REQUIRED if not env.get(name)]
if missing:
raise ConfigError("missing settings: " + ", ".join(missing))
config = {name: env[name] for name in REQUIRED}
config["LOG_LEVEL"] = env.get("LOG_LEVEL", "INFO") # harmless, so a default is fine
return config
def safe_to_log(config):
return {k: ("****" + v[-4:] if k in SECRETS else v) for k, v in config.items()}
test_vm = {"CRM_BASE_URL": "https://api.crmvendor.example/v2",
"CRM_TOKEN": "placeholder-token-7f3a",
"DATABASE_URL": "postgresql://claims:placeholder@db:5432/claims"}
print(safe_to_log(load_config(test_vm)))Code walkthrough
REQUIREDlists what the service cannot run without;SECRETSmarks what must never be printed.load_configreads the real environment by default; a dictionary makes it testable.missingcollects every absent name at once.LOG_LEVELmay have a default; a secret never does.safe_to_logprints'CRM_TOKEN': '****7f3a': enough to tell which token is loaded, not enough to use it.
Break it: start without the token
laptop = {k: v for k, v in test_vm.items() if k != "CRM_TOKEN"}
laptop["DATABASE_URL"] = ""
try:
load_config(laptop)
except ConfigError as err:
print("refused to start:", err)Code walkthrough: laptop is a typical half-filled .env: no token, empty database address. The output is refused to start: missing settings: CRM_TOKEN, DATABASE_URL. This is failing fast. Without it the service would start, look healthy, and fail on the first CRM call.
3. Linux basics: processes, permissions, logs, networking (Essential)
The VM has no desktop. You reach it over SSH, often with less access than you want, and when Mei calls, the answer is somewhere in a terminal.
# not run here (needs a Linux server), verified against each command's man page
ps aux | grep fastapi # running? one line per process, PID in column 2
ls -l .env # allowed? permissions, owner, group
journalctl -u claims --since "10 min ago" # what did it say? a service's recent logs
ss -ltnp # listening? ports and the process behind each
ulimit -n # how many files may a process open?Code walkthrough
- A process is a running program with an id, its PID.
grepkeeps matching lines. journalctl -ushows one service's logs;claimsis an example name.ss -ltnplists listening ports.127.0.0.1:8000means only the VM itself can connect.ulimit -nprints the open-files limit Slack hit.
Every file has an owner, a group and permissions: read, write and run, for owner, group and everyone else. A carelessly copied .env is often readable by every account. Colab runs on Linux, so try it.
import stat, tempfile
from pathlib import Path
env_file = Path(tempfile.mkdtemp(prefix="claims-")) / ".env"
env_file.write_text("CRM_TOKEN=placeholder-token-7f3a\n")
def show(path):
mode = path.stat().st_mode
return f"{stat.filemode(mode)} ({oct(mode & 0o777)[2:]})"
env_file.chmod(0o644)
print("as copied:", show(env_file))
env_file.chmod(0o600)
print("fixed: ", show(env_file))Code walkthrough
tempfile.mkdtempmakes a throwaway folder;stat.filemodegives the stringls -lprints.- Each digit adds read (4), write (2) and run (1). The output goes from
-rw-r--r-- (644)to-rw------- (600), owner only. On the VM:chmod 600 .env.
Break it: hit the open-files limit
Every open file and network connection counts against a per-process limit, the one Slack's provisioning service ran out of. A child process can hit it safely.
import subprocess, sys
child = r'''
import resource, tempfile
soft, hard = resource.getrlimit(resource.RLIMIT_NOFILE)
resource.setrlimit(resource.RLIMIT_NOFILE, (32, hard))
handles = []
try:
while True:
handles.append(tempfile.TemporaryFile())
except OSError as err:
print(f"limit 32: opened {len(handles)} files, then '{err.strerror}' (errno {err.errno})")
'''
out = subprocess.run([sys.executable, "-c", child], capture_output=True, text=True)
print(out.stdout.strip() or out.stderr)Code walkthrough
subprocess.runstarts a separate Python, so the lowered limit dies with it.setrlimitlowers the soft limit to 32; the loop opens files until refused.- The test run printed
opened 22 files, then 'Too many open files' (errno 24). Python already held about 10; your count may differ slightly. A web server at this wall fails new work while still looking alive.
4. Networking: DNS, TLS, proxies, firewalls (Essential)
"The tool cannot reach the CRM" is a sentence you will say in your first fortnight. Viktor can fix it in a day if you name the layer that failed.
DNS turns a name into an address. TCP opens a connection to a port. TLS encrypts and checks the server's certificate. Then HTTP carries the request. This function walks the first three and reports where it stopped.
import socket, ssl
def check(host, port=443, name=None, timeout=3):
try:
address = socket.getaddrinfo(host, port, type=socket.SOCK_STREAM)[0][4][0]
except socket.gaierror:
return f"{host}: stopped at DNS (name not found)"
try:
sock = socket.create_connection((address, port), timeout=timeout)
except ConnectionRefusedError:
return f"{host}: stopped at TCP (refused on port {port}: nothing listening)"
except OSError as err:
return f"{host}: stopped at TCP ({err}: firewall or route)"
try:
with ssl.create_default_context().wrap_socket(sock, server_hostname=name or host) as tls:
return f"{host}: DNS ok, TCP ok, TLS ok ({tls.version()})"
except (ssl.SSLError, OSError) as err:
sock.close()
return f"{host}: stopped at TLS ({getattr(err, 'verify_message', None) or err})"
print(check("pypi.org"))Code walkthrough
getaddrinfoasks DNS;create_connectionopens TCP with a timeout, so a silent firewall cannot hang it.ConnectionRefusedErrormeans the machine answered "no". OtherOSErrors, usually timeouts, point to a firewall or routing.create_default_context()checks the certificate against trusted authorities, for the name inserver_hostname.- The output is
pypi.org: DNS ok, TCP ok, TLS ok (TLSv1.3).
Break it: stop the request at each layer
print(check("claims.harbourline.invalid"))
probe = socket.socket()
probe.bind(("127.0.0.1", 0))
free_port = probe.getsockname()[1]
probe.close()
print(check("127.0.0.1", port=free_port))
print(check("pypi.org", name="claims.harbourline.example"))Code walkthrough
.invalidnames never resolve, so the first check stops at DNS.bind(("127.0.0.1", 0))borrows a free port, then closes it: TCP refused.- The last expects a Harbourline certificate from PyPI and stops at TLS: "Hostname mismatch".
A refusal is good news: the machine is reachable. A firewall is a set of rules that allow or block traffic, and most drop unwanted packets silently, so you see a timeout. "DNS resolves, TCP to port 443 times out from the VM" is a change Viktor can approve.
Companies often send outbound traffic through a proxy, a server that makes requests on your behalf. Some proxies inspect traffic and re-sign it with the company's own certificate, so every site fails with "certificate verify failed". The fix is to trust that certificate, never to switch verification off.
import os, urllib.request
os.environ["HTTPS_PROXY"] = "http://proxy.harbourline.example:8080"
os.environ["NO_PROXY"] = "localhost,127.0.0.1,.harbourline.example"
print("CRM goes via:", urllib.request.getproxies_environment()["https"])
for host in ["api.crmvendor.example", "claims.harbourline.example"]:
print(host, "skips proxy:", bool(urllib.request.proxy_bypass_environment(host)))
print("extra CA file setting:", ssl.get_default_verify_paths().openssl_cafile_env)
del os.environ["HTTPS_PROXY"], os.environ["NO_PROXY"]Code walkthrough
HTTPS_PROXYnames the proxy;NO_PROXYlists names that go direct. Both values are examples.- The CRM goes via the proxy (
False); the internal name skips it (True). SSL_CERT_FILEpoints Python'ssslat the company's certificate. The last line removes the examples.
5. One cloud well (Essential)
Harbourline's warehouse is in a cloud you did not choose. Know one cloud well enough to build in it, and translate the rest.
Harbourline runs on AWS (see the course guide), which is also the cloud in the opening case. The idea that transfers everywhere is least privilege: every person and service gets only the permissions its job needs. AWS writes permissions as JSON policies, and "by default, all requests are implicitly denied".
Mei's nightly files land in the S3 bucket harbourline-exports. The service needs to read the TMS files and nothing else. Below is a simplified model of AWS's rule, not the real evaluator.
from fnmatch import fnmatch
POLICY_DOC = {"Version": "2012-10-17",
"Statement": [{"Effect": "Allow", "Action": ["s3:GetObject"],
"Resource": ["arn:aws:s3:::harbourline-exports/tms/*"]}]}
def allowed(policy, action, resource):
decision = "deny (implicit)" # nothing has said yes yet
for st in policy["Statement"]:
if any(fnmatch(action, a) for a in st["Action"]) and \
any(fnmatch(resource, r) for r in st["Resource"]):
if st["Effect"] == "Deny":
return "deny (explicit)" # an explicit deny always wins
decision = "allow"
return decision
for action, key in [("s3:GetObject", "tms/2026-03-16.csv"),
("s3:PutObject", "tms/2026-03-16.csv"),
("s3:GetObject", "payroll/2026-03.csv")]:
print(f"{action:13} {key:20} -> {allowed(POLICY_DOC, action, 'arn:aws:s3:::harbourline-exports/' + key)}")Code walkthrough
POLICY_DOChas the shape of a real AWS policy:Effect,Action,Resource. The*matches any file undertms/.allowedstarts at deny and changes only when a statement matches action and resource; "an explicit deny overrides an explicit allow", as AWS's docs say.- Reading a TMS file prints
allow; writing it, or reading payroll,deny (implicit). Viktor can approve that this week; "admin on the account" starts a security review.
Clouds also have quotas: maximum numbers of resources per account or Region. AWS lets you "request increases", which "take some time to review, process, and approve". Before a busy day, list the quotas you depend on and check them.
6. CI/CD (Essential)
You will leave in June. The tests you run by hand leave with you, unless a machine runs them on every change.
Continuous integration (CI) runs the tests and build on every push; continuous delivery (CD) moves a passing build towards an environment the same way every time. Together they form a pipeline. Yours ends with a tested image; Harbourline's change process takes it from there (Module 6).
# .github/workflows/ci.yml, not run here, verified against https://docs.github.com/en/actions
name: ci
on:
push:
branches: [main]
pull_request:
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v7
- uses: actions/setup-python@v7
with: {python-version: "3.14"}
- run: pip install -r requirements.txt pytest
- run: pytest -q
- run: docker build -t claims-api:${{ github.sha }} .Code walkthrough
on: every push tomainand every pull request, each on a fresh Linux machine.- Version 7 of
checkoutandsetup-pythonwas the latest in October 2026. - Each
runis a shell command; any exit code but 0 "indicates the action failed" (GitHub's docs). - The commit id as image tag traces a container back to its code.
That exit-code rule is the heart of CI. This cell writes the late-credit rule and two tests, then runs them as the pipeline would.
import subprocess, sys, tempfile, textwrap
from pathlib import Path
repo = Path(tempfile.mkdtemp(prefix="claims-ci-"))
(repo / "claims.py").write_text(textwrap.dedent("""
def late_credit(freight_charge, working_days_late):
return round(0.10 * freight_charge, 2) if working_days_late > 2 else 0.0
"""))
(repo / "test_claims.py").write_text(textwrap.dedent("""
from claims import late_credit
def test_hf88301_three_days_late(): # freight 420.00, 3 working days late
assert late_credit(420.00, 3) == 42.00
def test_hf88233_one_day_late(): # freight 75.00, 1 working day late
assert late_credit(75.00, 1) == 0.0
"""))
def ci(command):
run = subprocess.run(command, shell=True, cwd=repo, capture_output=True, text=True)
return f"exit code {run.returncode}: {run.stdout.strip().splitlines()[-1]}"
print(ci(f"{sys.executable} -B -m pytest -q -p no:cacheprovider"))Code walkthrough
claims.pyholds the policy rule: 10% of freight when more than 2 working days late. The tests use HF-88301 (£42.00) and HF-88233 (nothing).ciruns a command and returns its exit code and pytest's last line.-Bstops Python caching compiled files, so the next cell's edit is seen. The output startsexit code 0: 2 passed.
Break it: a pipeline that cannot fail
(repo / "claims.py").write_text((repo / "claims.py").read_text().replace("> 2", "> 0"))
print(ci(f"{sys.executable} -B -m pytest -q -p no:cacheprovider"))
print(ci(f"{sys.executable} -B -m pytest -q -p no:cacheprovider || true"))Code walkthrough
- The edit makes any lateness earn a credit, a typical hurried slip. The run prints
exit code 1: 1 failed, 1 passed: red, correctly. || truemeans "if that failed, succeed anyway". Now it printsexit code 0over the same failure. A pipeline that cannot fail is worse than none, because people trust it.
7. Terraform or equivalent (Recommended)
A server built by clicking through a console is one nobody can rebuild exactly, including you in six months.
Infrastructure as code describes servers, networks and permissions in files kept in Git and reviewed. With Terraform, plan shows what would change and apply changes it. It records what it manages in a state file, by default local, which its docs say "risks losing workspace state"; teams keep state shared and locked. Ask Mei first. If Harbourline does not work this way, a runbook she can follow beats a tool her team does not run.
8. Kubernetes (Recommended)
You may hear "we deploy to our cluster". Know what that means, even if you never build one.
Kubernetes runs containers across a group of machines. You declare the state you want, say two copies of claims-api, and it keeps reality matching: its docs say it "restarts containers that fail". A pod is its smallest unit, one or more containers run together. For nine clerks, a VM with Compose is far easier for Mei's team to own. Use a cluster Harbourline already has; never leave them a new one.
In production
Slack's own follow-ups are a checklist. First, keep monitoring independent of what it watches: the dashboards failed "because they were running in a different VPC from their backend databases".
Second, load-test the machinery that adds capacity: Slack would "regularly load test provision-service". Third, know your ceilings before a busy day. At Harbourline, plan for the first working day after a bank holiday, when a long weekend's exceptions arrive together. Give the VM a health check Mei's monitoring can call, and agree with Viktor where alerts go.
What you added to the engagement
- A non-root
Dockerfileand acompose.yamlwith a health-checked database. - A config loader that fails fast and never prints a secret.
- A connection check that names the failing layer, and a least-privilege request.
- A CI pipeline that has been seen to go red.
Check yourself
- Inside the
apicontainer, why doeslocalhost:5432fail?
Localhost is the container itself. Use the service name, db:5432.
- Why should a secret never have a default value?
The service would start with a wrong credential and fail later. A missing secret should stop it at start-up.
- One connection times out; another is refused. What does each suggest?
A timeout usually means a firewall dropped the traffic. A refusal means the host answered and nothing listens on that port.
- Which two limits did Slack's provisioning service hit, and what would have found them earlier?
The Linux open files limit and an AWS quota. Load-testing the provisioning service.
Exercise
Put the config loader under CI. Restore the rule to > 2. Write load_config and ConfigError into repo / "config.py", and a repo / "test_config.py" testing that an empty CRM_TOKEN raises ConfigError (use pytest.raises). Run the check. Then delete the if missing: lines and watch it fail.
result = ci(f"{sys.executable} -B -m pytest -q -p no:cacheprovider")
checks = {
"config.py written": (repo / "config.py").exists(),
"test_config.py written": (repo / "test_config.py").exists(),
"pipeline green with 3 tests": result.startswith("exit code 0: 3 passed"),
}
for name, passed in checks.items():
print("PASS" if passed else "FAIL", name)
print(result)Code walkthrough: it runs the pipeline's command and reads the exit code and summary. Before you start, all three print FAIL.
Key terms
| Term | Meaning |
|---|---|
| Container image | Code plus everything it needs to run |
| Compose | One file and command for several containers |
| Secret | Configuration that grants access |
| Failing fast | Refusing to start when something required is missing |
| Permissions | Who may read, write or run a file |
| DNS / TLS | Names to addresses / encryption plus a certificate check |
| Proxy / firewall | Makes requests for you / allows or drops traffic |
| Least privilege | Only the permissions a job needs |
| Quota | A cloud's maximum for a resource |
| CI / CD | Tests on every change / repeatable delivery |
Sources
- Laura Nolan, Slack Engineering, "Slack's Outage on January 4th 2021", 2021. https://slack.engineering/slacks-outage-on-january-4th-2021/
- Docker, Dockerfile and Compose references, checked October 2026. https://docs.docker.com/reference/
- FastAPI, "FastAPI in Containers", checked October 2026. https://fastapi.tiangolo.com/deployment/docker/
- AWS, IAM policy evaluation logic, 2026. https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_evaluation-logic_policy-eval-denyallow.html
- AWS, What is Service Quotas, 2026. https://docs.aws.amazon.com/servicequotas/latest/userguide/intro.html
- GitHub, Setting exit codes for actions, 2026. https://docs.github.com/en/actions/how-tos/create-and-publish-actions/set-exit-codes
- HashiCorp, Terraform state, 2026. https://developer.hashicorp.com/terraform/language/state
- Kubernetes, Overview, 2026. https://kubernetes.io/docs/concepts/overview/