---
name: bare-metal-day-2-runbook
description: Use this skill when the user wants to execute or rehearse a Bare Metal Software Day 2 operational procedure against their own system - restoring from backup, rotating a secret, shipping a hotfix, onboarding or offboarding a customer, handling a vulnerable dependency, doing the quarterly toolchain upgrade, or adopting an abandoned dependency. Triggers on requests like "run the restore drill," "rotate our email provider key," "walk me through shipping this hotfix," "offboard this customer following the runbook," "we have a vulnerable dependency, what's the procedure," "do our quarterly dependency upgrade," or "this library looks abandoned, adopt it."
---

# Bare Metal Day 2 Runbook
This skill is Labor as Code™ for "Rehearse the Eight Things That Will Go Wrong Before They Happen" in *Bare Metal
Software*'s "Run It for Years" chapter. Read that chapter for why each
step exists — this skill only executes the eight procedures it
describes. It does not decide which procedure applies, whether this is
a rehearsal or a real incident, whether the user's actual stack
matches the book's assumptions, or whether a destructive step (deleting
a customer's data, revoking a key) is authorized. Those are always the
user's calls, confirmed before anything runs, never guessed from
context.

## Before running

Confirm all of these before executing a single command:

1. **Which of the eight procedures** (listed below) applies. If none
   of them cleanly fits what the user is dealing with, say so — this
   skill is not a general incident-response tool, only these eight.
2. **Drill or real incident.** A monthly restore drill against a
   scratch machine and a real production restore during an actual
   outage carry very different risk. Ask explicitly. Never treat a
   successful drill as proof the real procedure will work if it has
   never actually been run against production — the book's own point
   is that an unrehearsed runbook is a wish, not evidence.
3. **Does the user's actual stack match the book's assumptions** — one
   container holding the Rust program and Litestream (which starts the
   program and streams SQLite changes to object storage), SQLite on a
   named volume, TLS handled by the program, one Compose file, named ops scripts
   run by a restricted deploy account, secrets in one environment file
   on the host? If the real stack differs (files or PostgreSQL instead
   of SQLite, no containers, a different runtime), say so up front and adapt the checklist's *steps*
   without inventing command syntax you haven't confirmed exists in
   their environment. Ask for the equivalent commands rather than
   guessing them.
4. **Explicit go-ahead for anything irreversible, and for every
   production change** — deleting a customer's database and files at
   the end of offboarding, revoking an old secret, restarting or
   replacing a production container. The book has a person approve
   every ops-script run that changes production, and it never gives the
   code-writing session production credentials; run production steps
   only through the project's named ops scripts. Draft the plan,
   show it, and get a clear yes before executing the destructive step.
   This skill drafts and executes the mechanical steps; it does not
   decide that this is the moment to delete a customer's data.

## The eight procedures

Work through only the one(s) the user asked for. Each is the book's
own sequence, unchanged — do not skip a step because it seems
redundant in the moment; the sequence is short specifically because
nothing extra is in it.

### 1. Restore from backup

1. Stop the application container.
2. Pull the latest Litestream snapshot for the affected customer
   file(s), and the nightly files-folder copy, into a fresh volume.
3. Start the application container against the restored volume.
4. Run the health check and the full constitutional test suite against
   it.
5. Record the elapsed time from step 1 to a passing health check.

**Do not use `cp` against a live database.** A SQLite database in WAL
mode is three files (the database, its write-ahead log, and a shared-memory file); copying them
while the program is running can produce a copy that will not open.
Type this instead, while the program is stopped:

```
sqlite3 customer.db "VACUUM INTO 'restore/customer.db'"
```

If this is the monthly drill: run it against a scratch machine, using
last night's snapshot, and report the elapsed time against the
project's committed number (the book's own default target is under
fifteen minutes, at most one minute of writes lost — use the project's
own stated number if it differs). A restore that exceeds the committed
number is itself the finding — report it, do not quietly note it and
move on.

### 2. Rotate a secret

The order never changes — new key in, verify, old key out — because
reversing it is how a rotation locks you out instead of protecting
you:

1. Create the new key at the provider.
2. Put it in the secrets file (never in code, never in the database).
3. Restart the application container.
4. Confirm one real transaction succeeded with the new key — one email
   sent, one backup completed, whatever the secret is for.
5. Only after step 4 succeeds: revoke the old key at the provider.

Ask which provider and confirm the verification action (step 4) before
starting — "confirm it works" needs a concrete, checkable thing to
confirm, not a guess that it probably worked.

### 3. Ship a hotfix

1. Write the failing test first, directly from the bug report. Do not
   write the fix before the failing test exists.
2. Make the test pass, without weakening or deleting any protected
   (constitutional) test to do it. If passing the new test seems to
   require weakening a protected test, stop — that is a sign the fix
   is wrong, not that the protected test is.
3. Run the build script to produce the new container image.
4. Run the deploy script: it copies the image to the host and replaces
   the running container.
5. Watch the status page for at least two minutes after restart.
6. Record total elapsed time from bug report to verified fix. The
   book's own target for a small bug is under an hour, with the test
   staying in the suite permanently.

### 4. Onboard a customer

1. Create the customer row in the control database.
2. Have the program create the customer's database file and run the
   schema against it.
3. Create the owner account and send the sign-in link.
4. Configure the customer's settings rows.
5. Add the customer to the Litestream backup configuration — this
   should be generated by the program from the customer list, not
   hand-edited; if the user's project hand-edits this file, flag it as
   a gap against the book's default rather than silently reproducing
   the manual edit. Litestream reads its configuration when it starts,
   so restart the application container (with the user's go-ahead;
   it takes a few seconds).
6. Confirm the first backup for this customer actually landed in
   object storage before calling onboarding complete.

### 5. Offboard a customer

1. Export the customer's database file and files folder to a dated
   archive.
2. Hand the archive to the customer in a format they can actually use
   (the book names CSV/Parquet-style exports elsewhere — confirm the
   expected format with the user rather than assuming).
3. Record the handover in the audit table.
4. **Stop here and confirm the retention period** named in the
   project's own spec before deleting anything — do not delete on the
   same visit unless the spec's retention period has actually elapsed
   and the user confirms it.
5. After the retention period: delete the database file, the files,
   and the backups, and record the deletion in the audit table.

### 6. Handle a vulnerable dependency

1. Read the security advisory the build flagged.
2. Update or replace the affected dependency.
3. If the dependency is vendored (its source copied into the
   repository), update the vendored copy in the same commit — do not
   leave a stale vendored copy alongside an updated lock file.
4. Run the full test suite.
5. Report exactly what changed.
6. Ship it through the hotfix procedure above (Step 3), unless the
   advisory is in the language toolchain, the base image, or the
   operating system itself —
   then it goes through patch day, or an emergency patch day if the
   severity warrants one. Ask the user which applies before treating
   this as a normal hotfix.

### 7. Upgrade the language and libraries (quarterly)

1. Confirm this is the project's designated upgrade day (the spec
   should name one; ask if it doesn't and don't invent a date).
2. Move the pinned toolchain version, the container base image, and
   every dependency forward.
3. Build and run the full test suite.
4. Open one change for human review — do not merge this without
   review, even though it is routine. Small regular steps are cheap
   specifically because they stay small and reviewed; skipping review
   on the grounds that it's "just the quarterly bump" is how a project
   ends up on an unsupported version nobody dared touch.

### 8. Adopt an abandoned dependency

1. Confirm the signal the quarterly upgrade found: no release for a
   year, an issue tracker nobody answers, or a maintainer's farewell
   note. Name which one, with the evidence, rather than calling a
   library abandoned because it is quiet.
2. Move the vendored copy out of the vendor folder into a folder the
   spec owns.
3. Add the library's own test suite to the project's suite, and run
   the full suite.
4. Record in the decisions folder that this library is now generated
   code with a history. From then on it is maintained like the rest of
   the code, from the spec, against the tests.
5. If the library turns out to be too large to maintain that way, stop
   and tell the user: the dependency ledger was wrong about it, and the
   replacement path in its Exit row runs now, while the old copy still
   builds. Choosing to take the replacement path is the user's call;
   do not start it without a clear yes.

## The check that catches this chapter's own failure mode

Before telling the user a procedure is done, confirm every step above
was actually executed and its result actually observed — not assumed.
"A runbook nobody has run is a wish" is the book's own line, and it
applies as much to this skill's execution as to a human's: do not mark
the restore drill successful without having actually run the health
check and the test suite against the restored data, do not mark a
secret rotation complete without having actually confirmed a real
transaction succeeded on the new key, and do not report an elapsed
time you didn't actually measure. If any step couldn't be verified,
say so explicitly rather than reporting the procedure as complete.