Bare Metal Day 2 Runbook

Use this skill when the user wants to execute or rehearse a Bare Metal Software Day 2 operational procedure against their own system - restoring from backup, rotating a secret, shipping a hotfix, onboarding or offboarding a customer, handling a vulnerable…

This skill is Labor as Code™ for "Rehearse the Eight Things That Will Go Wrong Before They Happen" in Bare Metal Software's "Run It for Years" chapter. Read that chapter for why each step exists — this skill only executes the eight procedures it describes. It does not decide which procedure applies, whether this is a rehearsal or a real incident, whether the user's actual stack matches the book's assumptions, or whether a destructive step (deleting a customer's data, revoking a key) is authorized. Those are always the user's calls, confirmed before anything runs, never guessed from context.

Before running

Confirm all of these before executing a single command:

1. Which of the eight procedures (listed below) applies. If none of them cleanly fits what the user is dealing with, say so — this skill is not a general incident-response tool, only these eight.
2. Drill or real incident. A monthly restore drill against a scratch machine and a real production restore during an actual outage carry very different risk. Ask explicitly. Never treat a successful drill as proof the real procedure will work if it has never actually been run against production — the book's own point is that an unrehearsed runbook is a wish, not evidence.
3. Does the user's actual stack match the book's assumptions — one container holding the Rust program and Litestream (which starts the program and streams SQLite changes to object storage), SQLite on a named volume, TLS handled by the program, one Compose file, named ops scripts run by a restricted deploy account, secrets in one environment file on the host? If the real stack differs (files or PostgreSQL instead of SQLite, no containers, a different runtime), say so up front and adapt the checklist's steps without inventing command syntax you haven't confirmed exists in their environment. Ask for the equivalent commands rather than guessing them.
4. Explicit go-ahead for anything irreversible, and for every production change — deleting a customer's database and files at the end of offboarding, revoking an old secret, restarting or replacing a production container. The book has a person approve every ops-script run that changes production, and it never gives the code-writing session production credentials; run production steps only through the project's named ops scripts. Draft the plan, show it, and get a clear yes before executing the destructive step. This skill drafts and executes the mechanical steps; it does not decide that this is the moment to delete a customer's data.

The eight procedures

Work through only the one(s) the user asked for. Each is the book's own sequence, unchanged — do not skip a step because it seems redundant in the moment; the sequence is short specifically because nothing extra is in it.

1. Restore from backup

1. Stop the application container.
2. Pull the latest Litestream snapshot for the affected customer file(s), and the nightly files-folder copy, into a fresh volume.
3. Start the application container against the restored volume.
4. Run the health check and the full constitutional test suite against it.
5. Record the elapsed time from step 1 to a passing health check.

Do not use cp against a live database. A SQLite database in WAL mode is three files (the database, its write-ahead log, and a shared-memory file); copying them while the program is running can produce a copy that will not open. Type this instead, while the program is stopped:

sqlite3 customer.db "VACUUM INTO 'restore/customer.db'"

If this is the monthly drill: run it against a scratch machine, using last night's snapshot, and report the elapsed time against the project's committed number (the book's own default target is under fifteen minutes, at most one minute of writes lost — use the project's own stated number if it differs). A restore that exceeds the committed number is itself the finding — report it, do not quietly note it and move on.

2. Rotate a secret

The order never changes — new key in, verify, old key out — because reversing it is how a rotation locks you out instead of protecting you:

1. Create the new key at the provider.
2. Put it in the secrets file (never in code, never in the database).
3. Restart the application container.
4. Confirm one real transaction succeeded with the new key — one email sent, one backup completed, whatever the secret is for.
5. Only after step 4 succeeds: revoke the old key at the provider.

Ask which provider and confirm the verification action (step 4) before starting — "confirm it works" needs a concrete, checkable thing to confirm, not a guess that it probably worked.

3. Ship a hotfix

1. Write the failing test first, directly from the bug report. Do not write the fix before the failing test exists.
2. Make the test pass, without weakening or deleting any protected (constitutional) test to do it. If passing the new test seems to require weakening a protected test, stop — that is a sign the fix is wrong, not that the protected test is.
3. Run the build script to produce the new container image.
4. Run the deploy script: it copies the image to the host and replaces the running container.
5. Watch the status page for at least two minutes after restart.
6. Record total elapsed time from bug report to verified fix. The book's own target for a small bug is under an hour, with the test staying in the suite permanently.

4. Onboard a customer

1. Create the customer row in the control database.
2. Have the program create the customer's database file and run the schema against it.
3. Create the owner account and send the sign-in link.
4. Configure the customer's settings rows.
5. Add the customer to the Litestream backup configuration — this should be generated by the program from the customer list, not hand-edited; if the user's project hand-edits this file, flag it as a gap against the book's default rather than silently reproducing the manual edit. Litestream reads its configuration when it starts, so restart the application container (with the user's go-ahead; it takes a few seconds).
6. Confirm the first backup for this customer actually landed in object storage before calling onboarding complete.

5. Offboard a customer

1. Export the customer's database file and files folder to a dated archive.
2. Hand the archive to the customer in a format they can actually use (the book names CSV/Parquet-style exports elsewhere — confirm the expected format with the user rather than assuming).
3. Record the handover in the audit table.
4. Stop here and confirm the retention period named in the project's own spec before deleting anything — do not delete on the same visit unless the spec's retention period has actually elapsed and the user confirms it.
5. After the retention period: delete the database file, the files, and the backups, and record the deletion in the audit table.

6. Handle a vulnerable dependency

1. Read the security advisory the build flagged.
2. Update or replace the affected dependency.
3. If the dependency is vendored (its source copied into the repository), update the vendored copy in the same commit — do not leave a stale vendored copy alongside an updated lock file.
4. Run the full test suite.
5. Report exactly what changed.
6. Ship it through the hotfix procedure above (Step 3), unless the advisory is in the language toolchain, the base image, or the operating system itself — then it goes through patch day, or an emergency patch day if the severity warrants one. Ask the user which applies before treating this as a normal hotfix.

7. Upgrade the language and libraries (quarterly)

1. Confirm this is the project's designated upgrade day (the spec should name one; ask if it doesn't and don't invent a date).
2. Move the pinned toolchain version, the container base image, and every dependency forward.
3. Build and run the full test suite.
4. Open one change for human review — do not merge this without review, even though it is routine. Small regular steps are cheap specifically because they stay small and reviewed; skipping review on the grounds that it's "just the quarterly bump" is how a project ends up on an unsupported version nobody dared touch.

8. Adopt an abandoned dependency

1. Confirm the signal the quarterly upgrade found: no release for a year, an issue tracker nobody answers, or a maintainer's farewell note. Name which one, with the evidence, rather than calling a library abandoned because it is quiet.
2. Move the vendored copy out of the vendor folder into a folder the spec owns.
3. Add the library's own test suite to the project's suite, and run the full suite.
4. Record in the decisions folder that this library is now generated code with a history. From then on it is maintained like the rest of the code, from the spec, against the tests.
5. If the library turns out to be too large to maintain that way, stop and tell the user: the dependency ledger was wrong about it, and the replacement path in its Exit row runs now, while the old copy still builds. Choosing to take the replacement path is the user's call; do not start it without a clear yes.

The check that catches this chapter's own failure mode

Before telling the user a procedure is done, confirm every step above was actually executed and its result actually observed — not assumed. "A runbook nobody has run is a wish" is the book's own line, and it applies as much to this skill's execution as to a human's: do not mark the restore drill successful without having actually run the health check and the test suite against the restored data, do not mark a secret rotation complete without having actually confirmed a real transaction succeeded on the new key, and do not report an elapsed time you didn't actually measure. If any step couldn't be verified, say so explicitly rather than reporting the procedure as complete.