Deploy runbook — dudoxx-web-boot

**Rule first: deploys happen only on operator request.** A deploy, rollback, route change or boot setup runs from the main session only, after the operator writes an `OPERATOR_CONFIRMED` line for that target and action. Never from a worker, a push, a workflow, a runner, a git hook, a webhook, a timer or cron (`context/_invariants.md` rows 15 and 42, plan invariants 1-2). One chained ssh call at a time, never two in parallel. The engine is the scripts under `deploy/`; this page says when and in which order to call them. The detail of each step lives in the script header (`deploy/scripts/<name>.sh`) and in `./ddx-manage.sh remote help`. Ports are never written here: the one table is [ports.md](/en/docs/ports). Hosts, IPs and users come only from `~/.claude/DDX_LAN_INVENTORY.md`. ## 1. Targets | What | Where | |---|---| | Target registry (id, host, mode, rail) | `context/DEPLOY_TARGETS.md` | | One conf per target | `deploy/targets/<id>.conf` (schema in `context/DEPLOY_TARGETS.md`) | | Default target (no `--target`) | `deploy/targets/default` (today `forge`) | | App root layout on the target | `<app_root>/{repo,releases/<id>,current,shared/{env,cluster,logs,backups},.deploy-state}` | `forge` is a shared host: the scripts touch only the `pm2_namespace` and the `compose_project` of the conf, never another app. ## 2. Confirm forms | Action | Confirm line (env, set for the one command) | |---|---| | deploy | `DDX_DEPLOY_CONFIRM="OPERATOR_CONFIRMED: <id> deploy"` | | rollback | `DDX_DEPLOY_CONFIRM="OPERATOR_CONFIRMED: <id> rollback"` | | boot setup | `DDX_DEPLOY_CONFIRM="<id> boot-setup ..."` (the operator's confirm line) | | publish route | `OPERATOR_CONFIRMED="<id> publish-route"` | | remove route | `OPERATOR_CONFIRMED="<id> remove-route"` | Without the line a real run refuses and connects nowhere. `--dry-run` never needs it. ## 3. First deploy of a target (order) 1. Target prerequisites (one time): GitHub deploy key on the target for `<app_root>/repo`; BatchMode ssh from the target to its `backup_remote`; `deploy/` committed and pushed (the release archive must contain `deploy/scripts/deploy.sh`). 2. Preflight: `./ddx-manage.sh remote status --target <id>` and `deploy/scripts/deploy.sh --target <id> --ref <ref> --dry-run`. 3. Route: `./ddx-manage.sh remote route --target <id> --check`; publish only on request (section 6). 4. First deploy: `./ddx-manage.sh remote deploy --target <id> --ref <ref>` with the confirm line. `gen-secrets.sh` builds `shared/env/*.env` and `shared/cluster/*` on the target. 5. Fill the `<from-shared-infra>` keys that `gen-secrets.sh` listed as TODO (values from the shared-infra portal, never from the repo). 6. First boot (cluster admission) is automatic. Production keeps manual admission with enforced node auth (invariant 39); on an empty registry nobody can sign in, so the deploy runs the host-only ddx-api CLI `cluster:bootstrap` of the release (`ddx-api/src/cluster/CLAUDE.md`), never over HTTP: 1. `prepare`, after migrate: `mint` a join token (`shared/cluster/<id>.join-token`, 0600, TTL 3600 s) for the canary pair `api-prod-901` / `web-prod-901` and the live pair (`DDX_API_NODE_ID` / `DDX_WEB_NODE_ID`, paths from `DDX_*_NODE_JOIN_TOKEN_PATH`). A node with a registry record prints `known <id> <status>` and gets nothing. 2. After the canary pm2 start: `admit api-prod-901`, then `admit web-prod-901` (wait ≤ 60 s for `joining`). A failure stops `prepare`; the live release is untouched. 3. `swap`, after the live reload: `admit` the live api, then the live web node. A failure prints `STEP admit-live FAIL`; the live smoke then fails and rolls back. By hand on the target (cwd `<release>/ddx-api`, the ddx-api env loaded): `node dist/src/cluster/cluster-bootstrap-cli.js status`. The canary keys are made by `gen-secrets.sh`. Audit actor of every bootstrap write: `host-bootstrap`. 7. Boot persistence: `deploy/scripts/boot-setup.sh --target <id>` with the confirm line, then `--check` (section 8). 8. Smoke the public route: `deploy/scripts/smoke.sh --target <id> --public`. 9. Write the deployment report (section 10). ## 4. Deploy ``` DDX_DEPLOY_CONFIRM="OPERATOR_CONFIRMED: <id> deploy" ./ddx-manage.sh remote deploy --target <id> --ref <ref> ``` Also driven by `/ddx-deploy <target> <ref>` (main session). Flow (`deploy/scripts/deploy.sh`): preflight, phase A (fetch, `git archive` into a new `releases/<UTC ts>Z_<sha7>/`, prepare: gen-secrets, install, build, compose up, backup, migrate), canary smoke, phase C (atomic `current` swap, PM2 reload, `pm2 save`), live smoke, phase E (ok marker, prune, journal). The ref must be listed by `git ls-remote origin`. `--no-migrate` skips the migrate step. Seed policy: `seed: none` (only demo-user seeds exist; NEEDS_REVIEW in section 11). **In-flight marker.** `<app_root>/.deploy-state/inflight` holds `<release id> <user>@<host>:<pid> <UTC time>` from phase A to `finish` or `fail`. While it exists, a new deploy and a manual rollback refuse (exit 75). No auto-expiry. A crashed operator run leaves it: the refusal prints the path and the holder. Clear it by hand ONLY when no deploy runs (check `remote status` and the journal first), in one confirmed ssh call: `rm <app_root>/.deploy-state/inflight`. ## 5. Rollback ``` DDX_DEPLOY_CONFIRM="OPERATOR_CONFIRMED: <id> rollback" ./ddx-manage.sh remote rollback --target <id> [--to <release>] ``` `deploy/scripts/rollback.sh` walks the release stack: the target is `--to <release>`, else the newest release older than the live one that carries `.deploy-state/ok`. Run twice, it steps back two releases. Nothing is rebuilt. The database is not touched: migrations are forward-only, the older release stays compatible with the schema. A failed live smoke during a deploy rolls back by itself (exit 2; exit 3 when the rollback failed too). ## 6. Route `deploy/route/publish-route.sh --target <id>` (or `./ddx-manage.sh remote route`): | Mode | Effect | |---|---| | `--dry-run` | print every step, ssh line and rendered vhost; no ssh, no DNS call | | `--check` | read-only: CNAME, redirect, https health, configtest, vhosts, 503 page, UFW (own rule or an existing `allow <port>/tcp`) | | (none) | publish (same form as `sentinel.dudoxx.com`): CNAME, target vhost, UFW rule (skipped as COVERED when the port is already open), edge HTTP vhost + 503 page, cert, edge HTTPS vhost | | `--remove` | reverse order; restores a pre-existing vhost; the cert stays | Every vhost write keeps a backup and runs `apachectl configtest` before a reload. A failed configtest restores the backup (exit 3). A half-made route is undone with `--remove`. ## 7. Remote logs and status | Command | What it shows | |---|---| | `./ddx-manage.sh remote logs [api\|web\|all] --target <id>` | PM2 logs of this namespace, same filters as local logs (`--level`, `--scope`, `--req`, `--err`, `--lines`) | | `./ddx-manage.sh remote logs req <id> --target <id>` | one request across ddx-api and ddx-web, time order | | `./ddx-manage.sh remote status --target <id>` | PM2 rows, docker ps, disk, live release + sha, boot-setup check | | `./ddx-manage.sh remote pull-logs --target <id>` | rsync `shared/logs/` to `logs/remote/<target>/<UTC ts>/` | `remote logs --follow` streams until killed: run it in the background only. ## 8. Boot setup `deploy/scripts/boot-setup.sh --target <id>` makes PM2 and the prod containers come back after a host restart: PM2's own boot unit, `pm2 save`, docker enabled, restart policy on the data containers. `--with-logrotate` adds pm2-logrotate (global to the PM2 daemon: NEEDS_REVIEW on forge). `--check` is read-only and prints one `CHECK` row per item. **Never reboot a host to prove it**: `boot-setup.sh --check` is the proof. ### Node on a shared host (`node_dir`) forge is shared: other PM2 apps and the PM2 daemon run on the system Node 20. This app runs on its own Node 24 at `~/.local/opt/node-24` (conf key `node_dir=~/.local/opt/node-24/bin`). The system `node` stays as it is. - Install once per target: the official tarball `node-v24.x-linux-x64.tar.xz` (nodejs.org/dist/latest-v24.x/), `mkdir -p ~/.local/opt/node-24 && tar -xJf <tarball> --strip-components=1 -C ~/.local/opt/node-24`. `preflight.sh` FAILs the `node` row while `<node_dir>/node` is missing, is not 24.x or is not first on the ssh PATH. - Every ssh line puts `<node_dir>` before `~/.local/bin`, so pnpm, builds, gen-secrets and every remote script run on Node 24. - PM2: deploy.sh and rollback.sh export `DDX_DEPLOY_NODE_BIN=<node_dir>/node`; `deploy/ecosystem.config.cjs` sets it as the ddx-web interpreter and as the node that `wait-deps.sh` execs for ddx-api. `pm2 save` stores it, so a boot resurrect keeps Node 24. - The pm2 CLI itself never runs on Node 24. It talks to the shared PM2 daemon, can spawn it, restarts modules (`pm2 set`) and writes the boot unit (`pm2 startup`). Every pm2 call in the scripts goes through `ddx_pm2` (`deploy/scripts/lib.sh`): the ssh line exports `DDX_DEPLOY_NODE_DIR`, and `ddx_pm2` removes that dir from PATH for the one call. The host's default node then runs the CLI. `DDX_DEPLOY_NODE_BIN` stays in the env, so our apps still get Node 24. `boot-setup.sh --with-logrotate` runs `pm2 set` only for a key whose value differs from 50M / 14. - `boot-setup.sh --check` row `pm2-node` proves both apps run on `<node_dir>/node`. - A rollback to a release whose ecosystem file predates `DDX_DEPLOY_NODE_BIN` gives PM2 plain `node`; the `pm2-node` row then FAILs and names the app until the next deploy. ## 9. Secrets `deploy/scripts/gen-secrets.sh` runs ON the target (deploy.sh calls it). It renders `deploy/env/*.env.example` into `<app_root>/shared/env/{ddx-api,ddx-web,docker}.env` (files mode 600, folder 700) and the node keys into `shared/cluster/`. It never overwrites a value; it only appends missing keys. `--rotate KEY` renews one standalone key. Output holds key names only. No env file, key or token is ever committed. ## 10. Deployment report After each deploy or rollback, `deployment-specialist` (mode `report`) writes `context/deploy-reports/<target>/<YYYYMMDD-HHMM>_<from7>..<to7>.md` and appends one line to `context/deploy_journal_<host>.md`. A pull with no report is an unverified deploy. ## 11. Open NEEDS_REVIEW (owner) | Item | Decision needed | |---|---| | hms-1 `docker_subnet` | the pinned subnet is not in the inventory; confirm it is free on hms-1 | | hms-1 `backup_remote` | confirm `dudoxx-forge:backups/dudoxx-web-boot-hms-1` as the off-host copy | | pm2-logrotate on forge | approve `boot-setup.sh --with-logrotate` (rotates every app on the daemon) or accept a FAIL row in `--check` | | seed policy | `seed: none`; the super-admin seed stays a manual first-boot step | ## 12. Troubleshooting | Symptom | Check | Fix | |---|---|---| | preflight `node ... need 24.x` | `node -v` on the target (forge `/usr/bin/node` is 20) | install Node 24 user-scope (nvm/fnm in `$HOME`), its bin before `/usr/bin` | | preflight `pnpm missing or broken` | `pnpm -v`; corepack cannot run the pinned major | `npm install -g --prefix ~/.local pnpm@<major>` (major from root `package.json`) | | `pnpm` / `pm2` not found over ssh | `echo $PATH` in a non-login ssh shell | `lib.sh` puts `~/.local/bin` first; install tools with `npm -g --prefix ~/.local` | | backup step fails, no `pg_dump` | `command -v pg_dump` on the host is empty by design | nothing to install: `backup-db.sh` runs `pg_dump` inside the `<compose_project>-postgres` container | | route `--check` https fails, edge times out | `sudo ufw status` on the target (forge default DROP) | publish adds the rule for `edge_ip`; re-run `publish-route.sh` with its confirm line | | compose up fails, pool overlaps | `docker network ls` + subnets on the target | set a free `docker_subnet` in the conf (pinned `172.30.x`) | | deploy or rollback exit 75 | `<app_root>/.deploy-state/inflight` and the lock | wait for the running deploy; clear a stale marker by hand (section 4) | | `API_NODE_NOT_ADMITTED` in ddx-api log | `nodes` state of the node id | run the bootstrap (section 3 step 6) | | browser shows only "Loading the page…" | console: `Executing inline script violates ... script-src` | production CSP must keep `script-src 'self' 'unsafe-inline'` (`ddx-web/next.config.ts`): App Router flight data is inline; a nonce does not fit `cacheComponents`, SRI skips inline scripts | | sign-in / any POST answers 403 `CSRF_ORIGIN` | ddx-web log `Cross-origin state change refused origin=foreign` | behind a proxy `nextUrl.origin` is the bind address; `NEXT_PUBLIC_APP_URL` in `ddx-web.env` must be the public `https://<fqdn>` (the BFF origin gate accepts its origin; it is baked at build, so redeploy) | | one 502 right after a deploy | ddx-web log `Upstream not delivered` with `nodeId` `api-prod-901` | the deleted canary api is still in the pool for its lease (15 s); retry. Open item: drain the canary before delete | | a case-only file rename builds on the Mac, fails on the target | `git ls-files <dir>` vs the disk name | `git mv -f <Old> <new>` (macOS ignores case, Linux does not) |