ADR 0018 — Cluster node registry, node identity, hot plug

• Status: Accepted (plan bff-layered, tasks 024-034, 036, 037, 2026-10-24).
• Extends: ADR 0017 (the bridge and pool live in the BFF layers).
• Uses: ADR 0016 (Redis is already required; SSE replay moves streams on drain).
• Plan: plans/bff-layered/ (11-cluster-registry.md, 12-node-security.md, 13-env-contract.md, naming 10-naming.md).
• Security evidence: findings/bff-security/re-review.md.

Context

The ddx-api pool was a dotenv list. Adding or removing a backend needed a ddx-web restart. Production runs several ddx-web nodes against several ddx-api nodes over HA Postgres, Redis (Sentinel), distributed MinIO and Qdrant. Every node must be identified, secured, hot-pluggable and decommissionable.

Decision

1. The pool is data. A registry in the Redis the stack already runs, owned by ddx-api (ClusterRegistryService, keys ddx:cluster:<clusterId>:*). ddx-web nodes never touch Redis. BffNodeRegistry reads a snapshot from any ddx-api seed (GET /cluster/nodes) and holds one SSE watch (GET /cluster/nodes/watch) to rebuild BffNodePool copy-on-write. Dotenv holds only identity, seeds and secret file paths.
2. Lifecycle (ClusterLifecycle, pure table): joining → active → draining → decommissioned; any non-terminal → quarantined. Only active api nodes enter a pool. quarantined(health) recovers after 3 good heartbeats; quarantined(operator) sticks until admit. A decommissioned node id is never reused.
3. Node identity. Every node (web and api) owns an Ed25519 key, pinned in the registry at admission. Join: operator ./ddx-manage.sh nodes token → one-time token (Redis stores sha256(token), GETDEL on use) → POST /cluster/join with a proof signature and selfSig. Admit is manual in production; auto is development only.
4. Node token. A ddx-web → ddx-api call carries the user JWT AND x-ddx-node-token (EdDSA JWT via jose, exp − iat ≤ 60 s). BffNodeIdentity mints a NEW token per call (task 036); the jti is single-use (SET … NX PX), so a token is never cached. ApiNodeAuthMiddleware verifies it ahead of all guards. Modes DDX_CLUSTER_NODE_AUTH=off|warn|enforce; production accepts only enforce.
5. Heartbeat self-heal. A quarantined(lost|health) web node recovers through ClusterHeartbeatTokenGuard on POST /cluster/nodes/heartbeat only (task 037; isSelfHealing in api-node-heartbeat.service.ts). A non-healable quarantine answers 401 NODE_NOT_ADMITTED.
6. Drain. nodes drain <id> → draining; pools stop picking it; /health/ready answers 503; SSE clients get one system.reconnect frame (retry: 500), reconnect with Last-Event-ID, and BffStreamRelay picks another node and rewrites ddx_bff_sticky. Auto decommissioned at inflight=0 && streams=0 or after DDX_API_NODE_DRAIN_TIMEOUT_S.
7. Readiness per dependency (Postgres ×6, Redis, MinIO, Qdrant) on GET /health/ready; GET /health/live is the liveness alias.

Rejected: etcd / Consul (a fourth HA system for under 100 keys; Redis is already required). mTLS (cluster CA, per-hop TLS config, an L7 LB drops the client cert). A dotenv backend list (restart to change the pool). Shared service API keys (one leaked key admits any process, no per-node revocation). An ioredis subscriber inside ddx-web (new dependency, data-tier credentials on the edge tier).

Routes (ClusterAdminController, /cluster/*, skip org guards)

Environment (DDX_<APP>_<SCOPE>_<KEY>)

Full key table: ddx-web/CLAUDE.md §Environment and ddx-api/CLAUDE.md §Cluster. Schemas: packages/ddx-contracts/src/cluster.ts. DDX_API_URL is a deprecated static-pool fallback (ddx-web/src/env.ts:36) read for ONE release with warn ENV_DEPRECATED; removal release: the release after this plan ships (owner sets the number at deploy).

Development

Default DDX_BFF_POOL_SOURCE=static, node auth warn. Registry mode: ./ddx-manage.sh start api2 runs a second ddx-api on 16311 (api-dev-002). Static pool with NODE_ENV=production is a boot error BFF_POOL_STATIC_IN_PROD.

Consequences and accepted risks

• Web pin of an api node key is log-only (no TOFU enforcement); a forged record needs a Redis write or a compromised api node, all inside the clinic (re-review S1). Manual admit keeps a joined node joining, never pooled.
• ddx-web never sends a service/user API key, so key credentials skip the node token (S3).
• Invariants 37-40 in context/_invariants.md; gate ./ddx-manage.sh bff-layers.

Last reconciled: 2026-10-24 · plan bff-layered task 016

• ddx-documentation — dudoxx-web-boot

adr/

• ADR 0001 — Scaffold route: dudoxx-sandbox-creator with 163XX overrides
• ADR 0002 — Auth: better-auth 1.7 in ddx-api
• ADR 0003 — Route shape: org slug, role, [entity]/[id], ?view=
• ADR 0004 — Database split: one Postgres 18 instance, six databases
• ADR 0005 — Frameworks and the approved dependency table
• ADR 0006 — API clients come from one factory
• ADR 0007 — Typed service container (DI / IoC)
• ADR 0008 — Singleton scope: what may live at module level
• ADR 0009 — One variant API for every inventory component
• ADR 0010 — Media path: presigned URLs from ddx-api after the org check
• ADR 0011 — Media proxy: MinIO only through ddx-api
• ADR 0012 — Density names: compact | cosy | large
• ADR 0013 — Platform super-admin acts as org-admin of the target org
• ADR 0014 — Unified settings: one registry, three tiers, one resolve in ddx-api
• ADR 0015 — Entity kernel: one descriptor, one access seam, one projection, one proof chain
• ADR 0016 — SSE Bus: Redis Streams log, Redis pub/sub fan-out, BullMQ outbound queue
• ADR 0017 — Layered BFF: one route table, route classes, read models
• ADR 0018 — Cluster node registry, node identity, hot plug
• ADR 0019 — Tenant branding: settings keys, vendored palettes, public platform read
• Debug overlay

deploy/

• Deploy runbook — dudoxx-web-boot
• Entity kernel — guide
• Feature map — what each role can do
• Component inventory
• Naming constitution — dudoxx-web-boot
• Observability — one log layout, one key set
• Pattern reference — P1-P23 and groups A-K
• Port map — dudoxx-web-boot

research/

• RQ1 — Which release is "latest", and does it differ from the canon?
• RQ10 — SSE through the Next.js BFF route handler (desk half)
• RQ11 — Dev latency baseline: cold compile per route + next-server RSS
• RQ2 — Route group or dynamic org slug?
• RQ3 — better-auth 1.7 vs Auth.js v5: feature matrix
• RQ4 — Separate databases or separate schemas?
• RQ5 — Which frameworks beyond shadcn earn a place?
• RQ6 — How do DI, IoC and Singleton behave across RSC, client bundles and NestJS?
• RQ7 — Which data uses the server cache, which uses TanStack Query, how is each invalidated?
• RQ8 — Media path: presigned URLs vs streaming through the BFF
• Template triage — create-nextjs code in ddx-web
• Settings — guide
• Local setup — dudoxx-web-boot
• Shell tree — SiteShell and PortalShell
• SSE event catalog
• Version pins — dudoxx-web-boot