R&D documentation

ADR 0016 — SSE Bus: Redis Streams log, Redis pub/sub fan-out, BullMQ outbound queue

Source: ddx-documentation/adr/0016-sse-bus.md

Previous: ADR 0015 — Entity kernel: one descriptor, one access seam, one projection, one proof chain Next: ADR 0017 — Layered BFF: one route table, route classes, read models Inventory gallery

- **Status:** Accepted (plan `sse-bus`, task 001, 2026-10-23). - **Amends:** [ADR 0005](/en/docs/adr/0005-frameworks) approved dependency table (one row points here). - **Refines:** [ADR 0014](/en/docs/adr/0014-unified-settings) (`settings.changed` events), [ADR 0015](/en/docs/adr/0015-entity-kernel) (`entity.changed` events, `AccessPolicy` audience). - **Plan:** `plans/sse-bus/` (overview `00-overview.md`, invariants `_invariants.md`). - **Structure reference:** `findings/hms-sse-architecture/report.md` (HMS = structure only; no code or values copied). - **Ports:** [../ports.md](/en/docs/ports) (Redis host port 16340). ## Context - **Reconnect storm (owner log, lead note 2026-10-23).** The BFF logged `GET /api/v1/events 200 in 3ms` ×197 and ddx-api answered `429 ThrottlerException` on `/events`. The web manager reset its backoff counter on every open, so a stream that opened and closed at once retried every ~1 s, per tab. The lead fixed the counter (reset only after 10 s stable open). The root causes stay: every tab opens its own stream, the generic throttler guards `/events`, and every client retries in lockstep after an API restart. - **Reload hang (hang-fix, 2026-10-23).** Open SSE streams kept the old ddx-api process alive on a watch reload. `SseHub.closed$` + `takeUntil` and `return503OnClosing` fixed the hang. A restart still drops every stream at the same moment. - **In-memory replay, epoch resync.** The hub keeps the replay buffer in process memory. Event ids carry a process epoch. After a restart the buffer is empty and every client gets `resync` (`epoch-changed`), so every tab refetches everything at once — a second wave on top of the reconnect wave. - **One process, no queue.** Fan-out works only inside one ddx-api process. Nothing durable carries work out of the request (webhooks to other systems, retries). ## Decision 1. **Redis is the bus backbone.** One Redis server (image below) on host port 16340, container port 6379, owned by `ddx-docker-redis/`. AOF on; `maxmemory-policy noeviction` (BullMQ needs it). ddx-api reads `REDIS_URL` (`redis://127.0.0.1:16340` in dev). 2. **Redis Streams = the event log.** Each durable event is `XADD`ed (approximate `MAXLEN` cap) to a per-channel stream BEFORE it is published, so "logged ≥ delivered". The Stream entry id `<ms>-<seq>` IS the SSE `id`. Ids survive an API restart; the `epoch-changed` resync wave ends. Replay = `XRANGE (<Last-Event-ID> +`. A trimmed gap answers `resync` (`buffer-trimmed`); a Redis error answers `resync` (`replay-failed`), never an empty success. Volatile types (heartbeat, presence, system notices) never enter the log and never carry an id. 3. **Redis pub/sub = the fan-out.** `SseHub.publish` = validate → `XADD` (durable types) → `PUBLISH` → each hub filters locally with the one ACL seam (`isVisibleTo`, audience from `AccessPolicy.audienceOf`). Live and replayed frames pass the same filter. Domain code never calls Redis; only the hub and its transport do. Two connections per process (command + subscriber), both from one `platform/redis` provider. 4. **BullMQ = the outbound queue.** Outbound webhook deliveries run as BullMQ jobs on the same Redis: attempts + exponential backoff, dead-letter after the last attempt. The worker never re-enqueues its own job and never sends a user- or audience-scoped event to an outbound URL. 5. **HMAC webhooks.** Inbound `POST /webhooks/:source/:event`: HMAC-SHA256, timestamp skew check, per-source event whitelist, then a nonce claim (`SET NX PX`) — the nonce is claimed only AFTER the MAC verifies. Outbound deliveries are signed the same way. Header names, skew and TTLs live in `@ddx/contracts/sse-webhooks` (task 002), not here. 6. **Root-provider connection.** ddx-web opens ONE `EventSource` per browser, from `ddx-web/src/lib/sse/manager` only, owned by a provider at the root of the `(app)` group. The portal layout only sets the scope (org, user); a scope change reconnects. 7. **Leader tab.** Tabs elect one leader with Web Locks (`navigator.locks`); the leader holds the stream and relays frames to the other tabs over `BroadcastChannel`. Without those APIs each tab connects on its own (degraded, still correct). 8. **Load shaping.** `/events` skips the generic throttler and gets a per-principal connection limiter instead. Clients use full-jitter backoff and honour the server `retry:` hint; on shutdown the hub sends a `system.notice` with a jittered `retry:`. ## Approved dependencies Resolved live on **2026-10-23** (UTC 15:36) on dudoxxmac. `~/.claude/ddx-topic-flows/VERSIONS.md` has **no** `redis`, `ioredis` or `bullmq` row on 2026-10-23 (`rg -i 'redis|bullmq'` → no match), so the version source is the recorded command, per ADR 0005 Decision 4. Not copied from HMS. | Package / image | App | Kind | Approved version | Resolved | Version source (command, 2026-10-23) | |---|---|---|---|---|---| | `redis` Docker official image | `ddx-docker-redis/` | image | `redis:8.10.2-alpine3.23` **exact tag** | 8.10.2 (tag pushed 2026-10-22) | Docker Hub `GET /v2/repositories/library/redis/tags?ordering=last_updated` — newest GA alpine tag; `8.10-rc2-*` is an RC and is excluded | | `ioredis` | ddx-api | dep | `^6.0.0` | 6.0.0 (dist-tag `latest`, published 2026-07-31) | `pnpm view ioredis version` → `6.0.0` | | `bullmq` | ddx-api | dep | `^6.4.0` | 6.4.0 (dist-tag `latest`, published 2026-10-21) | `pnpm view bullmq version` → `6.4.0` | Notes: - `bullmq@6.4.0` declares `ioredis >=5.0.0` as a peer (`pnpm view bullmq@6.4.0 peerDependencies`), so `ioredis ^6.0.0` satisfies it. BullMQ docs require Redis ≥ 6.2 and recommend 7+; 8.10.2 meets both. - Ranges are caret within the major (ADR 0005 Decision 4). A major bump (bullmq 7, ioredis 7, redis 9) needs a new ADR. The Redis image is pinned to the exact tag; a tag bump is a one-line edit of this table plus the compose file. - `bullmq@6.4.0` is 2 days old on 2026-10-23. If the pnpm 12 release-age gate rejects it, the install task adds `bullmq@6.4.0` to `minimumReleaseAgeExclude` in `pnpm-workspace.yaml` and lists that file in its task files. - No other package is approved by this ADR (`redis` npm client, `bullmq-otel` and `pg` peers stay uninstalled). ## Rejected | Option | Why not | |---|---| | In-memory fallback when Redis is down | Silent split brain: events published while Redis is down reach only one process and never enter the log, so replay lies. Plan invariant: never fall back to in-memory fan-out silently. Redis down = `/health` not ready + a loud error. | | WebSocket | The traffic is server → browser only. SSE gives `Last-Event-ID` resume, plain HTTP through the BFF and the existing `api/v1` pipe. A socket adds a second protocol and its own auth path. | | Centrifugo / Mercure / NATS / `better-sse` | A second server (or a library hub) beside ddx-api: a second ACL seam outside `AccessPolicy`, a second port, a second auth path. HMS defect D1 (two hubs). `better-sse` keeps state in process — the defect this ADR removes. | | Kafka | Ops weight (cluster, topics, retention tuning) far beyond one dev workspace; Redis Streams give ordered ids, replay and trimming at this scale, and BullMQ needs Redis anyway. | ## Consequences - **Redis is a hard dependency.** ddx-api refuses to start without `REDIS_URL` (env validation marks it required, no default). `./ddx-manage.sh` starts the Redis container with Postgres and MinIO. - Good: event ids survive a restart; a resuming client gets replay, not a `resync` wave. - Good: fan-out is multi-instance ready (pub/sub); one ACL seam for live, replayed and bridged frames. - Good: one browser connection per user (leader tab) cuts the per-tab load and the 429 loop. - Cost: one more container, one more port (16340), one more thing that can be down in dev. - Risk: Stream memory. Each stream is capped (`MAXLEN ~`) and the cap is a named constant in the hub `POLICY.md`; a trimmed cursor answers `resync`, never silence. ## Validation ```sh rg -q 'bullmq' ddx-documentation/adr/0016-sse-bus.md && rg -q 'ioredis' ddx-documentation/adr/0016-sse-bus.md rg -q '0016' ddx-documentation/adr/0005-frameworks.md rg -q '16340' ddx-documentation/ports.md ``` --- Last reconciled: 2026-10-23 · task 001 (sse-bus)

All pages (45)

- [ddx-documentation — dudoxx-web-boot](/en/docs) **adr/** - [ADR 0001 — Scaffold route: dudoxx-sandbox-creator with 163XX overrides](/en/docs/adr/0001-scaffold-route) - [ADR 0002 — Auth: better-auth 1.7 in ddx-api](/en/docs/adr/0002-auth-better-auth) - [ADR 0003 — Route shape: org slug, role, `[entity]/[id]`, `?view=`](/en/docs/adr/0003-route-shape) - [ADR 0004 — Database split: one Postgres 18 instance, six databases](/en/docs/adr/0004-database-split) - [ADR 0005 — Frameworks and the approved dependency table](/en/docs/adr/0005-frameworks) - [ADR 0006 — API clients come from one factory](/en/docs/adr/0006-factory) - [ADR 0007 — Typed service container (DI / IoC)](/en/docs/adr/0007-di-container) - [ADR 0008 — Singleton scope: what may live at module level](/en/docs/adr/0008-singleton-scope) - [ADR 0009 — One variant API for every inventory component](/en/docs/adr/0009-variant-api) - [ADR 0010 — Media path: presigned URLs from ddx-api after the org check](/en/docs/adr/0010-media-path) - [ADR 0011 — Media proxy: MinIO only through ddx-api](/en/docs/adr/0011-media-proxy) - [ADR 0012 — Density names: compact | cosy | large](/en/docs/adr/0012-density-names) - [ADR 0013 — Platform super-admin acts as org-admin of the target org](/en/docs/adr/0013-super-admin-platform-role) - [ADR 0014 — Unified settings: one registry, three tiers, one resolve in ddx-api](/en/docs/adr/0014-unified-settings) - [ADR 0015 — Entity kernel: one descriptor, one access seam, one projection, one proof chain](/en/docs/adr/0015-entity-kernel) - [**ADR 0016 — SSE Bus: Redis Streams log, Redis pub/sub fan-out, BullMQ outbound queue**](/en/docs/adr/0016-sse-bus) - [ADR 0017 — Layered BFF: one route table, route classes, read models](/en/docs/adr/0017-bff-layered) - [ADR 0018 — Cluster node registry, node identity, hot plug](/en/docs/adr/0018-cluster-node-registry) - [ADR 0019 — Tenant branding: settings keys, vendored palettes, public platform read](/en/docs/adr/0019-tenant-branding) - [Debug overlay](/en/docs/debug-overlay) **deploy/** - [Deploy runbook — dudoxx-web-boot](/en/docs/deploy/README) - [Entity kernel — guide](/en/docs/entity-kernel) - [Feature map — what each role can do](/en/docs/features) - [Component inventory](/en/docs/inventory) - [Naming constitution — dudoxx-web-boot](/en/docs/naming-constitution) - [Observability — one log layout, one key set](/en/docs/observability) - [Pattern reference — P1-P23 and groups A-K](/en/docs/patterns) - [Port map — dudoxx-web-boot](/en/docs/ports) **research/** - [RQ1 — Which release is "latest", and does it differ from the canon?](/en/docs/research/rq1-versions) - [RQ10 — SSE through the Next.js BFF route handler (desk half)](/en/docs/research/rq10-sse-through-bff) - [RQ11 — Dev latency baseline: cold compile per route + next-server RSS](/en/docs/research/rq11-dev-latency) - [RQ2 — Route group or dynamic org slug?](/en/docs/research/rq2-route-shape) - [RQ3 — better-auth 1.7 vs Auth.js v5: feature matrix](/en/docs/research/rq3-auth-matrix) - [RQ4 — Separate databases or separate schemas?](/en/docs/research/rq4-db-split) - [RQ5 — Which frameworks beyond shadcn earn a place?](/en/docs/research/rq5-frameworks) - [RQ6 — How do DI, IoC and Singleton behave across RSC, client bundles and NestJS?](/en/docs/research/rq6-di-scope) - [RQ7 — Which data uses the server cache, which uses TanStack Query, how is each invalidated?](/en/docs/research/rq7-cache-classes) - [RQ8 — Media path: presigned URLs vs streaming through the BFF](/en/docs/research/rq8-media-path) - [Template triage — create-nextjs code in ddx-web](/en/docs/research/template-triage) - [Settings — guide](/en/docs/settings) - [Local setup — dudoxx-web-boot](/en/docs/setup) - [Shell tree — SiteShell and PortalShell](/en/docs/shell-tree) - [SSE event catalog](/en/docs/sse-events) - [Version pins — dudoxx-web-boot](/en/docs/versions)

R&D documentation

ADR 0016 — SSE Bus: Redis Streams log, Redis pub/sub fan-out, BullMQ outbound queue

Source: ddx-documentation/adr/0016-sse-bus.md

Previous: ADR 0015 — Entity kernel: one descriptor, one access seam, one projection, one proof chain Next: ADR 0017 — Layered BFF: one route table, route classes, read models Inventory gallery

Status: Accepted (plan sse-bus, task 001, 2026-10-23).
Amends: approved dependency table (one row points here).
Refines: (settings.changed events), (entity.changed events, AccessPolicy audience).
Plan: plans/sse-bus/ (overview 00-overview.md, invariants _invariants.md).
Structure reference: findings/hms-sse-architecture/report.md (HMS = structure only; no code or values copied).
Ports: (Redis host port 16340).

Context

Reconnect storm (owner log, lead note 2026-10-23). The BFF logged GET /api/v1/events 200 in 3ms ×197 and ddx-api answered 429 ThrottlerException on /events. The web manager reset its backoff counter on every open, so a stream that opened and closed at once retried every ~1 s, per tab. The lead fixed the counter (reset only after 10 s stable open). The root causes stay: every tab opens its own stream, the generic throttler guards /events, and every client retries in lockstep after an API restart.
Reload hang (hang-fix, 2026-10-23). Open SSE streams kept the old ddx-api process alive on a watch reload. SseHub.closed$ + takeUntil and return503OnClosing fixed the hang. A restart still drops every stream at the same moment.
In-memory replay, epoch resync. The hub keeps the replay buffer in process memory. Event ids carry a process epoch. After a restart the buffer is empty and every client gets resync (epoch-changed), so every tab refetches everything at once — a second wave on top of the reconnect wave.
One process, no queue. Fan-out works only inside one ddx-api process. Nothing durable carries work out of the request (webhooks to other systems, retries).

Decision

Redis is the bus backbone. One Redis server (image below) on host port 16340, container port 6379, owned by ddx-docker-redis/. AOF on; maxmemory-policy noeviction (BullMQ needs it). ddx-api reads REDIS_URL (redis://127.0.0.1:16340 in dev).
Redis Streams = the event log. Each durable event is XADDed (approximate MAXLEN cap) to a per-channel stream BEFORE it is published, so "logged ≥ delivered". The Stream entry id <ms>-<seq> IS the SSE id. Ids survive an API restart; the epoch-changed resync wave ends. Replay = XRANGE (<Last-Event-ID> +. A trimmed gap answers resync (buffer-trimmed); a Redis error answers resync (replay-failed), never an empty success. Volatile types (heartbeat, presence, system notices) never enter the log and never carry an id.
Redis pub/sub = the fan-out. SseHub.publish = validate → XADD (durable types) → PUBLISH → each hub filters locally with the one ACL seam (isVisibleTo, audience from AccessPolicy.audienceOf). Live and replayed frames pass the same filter. Domain code never calls Redis; only the hub and its transport do. Two connections per process (command + subscriber), both from one platform/redis provider.
BullMQ = the outbound queue. Outbound webhook deliveries run as BullMQ jobs on the same Redis: attempts + exponential backoff, dead-letter after the last attempt. The worker never re-enqueues its own job and never sends a user- or audience-scoped event to an outbound URL.
HMAC webhooks. Inbound POST /webhooks/:source/:event: HMAC-SHA256, timestamp skew check, per-source event whitelist, then a nonce claim (SET NX PX) — the nonce is claimed only AFTER the MAC verifies. Outbound deliveries are signed the same way. Header names, skew and TTLs live in @ddx/contracts/sse-webhooks (task 002), not here.
Root-provider connection. ddx-web opens ONE EventSource per browser, from ddx-web/src/lib/sse/manager only, owned by a provider at the root of the (app) group. The portal layout only sets the scope (org, user); a scope change reconnects.
Leader tab. Tabs elect one leader with Web Locks (navigator.locks); the leader holds the stream and relays frames to the other tabs over BroadcastChannel. Without those APIs each tab connects on its own (degraded, still correct).
Load shaping. /events skips the generic throttler and gets a per-principal connection limiter instead. Clients use full-jitter backoff and honour the server retry: hint; on shutdown the hub sends a system.notice with a jittered retry:.

Approved dependencies

Resolved live on 2026-10-23 (UTC 15:36) on dudoxxmac. ~/.claude/ddx-topic-flows/VERSIONS.md has no redis, ioredis or bullmq row on 2026-10-23 (rg -i 'redis|bullmq' → no match), so the version source is the recorded command, per ADR 0005 Decision 4. Not copied from HMS.

Package / image	App	Kind	Approved version	Resolved	Version source (command, 2026-10-23)
`redis` Docker official image	`ddx-docker-redis/`	image	`redis:8.10.2-alpine3.23` exact tag	8.10.2 (tag pushed 2026-10-22)	Docker Hub `GET /v2/repositories/library/redis/tags?ordering=last_updated` — newest GA alpine tag; `8.10-rc2-*` is an RC and is excluded
`ioredis`	ddx-api	dep	`^6.0.0`	6.0.0 (dist-tag `latest`, published 2026-07-31)	`pnpm view ioredis version` → `6.0.0`
`bullmq`	ddx-api	dep	`^6.4.0`	6.4.0 (dist-tag `latest`, published 2026-10-21)	`pnpm view bullmq version` → `6.4.0`

Notes:

bullmq@6.4.0 declares ioredis >=5.0.0 as a peer (pnpm view bullmq@6.4.0 peerDependencies), so ioredis ^6.0.0 satisfies it. BullMQ docs require Redis ≥ 6.2 and recommend 7+; 8.10.2 meets both.
Ranges are caret within the major (ADR 0005 Decision 4). A major bump (bullmq 7, ioredis 7, redis 9) needs a new ADR. The Redis image is pinned to the exact tag; a tag bump is a one-line edit of this table plus the compose file.
bullmq@6.4.0 is 2 days old on 2026-10-23. If the pnpm 12 release-age gate rejects it, the install task adds bullmq@6.4.0 to minimumReleaseAgeExclude in pnpm-workspace.yaml and lists that file in its task files.
No other package is approved by this ADR (redis npm client, bullmq-otel and pg peers stay uninstalled).

Rejected

Option	Why not
In-memory fallback when Redis is down	Silent split brain: events published while Redis is down reach only one process and never enter the log, so replay lies. Plan invariant: never fall back to in-memory fan-out silently. Redis down = `/health` not ready + a loud error.
WebSocket	The traffic is server → browser only. SSE gives `Last-Event-ID` resume, plain HTTP through the BFF and the existing `api/v1` pipe. A socket adds a second protocol and its own auth path.
Centrifugo / Mercure / NATS / `better-sse`	A second server (or a library hub) beside ddx-api: a second ACL seam outside `AccessPolicy`, a second port, a second auth path. HMS defect D1 (two hubs). `better-sse` keeps state in process — the defect this ADR removes.
Kafka	Ops weight (cluster, topics, retention tuning) far beyond one dev workspace; Redis Streams give ordered ids, replay and trimming at this scale, and BullMQ needs Redis anyway.

Consequences

Redis is a hard dependency. ddx-api refuses to start without REDIS_URL (env validation marks it required, no default). ./ddx-manage.sh starts the Redis container with Postgres and MinIO.
Good: event ids survive a restart; a resuming client gets replay, not a resync wave.
Good: fan-out is multi-instance ready (pub/sub); one ACL seam for live, replayed and bridged frames.
Good: one browser connection per user (leader tab) cuts the per-tab load and the 429 loop.
Cost: one more container, one more port (16340), one more thing that can be down in dev.
Risk: Stream memory. Each stream is capped (MAXLEN ~) and the cap is a named constant in the hub POLICY.md; a trimmed cursor answers resync, never silence.

Validation

rg -q 'bullmq' ddx-documentation/adr/0016-sse-bus.md && rg -q 'ioredis' ddx-documentation/adr/0016-sse-bus.mdrg -q '0016' ddx-documentation/adr/0005-frameworks.mdrg -q '16340' ddx-documentation/ports.md

Last reconciled: 2026-10-23 · task 001 (sse-bus)