Loading the page…
Loading the page…
Source: ddx-documentation/observability.md
ddx-api and ddx-web write the same pretty line, read the same `LOG_*` keys and keep the same non-blocking queue contract. Every fact below comes from a repo file named next to it. ## 1. Pretty layout One annotated line per record, in both apps: ``` HH:MM:SS.mmm INF ● ⇄ bff:bridge message key=value req=0f564cc9 ``` | Part | Rule | Source | |---|---|---| | Clock | local time, `HH:MM:SS.mmm` | `clockTime` | | Badge + icon | level label (`DBG` `INF` `WRN` `ERR` `FTL`) and level icon | `LEVEL_LABELS`, `LEVEL_ICONS` | | Scope | family icon + name, padded to 22 columns | `SCOPE_ICONS`, `SCOPE_WIDTH` | | Tail | `key=value` fields, then `req=<8>` (first 8 chars of the request id) | `fieldValue`, `shortReq` | | Wrap | long lines hang under the message column at the width below | ddx-web `server-format.ts`, ddx-api `pretty-line.outputter.ts` | | Stack | at most 8 app-only frames | `STACK_FRAMES` | An HTTP access record is one line: `method status path ms bytes authz=… req=<8> node=<id>`. Columns are 7 / 40 / 7 / 7 wide (`ACCESS_*_WIDTH`). Status class and latency class set the colour (`statusClass`, `latencyClass`). ddx-web adds `web=<node id>` and flags (stream, retry, breaker, error) only when they differ from the default. Level for an access record follows its status: 5xx → `error`, 4xx → `warn`, else `info`. `success` has no level of its own in the shared theme and prints with the `info` badge. `LOG_FORMAT=json` (or `NODE_ENV=production`) writes NDJSON with no colour. ## 2. Shared theme source `packages/ddx-contracts/src/log-theme.ts` (`@ddx/contracts/log-theme`) holds every token: scope families, icons, labels, `PAINT_SPECS`, paint order, column widths, status and latency classes and the width helpers (`padVisual`, `wrapVisual`, `truncateVisual`, `stripAnsi`). It imports no chalk and no `node:` module. Each app binds those specs to chalk 6 in a small adapter: - ddx-api: `src/core/logging/outputters/log-theme.ts` - ddx-web: `src/lib/logger/server-theme.ts` A colour or icon change is made once in contracts. The goldens in `ddx-api/src/core/logging/logging.spec.ts` and `ddx-web/src/lib/logger/server-format.spec.ts` pin the same byte output in both apps. ## 3. `LOG_*` keys Names, enums and limits: `packages/ddx-contracts/src/log-env.ts` (`LOG_ENV_KEYS`). The block is identical, in the same order, in `ddx-api/.env.example` and `ddx-web/.env.example`. An empty or absent key uses the default. A value outside its set stops `./ddx-manage.sh start|restart` (see §6). | Key | Values | Default | Read by | |---|---|---|---| | `LOG_LEVEL` | `trace` `debug` `info` `warn` `error` `fatal` | `info` | both | | `LOG_FORMAT` | `pretty` `json` | `pretty`; `json` when `NODE_ENV=production` | both | | `LOG_COLOR` | `auto` `always` `never` | `auto` (TTY, or the dev log file outside production); `NO_COLOR` turns `auto` off | both | | `LOG_SCOPES` | csv of scope names, `*` = prefix (`bff:*,http`) | empty = all | both | | `LOG_QUEUE_MAX` | 1..1000000 | 10000 | both | | `LOG_FLUSH_MS` | 1..10000 | 50 | both | | `DDX_API_LOG_WIDTH` | 60..240 | terminal width from `ddx-manage.sh`, else 100 | ddx-api (and `api2`) | | `DDX_WEB_LOG_WIDTH` | 60..240 | same | ddx-web | | `DDX_API_LOG_COLOR` | `always` `never` (also `true`/`false`, `1`/`0`, `on`/`off`, `yes`/`no`) | unset | ddx-api only; beats `LOG_COLOR` and `NO_COLOR` | | `DDX_API_LOG_CLEAR` | `on` `off` | on for pretty + colour outside production | ddx-api only; clears the terminal once at boot | | `LOG_HTTP_HEADERS` | `none` `safe` `all` | `none` | ddx-api (§5) | Rules: - Browser `debug` records and the debug overlay share one flag, `NEXT_PUBLIC_UI_DEBUG` (`0` default, `1` on); no other debug flag exists ([debug-overlay.md](/en/docs/debug-overlay)). ddx-api scopes use the form `api:<layer>` (for example `api:redis`). - `LOG_SCOPES` filters records below `warn` only. `warn` and above always print. An unscoped ddx-web record counts as scope `web`. - ddx-web has no `LOG_TRANSPORT`; the key is read nowhere. - ddx-api prints one `logging` warn at boot per invalid value that the app itself resolves (for example an unknown `LOG_HTTP_HEADERS` falls back to `none`). `./ddx-manage.sh` stops earlier on the same values. - `.env` is operator-owned. For the dev ddx-web, set `LOG_FORMAT=pretty` and delete `LOG_TRANSPORT`, then `./ddx-manage.sh restart web`. ## 4. Non-blocking queue A log call never writes to a stream. It puts a raw record in a bounded ring; the drain formats and writes. | Guarantee | ddx-api | ddx-web | |---|---|---| | Enqueue cost | O(1), no format, no I/O | same | | Bound | ring of `LOG_QUEUE_MAX` | same | | Overflow | drop the oldest record; one counted `log.dropped` warn (`dropped=N`) at the head of the next batch | same | | Drain | `LOG_FLUSH_MS` timer (unref'd); sooner on a full batch or an `error`/`fatal` record | same; batch of 512 | | Backpressure | `write()` false → pause until `drain` | same | | Source | `writers/queued-writer.ts` | `src/lib/logger/server-queue.ts` | The ddx-web queue is a process singleton on `globalThis`, so HMR keeps one ring and only swaps the formatter. It reads no clock. **RequestLog rows (ddx-api).** The `request_log` insert uses the same shape (`src/tenancy/request-log-queue.service.ts`): `push` is O(1) into a ring of `LOG_QUEUE_MAX`; a batch is written with one `createMany` at 100 rows or after `LOG_FLUSH_MS × 20`; one write is in flight at a time; a failed batch logs one warn with the row count and no row data; overflow drops the oldest and reports `log.dropped` with `sink=request_log`. Each row keeps the time of the request, not of the flush. **Shutdown order (ddx-api).** 1. `RequestLogQueueService.beforeApplicationShutdown` flushes the buffered rows with `createMany`. 2. `DdxLogger.onApplicationShutdown` awaits `writer.flush()` (ring → stream drain → sink, bounded 2 s; on timeout a synchronous `fs.writeSync` drain). 3. `PrismaLogsService.onApplicationShutdown` runs `$disconnect`. A final synchronous drain also runs on process `exit`. No log code adds a SIGTERM/SIGINT listener in ddx-api. **Shutdown (ddx-web).** `beforeExit` runs an async flush; `exit` runs a synchronous `fs.writeSync` drain; SIGTERM/SIGINT drain synchronously, then re-raise the signal when no other listener remains, so the default termination still happens. ## 5. `LOG_HTTP_HEADERS` and redaction Header lines under an access record are opt-in and independent of `LOG_LEVEL`. | Mode | Under the access line | |---|---| | `none` (default) | nothing | | `safe` | an allowlist only: `content-type`, `content-length`, `accept`, `accept-encoding`, `user-agent`, `cache-control`, `etag` and the BFF request id, node id and authz headers | | `all` | every request header | Redaction runs in every mode. A header on the redacted list (`authorization`, `cookie`, node token, `set-auth-jwt`, …) is dropped by `safe` even when someone adds it to the allowlist, and prints as `[redacted]` under `all`. List and mode logic: `ddx-api/src/tenancy/request-log.middleware.ts` (`REDACTED_HEADERS`, `SAFE_HEADERS`, `selectHeaders`). An invalid value falls back to `none`. ## 6. Reading logs `./ddx-manage.sh logs` (source: `LOGS_USAGE` in `ddx-manage.sh`): | Form | Result | |---|---| | `logs web\|api\|api2\|docker\|mem\|all [-n N] [--no-follow] [--raw]` | tail the app logs; `all` includes `ddx-api2.log` | | `--level L` | minimum level, for web, api, api2 | | `--scope P` | scope prefix match | | `--req ID` | records of one request (first 8 chars) | | `logs req <id>` | one request across web, api and api2 in time order, tagged `[web]` `[api]` `[api2]`; never follows | The filters keep ANSI colour; an indented line (stack, headers) follows its record. Without `--no-follow` the verb uses `tail -F` and blocks; agents read `logs/ddx-web.log` and `logs/ddx-api.log` with `tail -n` or use `--no-follow` and `logs req`. Request id: the BFF forwards it, ddx-api prints it as `req=<8>` and ddx-web prints the same 8 characters, so one id finds both sides. **Preboot gate.** `./ddx-manage.sh preboot [api|web]` (no target = both) checks the env keys. `start` and `restart` of ddx-api and ddx-web run it first. Its `LOG_*` value gate rejects a value outside the enums or ranges in §3 (`LOG_LEVEL`, `LOG_FORMAT`, `LOG_COLOR`, `LOG_HTTP_HEADERS`, widths, queue limits, `LOG_SCOPES` characters) with exit 1 and a message naming the key. `restart web` runs the gate before it stops the app. ## 6a. Remote logs Production logs come from PM2 on the target, not from `logs/*.log`. Source and all verbs: [deploy/README.md](/en/docs/deploy/README) §7. | Form | Result | |---|---| | `./ddx-manage.sh remote logs [api\|web\|all] --target <id>` | PM2 logs of the app namespace; same filters as local (`--level`, `--scope`, `--req`, `--err`, `--lines`) | | `./ddx-manage.sh remote logs req <id> --target <id>` | one request across ddx-web and ddx-api, in time order | | `./ddx-manage.sh remote pull-logs --target <id>` | copy `shared/logs/` to `logs/remote/<target>/<UTC ts>/` | `remote logs --follow` blocks until killed: run it in the background only. ## 7. Next.js request lines `ddx-web/next.config.ts` sets `logging.incomingRequests.ignore` for `/api/v1/*` and `/_next/*`, so Next prints no request line for BFF calls; the `bff:bridge` access line is the one record. `logging.fetches` is unset, so bridge fetches print nothing. ## 8. Deferred - Runtime per-node log level. The settings tiers are platform, org and user (`packages/ddx-contracts/src/settings.ts`); no ddx-api node subscribes to `settings.changed`. Today a level change is an `.env` edit plus a restart. A node tier needs a decision record. - External log shipping. - One `LOG_WIDTH` key instead of `DDX_API_LOG_WIDTH` / `DDX_WEB_LOG_WIDTH` (needs a decision). --- Last reconciled: 2026-10-25 · plan deploy-pipeline task 022
ddx-api and ddx-web write the same pretty line, read the same LOG_* keys and keep the same
non-blocking queue contract. Every fact below comes from a repo file named next to it.
One annotated line per record, in both apps:
An HTTP access record is one line: method status path ms bytes authz=… req=<8> node=<id>.
Columns are 7 / 40 / 7 / 7 wide (ACCESS_*_WIDTH). Status class and latency class set the
colour (statusClass, latencyClass). ddx-web adds web=<node id> and flags
(stream, retry, breaker, error) only when they differ from the default.
Level for an access record follows its status: 5xx → error, 4xx → warn, else info.
success has no level of its own in the shared theme and prints with the info badge.
LOG_FORMAT=json (or NODE_ENV=production) writes NDJSON with no colour.
packages/ddx-contracts/src/log-theme.ts (@ddx/contracts/log-theme) holds every token:
scope families, icons, labels, PAINT_SPECS, paint order, column widths, status and latency
classes and the width helpers (padVisual, wrapVisual, truncateVisual, stripAnsi).
It imports no chalk and no node: module.
Each app binds those specs to chalk 6 in a small adapter:
src/core/logging/outputters/log-theme.tssrc/lib/logger/server-theme.tsA colour or icon change is made once in contracts. The goldens in
ddx-api/src/core/logging/logging.spec.ts and ddx-web/src/lib/logger/server-format.spec.ts
pin the same byte output in both apps.
LOG_* keysNames, enums and limits: packages/ddx-contracts/src/log-env.ts (LOG_ENV_KEYS). The block
is identical, in the same order, in ddx-api/.env.example and ddx-web/.env.example. An empty
or absent key uses the default. A value outside its set stops ./ddx-manage.sh start|restart
(see §6).
Rules:
debug records and the debug overlay share one flag, NEXT_PUBLIC_UI_DEBUG (0 default, 1 on); no other debug flag exists (). ddx-api scopes use the form api:<layer> (for example api:redis).LOG_SCOPES filters records below warn only. warn and above always print.
An unscoped ddx-web record counts as scope web.LOG_TRANSPORT; the key is read nowhere.logging warn at boot per invalid value that the app itself resolves
(for example an unknown LOG_HTTP_HEADERS falls back to none). ./ddx-manage.sh stops
earlier on the same values..env is operator-owned. For the dev ddx-web, set LOG_FORMAT=pretty and delete
LOG_TRANSPORT, then ./ddx-manage.sh restart web.A log call never writes to a stream. It puts a raw record in a bounded ring; the drain formats and writes.
The ddx-web queue is a process singleton on globalThis, so HMR keeps one ring and only swaps
the formatter. It reads no clock.
RequestLog rows (ddx-api). The request_log insert uses the same shape
(src/tenancy/request-log-queue.service.ts): push is O(1) into a ring of LOG_QUEUE_MAX;
a batch is written with one createMany at 100 rows or after LOG_FLUSH_MS × 20; one write is
in flight at a time; a failed batch logs one warn with the row count and no row data; overflow
drops the oldest and reports log.dropped with sink=request_log. Each row keeps the time of
the request, not of the flush.
Shutdown order (ddx-api).
RequestLogQueueService.beforeApplicationShutdown flushes the buffered rows with createMany.DdxLogger.onApplicationShutdown awaits writer.flush() (ring → stream drain → sink,
bounded 2 s; on timeout a synchronous fs.writeSync drain).PrismaLogsService.onApplicationShutdown runs $disconnect.A final synchronous drain also runs on process exit. No log code adds a SIGTERM/SIGINT
listener in ddx-api.
Shutdown (ddx-web). beforeExit runs an async flush; exit runs a synchronous
fs.writeSync drain; SIGTERM/SIGINT drain synchronously, then re-raise the signal when no
other listener remains, so the default termination still happens.
LOG_HTTP_HEADERS and redactionHeader lines under an access record are opt-in and independent of LOG_LEVEL.
Redaction runs in every mode. A header on the redacted list (authorization, cookie,
node token, set-auth-jwt, …) is dropped by safe even when someone adds it to the allowlist,
and prints as [redacted] under all. List and mode logic:
ddx-api/src/tenancy/request-log.middleware.ts (REDACTED_HEADERS, SAFE_HEADERS,
selectHeaders). An invalid value falls back to none.
./ddx-manage.sh logs (source: LOGS_USAGE in ddx-manage.sh):
The filters keep ANSI colour; an indented line (stack, headers) follows its record.
Without --no-follow the verb uses tail -F and blocks; agents read logs/ddx-web.log and
logs/ddx-api.log with tail -n or use --no-follow and logs req.
Request id: the BFF forwards it, ddx-api prints it as req=<8> and ddx-web prints the same 8
characters, so one id finds both sides.
Preboot gate. ./ddx-manage.sh preboot [api|web] (no target = both) checks the env keys.
start and restart of ddx-api and ddx-web run it first. Its LOG_* value gate rejects a
value outside the enums or ranges in §3 (LOG_LEVEL, LOG_FORMAT, LOG_COLOR,
LOG_HTTP_HEADERS, widths, queue limits, LOG_SCOPES characters) with exit 1 and a message
naming the key. restart web runs the gate before it stops the app.
Production logs come from PM2 on the target, not from logs/*.log. Source and all verbs:
§7.
remote logs --follow blocks until killed: run it in the background only.
ddx-web/next.config.ts sets logging.incomingRequests.ignore for /api/v1/* and /_next/*,
so Next prints no request line for BFF calls; the bff:bridge access line is the one record.
logging.fetches is unset, so bridge fetches print nothing.
packages/ddx-contracts/src/settings.ts); no ddx-api node subscribes to settings.changed.
Today a level change is an .env edit plus a restart. A node tier needs a decision record.LOG_WIDTH key instead of DDX_API_LOG_WIDTH / DDX_WEB_LOG_WIDTH (needs a decision).Last reconciled: 2026-10-25 · plan deploy-pipeline task 022
adr/
deploy/
research/
HH:MM:SS.mmm INF ● ⇄ bff:bridge message key=value req=0f564cc9