--- id: "omnitron/diagnostics" title: "Operational diagnostics" --- A diagnostic must say what failed, what was observed, and what can restore service. Omnitron uses the same finding contract in commands, runtime health and notification adapters. Capture does not mean recovery, and broker acceptance does not mean a person received a message. `doctor` checks the complete internal database schema, including Titan’s `metrics_samples` table. The normal test typecheck verifies that the required table list covers inherited module tables as well as Omnitron’s own tables. A missing metrics table is a database finding, not a healthy empty chart. ## Investigate a failure ```bash omnitron --json health omnitron --json doctor omnitron logs omnitron -n 100 omnitron inspect project/stack/application omnitron migrate status --daemon ``` The Console's node diagnostics show the same health indicators as the CLI. The `runtime-diagnostics` indicator retains unresolved runtime failures. Expand **Evidence and recovery** to see causes, occurrence counts and next steps; the log link preserves the selected node. Switching nodes clears the previous node’s readings immediately. The dialog displays loading state until the selected node answers, and an older request cannot overwrite a newer check. | Failure boundary | Evidence and response | | ---------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- | | Application process | Titan classifies operational errors and controls shutdown; Omnitron supervises process state and captures logs | | Daemon process | Unhandled rejections, uncaught exceptions and startup failures produce structured, redacted diagnostics | | Scheduled work | Session cleanup, metrics, alert evaluation, fleet heartbeat, log rotation and replication retain failure state until a successful run | | Startup dependencies | Failed migrations, missing scheduler registration, telemetry, traces and leader election remain visible in health | | Database | Query rejection, nested causes, driver codes, transaction/rollback failures and migration integrity remain distinguishable | | Event observers | Async regular and wildcard failures reach the caller; notification lifecycle observer failures degrade health without resending delivered messages | | Alert delivery | The existing alert outbox retains pending transitions and records delivery failures, including absent consumers | | Notification transport | The diagnostic adapter distinguishes queued, duplicate, unavailable consumer and failed publication | Runtime health is a bounded projection of this daemon process, not a second persistent event store. It keeps at most 100 active diagnostic codes and counts repeats. The daemon log is the persistent record. Evicted records are reported as lost projection coverage; restarting clears process-local state, not the log. Unhandled failures are not automatically declared recovered. Periodic jobs clear their own finding only after their next successful execution. The emergency stderr boundary uses native synchronous writes. A full disk, closed pipe or unwritable descriptor cannot emit an unhandled stream error and terminate the owner while it is reporting another failure. Failed writes retain their count, last error and timestamp in `runtime-diagnostics` health, even after the original finding resolves. These records remain in bounded memory; unavailable storage means they are not durably recorded. The next successful output attempt clears the current output error, while the failure count remains. Titan logger sink health and the durable log journal report their own asynchronous failures and commit acknowledgments; accepting a log call does not prove persistence. The process monitor observes exceptions without replacing Titan's shutdown policy. A fatal exception is not made safe by logging it. A healthy status also does not prove application correctness: register domain health checks and monitor their coverage. Older daemons without the runtime indicator appear as unexamined coverage in `doctor`. ## Process, health and start prerequisites `stack list`, `stack status` and Console show process lifecycle separately from application health. **Running** means the expected processes were observed online; it does not prove their domain checks passed. **Healthy**, **Health warning**, **Unhealthy** and **Health unknown** describe those checks. A deliberately stopped application has **Not running** health, rather than an invented probe failure. Stack detail preserves the application, physical node and failed or warning checks. A healthy replica cannot hide an unobserved replica on another node. Missing measurements and failed owner/node reads remain unknown. Concurrent local stack readers share one owner observation, and a process replaced during that read cannot inherit the previous process's health. CLI and Console use the same reading. **Start prerequisites** (`stack diagnose`) is a separate, read-only preflight: it explains what starting would require or change. It does not start applications and cannot establish the health of a future launch. ## Stack infrastructure ownership `omnitron stack diagnose ` and Console's stack diagnosis inspect provisioning authority before checking inferred services. For `controller` and `external` infrastructure ownership, **Infrastructure ownership** explains who owns provisioning/migrations and **Application connections** remains **unknown** until endpoint reachability, credentials and schema have actually been checked. The node does not create substitute databases or containers to make that reading green. Conflicting infrastructure declarations or a missing controller source binding block diagnosis with a configuration reason. See [infrastructure ownership](/docs/omnitron/infrastructure#infrastructure-ownership). ## RPC connection and replication Run `omnitron node check ` or open **Nodes → Diagnostics**. Both use the same connection reading. A running application, an SSH-reachable machine and a working authenticated RPC session are separate observations. Stack and release views report **Inventory observed** or **Inventory unavailable**. The latter means the controller could not read application ownership; it does not prove the host or transport is offline. The topology preserves that node as **unknown**, and `stack status` reports its inventory as **unobserved**. Use the node diagnostics to distinguish connection failure from an unavailable process owner; replication can remain active while application inventory is unavailable. | Observation | Meaning and next step | | -------------------------------- | -------------------------------------------------------------------------------------------- | | Connected directly / through SSH | Recent RPC heartbeat and a verified service credential; check replication separately | | Opening / authenticating | Connection establishment is in progress | | Disconnected / retrying | Read the transport error, cleanup error, next retry time and attempt number | | Stale heartbeat | Current reachability is unknown; an old successful ping is not current health | | Catching up | Delivery is active, but the oldest pending entry is at least 60 seconds old | | Delivery active, age unavailable | An older node reports acknowledgements but cannot prove queue age; upgrade it to verify lag | | Replication behind | Entries are waiting and no acknowledgement has arrived for 60 seconds | | Replication failing | Inspect the reported read, ingestion or acknowledgement error; queued entries remain pending | | Replication unknown | No recent authenticated connection or fresh replication reading is available | The master normally pulls every 15 seconds. **In sync** means an empty queue. **Keeping up** requires both a recent acknowledgement and an oldest pending entry under 60 seconds. A fresh acknowledgement alone cannot prove this: a large old backlog can still be catching up. Older nodes without queue-age evidence show **delivery active, queue age unavailable**, rather than a green sync state. The diagnostic detail includes `oldestPendingAgeMs`, node-local `lastSyncAgeMs` (to avoid clock skew), and `deliveryRatePerSecond` over a rolling 60-second window, including idle time. Only newly acknowledged rows contribute; duplicate acknowledgements do not refresh progress. Database read failures propagate as unavailable, never as an empty queue. The slave's legacy `connected` field represents its optional push channel and is not evidence about master-driven replication. A failed status request is shown as **unavailable**, with its reason. The optional telemetry relay has its own explicit observation. A controller using a local database aggregator reports **local database** without requiring a remote transport. A stopped relay, missing destination, transport failure or dropped entries remain visible. Workers report **not applicable** because their logs and metrics use the durable replication outbox; inspect its acknowledgement evidence above. This is an explicit node reply, never an inference from a missing service error. Older nodes without this contract, failed authentication and failed reads remain **unavailable** with a reason, without fabricated zero-loss counters. Netron disconnect events and heartbeat failures trigger reconnect with backoff from 5 seconds to 2 minutes. Peer and SSH tunnel cleanup run independently and have a five-second combined deadline, so a stuck close cannot disable retries. A replacement connection authenticates again. `authenticated` becomes true only after that peer accepts the service credential, and resets on disconnect. Responses from the old peer cannot overwrite its state or acknowledge new work. RPC operations are not replayed implicitly; only explicitly retryable, idempotent operations opt into retry. `OmnitronNodes.getMeshStatus()` includes `lastHeartbeat`, `lastError`, `reconnectAt`, `reconnectAttempt`, `cleanupError`, `syncStatus`, `syncStatusAt`, `lastSyncStatusError`, `lastPullError` and `lastPullAt`. These timestamps describe observations, not guarantees between polls. Console refreshes an open diagnostic dialog every five seconds and pauses requests while it is hidden or busy. ## Classic services and completed work Applications launched through `script` declare `readiness` in their ecosystem entry. Without it they remain `degraded` with `readiness_not_configured`: `online` alone does not establish readiness. Native bootstrap applications use `hooks.onHealthCheck`; optional HTTP probes can supplement that verdict. ```typescript { name: 'service', script: './service.mjs', readiness: { intervalMs: 5_000, timeoutMs: 3_000, probes: [{ name: 'service HTTP', type: 'http', url: 'http://127.0.0.1:4720/health', expect: { status: 'ok', revision: expectedRevision }, }], }, } ``` An HTTP probe requires the expected status (200 by default), refuses redirects and optionally matches exact top-level JSON fields. This checks a declared contract, not all business operations. Use `/` without `expect` for a frontend; use domain fields for a service. Responses are bounded to 64 KiB; diagnostics never include response bodies or URL query strings. URL credentials are refused. For a classic background worker, declare one `heartbeat` probe: ```typescript readiness: { probes: [{ name: 'delivery', type: 'heartbeat', file: '.local/health/delivery.json', maxAgeMs: 60_000 }], } ``` Report **after real completed work**, not on a timer independent of that work: ```typescript import { writeHealthHeartbeat } from '@omnitron-dev/omnitron/health'; await writeHealthHeartbeat({ status: 'healthy', checks: [{ name: 'delivery', status: 'pass', message: 'Batch committed.' }], }); ``` Failures should publish `unhealthy` with named failed checks and a recovery hint. An initial `degraded` report may explain that the first cycle is still pending. Await each write; the helper atomically replaces a mode-0600 file. It is a no-op outside the supervisor when no report path is configured. There is no automatic healthy heartbeat. Set the freshness limit to cover actual work plus its normal idle interval; a missing, stale, malformed or future report fails readiness. Omnitron passes `OMNITRON_HEALTH_FILE` (resolved against the application's `cwd`) and a fresh `OMNITRON_RUN_ID` to each classic launch. Wrappers must inherit these variables. A report from an earlier launch is rejected. Other languages can atomically publish the same UTF-8 JSON contract: ```json {"schemaVersion":1,"timestamp":1793320000000,"runId":"supervisor-provided-id","status":"healthy","checks":[{"name":"delivery","status":"pass","message":"Batch committed."}]} ``` `timestamp` is the Unix time in milliseconds at completion; use the actual time, not this example. Reports accept 1–32 named checks and at most 64 KiB. Symlinks and special files are refused. An overall failure cannot be hidden by passing checks. Bootstrap workers use Titan's health hook instead of heartbeat files. Measurements are demand-driven by health readers. Concurrent reads share a probe; results are cached for `intervalMs` (default 5 seconds). A changed configuration or launch invalidates that cache. State transitions produce `application.readiness.changed` daemon log events. CLI `health`, declared `health-check` probes and Console use these same measurements. After adding a heartbeat declaration, restart its worker to receive the channel variables. ### Immutable releases An operator can add HTTP readiness without editing an installed release. In the node's `~/.omnitron/config.json`, `applicationReadiness` maps exact qualified app names to the same probe contract. For example: ```json { "applicationReadiness": { "project/production/service": { "probes": [{ "name": "service HTTP", "type": "http", "url": "http://127.0.0.1:4720/health", "expect": { "status": "ok" } }] } } } ``` Merge this field into the existing daemon configuration; do not replace other settings. A controlled daemon restart activates it. Wildcards and heartbeat injection are refused. This is a fallback when the application has no readiness declaration; an application-owned declaration takes precedence. Native domain health is always preserved. A new application release should declare its own contract, allowing this operator fallback to be removed during normal rollout. ## Container state and stack intent The Docker indicator reads Docker and the local stack registry. Containers of an enabled or running local stack are required: missing or stopped containers fail health. Unhealthy/starting healthchecks remain visible. A cleanly stopped container of a known disabled local stack is reported as intentionally inactive; its old failing healthcheck is not treated as a current failure. This never starts, deletes or recreates containers. A stopped container omitted from its active stack's **current reconciler inventory** is reported as retired. A matching stack name alone cannot prove retirement. A retired container still running remains a warning, and missing required containers still fail. Unknown ownership is a warning, not proof that a container is unnecessary. Unloaded configuration, unregistered stacks and failed desired-state reads remain explicit. Inspect project configuration, stack startup and container logs before changing intent. Docker unavailability fails the indicator even when a cached reconciler state claimed readiness. All simultaneous failure reasons are retained. ## Domain readiness Process state and readiness answer different questions. `online` means the supervised process is running. A worker with a working event loop can still be unable to commit data or complete its business operation. Implement `hooks.onHealthCheck(app)` in the bootstrap and return the public `AppHealthAnswer` contract: ```typescript import type { AppHealthAnswer } from '@omnitron-dev/omnitron'; const report: AppHealthAnswer = { status: 'unhealthy', checks: [{ name: 'ingestion', status: 'fail', message: 'source_stale: no committed observations in the required window. Check collector errors and queue write capacity.', }], }; ``` Name the failed dependency or stream. Include the stable error code, the observed condition, the effect on service and the next useful check. Return these in `checks[].message`; a private `details` object is not forwarded as diagnostic checks. The daemon prefixes application/process names and carries the same messages to `omnitron health ` and the Console's application health view. - Probe successful work and freshness, not connectivity alone. Redis can answer `PING` while rejecting writes with OOM; a collector can receive events without committing any of them. - Account for every required stream. Missing results, stale cycles and unreadable answers must not become healthy through an empty list or a default status. - Preserve a worker's failure verdict when adapting an older health shape. Do not promote it because the adapter cannot identify the cause. - Log failure and recovery transitions with Titan's structured logger. Include resource identity and evidence, suppress unchanged healthy polls, and avoid secrets and raw request payloads. - Restore health only after the required work succeeds. Reconnecting a source may leave a data window incomplete for several minutes; retain that explanation until the window is usable. Do not erase failure evidence or invent missing data. For registered experiments, place diagnostic adapters outside immutable worker artifacts. Verify the pinned artifacts before and after activation, and restart only the affected worker. Operational recovery must not silently create a new implementation under an existing experiment identity. ## Supervise maintenance with the same health contract Declare infrastructure maintenance as an ecosystem application when it needs an independent lifecycle. Use Titan Scheduler for intervals, overlap prevention and shutdown. Return named checks through `onHealthCheck`, so the CLI and Console report the same policy, last completed attempt, dependency capacity and recovery action. A scheduler tick alone is not proof that its operation succeeded. For a durable stream pipeline, a safe cleanup policy needs both an age boundary and acknowledgement by every consumer group. Inspect group positions and pending entries atomically with deletion; a boundary computed by a previous observation can already be stale. Refuse cleanup when the required journal consumer is missing. Bound work per cycle and retry by recomputing the boundary after a lost reply. Default a reusable maintenance component to observation mode; make active deletion an explicit deployment policy. Capacity recovery must remain possible when normal writes are rejected. For example, inline stream trimming attached to `XADD` cannot free memory when Redis rejects the entire write with OOM. Test the independent cleanup under that actual failure, as well as pending entries, unread backlog, multiple consumers and consumer creation between observation and execution. A capacity indicator must read the write budget and preserve the failure until useful work resumes. Log state transitions and bounded work summaries, not every successful poll. Show observation-only mode and blocked cleanup as degraded readiness, with the reason data was retained. Keep maintenance separate from immutable application artifacts when changing operational policy must not change an experiment's registered implementation. Document the producer's own retention limits too; a maintenance guard cannot promise stronger guarantees than the entire pipeline. ## One transport-independent contract Import from `@omnitron-dev/omnitron/diagnostics`, or from `@omnitron-dev/titan/diagnostics` in a standalone Titan application. - `DiagnosticFinding`: stable `id`, severity, title, evidence and optional remedy. - `DiagnosticEvent`: a finding plus `eventId`, timestamp, source, raised/resolved state, and optional correlation/resource identifiers. - `DiagnosticSink.deliver(event)`: returns a queued/duplicate receipt or rejects. - `describeError`: bounded, cycle-safe nested cause formatting, including `AggregateError` and database driver codes. `describeError(error, 0, sensitiveValues)` also redacts supplied credential values before truncating messages or nested causes; use it when a subprocess error can include credentials as positional arguments. - `sanitizeDiagnostic`: validates the contract, allowlists fields, redacts common credential forms and bounds text/evidence before delivery. Only send operational facts. Credential redaction is a defense, not permission to include request bodies, personal data, SQL literals or arbitrary metadata. Raw stacks, headers and payload objects are not forwarded by the adapter. ## Optional Titan notification adapter Messenger-specific integrations are deliberately separate. The universal bridge uses a configured `titan-notifications` transport, its queue, retry policy and DLQ. It creates no competing delivery engine and sends nothing until explicitly wired and invoked. ```typescript import { NotificationsModule } from '@omnitron-dev/titan-notifications'; import { DiagnosticNotificationsModule, DIAGNOSTIC_NOTIFICATION_SINK, } from '@omnitron-dev/titan-notifications/diagnostics'; const diagnosticsModule = DiagnosticNotificationsModule.forRoot({ notifications: NotificationsModule.forRoot({ redis: process.env.REDIS_URL! }), channel: 'diagnostics.operational', }); // Import diagnosticsModule in your Titan module and inject // DIAGNOSTIC_NOTIFICATION_SINK where diagnostics are published. ``` Register a consumer before publishing. Messages use type `operational.diagnostic` and metadata `schemaVersion: 1`. The default channel is `diagnostics.operational`; critical/error events request urgent/high priority. Deduplication uses the event identity and transition with a 24-hour transport window. Reuse the same identity across retries. Delivery outside that window can duplicate, and receivers must remain idempotent. A queued receipt acknowledges the broker only. Consumer acknowledgement, retries and DLQ handling determine subsequent delivery; no end-to-end exactly-once guarantee is claimed. `DiagnosticDeliveryError` means no consumer or an unconfirmed publication. A failed stream rejects the entire fan-out, even if other streams accepted it. Retry with the same event identity: confirmed streams deduplicate the retry, while failed streams can accept it. Redis failures and invalid acknowledgements never count as duplicates. Transport causes are retained with credential redaction, including an error observer failing alongside publication. This lets an existing durable producer keep its pending record until all matching streams acknowledge the event. Explicit deduplication keys preserve punctuation through percent encoding. When upgrading, keys previously changed by underscore substitution start a new broker deduplication window. Consumer idempotency must cover retries that cross that deployment boundary. ## Connect the existing Omnitron alert outbox `createAlertDiagnosticHandler(sink)` converts the payload of `monitoring.alertSink` into the shared contract. Call it from the exposed method configured as your alert sink. It preserves stable identities across retries and produces separate firing/resolved event identities. ```typescript import { createAlertDiagnosticHandler } from '@omnitron-dev/omnitron/diagnostics'; import { DiagnosticNotificationSink } from '@omnitron-dev/titan-notifications/diagnostics'; const handleAlert = createAlertDiagnosticHandler( new DiagnosticNotificationSink(configuredTransport), ); // Your exposed alert method returns handleAlert(payload). ``` The alert outbox currently delivers critical alert transitions to configured sinks. This adapter does not automatically forward all log lines or runtime findings. Its timestamp records receipt at the adapter; the original alert identity remains available for correlation. Messenger routing, credentials, recipient policy and escalation are consumer responsibilities to add separately. ## Failure-injection checklist Test denied access, an unreachable database, a rollback failure, a missing alert consumer, a rejected event handler and recovery after each transient failure. Check the original cause, health state and exit code, then verify that retries do not repeat committed effects. [Migration guarantees](/docs/omnitron/migrations) describe the transaction and checksum boundaries used by all framework layers. ## Storage exhaustion and execution ownership A failed background state write is observed immediately and retained by the resident owner. It does not become an unhandled promise rejection. Owner snapshots report pending writes, failed-write count and the current persistence error; `execution.state-persistence` makes the same finding available in CLI and Console health. A later successful **full** snapshot or clear can recover current state durability. Until then, the owner's state flush rejects and clean shutdown cannot be certified. The `log-sinks` health indicator reports backend destination failures and byte gaps. `execution.log-sinks` reports the resident owner's equivalent evidence. Healthy journals continue receiving records when another destination is full. Successful later writes do not erase earlier unconfirmed or dropped bytes, and custom transport error text is excluded from the health response. Inspect sink capacity and retained journals before making a historical completeness claim. A zero backend exit code alone does not prove an intentional platform shutdown. When the owner is gone, the supervisor requires a `clean:true` receipt matching its observed owner PID and current host boot before exiting as stopped. A dirty, missing or stale receipt keeps the guardian blocked with a reconciliation reason; it never launches another owner over surviving application processes. If the supervisor's own state file cannot be written, its private status retains that persistence failure in memory.