Operational diagnostics

A diagnostic must say what failed, what was observed, and what can restore service. Omnitron uses the same finding contract in commands, runtime health and notification adapters. Capture does not mean recovery, and broker acceptance does not mean a person received a message.

doctor checks the complete internal database schema, including Titan’s metrics_samples table. The normal test typecheck verifies that the required table list covers inherited module tables as well as Omnitron’s own tables. A missing metrics table is a database finding, not a healthy empty chart.

Investigate a failure

omnitron --json health
omnitron --json doctor
omnitron logs omnitron -n 100
omnitron inspect project/stack/application
omnitron migrate status --daemon

The Console's node diagnostics show the same health indicators as the CLI. The runtime-diagnostics indicator retains unresolved runtime failures. Expand Evidence and recovery to see causes, occurrence counts and next steps; the log link preserves the selected node. Switching nodes clears the previous node’s readings immediately. The dialog displays loading state until the selected node answers, and an older request cannot overwrite a newer check.

Runtime health is a bounded projection of this daemon process, not a second persistent event store. It keeps at most 100 active diagnostic codes and counts repeats. The daemon log is the persistent record. Evicted records are reported as lost projection coverage; restarting clears process-local state, not the log. Unhandled failures are not automatically declared recovered. Periodic jobs clear their own finding only after their next successful execution.

The emergency stderr boundary uses native synchronous writes. A full disk, closed pipe or unwritable descriptor cannot emit an unhandled stream error and terminate the owner while it is reporting another failure. Failed writes retain their count, last error and timestamp in runtime-diagnostics health, even after the original finding resolves. These records remain in bounded memory; unavailable storage means they are not durably recorded. The next successful output attempt clears the current output error, while the failure count remains. Titan logger sink health and the durable log journal report their own asynchronous failures and commit acknowledgments; accepting a log call does not prove persistence.

The process monitor observes exceptions without replacing Titan's shutdown policy. A fatal exception is not made safe by logging it. A healthy status also does not prove application correctness: register domain health checks and monitor their coverage. Older daemons without the runtime indicator appear as unexamined coverage in doctor.

Process, health and start prerequisites

stack list, stack status and Console show process lifecycle separately from application health. Running means the expected processes were observed online; it does not prove their domain checks passed. Healthy, Health warning, Unhealthy and Health unknown describe those checks. A deliberately stopped application has Not running health, rather than an invented probe failure.

Stack detail preserves the application, physical node and failed or warning checks. A healthy replica cannot hide an unobserved replica on another node. Missing measurements and failed owner/node reads remain unknown. Concurrent local stack readers share one owner observation, and a process replaced during that read cannot inherit the previous process's health. CLI and Console use the same reading.

Start prerequisites (stack diagnose) is a separate, read-only preflight: it explains what starting would require or change. It does not start applications and cannot establish the health of a future launch.

Stack infrastructure ownership

omnitron stack diagnose <project> <stack> and Console's stack diagnosis inspect provisioning authority before checking inferred services. For controller and external infrastructure ownership, Infrastructure ownership explains who owns provisioning/migrations and Application connections remains unknown until endpoint reachability, credentials and schema have actually been checked. The node does not create substitute databases or containers to make that reading green. Conflicting infrastructure declarations or a missing controller source binding block diagnosis with a configuration reason. See infrastructure ownership.

RPC connection and replication

Run omnitron node check <id> or open Nodes → Diagnostics. Both use the same connection reading. A running application, an SSH-reachable machine and a working authenticated RPC session are separate observations.

Stack and release views report Inventory observed or Inventory unavailable. The latter means the controller could not read application ownership; it does not prove the host or transport is offline. The topology preserves that node as unknown, and stack status reports its inventory as unobserved. Use the node diagnostics to distinguish connection failure from an unavailable process owner; replication can remain active while application inventory is unavailable.

The master normally pulls every 15 seconds. In sync means an empty queue. Keeping up requires both a recent acknowledgement and an oldest pending entry under 60 seconds. A fresh acknowledgement alone cannot prove this: a large old backlog can still be catching up. Older nodes without queue-age evidence show delivery active, queue age unavailable, rather than a green sync state. The diagnostic detail includes oldestPendingAgeMs, node-local lastSyncAgeMs (to avoid clock skew), and deliveryRatePerSecond over a rolling 60-second window, including idle time. Only newly acknowledged rows contribute; duplicate acknowledgements do not refresh progress. Database read failures propagate as unavailable, never as an empty queue. The slave's legacy connected field represents its optional push channel and is not evidence about master-driven replication. A failed status request is shown as unavailable, with its reason.

The optional telemetry relay has its own explicit observation. A controller using a local database aggregator reports local database without requiring a remote transport. A stopped relay, missing destination, transport failure or dropped entries remain visible. Workers report not applicable because their logs and metrics use the durable replication outbox; inspect its acknowledgement evidence above. This is an explicit node reply, never an inference from a missing service error. Older nodes without this contract, failed authentication and failed reads remain unavailable with a reason, without fabricated zero-loss counters.

Netron disconnect events and heartbeat failures trigger reconnect with backoff from 5 seconds to 2 minutes. Peer and SSH tunnel cleanup run independently and have a five-second combined deadline, so a stuck close cannot disable retries. A replacement connection authenticates again. authenticated becomes true only after that peer accepts the service credential, and resets on disconnect. Responses from the old peer cannot overwrite its state or acknowledge new work. RPC operations are not replayed implicitly; only explicitly retryable, idempotent operations opt into retry.

OmnitronNodes.getMeshStatus() includes lastHeartbeat, lastError, reconnectAt, reconnectAttempt, cleanupError, syncStatus, syncStatusAt, lastSyncStatusError, lastPullError and lastPullAt. These timestamps describe observations, not guarantees between polls. Console refreshes an open diagnostic dialog every five seconds and pauses requests while it is hidden or busy.

Classic services and completed work

Applications launched through script declare readiness in their ecosystem entry. Without it they remain degraded with readiness_not_configured: online alone does not establish readiness. Native bootstrap applications use hooks.onHealthCheck; optional HTTP probes can supplement that verdict.

{
  name: 'service',
  script: './service.mjs',
  readiness: {
    intervalMs: 5_000,
    timeoutMs: 3_000,
    probes: [{
      name: 'service HTTP', type: 'http',
      url: 'http://127.0.0.1:4720/health',
      expect: { status: 'ok', revision: expectedRevision },
    }],
  },
}

An HTTP probe requires the expected status (200 by default), refuses redirects and optionally matches exact top-level JSON fields. This checks a declared contract, not all business operations. Use / without expect for a frontend; use domain fields for a service. Responses are bounded to 64 KiB; diagnostics never include response bodies or URL query strings. URL credentials are refused.

For a classic background worker, declare one heartbeat probe:

readiness: {
  probes: [{ name: 'delivery', type: 'heartbeat',
    file: '.local/health/delivery.json', maxAgeMs: 60_000 }],
}

Report after real completed work, not on a timer independent of that work:

import { writeHealthHeartbeat } from '@omnitron-dev/omnitron/health';

await writeHealthHeartbeat({
  status: 'healthy',
  checks: [{ name: 'delivery', status: 'pass', message: 'Batch committed.' }],
});

Failures should publish unhealthy with named failed checks and a recovery hint. An initial degraded report may explain that the first cycle is still pending. Await each write; the helper atomically replaces a mode-0600 file. It is a no-op outside the supervisor when no report path is configured. There is no automatic healthy heartbeat. Set the freshness limit to cover actual work plus its normal idle interval; a missing, stale, malformed or future report fails readiness.

Omnitron passes OMNITRON_HEALTH_FILE (resolved against the application's cwd) and a fresh OMNITRON_RUN_ID to each classic launch. Wrappers must inherit these variables. A report from an earlier launch is rejected. Other languages can atomically publish the same UTF-8 JSON contract:

{"schemaVersion":1,"timestamp":1793320000000,"runId":"supervisor-provided-id","status":"healthy","checks":[{"name":"delivery","status":"pass","message":"Batch committed."}]}

timestamp is the Unix time in milliseconds at completion; use the actual time, not this example. Reports accept 1–32 named checks and at most 64 KiB. Symlinks and special files are refused. An overall failure cannot be hidden by passing checks. Bootstrap workers use Titan's health hook instead of heartbeat files.

Measurements are demand-driven by health readers. Concurrent reads share a probe; results are cached for intervalMs (default 5 seconds). A changed configuration or launch invalidates that cache. State transitions produce application.readiness.changed daemon log events. CLI health, declared health-check probes and Console use these same measurements. After adding a heartbeat declaration, restart its worker to receive the channel variables.

Immutable releases

An operator can add HTTP readiness without editing an installed release. In the node's ~/.omnitron/config.json, applicationReadiness maps exact qualified app names to the same probe contract. For example:

{
  "applicationReadiness": {
    "project/production/service": {
      "probes": [{ "name": "service HTTP", "type": "http", "url": "http://127.0.0.1:4720/health", "expect": { "status": "ok" } }]
    }
  }
}

Merge this field into the existing daemon configuration; do not replace other settings. A controlled daemon restart activates it. Wildcards and heartbeat injection are refused. This is a fallback when the application has no readiness declaration; an application-owned declaration takes precedence. Native domain health is always preserved. A new application release should declare its own contract, allowing this operator fallback to be removed during normal rollout.

Container state and stack intent

The Docker indicator reads Docker and the local stack registry. Containers of an enabled or running local stack are required: missing or stopped containers fail health. Unhealthy/starting healthchecks remain visible. A cleanly stopped container of a known disabled local stack is reported as intentionally inactive; its old failing healthcheck is not treated as a current failure. This never starts, deletes or recreates containers. A stopped container omitted from its active stack's current reconciler inventory is reported as retired. A matching stack name alone cannot prove retirement. A retired container still running remains a warning, and missing required containers still fail.

Unknown ownership is a warning, not proof that a container is unnecessary. Unloaded configuration, unregistered stacks and failed desired-state reads remain explicit. Inspect project configuration, stack startup and container logs before changing intent. Docker unavailability fails the indicator even when a cached reconciler state claimed readiness. All simultaneous failure reasons are retained.

Domain readiness

Process state and readiness answer different questions. online means the supervised process is running. A worker with a working event loop can still be unable to commit data or complete its business operation.

Implement hooks.onHealthCheck(app) in the bootstrap and return the public AppHealthAnswer contract:

import type { AppHealthAnswer } from '@omnitron-dev/omnitron';

const report: AppHealthAnswer = {
  status: 'unhealthy',
  checks: [{
    name: 'ingestion',
    status: 'fail',
    message: 'source_stale: no committed observations in the required window. Check collector errors and queue write capacity.',
  }],
};

Name the failed dependency or stream. Include the stable error code, the observed condition, the effect on service and the next useful check. Return these in checks[].message; a private details object is not forwarded as diagnostic checks. The daemon prefixes application/process names and carries the same messages to omnitron health <app> and the Console's application health view.

• Probe successful work and freshness, not connectivity alone. Redis can answer PING while rejecting writes with OOM; a collector can receive events without committing any of them.

• Account for every required stream. Missing results, stale cycles and unreadable answers must not become healthy through an empty list or a default status.

• Preserve a worker's failure verdict when adapting an older health shape. Do not promote it because the adapter cannot identify the cause.

• Log failure and recovery transitions with Titan's structured logger. Include resource identity and evidence, suppress unchanged healthy polls, and avoid secrets and raw request payloads.

• Restore health only after the required work succeeds. Reconnecting a source may leave a data window incomplete for several minutes; retain that explanation until the window is usable. Do not erase failure evidence or invent missing data.

For registered experiments, place diagnostic adapters outside immutable worker artifacts. Verify the pinned artifacts before and after activation, and restart only the affected worker. Operational recovery must not silently create a new implementation under an existing experiment identity.

Supervise maintenance with the same health contract

Declare infrastructure maintenance as an ecosystem application when it needs an independent lifecycle. Use Titan Scheduler for intervals, overlap prevention and shutdown. Return named checks through onHealthCheck, so the CLI and Console report the same policy, last completed attempt, dependency capacity and recovery action. A scheduler tick alone is not proof that its operation succeeded.

For a durable stream pipeline, a safe cleanup policy needs both an age boundary and acknowledgement by every consumer group. Inspect group positions and pending entries atomically with deletion; a boundary computed by a previous observation can already be stale. Refuse cleanup when the required journal consumer is missing. Bound work per cycle and retry by recomputing the boundary after a lost reply. Default a reusable maintenance component to observation mode; make active deletion an explicit deployment policy.

Capacity recovery must remain possible when normal writes are rejected. For example, inline stream trimming attached to XADD cannot free memory when Redis rejects the entire write with OOM. Test the independent cleanup under that actual failure, as well as pending entries, unread backlog, multiple consumers and consumer creation between observation and execution. A capacity indicator must read the write budget and preserve the failure until useful work resumes.

Log state transitions and bounded work summaries, not every successful poll. Show observation-only mode and blocked cleanup as degraded readiness, with the reason data was retained. Keep maintenance separate from immutable application artifacts when changing operational policy must not change an experiment's registered implementation. Document the producer's own retention limits too; a maintenance guard cannot promise stronger guarantees than the entire pipeline.

One transport-independent contract

Import from @omnitron-dev/omnitron/diagnostics, or from @omnitron-dev/titan/diagnostics in a standalone Titan application.

• DiagnosticFinding: stable id, severity, title, evidence and optional remedy.

• DiagnosticEvent: a finding plus eventId, timestamp, source, raised/resolved state, and optional correlation/resource identifiers.

• DiagnosticSink.deliver(event): returns a queued/duplicate receipt or rejects.

• describeError: bounded, cycle-safe nested cause formatting, including AggregateError and database driver codes. describeError(error, 0, sensitiveValues) also redacts supplied credential values before truncating messages or nested causes; use it when a subprocess error can include credentials as positional arguments.

• sanitizeDiagnostic: validates the contract, allowlists fields, redacts common credential forms and bounds text/evidence before delivery.

Only send operational facts. Credential redaction is a defense, not permission to include request bodies, personal data, SQL literals or arbitrary metadata. Raw stacks, headers and payload objects are not forwarded by the adapter.

Optional Titan notification adapter

Messenger-specific integrations are deliberately separate. The universal bridge uses a configured titan-notifications transport, its queue, retry policy and DLQ. It creates no competing delivery engine and sends nothing until explicitly wired and invoked.

import { NotificationsModule } from '@omnitron-dev/titan-notifications';
import {
  DiagnosticNotificationsModule, DIAGNOSTIC_NOTIFICATION_SINK,
} from '@omnitron-dev/titan-notifications/diagnostics';

const diagnosticsModule = DiagnosticNotificationsModule.forRoot({
  notifications: NotificationsModule.forRoot({ redis: process.env.REDIS_URL! }),
  channel: 'diagnostics.operational',
});
// Import diagnosticsModule in your Titan module and inject
// DIAGNOSTIC_NOTIFICATION_SINK where diagnostics are published.

Register a consumer before publishing. Messages use type operational.diagnostic and metadata schemaVersion: 1. The default channel is diagnostics.operational; critical/error events request urgent/high priority. Deduplication uses the event identity and transition with a 24-hour transport window. Reuse the same identity across retries. Delivery outside that window can duplicate, and receivers must remain idempotent. A queued receipt acknowledges the broker only. Consumer acknowledgement, retries and DLQ handling determine subsequent delivery; no end-to-end exactly-once guarantee is claimed.

DiagnosticDeliveryError means no consumer or an unconfirmed publication. A failed stream rejects the entire fan-out, even if other streams accepted it. Retry with the same event identity: confirmed streams deduplicate the retry, while failed streams can accept it. Redis failures and invalid acknowledgements never count as duplicates. Transport causes are retained with credential redaction, including an error observer failing alongside publication. This lets an existing durable producer keep its pending record until all matching streams acknowledge the event.

Explicit deduplication keys preserve punctuation through percent encoding. When upgrading, keys previously changed by underscore substitution start a new broker deduplication window. Consumer idempotency must cover retries that cross that deployment boundary.

Connect the existing Omnitron alert outbox

createAlertDiagnosticHandler(sink) converts the payload of monitoring.alertSink into the shared contract. Call it from the exposed method configured as your alert sink. It preserves stable identities across retries and produces separate firing/resolved event identities.

import { createAlertDiagnosticHandler } from '@omnitron-dev/omnitron/diagnostics';
import { DiagnosticNotificationSink } from '@omnitron-dev/titan-notifications/diagnostics';

const handleAlert = createAlertDiagnosticHandler(
  new DiagnosticNotificationSink(configuredTransport),
);
// Your exposed alert method returns handleAlert(payload).

The alert outbox currently delivers critical alert transitions to configured sinks. This adapter does not automatically forward all log lines or runtime findings. Its timestamp records receipt at the adapter; the original alert identity remains available for correlation. Messenger routing, credentials, recipient policy and escalation are consumer responsibilities to add separately.

Failure-injection checklist

Test denied access, an unreachable database, a rollback failure, a missing alert consumer, a rejected event handler and recovery after each transient failure. Check the original cause, health state and exit code, then verify that retries do not repeat committed effects. Migration guarantees describe the transaction and checksum boundaries used by all framework layers.

Storage exhaustion and execution ownership

A failed background state write is observed immediately and retained by the resident owner. It does not become an unhandled promise rejection. Owner snapshots report pending writes, failed-write count and the current persistence error; execution.state-persistence makes the same finding available in CLI and Console health. A later successful full snapshot or clear can recover current state durability. Until then, the owner's state flush rejects and clean shutdown cannot be certified.

The log-sinks health indicator reports backend destination failures and byte gaps. execution.log-sinks reports the resident owner's equivalent evidence. Healthy journals continue receiving records when another destination is full. Successful later writes do not erase earlier unconfirmed or dropped bytes, and custom transport error text is excluded from the health response. Inspect sink capacity and retained journals before making a historical completeness claim.

A zero backend exit code alone does not prove an intentional platform shutdown. When the owner is gone, the supervisor requires a clean:true receipt matching its observed owner PID and current host boot before exiting as stopped. A dirty, missing or stale receipt keeps the guardian blocked with a reconciliation reason; it never launches another owner over surviving application processes. If the supervisor's own state file cannot be written, its private status retains that persistence failure in memory.