Observability

Supervised applications can emit logs, metrics, traces and health signals through Titan and Omnitron. Available signals depend on the application modules, instrumentation and configured storage. A running daemon does not imply complete instrumentation. The baseline does not require a separate Prometheus, Loki or Tempo deployment.

This page describes those pipelines end-to-end so you know which knob to turn and which storage to inspect when something goes wrong.

Verified against apps/omnitron/src/{observability,monitoring,services}/.

The four pipelines

Logs

Capture, persistence and replay

sequenceDiagram
  participant App as Fork or worker thread
  participant PM as Titan PM capture
  participant Journal as Private SQLite journal
  participant Backend as Omnitron backend
  participant DB as PostgreSQL or node SQLite
  participant Files as Application files
  App->>PM: stdout / stderr, including startup failures
  PM->>Journal: ID + source + capture time + message bytes
  Note over Journal: WAL + synchronous=FULL<br/>unfinished multiline records persist too
  Journal->>Files: Append with a durable file offset<br/>fsync
  Files-->>Journal: File projection confirmed
  Backend->>Journal: Read a bounded page
  Backend->>DB: Commit logs and node replication outbox together
  DB-->>Backend: Transaction committed
  Backend->>Journal: Confirm exact IDs and payload digests
  Note over Journal: Release only after both destinations confirm

Titan PM installs output capture before readiness, including for workers replaced by a pool. One UTF-8 reader handles each pipe from spawn through EOF. A failed initialization does not require a WorkerHandle to exist before output reaches storage. Stdout and stderr from different process instances are assembled separately. The same canonical record feeds files, the database and replication.

The local delivery files are data/application-logs.sqlite and data/backend-logs.sqlite under the configured Omnitron storage directory. They use SQLite WAL with synchronous=FULL and private permissions. Pending messages and finalized records survive process replacement. No oldest-first eviction is applied to unacknowledged logs. Recovery checks the writer's host boot and process start identity, not just its PID. It periodically revisits unfinished records, so a previous backend that dies after handoff does not strand them until another restart. If the original writer's lifetime cannot be verified, those bytes remain pending with an explicit diagnostic; they are not finalized while a writer may still be assembling them. On macOS, all process readers use the boot-session UUID and a birth timestamp explicitly captured in UTC. Legacy timestamps did not retain their timezone: a changed rendering or a mixed legacy/UTC identity does not prove that the writer exited. Its unfinished bytes remain pending until native death or a verified change of host boot is observed. If capture cannot write its journal, the pipe is paused and the same frame is retried. Monitor storage availability: backpressure can eventually block an application that continues writing to a full pipe.

A database response is acknowledged only after the enclosing transaction commits. PostgreSQL confirmation requires synchronous_commit=on, fsync=on, full_page_writes=on and a logged logs table. Persistent SQLite requires WAL with FULL/EXTRA, or a disk rollback journal with EXTRA. A slave's local rows and replication outbox share one transaction. Retries keep the original IDs; conflicting content under an existing ID is retained and reported, never overwritten.

Database and outbox failures use Titan's bounded diagnostic formatter, including the underlying connection failures of an AggregateError with an empty message. Common credential forms are redacted. A failed confirmation rejects the current drain and retains the same records for a later retry; it does not immediately loop during owner acknowledgment or shutdown. CLI ingestion status and Console use the same reading and show ingestion, local-journal and owner-journal failures separately. An empty error reported by an older backend is still a failure, with an explicit notice that its details are unavailable.

File projection has its own acknowledgment. Before appending, the journal records the file identity and original offset. After a crash, matching complete bytes are accepted once; a proven partial append is replayed from that offset. Unrelated or replaced bytes cause a diagnostic and remain intact. Rotation and compression are serialized with appends. File retention may remove acknowledged historical archives according to maxSize/maxFiles; it does not evict pending delivery records.

Time and multiline messages

• A structured message keeps its supplied timestamp. PostgreSQL stores microseconds; the original timestamp also survives in capture metadata. Ordering never manufactures fractional seconds. Capture sequence and stable IDs are separate from event time.

• Unstructured messages carry the parent's time of first read from the pipe. That is capture time, not proof of when the child generated the text. The record detail identifies reported, captured, inferred or legacy observed time.

• JSON lines define explicit boundaries. Indented continuations and contiguous unstructured stderr are assembled per process/stream until the next structured boundary or 100 ms of idle time. Arbitrary plain-text output has no universal message-boundary protocol; use newline-delimited structured JSON for exact application-defined boundaries.

• A pending multiline record is journaled before it is considered accepted. An interrupted owner leaves those bytes recoverable. Physical lines above 256 KiB use linked fragments; oversized multiline records retain a continuation reference. Fragment boundaries are explicit metadata, not discarded text.

• Time-of-day-only pretty headers require a date inference. Historical undated output remains uncertain; this implementation does not retroactively invent timestamps for old files.

Files and startup failures

Standalone apps write logs/{app}/app.log; project apps write projects/{project}/{stack}/logs/{app}/app.log. Error and fatal records also go to error.log. JSON files include _omnitron capture metadata. The daemon writes logs/omnitron.log. Defaults remain level: info, maxSize: 50mb, maxFiles: 10, compress: true; per-app overrides use observability.logging.

Native entry guards cover imports and initialization before Titan is available. They synchronously report one structured failure with UTC time, ID, component and stack to stderr, and preserve a private source file under $OMNITRON_HOME/data/startup-failures. The next working backend imports those files through its durable journal with an atomic import checkpoint. Repeated startup does not duplicate the row. Keep these source files as evidence; a storage failure can still leave stderr as the final available destination.

The in-memory tail is a bounded convenience view. Database history, durable pending delivery and rotated files have distinct retention and availability; a full live window is not evidence that no other records exist.

File projection during LogManager.dispose() has a default five-second budget, configurable through LogManagerConfig.shutdownProjectionTimeoutMs. An in-flight record finishes before the journal closes; the next record is deferred when the budget expires. This bounds backlog draining, not a stuck operating-system I/O call. Deferred records and append intents remain in SQLite and replay after the next start with the same IDs and offsets. Concurrent disposal callers await the same completion. This prevents a large derived-file backlog from indefinitely holding the owner shutdown receipt; it does not discard or acknowledge unwritten records. An explicit flush() outside disposal still waits for the file projection.

Owner and supervisor journals

The resident execution owner owns worker pools even when the backend is replaced. Its structured stdout includes worker replacement reasons (max_lifetime, request_count, memory_limit, memory_warning) and process exits. Retaining only stderr loses these events, including errors emitted by the JSON logger.

Generated launchd jobs retain both stdout and stderr. In resident mode the owner writes to $OMNITRON_HOME/execution/owner-service.log and the backend guardian to $OMNITRON_HOME/execution/supervisor.log (under the configured execution directory when overridden). Programmatic unit rendering may set a separate stdoutLog; otherwise both streams share stderrLog. Discarded, relative and control-character paths are refused. Generated systemd units explicitly route both streams to the journal:

journalctl -u omnitron-owner.service -u omnitron-daemon.service --since '10 minutes ago'
# For user services, add --user.

These native supervisor journals are separate from the application log archive and its retention policy. Apply host journal retention or file rotation to them. Editing an installed unit does not reconnect a running process's file descriptors. Review the old and new definitions, preserve the original runtime and private configuration, and schedule a controlled owner restart; it restarts the owner's applications. A backend-only handoff leaves owner logging unchanged. Records already discarded by an older unit cannot be reconstructed.

OmnitronLogs RPC

streamLogs is a poll-friendly "recent entries" call rather than a push subscription. since includes its timestamp boundary; deduplicate by row ID. The response is bounded by tail, so use queryLogs to inspect complete historical windows or bursts larger than the live tail.

CLI

omnitron logs --status              # same persistence verdict as Console Logs
omnitron --json logs --status       # structured queue, capability and commit evidence
omnitron logs                       # daemon log
omnitron logs api                   # one app
omnitron logs api -f -l warn -g err # follow + level filter + regex
omnitron logs api --file            # read directly from files (bypass daemon)
omnitron --json logs api            # JSON envelope with source and entries
omnitron logs sample/production/mail --node worker-1 -n 100

Interpreting diagnostics

Console Logs and omnitron logs --status use the same reading of OmnitronLogs.getIngestionStats(). The status describes the connected backend; selecting a remote node in the log table does not turn a local ingestion counter into remote evidence. Inspect node replication diagnostics for the remote delivery path.

lastCommitAt is a successful database commit, not the newest event timestamp. journal.pending includes records awaiting delivery, and journal.assembling identifies messages still being joined. ownerPending is the observed owner backlog. lastError, ownerError and journal errors explain failure; a zero in an unavailable or older backend is never substituted for evidence. Counters describe this collector's lifetime, not proof that historical data was never lost.

capture: legacy-owner explicitly means the running owner lacks the durable protocol. A backend-only update cannot repair its old truncation, volatile event backlog or missing pipe timestamps. Updating the owner requires an approved application restart. Do not use a legacy backend with a new owner as a silent rollback: restore a matching owner/backend pair, preserving all journals.

Before replacing a legacy owner, save the exact owner/backend release pair, supervision configuration, application inventory, restart configuration and online SQLite backups. Prepare the candidate and rollback with the same dependency closure. Before stopping applications, read their bootstrap declarations in isolated processes using the candidate, the host’s actual Node executable and preload arguments, and the daemon’s working directory. A package import check alone does not exercise child-process loading. Framework preloads must resolve from the installed runtime even when the daemon changes its working directory; preserve the directory seen by application definitions. Drain applications gracefully, stop both supervisors, replace the owner and backend as one reviewed pair, then restore the saved inventory and watch state. An owner restart cannot transfer existing pipes and worker threads; it needs a controlled application maintenance window even when ordinary backend handoffs are seamless.

Accept the transition only after CLI and Console report durable capture, the database has a confirmed commit, owner/journal errors are absent and pending records are acknowledged. Verify application health, RPC/replication progress and immutable experiment identities as separate checks. A running process alone is insufficient. Rollback restores the saved release pair and supervision configuration, retaining all current data and delivery journals; do not restore an old database snapshot over newer writes. Historical records without original capture evidence retain that limitation after the upgrade; their original timestamps cannot be reconstructed by relabeling them.

The compact log rows show event time; expanded metadata includes the capture ID, process, stream, sequence, full timestamp and time basis. CLI JSON includes the same metadata. LogEntryDto.timestamp remains epoch milliseconds for compatibility; use metadata to inspect the original fractional precision.

Metrics

Flow

The daemon collects process observations through Titan. Worker samples enter local history and the replication outbox in one SQLite transaction. The controller pulls pending entries and writes the resulting history to PostgreSQL. CLI and Console read the same storage diagnostics. See the metrics data flow.

Metric persistence and confirmation

The daemon's metric history uses Titan's SQL adapters: PostgreSQL on masters and SQLite on updated slave daemons. Console history queries, CLI diagnostics and the metrics-storage health indicator refer to that same store. No second write-only metric table is maintained.

• A local collection call queues a measurement. flush() confirms that the selected storage completed the queued writes; recordDurable() returns a persistent commit receipt or rejects.

• A slave commits each metric and its replication outbox entry atomically. Its pending queue survives daemon replacement and database reconnection.

• A master commits the received metric and sync_ingested outcome atomically. A missing response leaves the same source entry pending; replay returns the existing outcome without writing the metric twice.

• Original sample timestamps and source-node labels survive transport. The live Prometheus registry is updated after commit and is not the history database.

omnitron metrics --json
omnitron health --json
omnitron node check <node>

The first command includes storage.status and the same summary/detail shown on the Console Metrics page. It distinguishes persistent, volatile and unknown storage, queued samples/bytes, completed writes, errors, retention failures and known losses. Collection/write counters describe the current MetricsService instance, including confirmed retries. Imported history and incoming replication commits have their own evidence; replication receipt counters retain their tracking epoch on disk. A successful RPC call or an empty queue is not substituted for database confirmation.

Migration 015_persistent_metrics creates Titan's metrics_samples table and indexes through Omnitron's existing Kysera-backed migration lifecycle. For a resident backend, prepare/apply the reviewed additive schema transition before handoff; retain its exact rollback runtime and receipt. Do not run a down migration to roll back the backend. Slave updates use the local SQLite schema initializer. Applications do not need a direct Kysera dependency.

Old unverified receipts are retained as historical evidence. An upgrade cannot certify them. omnitron --json node reconcile <node> compares retained metric payloads with the database; --restore explicitly repairs missing samples and records new commit proofs. Original claims remain unchanged. Console and CLI show later verification/restoration separately; see reconciliation limits and recovery. A master update improves confirmation for new deliveries immediately; the source polling producer changes to the atomic outbox only after its backend is updated.

Ordinary collection buffers are bounded and volatile before commit. The default is 50,000 samples / 16 MiB; overflow is reported, never acknowledged as persisted. Chart history uses monitoring.metrics.historyMaxAge (default 7d), passed to Titan's retention policy. For example, historyMaxAge: '24h' keeps one day. The legacy numeric monitoring.metrics.retention field does not control SQL history. Pruning chart history and pruning delivered replication entries are separate operations. See the Titan metrics contract for durable calls, database settings, query budgets and optional storage adapters.

Producer side — MetricsBridge

apps/omnitron/src/observability/metrics-bridge.ts is the in-daemon bridge that:

• Hosts the MetricsService shared by daemon services.

• Pushes daemon-internal metrics (RPC call counts, app supervision events, infra reconciler outcomes) to storage on the same cadence as app-side metrics.

• Generates daemon-level omnitron_* series alongside per-app series.

Application metrics and worker replication

Import TitanMetricsModule for application-specific instruments. An in-memory application registry retains samples in that process. Configure a persistent storage adapter when the application needs its own retained history. The Titan metrics guide covers recording, querying and flushing samples.

The daemon samples the processes it supervises. A worker writes samples and replication entries in one SQLite transaction. The controller pulls entries over the authenticated mesh, commits delivery evidence and acknowledges the source. Use omnitron --json metrics and node diagnostics to inspect persistence and the replication backlog. Older sources may have different evidence coverage.

Read side — OmnitronMetrics (from titan-metrics)

The daemon registers its role-protected OmnitronMetricsRpcService facade over Titan's metrics service. The webapp and CLI use the same contracts.

See titan-metrics docs for the full API — getSnapshot, querySeries, getPrometheusText, getStorageStatus and restricted maintenance operations.

Prometheus scrape endpoint — MetricsServer

apps/omnitron/src/observability/metrics-server.ts runs a tiny standalone HTTP server alongside the RPC surface, so an existing Prometheus can scrape the daemon directly:

• GET /metrics → text/plain Prometheus exposition (metrics.getPrometheusText(), after refreshing per-app gauges).

• GET /healthz (and /health) → ok for liveness probes.

• Anything else → 404; non-GET → 405.

Port defaults to httpPort + 3 (≈ 9803); it binds to 127.0.0.1 by default so the aggregate counters aren't exposed network-wide. To scrape remotely, bind 0.0.0.0 and set a bearer token (authToken) — /metrics then requires Authorization: Bearer <token> (constant-time compared); /healthz stays public. Binding non-loopback without a token logs a loud warning.

So the "no separate Prometheus to deploy" baseline still holds — but if you do run Prometheus, this is the endpoint to point it at, rather than the RPC read path.

Aggregated metrics — AggregatedMetricsDto

OmnitronDaemon.getMetrics({name?}) returns:

interface AggregatedMetricsDto {
  timestamp: number;
  apps: Record<string, {
    cpu:        number;
    memory:     number;
    requests?:  number;
    errors?:    number;
    latency?:   { p50, p95, p99, mean };
  }>;
  totals: {
    cpu:    number;
    memory: number;
  };
}

This is the canonical "what's everyone using" snapshot — used by the dashboard's overview tiles.

CLI

omnitron metrics                    # all apps
omnitron metrics api                # one app
omnitron --json metrics             # NDJSON

For time-series queries (OmnitronMetrics.querySeries), use the webapp — OmnitronMetrics is a daemon-hosted service, so it isn't reachable through omnitron exec (which targets services running inside a managed app).

Traces

Flow

sequenceDiagram
  participant App as App process
  participant Producer as Authorized trace producer
  participant TC as OmnitronTraces service
  participant Store as Postgres (OmnitronDatabase)
  participant UI as Webapp / CLI

  App->>Producer: Finished application span
  Producer->>TC: ingestSpan or ingestBatch
  TC->>TC: buffer span (flush at 500 or every 5s)
  TC->>Store: flush batch (INSERT)
  UI->>TC: getTrace({traceId}) | queryTraces(filter) | getServiceMap()
  TC->>Store: SQL query
  Store-->>UI: trace(s) / service map

OmnitronTraces RPC

Storage is Postgres-backed, not an in-memory ring: TraceCollectorService write-buffers spans (flushing to OmnitronDatabase whenever the buffer hits 500 spans or every 5 s), and getTrace / queryTraces / getServiceMap run SQL queries against that table.

TraceFilter

interface TraceFilter {
  service?:    string;
  operation?:  string;
  minDuration?: number;        // ms
  maxDuration?: number;
  from?:       string;         // ISO timestamp
  to?:         string;
  tags?:       Record<string, string>;
  limit?:      number;
}

CLI

No dedicated omnitron traces command. OmnitronTraces is a daemon-hosted RPC service, and omnitron exec only reaches services inside a managed app (it routes by app name) — so the trace API is consumed by the webapp's /traces page rather than from the CLI.

Operational failures

The diagnostics guide describes failure capture, recovery state in CLI and Console, and optional delivery through Titan notifications. The runtime indicator reports unresolved daemon failures independently of app process and infrastructure health.

Health

Two levels: app/infra health (on-demand probes via OmnitronHealth) and node health (cross-machine availability). On top of those sit titan-health indicators that roll up live state for the daemon's own /health summary.

OmnitronHealth — active probing

HealthCheckService (apps/omnitron/src/services/health-check.service.ts, exposed as OmnitronHealth) combines the orchestrator's application health with active HTTP/TCP probes. CLI, Console and agent tools read this shared service:

• Application health — domain checks returned by the supervised worker. A failed check or failed RPC stays visible even if the process is online.

• Declared readiness — configured readiness measurements replace an assumed HTTP endpoint when supplied by the application.

• HTTP — GET /health on the application's published port; expect HTTP 200.

• TCP — connect to the published ports of declared infrastructure services.

• Process liveness — reported separately; it does not override failed checks.

Health hooks should return { status, checks: [{ name, status, message }] }. Custom details fields are not part of this contract: put the actionable reason in a check's message. A hook that returns only a degraded or unhealthy verdict still causes the aggregate report to fail. omnitron --json health-check [app] returns the same checks with a nonzero exit code for an unhealthy or degraded report.

sequenceDiagram
  participant Caller as CLI / webapp
  participant HC as OmnitronHealth (HealthCheckService)
  participant App as App process / container

  Caller->>HC: checkApp / checkApps / checkInfrastructure / checkAll
  HC->>App: HTTP GET health endpoint / TCP connect
  App-->>HC: 2xx / socket accepted
  HC-->>Caller: HealthReport { overall, checks[] }

AppHealthIndicator

apps/omnitron/src/monitoring/app-health.indicator.ts is a titan-health indicator (name 'apps') — not a network prober. It reads orchestrator.list() and grades by process status: healthy when all managed apps are online, degraded when some are crashed/errored, unhealthy when a critical app is down. It feeds the daemon's aggregated /health summary, not the per-app HTTP/TCP probe path above.

DockerHealthIndicator

apps/omnitron/src/monitoring/docker-health.indicator.ts is the infra counterpart (name 'docker'). It reads InfrastructureService.getState() and aggregates each managed container's reported status / health — it does not itself run Docker healthchecks or fall back to a TCP probe; it reports the infra service's cached state.

OmnitronHealth RPC

OmnitronNodes health (cross-machine)

For fleet nodes (registered in OmnitronNodes), the health-monitor runs separately:

Per-node history is stored for 90 days and surfaced as the green/yellow/red uptime bars in the webapp.

CLI

omnitron health [app]              # daemon-level health summary
omnitron health-check [app]        # detailed HTTP/TCP probe report
omnitron node check [id]           # node connectivity probe

Putting it together — what the daemon stores

The default storage directory is ~/.omnitron/; OMNITRON_HOME and explicit storage settings can relocate it. Selected files are shown below. Application names and rotated archive names depend on the registered stack and retention settings.

Controller logs, metrics and traces persist in its configured database. Worker history and pending replication use local SQLite storage. These query stores are distinct from the private delivery journals; see the pipeline descriptions above for their commit and acknowledgment semantics.

Retention

Tune retention to match storage budget; defaults are sensible for small-to-medium fleets.

Wiring telemetry from your apps

A supervised application's stdout and stderr already reach the owner log capture. To add application-specific metrics, register the metrics module and choose storage based on the required retention. This example uses a process-local registry.

import { Module } from '@omnitron-dev/titan/decorators';
import { TitanMetricsModule } from '@omnitron-dev/titan-metrics';

@Module({
  imports: [
    TitanMetricsModule.forRoot({
      appName: 'api',
      collection: { enabled: true },
      storage: { type: 'memory' },
    }),
  ],
})
class AppModule {}

Daemon process metrics and replication work independently of that registry. For app-specific readiness checks, configure Titan health.

Anti-patterns

• High-cardinality metric labels. userId as a label multiplies series count by user count. Use buckets / tiers, not per-entity ids.

• Treating a file tail as a persistence receipt. --file supports filters and offline recovery, but a bounded tail does not prove database delivery or complete historical coverage. Check logs --status and the requested history window.

• No retention on the persistent metrics store. Disk fills; queries get slow. Default '7d' is sensible.

• Tracing every span at 100% sampling. Volume kills the daemon. Use head sampling at the producer (titan-tracing has built-in samplers).

• Custom HTTP health endpoint that talks to the database on every probe. Health probes run every 15 s — that's a lot of pointless queries. Use a cached health summary.

• Treating getMetrics() as a high-frequency call. It aggregates across apps; if you need real-time per-app data, query OmnitronMetrics.querySeries with the right filter.

See also

• titan-metrics — producer-side detail

• titan-telemetry-relay — store-and-forward for offline producers

• titan-health — indicator framework

• Services reference — OmnitronLogs, OmnitronTelemetry, OmnitronTraces, OmnitronHealth, OmnitronNodes

• Architecture — where these pipelines sit

• CLI — omnitron logs / metrics / health

Bounded queries and agent evidence

Log filters are validated in the collector before SQL construction, so CLI, Console and RPC consumers share the bounds: limit 1–1000, recent tail 1–10000, offset 0–100000, valid levels/timestamps, ordered time ranges and bounded strings/labels. These input bounds do not by themselves impose a database execution deadline. Prefer a narrow application and time window over deep offset pagination.

The MCP monitoring readers reuse the same services:

Each source reports available, unavailable or limited, with collection time and duration. A failed or oversized source makes the combined answer partial. Each source has a 64 KiB response budget and a five-second diagnostic deadline; the underlying transport still owns RPC cancellation. An oversized source is withheld with a request to narrow the scope, rather than presented as a complete subset.

Collection time is not event time or evidence of freshness. Inspect source timestamps and ingestion loss. Zero dropped records since daemon startup does not prove that older data was never lost. A persisted alert summary does not prove that its evaluator is currently connected. Sampled counts must not be presented as exact population totals.

These tools currently run through the local administrative MCP connection. They do not provide delegated agent scopes, bounded database query cost, resumable subscriptions or guaranteed evidence retention. The agent monitoring design specifies those next contracts and their failure tests.

Treat log messages, traces and tool output as untrusted data. Embedded instructions cannot authorize infrastructure changes. Read permission never implies permission to acknowledge an incident, silence a rule or run remediation.

Auditing history before receipts

Current replication health and historical completeness are separate readings. For data predating delivery evidence, use the retained-history recovery procedure. Node diagnostics in CLI and Console expose the same source coverage, verification time, restoration counts and unresolved gaps. Historical imports do not update current gauges. Records intentionally outside retention remain identifiable in their preserved source archives; they are not reported as a transport failure.

A later recovery keeps the first audit intact and links a new committed proof to its exact gap fingerprint. Diagnostics distinguish the resolved finding from still unknown history. Group repeated stack frames into error reports before counting events; a plain stack trace or a time-of-day-only header cannot be dated from the audit time. Neighboring absolute timestamps can establish context, but do not supply an exact event time on their own.