--- id: "omnitron/observability" title: "Observability" description: "How logs, metrics, traces, and health flow through Omnitron." --- # Observability Supervised applications can emit logs, metrics, traces and health signals through Titan and Omnitron. Available signals depend on the application modules, instrumentation and configured storage. A running daemon does not imply complete instrumentation. The baseline does not require a separate Prometheus, Loki or Tempo deployment. This page describes those pipelines end-to-end so you know which knob to turn and which storage to inspect when something goes wrong. Verified against `apps/omnitron/src/{observability,monitoring,services}/`. ## The four pipelines | Signal | Owner | Storage | Exposure | | ----------- | ---------------------------------------------------------------------------- | ------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------- | | **Logs** | Titan PM capture + `LogManager` + `LogCollectorService` | Private SQLite delivery journals; per-app files; PostgreSQL/SQLite `logs` | RPC: `OmnitronLogs.queryLogs` / `streamLogs` | | **Metrics** | `MetricsBridge`, Titan storage and `OmnitronSync` | PostgreSQL on controllers, SQLite history and outbox on workers | CLI and Console query `OmnitronMetrics` | | **Traces** | `OmnitronTraces` (`TraceCollectorService`) | Postgres (`OmnitronDatabase`), write-buffered | RPC: `ingestSpan`, `ingestBatch`, `getTrace`, `queryTraces`, `getServiceMap` | | **Health** | `OmnitronHealth` (app/infra probes) + `OmnitronNodes` health monitor (fleet) | rolling window in state-store / node history | RPC: `OmnitronHealth.checkApp` / `checkAll`; `OmnitronNodes.getCheckHistory` / `getUptimeBar` | ## Logs ### Capture, persistence and replay ```mermaid sequenceDiagram participant App as Fork or worker thread participant PM as Titan PM capture participant Journal as Private SQLite journal participant Backend as Omnitron backend participant DB as PostgreSQL or node SQLite participant Files as Application files App->>PM: stdout / stderr, including startup failures PM->>Journal: ID + source + capture time + message bytes Note over Journal: WAL + synchronous=FULL
unfinished multiline records persist too Journal->>Files: Append with a durable file offset
fsync Files-->>Journal: File projection confirmed Backend->>Journal: Read a bounded page Backend->>DB: Commit logs and node replication outbox together DB-->>Backend: Transaction committed Backend->>Journal: Confirm exact IDs and payload digests Note over Journal: Release only after both destinations confirm ``` Titan PM installs output capture before readiness, including for workers replaced by a pool. One UTF-8 reader handles each pipe from spawn through EOF. A failed initialization does not require a `WorkerHandle` to exist before output reaches storage. Stdout and stderr from different process instances are assembled separately. The same canonical record feeds files, the database and replication. The local delivery files are `data/application-logs.sqlite` and `data/backend-logs.sqlite` under the configured Omnitron storage directory. They use SQLite WAL with `synchronous=FULL` and private permissions. Pending messages and finalized records survive process replacement. No oldest-first eviction is applied to unacknowledged logs. Recovery checks the writer's host boot and process start identity, not just its PID. It periodically revisits unfinished records, so a previous backend that dies after handoff does not strand them until another restart. If the original writer's lifetime cannot be verified, those bytes remain pending with an explicit diagnostic; they are not finalized while a writer may still be assembling them. On macOS, all process readers use the boot-session UUID and a birth timestamp explicitly captured in UTC. Legacy timestamps did not retain their timezone: a changed rendering or a mixed legacy/UTC identity does not prove that the writer exited. Its unfinished bytes remain pending until native death or a verified change of host boot is observed. If capture cannot write its journal, the pipe is paused and the same frame is retried. Monitor storage availability: backpressure can eventually block an application that continues writing to a full pipe. A database response is acknowledged only after the enclosing transaction commits. PostgreSQL confirmation requires `synchronous_commit=on`, `fsync=on`, `full_page_writes=on` and a logged `logs` table. Persistent SQLite requires WAL with `FULL`/`EXTRA`, or a disk rollback journal with `EXTRA`. A slave's local rows and replication outbox share one transaction. Retries keep the original IDs; conflicting content under an existing ID is retained and reported, never overwritten. Database and outbox failures use Titan's bounded diagnostic formatter, including the underlying connection failures of an `AggregateError` with an empty message. Common credential forms are redacted. A failed confirmation rejects the current drain and retains the same records for a later retry; it does not immediately loop during owner acknowledgment or shutdown. CLI ingestion status and Console use the same reading and show ingestion, local-journal and owner-journal failures separately. An empty error reported by an older backend is still a failure, with an explicit notice that its details are unavailable. File projection has its own acknowledgment. Before appending, the journal records the file identity and original offset. After a crash, matching complete bytes are accepted once; a proven partial append is replayed from that offset. Unrelated or replaced bytes cause a diagnostic and remain intact. Rotation and compression are serialized with appends. File retention may remove acknowledged historical archives according to `maxSize`/`maxFiles`; it does not evict pending delivery records. ### Time and multiline messages - A structured message keeps its supplied timestamp. PostgreSQL stores microseconds; the original timestamp also survives in capture metadata. Ordering never manufactures fractional seconds. Capture sequence and stable IDs are separate from event time. - Unstructured messages carry the parent's time of first read from the pipe. That is capture time, not proof of when the child generated the text. The record detail identifies `reported`, `captured`, `inferred` or legacy `observed` time. - JSON lines define explicit boundaries. Indented continuations and contiguous unstructured stderr are assembled per process/stream until the next structured boundary or 100 ms of idle time. Arbitrary plain-text output has no universal message-boundary protocol; use newline-delimited structured JSON for exact application-defined boundaries. - A pending multiline record is journaled before it is considered accepted. An interrupted owner leaves those bytes recoverable. Physical lines above 256 KiB use linked fragments; oversized multiline records retain a continuation reference. Fragment boundaries are explicit metadata, not discarded text. - Time-of-day-only pretty headers require a date inference. Historical undated output remains uncertain; this implementation does not retroactively invent timestamps for old files. ### Files and startup failures Standalone apps write `logs/{app}/app.log`; project apps write `projects/{project}/{stack}/logs/{app}/app.log`. Error and fatal records also go to `error.log`. JSON files include `_omnitron` capture metadata. The daemon writes `logs/omnitron.log`. Defaults remain `level: info`, `maxSize: 50mb`, `maxFiles: 10`, `compress: true`; per-app overrides use `observability.logging`. Native entry guards cover imports and initialization before Titan is available. They synchronously report one structured failure with UTC time, ID, component and stack to stderr, and preserve a private source file under `$OMNITRON_HOME/data/startup-failures`. The next working backend imports those files through its durable journal with an atomic import checkpoint. Repeated startup does not duplicate the row. Keep these source files as evidence; a storage failure can still leave stderr as the final available destination. The in-memory tail is a bounded convenience view. Database history, durable pending delivery and rotated files have distinct retention and availability; a full live window is not evidence that no other records exist. File projection during `LogManager.dispose()` has a default five-second budget, configurable through `LogManagerConfig.shutdownProjectionTimeoutMs`. An in-flight record finishes before the journal closes; the next record is deferred when the budget expires. This bounds backlog draining, not a stuck operating-system I/O call. Deferred records and append intents remain in SQLite and replay after the next start with the same IDs and offsets. Concurrent disposal callers await the same completion. This prevents a large derived-file backlog from indefinitely holding the owner shutdown receipt; it does not discard or acknowledge unwritten records. An explicit `flush()` outside disposal still waits for the file projection. ### Owner and supervisor journals The resident execution owner owns worker pools even when the backend is replaced. Its structured stdout includes worker replacement reasons (`max_lifetime`, `request_count`, `memory_limit`, `memory_warning`) and process exits. Retaining only stderr loses these events, including errors emitted by the JSON logger. Generated launchd jobs retain both stdout and stderr. In resident mode the owner writes to `$OMNITRON_HOME/execution/owner-service.log` and the backend guardian to `$OMNITRON_HOME/execution/supervisor.log` (under the configured execution directory when overridden). Programmatic unit rendering may set a separate `stdoutLog`; otherwise both streams share `stderrLog`. Discarded, relative and control-character paths are refused. Generated systemd units explicitly route both streams to the journal: ```bash journalctl -u omnitron-owner.service -u omnitron-daemon.service --since '10 minutes ago' # For user services, add --user. ``` These native supervisor journals are separate from the application log archive and its retention policy. Apply host journal retention or file rotation to them. Editing an installed unit does not reconnect a running process's file descriptors. Review the old and new definitions, preserve the original runtime and private configuration, and schedule a controlled owner restart; it restarts the owner's applications. A backend-only handoff leaves owner logging unchanged. Records already discarded by an older unit cannot be reconstructed. ### `OmnitronLogs` RPC | Method | Effect | | --------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------- | | `queryLogs({app?, nodeId?, level?, search?, labels?, traceId?, from?, to?, limit?, offset?})` | Filter & return entries (note: full-text filter is `search`, not `grep`; page with `limit`/`offset`) | | `streamLogs({app?, nodeId?, level?, search?, labels?, tail?, since?})` | Return the last `tail` entries (default 100) in chronological order — poll for near-real-time tailing | | `getLogStats()` | Exact stored counts and archive range; an analytic query, not a polling health check | | `getIngestionStats()` | Capture capability, durable pending records/bytes, owner errors, last confirmed DB commit and last write failure; inexpensive to poll | `streamLogs` is a poll-friendly "recent entries" call rather than a push subscription. `since` includes its timestamp boundary; deduplicate by row ID. The response is bounded by `tail`, so use `queryLogs` to inspect complete historical windows or bursts larger than the live tail. ### CLI ```bash omnitron logs --status # same persistence verdict as Console Logs omnitron --json logs --status # structured queue, capability and commit evidence omnitron logs # daemon log omnitron logs api # one app omnitron logs api -f -l warn -g err # follow + level filter + regex omnitron logs api --file # read directly from files (bypass daemon) omnitron --json logs api # JSON envelope with source and entries omnitron logs sample/production/mail --node worker-1 -n 100 ``` ### Interpreting diagnostics Console **Logs** and `omnitron logs --status` use the same reading of `OmnitronLogs.getIngestionStats()`. The status describes the connected backend; selecting a remote node in the log table does not turn a local ingestion counter into remote evidence. Inspect node replication diagnostics for the remote delivery path. `lastCommitAt` is a successful database commit, not the newest event timestamp. `journal.pending` includes records awaiting delivery, and `journal.assembling` identifies messages still being joined. `ownerPending` is the observed owner backlog. `lastError`, `ownerError` and journal errors explain failure; a zero in an unavailable or older backend is never substituted for evidence. Counters describe this collector's lifetime, not proof that historical data was never lost. `capture: legacy-owner` explicitly means the running owner lacks the durable protocol. A backend-only update cannot repair its old truncation, volatile event backlog or missing pipe timestamps. Updating the owner requires an approved application restart. Do not use a legacy backend with a new owner as a silent rollback: restore a matching owner/backend pair, preserving all journals. Before replacing a legacy owner, save the exact owner/backend release pair, supervision configuration, application inventory, restart configuration and online SQLite backups. Prepare the candidate and rollback with the same dependency closure. Before stopping applications, read their bootstrap declarations in isolated processes using the candidate, the host’s actual Node executable and preload arguments, and the daemon’s working directory. A package import check alone does not exercise child-process loading. Framework preloads must resolve from the installed runtime even when the daemon changes its working directory; preserve the directory seen by application definitions. Drain applications gracefully, stop both supervisors, replace the owner and backend as one reviewed pair, then restore the saved inventory and watch state. An owner restart cannot transfer existing pipes and worker threads; it needs a controlled application maintenance window even when ordinary backend handoffs are seamless. Accept the transition only after CLI and Console report durable capture, the database has a confirmed commit, owner/journal errors are absent and pending records are acknowledged. Verify application health, RPC/replication progress and immutable experiment identities as separate checks. A running process alone is insufficient. Rollback restores the saved release pair and supervision configuration, retaining all current data and delivery journals; do not restore an old database snapshot over newer writes. Historical records without original capture evidence retain that limitation after the upgrade; their original timestamps cannot be reconstructed by relabeling them. The compact log rows show event time; expanded metadata includes the capture ID, process, stream, sequence, full timestamp and time basis. CLI JSON includes the same metadata. `LogEntryDto.timestamp` remains epoch milliseconds for compatibility; use metadata to inspect the original fractional precision. ## Metrics ### Flow The daemon collects process observations through Titan. Worker samples enter local history and the replication outbox in one SQLite transaction. The controller pulls pending entries and writes the resulting history to PostgreSQL. CLI and Console read the same storage diagnostics. See the [metrics data flow](/docs/omnitron/architecture#data-flow--metrics-aggregation). ### Metric persistence and confirmation The daemon's metric history uses Titan's SQL adapters: PostgreSQL on masters and SQLite on updated slave daemons. Console history queries, CLI diagnostics and the `metrics-storage` health indicator refer to that same store. No second write-only metric table is maintained. - A local collection call queues a measurement. `flush()` confirms that the selected storage completed the queued writes; `recordDurable()` returns a persistent commit receipt or rejects. - A slave commits each metric and its replication outbox entry atomically. Its pending queue survives daemon replacement and database reconnection. - A master commits the received metric and `sync_ingested` outcome atomically. A missing response leaves the same source entry pending; replay returns the existing outcome without writing the metric twice. - Original sample timestamps and source-node labels survive transport. The live Prometheus registry is updated after commit and is not the history database. ```bash omnitron metrics --json omnitron health --json omnitron node check ``` The first command includes `storage.status` and the same summary/detail shown on the Console Metrics page. It distinguishes persistent, volatile and unknown storage, queued samples/bytes, completed writes, errors, retention failures and known losses. Collection/write counters describe the current MetricsService instance, including confirmed retries. Imported history and incoming replication commits have their own evidence; replication receipt counters retain their tracking epoch on disk. A successful RPC call or an empty queue is not substituted for database confirmation. Migration `015_persistent_metrics` creates Titan's `metrics_samples` table and indexes through Omnitron's existing Kysera-backed migration lifecycle. For a resident backend, prepare/apply the reviewed additive schema transition before handoff; retain its exact rollback runtime and receipt. Do not run a down migration to roll back the backend. Slave updates use the local SQLite schema initializer. Applications do not need a direct Kysera dependency. Old unverified receipts are retained as historical evidence. An upgrade cannot certify them. `omnitron --json node reconcile ` compares retained metric payloads with the database; `--restore` explicitly repairs missing samples and records new commit proofs. Original claims remain unchanged. Console and CLI show later verification/restoration separately; see [reconciliation limits and recovery](/docs/omnitron/cluster#reconcile-historical-metric-receipts). A master update improves confirmation for new deliveries immediately; the source polling producer changes to the atomic outbox only after its backend is updated. Ordinary collection buffers are bounded and volatile before commit. The default is 50,000 samples / 16 MiB; overflow is reported, never acknowledged as persisted. Chart history uses `monitoring.metrics.historyMaxAge` (default `7d`), passed to Titan's retention policy. For example, `historyMaxAge: '24h'` keeps one day. The legacy numeric `monitoring.metrics.retention` field does not control SQL history. Pruning chart history and pruning delivered replication entries are separate operations. See the [Titan metrics contract](/docs/titan/modules/metrics) for durable calls, database settings, query budgets and optional storage adapters. ### Producer side — `MetricsBridge` `apps/omnitron/src/observability/metrics-bridge.ts` is the in-daemon bridge that: - Hosts the `MetricsService` shared by daemon services. - Pushes daemon-internal metrics (RPC call counts, app supervision events, infra reconciler outcomes) to storage on the same cadence as app-side metrics. - Generates daemon-level `omnitron_*` series alongside per-app series. ### Application metrics and worker replication Import `TitanMetricsModule` for application-specific instruments. An in-memory application registry retains samples in that process. Configure a persistent storage adapter when the application needs its own retained history. The [Titan metrics guide](/docs/titan/modules/metrics) covers recording, querying and flushing samples. The daemon samples the processes it supervises. A worker writes samples and replication entries in one SQLite transaction. The controller pulls entries over the authenticated mesh, commits delivery evidence and acknowledges the source. Use `omnitron --json metrics` and node diagnostics to inspect persistence and the replication backlog. Older sources may have different evidence coverage. ### Read side — `OmnitronMetrics` (from `titan-metrics`) The daemon registers its role-protected `OmnitronMetricsRpcService` facade over Titan's metrics service. The webapp and CLI use the same contracts. See [titan-metrics docs](/docs/titan/modules/metrics) for the full API — `getSnapshot`, `querySeries`, `getPrometheusText`, `getStorageStatus` and restricted maintenance operations. ### Prometheus scrape endpoint — `MetricsServer` `apps/omnitron/src/observability/metrics-server.ts` runs a tiny standalone HTTP server *alongside* the RPC surface, so an existing Prometheus can scrape the daemon directly: - `GET /metrics` → text/plain Prometheus exposition (`metrics.getPrometheusText()`, after refreshing per-app gauges). - `GET /healthz` (and `/health`) → `ok` for liveness probes. - Anything else → 404; non-GET → 405. Port defaults to `httpPort + 3` (≈ `9803`); it binds to **`127.0.0.1` by default** so the aggregate counters aren't exposed network-wide. To scrape remotely, bind `0.0.0.0` **and** set a bearer token (`authToken`) — `/metrics` then requires `Authorization: Bearer ` (constant-time compared); `/healthz` stays public. Binding non-loopback without a token logs a loud warning. So the "no separate Prometheus to deploy" baseline still holds — but if you *do* run Prometheus, this is the endpoint to point it at, rather than the RPC read path. ### Aggregated metrics — `AggregatedMetricsDto` `OmnitronDaemon.getMetrics({name?})` returns: ```typescript interface AggregatedMetricsDto { timestamp: number; apps: Record; totals: { cpu: number; memory: number; }; } ``` This is the canonical "what's everyone using" snapshot — used by the dashboard's overview tiles. ### CLI ```bash omnitron metrics # all apps omnitron metrics api # one app omnitron --json metrics # NDJSON ``` For time-series queries (`OmnitronMetrics.querySeries`), use the webapp — `OmnitronMetrics` is a daemon-hosted service, so it isn't reachable through `omnitron exec` (which targets services running inside a managed app). ## Traces ### Flow ```mermaid sequenceDiagram participant App as App process participant Producer as Authorized trace producer participant TC as OmnitronTraces service participant Store as Postgres (OmnitronDatabase) participant UI as Webapp / CLI App->>Producer: Finished application span Producer->>TC: ingestSpan or ingestBatch TC->>TC: buffer span (flush at 500 or every 5s) TC->>Store: flush batch (INSERT) UI->>TC: getTrace({traceId}) | queryTraces(filter) | getServiceMap() TC->>Store: SQL query Store-->>UI: trace(s) / service map ``` ### `OmnitronTraces` RPC | Method | Effect | | ---------------------- | ----------------------------------------------- | | `ingestSpan(span)` | Push one span | | `ingestBatch({spans})` | Push many | | `getTrace({traceId})` | One trace (all its spans) | | `queryTraces(filter)` | Filter by service / op / duration / tags / time | | `getServiceMap()` | Derived `service_a → service_b` call graph | Storage is **Postgres-backed**, not an in-memory ring: `TraceCollectorService` write-buffers spans (flushing to `OmnitronDatabase` whenever the buffer hits 500 spans or every 5 s), and `getTrace` / `queryTraces` / `getServiceMap` run SQL queries against that table. ### `TraceFilter` ```typescript interface TraceFilter { service?: string; operation?: string; minDuration?: number; // ms maxDuration?: number; from?: string; // ISO timestamp to?: string; tags?: Record; limit?: number; } ``` ### CLI No dedicated `omnitron traces` command. `OmnitronTraces` is a daemon-hosted RPC service, and `omnitron exec` only reaches services *inside a managed app* (it routes by app name) — so the trace API is consumed by the webapp's `/traces` page rather than from the CLI. ## Operational failures The [diagnostics guide](/docs/omnitron/diagnostics) describes failure capture, recovery state in CLI and Console, and optional delivery through Titan notifications. The runtime indicator reports unresolved daemon failures independently of app process and infrastructure health. ## Health Two levels: **app/infra health** (on-demand probes via `OmnitronHealth`) and **node health** (cross-machine availability). On top of those sit **titan-health indicators** that roll up live state for the daemon's own `/health` summary. ### `OmnitronHealth` — active probing `HealthCheckService` (`apps/omnitron/src/services/health-check.service.ts`, exposed as `OmnitronHealth`) combines the orchestrator's application health with active HTTP/TCP probes. CLI, Console and agent tools read this shared service: - **Application health** — domain checks returned by the supervised worker. A failed check or failed RPC stays visible even if the process is online. - **Declared readiness** — configured readiness measurements replace an assumed HTTP endpoint when supplied by the application. - **HTTP** — GET `/health` on the application's published port; expect HTTP 200. - **TCP** — connect to the published ports of declared infrastructure services. - **Process liveness** — reported separately; it does not override failed checks. Health hooks should return `{ status, checks: [{ name, status, message }] }`. Custom `details` fields are not part of this contract: put the actionable reason in a check's `message`. A hook that returns only a degraded or unhealthy verdict still causes the aggregate report to fail. `omnitron --json health-check [app]` returns the same checks with a nonzero exit code for an unhealthy or degraded report. ```mermaid sequenceDiagram participant Caller as CLI / webapp participant HC as OmnitronHealth (HealthCheckService) participant App as App process / container Caller->>HC: checkApp / checkApps / checkInfrastructure / checkAll HC->>App: HTTP GET health endpoint / TCP connect App-->>HC: 2xx / socket accepted HC-->>Caller: HealthReport { overall, checks[] } ``` ### `AppHealthIndicator` `apps/omnitron/src/monitoring/app-health.indicator.ts` is a titan-health indicator (name `'apps'`) — **not** a network prober. It reads `orchestrator.list()` and grades by *process status*: `healthy` when all managed apps are online, `degraded` when some are crashed/errored, `unhealthy` when a `critical` app is down. It feeds the daemon's aggregated `/health` summary, not the per-app HTTP/TCP probe path above. ### `DockerHealthIndicator` `apps/omnitron/src/monitoring/docker-health.indicator.ts` is the infra counterpart (name `'docker'`). It reads `InfrastructureService.getState()` and aggregates each managed container's reported `status` / `health` — it does not itself run Docker healthchecks or fall back to a TCP probe; it reports the infra service's cached state. ### `OmnitronHealth` RPC | Method | Effect | | ---------------------------- | --------------------------- | | `checkApp({appName, port?})` | Probe one app on demand | | `checkApps()` | Probe every app | | `checkInfrastructure()` | Probe every infra container | | `checkAll()` | Full platform health report | ### `OmnitronNodes` health (cross-machine) For fleet nodes (registered in `OmnitronNodes`), the **health-monitor** runs separately: | Parameter | Default | | ---------------------- | ----------------------------------------- | | `intervalMs` | `60_000` (1/min) | | `concurrency` | `20` | | `offlineTimeoutMs` | `90_000` | | `pingEnabled` | `true` (ICMP) | | `pingTimeout` | `5_000` | | `sshTimeout` | `10_000` | | `omnitronCheckTimeout` | `15_000` | | `retentionDays` | `90` | | `uptimeIntervalMs` | `86_400_000` (24h per uptime bar segment) | Per-node history is stored for 90 days and surfaced as the green/yellow/red **uptime bars** in the webapp. ### CLI ```bash omnitron health [app] # daemon-level health summary omnitron health-check [app] # detailed HTTP/TCP probe report omnitron node check [id] # node connectivity probe ``` ## Putting it together — what the daemon stores The default storage directory is `~/.omnitron/`; `OMNITRON_HOME` and explicit storage settings can relocate it. Selected files are shown below. Application names and rotated archive names depend on the registered stack and retention settings. **\~/.omnitron/** - `data/` - `daemon-state.db` — Local registries, execution intent and state - `application-logs.sqlite` — Durable application output journal - `backend-logs.sqlite` — Durable backend output journal - `logs/` - `omnitron.log` — Backend log - `omnitron.error.log` — Backend errors - `api/` — Example standalone application - `app.log` — Combined application output - `error.log` — Error output - `app.log.1.gz` — Example retained archive - `projects/` - `{project}/` - `{stack}/` - `logs/` - `{app}/` — Project application logs and rotated archives - `execution/` — Resident owner and guardian journals - `secrets.enc` — Encrypted file-provider secrets Controller logs, metrics and traces persist in its configured database. Worker history and pending replication use local SQLite storage. These query stores are distinct from the private delivery journals; see the pipeline descriptions above for their commit and acknowledgment semantics. ## Retention | Signal | Retention default | | ------------------------------- | ------------------------------------------------------------------------------------------------------- | | Log files | `logging.maxFiles` × `logging.maxSize` per stream | | Log database | `logging.databaseRetentionDays` (default 14 days; nonpositive disables pruning) | | Unacknowledged logs | Retained in the local journal until both destinations confirm; never evicted by file/database retention | | Omnitron metric history | `monitoring.metrics.historyMaxAge` (default `7d`, hourly sweep) | | Standalone Titan metric history | `retention.maxAge` (default `7d`) | | Traces (write buffer) | flushed to Postgres at 500 spans or every 5 s | | Traces (persistent) | Postgres (`OmnitronDatabase`) — trim/retain at the DB layer | | Health history | `healthMonitor.retentionDays` (default 90 days) | | Alert events | from `OmnitronAlerts` per-rule retention | Tune retention to match storage budget; defaults are sensible for small-to-medium fleets. ## Wiring telemetry from your apps A supervised application's stdout and stderr already reach the owner log capture. To add application-specific metrics, register the metrics module and choose storage based on the required retention. This example uses a process-local registry. ```typescript import { Module } from '@omnitron-dev/titan/decorators'; import { TitanMetricsModule } from '@omnitron-dev/titan-metrics'; @Module({ imports: [ TitanMetricsModule.forRoot({ appName: 'api', collection: { enabled: true }, storage: { type: 'memory' }, }), ], }) class AppModule {} ``` Daemon process metrics and replication work independently of that registry. For app-specific readiness checks, configure [Titan health](/docs/titan/modules/health). ## Anti-patterns - **High-cardinality metric labels.** `userId` as a label multiplies series count by user count. Use buckets / tiers, not per-entity ids. - **Treating a file tail as a persistence receipt.** `--file` supports filters and offline recovery, but a bounded tail does not prove database delivery or complete historical coverage. Check `logs --status` and the requested history window. - **No retention on the persistent metrics store.** Disk fills; queries get slow. Default `'7d'` is sensible. - **Tracing every span at 100% sampling.** Volume kills the daemon. Use head sampling at the producer (titan-tracing has built-in samplers). - **Custom HTTP health endpoint that talks to the database on every probe.** Health probes run every 15 s — that's a lot of pointless queries. Use a cached health summary. - **Treating `getMetrics()` as a high-frequency call.** It aggregates across apps; if you need real-time per-app data, query `OmnitronMetrics.querySeries` with the right filter. ## See also - [`titan-metrics`](/docs/titan/modules/metrics) — producer-side detail - [`titan-telemetry-relay`](/docs/titan/modules/telemetry-relay) — store-and-forward for offline producers - [`titan-health`](/docs/titan/modules/health) — indicator framework - [Services reference](/docs/omnitron/services-reference) — `OmnitronLogs`, `OmnitronTelemetry`, `OmnitronTraces`, `OmnitronHealth`, `OmnitronNodes` - [Architecture](/docs/omnitron/architecture) — where these pipelines sit - [CLI](/docs/omnitron/cli#monitoring) — `omnitron logs` / `metrics` / `health` ## Bounded queries and agent evidence Log filters are validated in the collector before SQL construction, so CLI, Console and RPC consumers share the bounds: `limit` 1–1000, recent `tail` 1–10000, `offset` 0–100000, valid levels/timestamps, ordered time ranges and bounded strings/labels. These input bounds do not by themselves impose a database execution deadline. Prefer a narrow application and time window over deep offset pagination. The MCP monitoring readers reuse the same services: | Tool | Reading and budget | | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------- | | `monitoring.snapshot` | Metrics snapshot, alert summary and log ingestion counters in parallel; no archive counts or active health probes | | `metrics.series` | 1–8 explicit applications and metric names, a positive window up to 24 hours, 1m/5m/15m/1h aggregation, at most 1000 requested points | | `alerts.active` | At most 100 firing alerts; does not acknowledge or silence them | Each source reports `available`, `unavailable` or `limited`, with collection time and duration. A failed or oversized source makes the combined answer `partial`. Each source has a 64 KiB response budget and a five-second diagnostic deadline; the underlying transport still owns RPC cancellation. An oversized source is withheld with a request to narrow the scope, rather than presented as a complete subset. Collection time is **not** event time or evidence of freshness. Inspect source timestamps and ingestion loss. Zero dropped records since daemon startup does not prove that older data was never lost. A persisted alert summary does not prove that its evaluator is currently connected. Sampled counts must not be presented as exact population totals. These tools currently run through the local administrative MCP connection. They do not provide delegated agent scopes, bounded database query cost, resumable subscriptions or guaranteed evidence retention. The [agent monitoring design](/docs/omnitron/agents#monitoring-as-evidence) specifies those next contracts and their failure tests. Treat log messages, traces and tool output as untrusted data. Embedded instructions cannot authorize infrastructure changes. Read permission never implies permission to acknowledge an incident, silence a rule or run remediation. ### Auditing history before receipts Current replication health and historical completeness are separate readings. For data predating delivery evidence, use the [retained-history recovery procedure](/docs/omnitron/cluster#history-before-delivery-receipts). Node diagnostics in CLI and Console expose the same source coverage, verification time, restoration counts and unresolved gaps. Historical imports do not update current gauges. Records intentionally outside retention remain identifiable in their preserved source archives; they are not reported as a transport failure. A later recovery keeps the first audit intact and links a new committed proof to its exact gap fingerprint. Diagnostics distinguish the resolved finding from still unknown history. Group repeated stack frames into error reports before counting events; a plain stack trace or a time-of-day-only header cannot be dated from the audit time. Neighboring absolute timestamps can establish context, but do not supply an exact event time on their own.