---
id: "titan/modules/observability-matrix"
title: "Observability matrix"
description: "What every module logs, raises as events, and writes as metrics."
---

# Observability matrix

Operating a Titan app means knowing *what to grep for* and *what
to alert on*. This page lists — per module — the events emitted,
notable log keys, and metric/health signals you can rely on.

All claims here are verified against module source. If a row is
empty, the module is deliberately quiet at that layer.

## At-a-glance

<span class="module-badge">official · @omnitron-dev/titan-*</span>

The **Metrics surface** column distinguishes metrics actively *pushed*
into `titan-metrics` from stats you read on demand (*pull* — a
`getStats()` / `getMetrics()` / `stats()` accessor).

| Module                  |              Emits events              |        Notable logs        |             Metrics surface            |    Health indicator    |
| ----------------------- | :------------------------------------: | :------------------------: | :------------------------------------: | :--------------------: |
| `titan-auth`            |                    —                   |     JWT verify failures    |                    —                   |            —           |
| `titan-cache`           |                    —                   |    L2 fallback / errors    |  hit/miss/evict *(pull: `getStats()`)* |            —           |
| `titan-database`        |                    —                   |     slow query, retries    |                    —                   | yes (via titan-health) |
| `titan-discovery`       |            `discovery:event`           | register, heartbeat, dereg |                    —                   |            —           |
| `titan-events`          |                  proxy                 |       dispatch errors      |                    —                   |            —           |
| `titan-health`          |                    —                   |     indicator failures     |                    —                   |      n/a (itself)      |
| `titan-lock`            |                    —                   |   failure-tracker windows  |                    —                   |            —           |
| `titan-metrics`         |                    —                   |       quiet by design      |        process/system collectors       |            —           |
| `titan-notifications`   | `notifications.*` *(via titan-events)* |         send + DLQ         |                    —                   |           yes          |
| `titan-pm`              |             many *(below)*             |   spawn / crash / restart  |              process stats             | yes (via titan-health) |
| `titan-ratelimit`       |                    —                   |       reject reasons       |    allow/deny *(pull: `getStats()`)*   |            —           |
| `titan-redis`           |                    —                   |     reconnect / errors     |                    —                   | yes (via titan-health) |
| `titan-scheduler`       |                    —                   |     job start / finish     | runs/failures *(pull: `getMetrics()`)* |            —           |
| `titan-telemetry-relay` |                    —                   |         WAL events         |             pull: `stats()`            |            —           |

<span class="module-badge">built-in · @omnitron-dev/titan</span>

| Module   |  Change notifications |     Notable logs    | Pushes metrics |
| -------- | :-------------------: | :-----------------: | :------------: |
| `config` | `onChange()` callback | reload success/fail |        —       |
| `logger` |           —           | (the logger itself) |        —       |

## Per-module reference

### `titan-pm` — the most chatty module

<span class="module-badge">official · @omnitron-dev/titan-pm</span>

Process manager is the loudest module by design — every supervised
process / pool / worker raises events you can subscribe to.

| Event                     | Args                       | Meaning                                            |
| ------------------------- | -------------------------- | -------------------------------------------------- |
| `process:spawn`           | `processInfo`              | Child process started                              |
| `process:ready`           | `processInfo`              | Child responded to ready handshake                 |
| `process:stop`            | `processInfo`              | Child exited cleanly                               |
| `process:crash`           | `processInfo, error`       | Child died unexpectedly                            |
| `child:started`           | `name`                     | Supervised child entry started                     |
| `child:stopped`           | `name`                     | Supervised child stopped                           |
| `child:start-failed`      | `name, error`              | Initial start raised before ready                  |
| `child:crash`             | `name, error`              | Supervised child crashed mid-run                   |
| `child:restart`           | `name, count`              | Child restarted; `count` = total restarts          |
| `pool:initialized`        | `{ size, class }`          | Worker pool warmed up                              |
| `pool:scaled`             | `{ from, to, class }`      | Pool grew / shrank                                 |
| `pool:drained`            | `{ class }`                | Pool finished draining queued work                 |
| `pool:destroyed`          | `{ class }`                | Pool released                                      |
| `pool:memory`             | `{ workerId, rssMB }`      | Memory limit exceeded                              |
| `worker:spawned`          | `{ workerId, class }`      | Pool worker forked                                 |
| `worker:shutdown`         | `{ workerId, class }`      | Pool worker reaped                                 |
| `worker:unhealthy`        | `{ workerId }`             | Worker missed a ping                               |
| `worker:unresponsive`     | `{ workerId }`             | Worker still silent after retries                  |
| `worker:replaced`         | `{ workerId }`             | Unresponsive worker replaced                       |
| `request:queued`          | `{ poolClass, queueSize }` | Pool saturated; work queued                        |
| `circuitbreaker:open`     | —                          | Breaker tripped                                    |
| `circuitbreaker:halfopen` | —                          | Probe window started                               |
| `circuitbreaker:close`    | —                          | Breaker reset                                      |
| `escalate`                | `name, error`              | Restart limit exceeded; failure escalated upstream |
| `shutdown`                | —                          | Supervisor shutdown                                |
| `exit`                    | `info`                     | Process exited                                     |
| `health:change`           | `processId, IHealthStatus` | Per-process health flipped                         |
| `health:critical`         | `processId, IHealthStatus` | Per-process health became critical                 |

Subscribe with the standard event emitter API:

```typescript
pmService.on('process:crash', (info, error) => {
  logger.error({ pid: info.id, error }, 'child crashed');
});
```

### `titan-notifications`

<span class="module-badge">official · @omnitron-dev/titan-notifications</span>

Lifecycle events are emitted through `titan-events` under the
`NOTIFICATIONS_EVENTS` topic names (only when an `EventsService` is
available). Selected topics:

| Topic constant             | Event name                               | Meaning                     |
| -------------------------- | ---------------------------------------- | --------------------------- |
| `NOTIFICATION_SENT`        | `notifications.notification.sent`        | Send completed              |
| `NOTIFICATION_FAILED`      | `notifications.notification.failed`      | Send failed                 |
| `CHANNEL_DELIVERY_SUCCESS` | `notifications.channel.delivery.success` | Per-channel delivery ok     |
| `CHANNEL_DELIVERY_FAILED`  | `notifications.channel.delivery.failed`  | Per-channel delivery failed |
| `DLQ_MESSAGE_ADDED`        | `notifications.dlq.message.added`        | Message routed to DLQ       |

Dead-lettered messages are also inspectable on the service via
`getDLQStats()` / `getDLQMessages(options?)` and requeued with
`requeueFromDLQ(count?)`.

Health: the module registers `NotificationsHealthIndicator`
(`NOTIFICATIONS_HEALTH`) — automatically picked up by
`titan-health` if both modules are loaded.

### `titan-discovery`

<span class="module-badge">official · @omnitron-dev/titan-discovery</span>

| Event             | Payload                        | Meaning                        |
| ----------------- | ------------------------------ | ------------------------------ |
| `discovery:event` | `{ type, node, service?, ts }` | Node added / removed / updated |

`type` is one of `node.added`, `node.removed`, `node.updated`.

Notable logs (all under the service's own logger namespace):

| Level   | Pattern                                                 |
| ------- | ------------------------------------------------------- |
| `info`  | `Node 'X' registered`, `DiscoveryService started`       |
| `info`  | `Initiating graceful shutdown for node 'X'`             |
| `warn`  | `Heartbeat attempt failed`                              |
| `error` | `All N heartbeat attempts failed`                       |
| `error` | `Failed to publish discovery event`                     |
| `debug` | `Received discovery event`, `Cleaned up inactive nodes` |

### `titan-events`

<span class="module-badge">official · @omnitron-dev/titan-events</span>

`titan-events` *is* the event bus; it doesn't emit framework-level
events of its own. Per-emitter introspection is available via
`EventHistoryService` if `enableHistory: true`.

### `titan-scheduler`

<span class="module-badge">official · @omnitron-dev/titan-scheduler</span>

Doesn't expose a hot event stream — `IJobListener` is the
extension point. Implement one to observe job execution:

```typescript
class MyJobListener implements IJobListener {
  onJobStart?(job, context)         { /* ... */ }
  onJobComplete?(job, result)       { /* ... */ }
  onJobError?(job, error, context)  { /* ... */ }
  onJobRetry?(job, attempt, error)  { /* ... */ }
  onJobCancelled?(job, reason?)     { /* ... */ }
}
```

All hooks are optional. Register listeners via the
`listeners` module option or the `SCHEDULER_LISTENERS_TOKEN`.

### `titan-cache`

<span class="module-badge">official · @omnitron-dev/titan-cache</span>

Cache hit/miss/eviction metrics are exposed through the
`@Cached` decorator metadata and the `CacheService.getStats()` API.

```typescript
const stats = cache.getStats();
// { hits, misses, evictions, size, hitRate, ... }
```

No native event stream — wrap calls if you need per-key
observability.

### `titan-lock`

<span class="module-badge">official · @omnitron-dev/titan-lock</span>

Uses the framework's `FailureTracker` primitive: instead of
logging every failure, it collapses repeated failures of the same
operation into windowed "X failing" warnings. This keeps logs
quiet when Redis briefly hiccups.

| Level   | Pattern                               |
| ------- | ------------------------------------- |
| `warn`  | `[DistributedLock] X started failing` |
| `debug` | `[DistributedLock] X still failing`   |
| `info`  | `[DistributedLock] X recovered`       |

### `titan-database`

<span class="module-badge">official · @omnitron-dev/titan-database</span>

| Level   | Pattern                                    |
| ------- | ------------------------------------------ |
| `warn`  | Slow query (over the configured threshold) |
| `warn`  | Transient error retried (via `withRetry`)  |
| `error` | Pool exhaustion / connection lost          |
| `info`  | Migration applied                          |

The module exposes `DatabaseHealthIndicator` for k8s probes.

### `titan-redis`

<span class="module-badge">official · @omnitron-dev/titan-redis</span>

Inherits ioredis's event model (`connect`, `ready`, `error`,
`close`, `reconnecting`, `end`). The manager logs reconnect
attempts and surfaces a `RedisHealthIndicator` via
`titan-health`.

### `titan-auth`

<span class="module-badge">official · @omnitron-dev/titan-auth</span>

Quiet by design — token-verification details should not be
logged at info level (PII/tokens). Failures log at `warn` with
`error.code` populated; successful verifications are silent.

### `titan-ratelimit`

<span class="module-badge">official · @omnitron-dev/titan-ratelimit</span>

Statistics are pull-based:

```typescript
const stats = rate.getStats();
// { totalChecks, totalAllowed, totalDenied, activeKeys, byTier? }
```

Denied requests are not logged automatically (volume would be
high) — count them via `metrics.recordTyped('counter', ...)` from
the caller if you need to.

### `titan-metrics`

<span class="module-badge">official · @omnitron-dev/titan-metrics</span>

`MetricsService.getStorageStatus()` reports durability, queue samples/bytes,
completed writes, retry failures, known losses and retention/callback errors.
Background errors also reach `onError`; explicit write/flush failures reject.
Omnitron records rate-limited structured failures and exposes the same reading
through daemon health, CLI and Console. Normal successful flushes stay quiet.

The process collector (`collection: { process: true }`) records, for
the daemon's own process: `heap_used_bytes`, `heap_total_bytes`,
`rss_bytes`, `external_bytes`, `cpu_percent`, `uptime_seconds`. The
child/orchestrator collector (`system: true`) additionally records
`memory_bytes`, `rpc_requests_total`, `rpc_errors_total`, and
`app_status` per supervised app. Event-loop lag is not collected by
the built-in collector.

### `titan-telemetry-relay`

<span class="module-badge">official · @omnitron-dev/titan-telemetry-relay</span>

Exposes a pull-based `stats()` snapshot —
`{ totalEmitted, totalSent, totalFailed, totalReceived,
transportConnected, buffer, wal }` — where `buffer`
(`{ size, totalPushed, totalDropped, totalFlushed }`) and `wal`
(`{ segments, totalSize, totalWritten, currentSegment }`) carry the
queue-depth and throughput signals. Surface these through your metrics
module at the leader node; the relay does not push them itself.

### `titan-health`

<span class="module-badge">official · @omnitron-dev/titan-health</span>

| Level   | Pattern                       |
| ------- | ----------------------------- |
| `error` | Indicator threw during check  |
| `warn`  | Indicator returned `degraded` |
| `debug` | Probe cache hit / refresh     |

Health results carry per-check timing in their own result objects;
the module does not push check latency into `titan-metrics`.

## Built-in modules

### `config`

<span class="module-badge">built-in · @omnitron-dev/titan/module/config</span>

Change notifications are delivered through a callback subscription,
not a named event-bus topic:

```typescript
const unsubscribe = config.onChange((event: IConfigChangeEvent) => {
  // event: { path, oldValue, newValue, source, timestamp }
});
```

Notable logs:

- `info` — `Configuration reloaded due to file change` (after a watched source changes)
- `error` — `Failed to reload configuration`

### `logger`

<span class="module-badge">built-in · @omnitron-dev/titan/module/logger</span>

The logger doesn't log about itself except at fatal failures
(e.g., transport open failed at boot). Use `createNullLogger()` in
tests when you want absolute silence.

## Cross-cutting recommendations

### Subscribe instead of poll

The pm/discovery event streams are hot — subscribing is cheaper
and more responsive than polling state. For per-process autoscaling
or alerting, hook the events:

```typescript
pm.on('pool:memory',      handleMemorySpike);
pm.on('worker:unresponsive', handleWorkerHang);
pm.on('circuitbreaker:open', notifyOncall);
```

### Tie metrics to events

For dashboards, route key events through `titan-metrics`:

```typescript
pm.on('process:crash', (info, error) => {
  metrics.recordTyped('counter', 'pm.process.crash.total',
    { class: info.processName }, 1);
});
```

### Alert thresholds — common starting points

| Signal                        | Suggested alert              |
| ----------------------------- | ---------------------------- |
| `rss_bytes` growth rate       | > +50 MB/min sustained       |
| `cpu_percent`                 | sustained near 100 for 2 min |
| `pool:scaled` to/from limit   | repeated within 5 min        |
| `worker:replaced` count       | > 3 in 5 min for same pool   |
| `circuitbreaker:open`         | any open lasting > 30 s      |
| `app_status` flips to 0       | any supervised app offline   |
| Health `degraded`/`unhealthy` | sustained > 1 probe window   |

### Don't log per-call by default

Several modules (`titan-ratelimit`, `titan-cache`, `titan-auth`)
intentionally stay quiet at info level. Per-call logging at request
rate is a recipe for disk/SIEM saturation. Use counters and
sampled debug logging instead.

## See also

- [Best Practices / Observability](/docs/titan/best-practices/observability) — logs, metrics, traces together
- [`titan-metrics`](/docs/titan/modules/metrics) — recording API
- [`titan-health`](/docs/titan/modules/health) — indicator registry
- [`titan-telemetry-relay`](/docs/titan/modules/telemetry-relay) — store-and-forward shipping
- [Lifecycle reference](/docs/titan/modules/lifecycle-reference) — which hook each module uses