Observability matrix

Operating a Titan app means knowing what to grep for and what to alert on. This page lists — per module — the events emitted, notable log keys, and metric/health signals you can rely on.

All claims here are verified against module source. If a row is empty, the module is deliberately quiet at that layer.

At-a-glance

Official@omnitron-dev/titan-*

Maintained by the Omnitron team. Independent npm package.

The Metrics surface column distinguishes metrics actively pushed into titan-metrics from stats you read on demand (pull — a getStats() / getMetrics() / stats() accessor).

Module	Emits events	Notable logs	Metrics surface	Health indicator
titan-auth	—	JWT verify failures	—	—
titan-cache	—	L2 fallback / errors	hit/miss/evict (pull: getStats())	—
titan-database	—	slow query, retries	—	yes (via titan-health)
titan-discovery	discovery:event	register, heartbeat, dereg	—	—
titan-events	proxy	dispatch errors	—	—
titan-health	—	indicator failures	—	n/a (itself)
titan-lock	—	failure-tracker windows	—	—
titan-metrics	—	quiet by design	process/system collectors	—
titan-notifications	notifications.* (via titan-events)	send + DLQ	—	yes
titan-pm	many (below)	spawn / crash / restart	process stats	yes (via titan-health)
titan-ratelimit	—	reject reasons	allow/deny (pull: getStats())	—
titan-redis	—	reconnect / errors	—	yes (via titan-health)
titan-scheduler	—	job start / finish	runs/failures (pull: getMetrics())	—
titan-telemetry-relay	—	WAL events	pull: stats()	—

Built-in@omnitron-dev/titan

Ships inside @omnitron-dev/titan. No additional install required.

Module	Change notifications	Notable logs	Pushes metrics
config	onChange() callback	reload success/fail	—
logger	—	(the logger itself)	—

Per-module reference

titan-pm — the most chatty module

Official@omnitron-dev/titan-pm

Maintained by the Omnitron team. Independent npm package.

Process manager is the loudest module by design — every supervised process / pool / worker raises events you can subscribe to.

Event	Args	Meaning
process:spawn	processInfo	Child process started
process:ready	processInfo	Child responded to ready handshake
process:stop	processInfo	Child exited cleanly
process:crash	processInfo, error	Child died unexpectedly
child:started	name	Supervised child entry started
child:stopped	name	Supervised child stopped
child:start-failed	name, error	Initial start raised before ready
child:crash	name, error	Supervised child crashed mid-run
child:restart	name, count	Child restarted; count = total restarts
pool:initialized	{ size, class }	Worker pool warmed up
pool:scaled	{ from, to, class }	Pool grew / shrank
pool:drained	{ class }	Pool finished draining queued work
pool:destroyed	{ class }	Pool released
pool:memory	{ workerId, rssMB }	Memory limit exceeded
worker:spawned	{ workerId, class }	Pool worker forked
worker:shutdown	{ workerId, class }	Pool worker reaped
worker:unhealthy	{ workerId }	Worker missed a ping
worker:unresponsive	{ workerId }	Worker still silent after retries
worker:replaced	{ workerId }	Unresponsive worker replaced
request:queued	{ poolClass, queueSize }	Pool saturated; work queued
circuitbreaker:open	—	Breaker tripped
circuitbreaker:halfopen	—	Probe window started
circuitbreaker:close	—	Breaker reset
escalate	name, error	Restart limit exceeded; failure escalated upstream
shutdown	—	Supervisor shutdown
exit	info	Process exited
health:change	processId, IHealthStatus	Per-process health flipped
health:critical	processId, IHealthStatus	Per-process health became critical

Subscribe with the standard event emitter API:

pmService.on('process:crash', (info, error) => {
  logger.error({ pid: info.id, error }, 'child crashed');
});

titan-notifications

Official@omnitron-dev/titan-notifications

Maintained by the Omnitron team. Independent npm package.

Lifecycle events are emitted through titan-events under the NOTIFICATIONS_EVENTS topic names (only when an EventsService is available). Selected topics:

Topic constant	Event name	Meaning
NOTIFICATION_SENT	notifications.notification.sent	Send completed
NOTIFICATION_FAILED	notifications.notification.failed	Send failed
CHANNEL_DELIVERY_SUCCESS	notifications.channel.delivery.success	Per-channel delivery ok
CHANNEL_DELIVERY_FAILED	notifications.channel.delivery.failed	Per-channel delivery failed
DLQ_MESSAGE_ADDED	notifications.dlq.message.added	Message routed to DLQ

Dead-lettered messages are also inspectable on the service via getDLQStats() / getDLQMessages(options?) and requeued with requeueFromDLQ(count?).

Health: the module registers NotificationsHealthIndicator (NOTIFICATIONS_HEALTH) — automatically picked up by titan-health if both modules are loaded.

titan-discovery

Official@omnitron-dev/titan-discovery

Maintained by the Omnitron team. Independent npm package.

Event	Payload	Meaning
discovery:event	{ type, node, service?, ts }	Node added / removed / updated

type is one of node.added, node.removed, node.updated.

Notable logs (all under the service's own logger namespace):

Level	Pattern
info	Node 'X' registered, DiscoveryService started
info	Initiating graceful shutdown for node 'X'
warn	Heartbeat attempt failed
error	All N heartbeat attempts failed
error	Failed to publish discovery event
debug	Received discovery event, Cleaned up inactive nodes

titan-events

Official@omnitron-dev/titan-events

Maintained by the Omnitron team. Independent npm package.

titan-events is the event bus; it doesn't emit framework-level events of its own. Per-emitter introspection is available via EventHistoryService if enableHistory: true.

titan-scheduler

Official@omnitron-dev/titan-scheduler

Maintained by the Omnitron team. Independent npm package.

Doesn't expose a hot event stream — IJobListener is the extension point. Implement one to observe job execution:

class MyJobListener implements IJobListener {
  onJobStart?(job, context)         { /* ... */ }
  onJobComplete?(job, result)       { /* ... */ }
  onJobError?(job, error, context)  { /* ... */ }
  onJobRetry?(job, attempt, error)  { /* ... */ }
  onJobCancelled?(job, reason?)     { /* ... */ }
}

All hooks are optional. Register listeners via the listeners module option or the SCHEDULER_LISTENERS_TOKEN.

titan-cache

Official@omnitron-dev/titan-cache

Maintained by the Omnitron team. Independent npm package.

Cache hit/miss/eviction metrics are exposed through the @Cached decorator metadata and the CacheService.getStats() API.

const stats = cache.getStats();
// { hits, misses, evictions, size, hitRate, ... }

No native event stream — wrap calls if you need per-key observability.

titan-lock

Official@omnitron-dev/titan-lock

Maintained by the Omnitron team. Independent npm package.

Uses the framework's FailureTracker primitive: instead of logging every failure, it collapses repeated failures of the same operation into windowed "X failing" warnings. This keeps logs quiet when Redis briefly hiccups.

Level	Pattern
warn	[DistributedLock] X started failing
debug	[DistributedLock] X still failing
info	[DistributedLock] X recovered

titan-database

Official@omnitron-dev/titan-database

Maintained by the Omnitron team. Independent npm package.

Level	Pattern
warn	Slow query (over the configured threshold)
warn	Transient error retried (via withRetry)
error	Pool exhaustion / connection lost
info	Migration applied

The module exposes DatabaseHealthIndicator for k8s probes.

titan-redis

Official@omnitron-dev/titan-redis

Maintained by the Omnitron team. Independent npm package.

Inherits ioredis's event model (connect, ready, error, close, reconnecting, end). The manager logs reconnect attempts and surfaces a RedisHealthIndicator via titan-health.

titan-auth

Official@omnitron-dev/titan-auth

Maintained by the Omnitron team. Independent npm package.

Quiet by design — token-verification details should not be logged at info level (PII/tokens). Failures log at warn with error.code populated; successful verifications are silent.

titan-ratelimit

Official@omnitron-dev/titan-ratelimit

Maintained by the Omnitron team. Independent npm package.

Statistics are pull-based:

const stats = rate.getStats();
// { totalChecks, totalAllowed, totalDenied, activeKeys, byTier? }

Denied requests are not logged automatically (volume would be high) — count them via metrics.recordTyped('counter', ...) from the caller if you need to.

titan-metrics

Official@omnitron-dev/titan-metrics

Maintained by the Omnitron team. Independent npm package.

The MetricsService itself is quiet — it neither logs flush activity at info level nor records meta-metrics about its own flushing. (The MetricsCollector keeps an internal totalDropped counter for buffer- pressure diagnostics, surfaced by the omnitron metrics-bridge rather than as a self-instrumented metric.)

The process collector (collection: { process: true }) records, for the daemon's own process: heap_used_bytes, heap_total_bytes, rss_bytes, external_bytes, cpu_percent, uptime_seconds. The child/orchestrator collector (system: true) additionally records memory_bytes, rpc_requests_total, rpc_errors_total, and app_status per supervised app. Event-loop lag is not collected by the built-in collector.

titan-telemetry-relay

Official@omnitron-dev/titan-telemetry-relay

Maintained by the Omnitron team. Independent npm package.

Exposes a pull-based stats() snapshot — { totalEmitted, totalSent, totalFailed, totalReceived, transportConnected, buffer, wal } — where buffer ({ size, totalPushed, totalDropped, totalFlushed }) and wal ({ segments, totalSize, totalWritten, currentSegment }) carry the queue-depth and throughput signals. Surface these through your metrics module at the leader node; the relay does not push them itself.

titan-health

Official@omnitron-dev/titan-health

Maintained by the Omnitron team. Independent npm package.

Level	Pattern
error	Indicator threw during check
warn	Indicator returned degraded
debug	Probe cache hit / refresh

Health results carry per-check timing in their own result objects; the module does not push check latency into titan-metrics.

Built-in modules

config

Built-in@omnitron-dev/titan/module/config

Ships inside @omnitron-dev/titan. No additional install required.

Change notifications are delivered through a callback subscription, not a named event-bus topic:

const unsubscribe = config.onChange((event: IConfigChangeEvent) => {
  // event: { path, oldValue, newValue, source, timestamp }
});

Notable logs:

• info — Configuration reloaded due to file change (after a watched source changes)
• error — Failed to reload configuration

logger

Built-in@omnitron-dev/titan/module/logger

Ships inside @omnitron-dev/titan. No additional install required.

The logger doesn't log about itself except at fatal failures (e.g., transport open failed at boot). Use createNullLogger() in tests when you want absolute silence.

Cross-cutting recommendations

Subscribe instead of poll

The pm/discovery event streams are hot — subscribing is cheaper and more responsive than polling state. For per-process autoscaling or alerting, hook the events:

pm.on('pool:memory',      handleMemorySpike);
pm.on('worker:unresponsive', handleWorkerHang);
pm.on('circuitbreaker:open', notifyOncall);

Tie metrics to events

For dashboards, route key events through titan-metrics:

pm.on('process:crash', (info, error) => {
  metrics.recordTyped('counter', 'pm.process.crash.total',
    { class: info.processName }, 1);
});

Alert thresholds — common starting points

Signal	Suggested alert
rss_bytes growth rate	> +50 MB/min sustained
cpu_percent	sustained near 100 for 2 min
pool:scaled to/from limit	repeated within 5 min
worker:replaced count	> 3 in 5 min for same pool
circuitbreaker:open	any open lasting > 30 s
app_status flips to 0	any supervised app offline
Health degraded/unhealthy	sustained > 1 probe window

Don't log per-call by default

Several modules (titan-ratelimit, titan-cache, titan-auth) intentionally stay quiet at info level. Per-call logging at request rate is a recipe for disk/SIEM saturation. Use counters and sampled debug logging instead.

See also

• Best Practices / Observability — logs, metrics, traces together
• titan-metrics — recording API
• titan-health — indicator registry
• titan-telemetry-relay — store-and-forward shipping
• Lifecycle reference — which hook each module uses