---
id: "omnitron/daemon"
title: "Daemon"
description: "The long-running supervisor — its lifecycle, state, and RPC surface."
---
# Daemon
The backend serves the CLI, Console and mesh. In the default coupled mode it
also owns application processes. Opt-in resident execution keeps those processes
in a separate owner so the backend can be replaced without restarting them.
See [backend replacement](/docs/omnitron/backend-handoff) for the initial maintenance
window, supported environments, artifact validation and recovery.
The lifecycle below describes coupled mode. In resident mode, backend retirement
preserves the owner; an explicit shutdown still stops applications.
Verified against `src/daemon/`:
**daemon/**
- `daemon.ts` — main daemon class
- `daemon.rpc-service.ts` — Netron-facing service
- `daemon.module.ts` — Titan module wiring
- `daemon-client.ts` — Typed RPC client (used by CLI)
- `daemon-entry.ts` — Standalone entrypoint
- `daemon-scheduler.ts` — Periodic task scheduler
- `pid-manager.ts` — PID file lock + liveness sweep
- `daemon-state-store.service.ts` — Transactional SQLite state storage
- `state-store.ts` — Application snapshots and execution intent
- `execution-owner.ts` — Independent application owner
- `backend-supervisor.ts` — Backend guardian
- `backend-upgrade.ts` — Validated backend handoff
## Lifecycle
```mermaid
sequenceDiagram
participant Init as omnitron up
participant PM as PID manager
participant SS as State store
participant N as Netron bus
participant SVC as RPC services
participant Orch as Orchestrator
participant Apps
Init->>PM: acquire ~/.omnitron/daemon.pid
alt lock held by live process
PM-->>Init: error — already running
Init->>Init: exit 1
end
Init->>SS: open durable state and project registry
Init->>Init: register cleanup before signal handlers
Init->>N: bind Unix socket + [TCP / HTTP]
Init->>SVC: register all services
Init->>Orch: read registered projects and enabled stacks
loop per enabled stack
Orch->>Apps: resume selected release or development configuration
end
Init-->>Init: ready
Note over Init: serve indefinitely
Note over Init: SIGTERM
Init->>Init: cancel remaining startup steps, join in-flight work
Init->>Orch: stopAll (coupled mode)
Init->>SS: persist final state
Init->>PM: release lock
Init->>Init: exit 0
```
## Startup and node diagnostics
A master prepares its control-plane database before Titan eagerly initializes
providers. Foreground CLI, detached CLI and native supervision share this path.
A configured `OMNITRON_DATABASE_URL` uses the supplied database without creating a
local PostgreSQL container. `--no-infra` skips Docker provisioning, but still
initializes the master schema through the shared Kysera migration runner. A
slave keeps its SQLite lifecycle and does not connect to PostgreSQL.
During backend handoff, this stage only verifies schema compatibility; it does
not provision or migrate. Database or migration failures stop startup before
claiming the resident owner or publishing listeners, preserving the concrete
cause in the boot diagnostic. Prepare schema changes before replacing a backend.
`omnitron up` launches a detached daemon and releases both the child-process
and IPC event-loop references. The CLI exits after its readiness checks while
the daemon keeps running. Startup stderr is appended to the boot log reported
by the command, including failures before RPC becomes available.
The Nodes API exposes `getNodeDaemonStatus`, `getNodeDaemonHealth` and
`getNodeDaemonMetrics` for every registered node. Local requests use the same
live daemon readers as the CLI; remote requests use the authenticated mesh.
`reachable: false` includes the failure reason and no answer. It is not a
health verdict, and a remote failure never falls back to the local node's data.
### Shutdown during startup
The daemon registers cleanup before Titan installs its signal handlers. If
`SIGTERM` or `SIGINT` arrives during startup, it stops beginning new startup
phases and enabled stacks. Titan's first shutdown task waits for the current
startup operation and any in-flight health-monitor spawn or restart before
cleanup closes their resources. The existing Titan shutdown deadline still
applies: an operation that cannot finish is reported as a shutdown failure,
not successful cleanup.
An interrupted startup does not announce `daemon.started` or a replacement
backend's readiness. The startup diagnostic remains an error if an actual
operation failed while shutdown was pending. A resident owner's applications
continue running when only its backend terminates; explicit `omnitron down`
still requests application shutdown. Event-loop monitoring and the reload
signal handler are disposed with the backend.
## PID manager — `pid-manager.ts`
Owns the daemon lock file `~/.omnitron/daemon.pid`. Responsibilities:
- **Acquire** on boot: write `process.pid` to the file with an
exclusive open. If the file exists and the PID inside is still
alive, abort.
- **Signature verification.** The file carries a process-start
signature (PID + start time fingerprint). A stale PID with the
same numeric value as a recycled new process is detected and
treated as dead.
- **Liveness sweep.** A periodic check ensures the lock owner is
still alive; reclaims if the holder has crashed without
releasing.
- **Release** on `SIGTERM` / `SIGINT` / normal exit.
When `omnitron up` reports "daemon already running", this is the
check that fired.
## State store — `state-store.ts`
Persists daemon intent to a **SQLite-backed** key/value row
(`DaemonStateStore`, `state_kv` key `orchestrator:persisted-state`)
co-located with the daemon's other state surfaces (PID lock,
project registry, node registry, backup index). SQLite WAL-mode
transactions remove the torn-write risk that the legacy
`fsync` + atomic-rename JSON recipe guarded against. Writes are
serialised through an internal promise chain so back-to-back
`save()` calls never interleave; callers use a synchronous API
and the actual write is awaited in the background.
Each persisted app entry records `name`, `pid`, `status`,
`mode` (`classic` | `bootstrap`), `startedAt`, `restarts`, and
`port`:
| Mode | When | Layout |
| ----------- | ---------------------------------------------------- | ----------------------------------------------- |
| `classic` | App was launched via classic launcher (single fork) | One PID + status per app |
| `bootstrap` | App was launched via the PM supervisor (per-process) | Per-process PID + status, parent app aggregates |
**Legacy migration.** On the first boot of the SQLite-backed
code, if an old `~/.omnitron/state.json` exists it is read once,
migrated into the SQLite row, and unlinked. After that the JSON
file never reappears.
State is persisted on every status transition (start, stop,
crash, restart). A crash of the daemon itself leaves the row in a
consistent state — on next boot, the daemon rehydrates and
relaunches whatever was running.
### File permissions and SQLite locks
State, recovery certificates and the log journal use owner-only directories
(`0700`) and files (`0600`) on POSIX systems. Permission checks on an open
SQLite database inspect metadata without opening or closing another descriptor.
Closing any descriptor for the same inode can cancel the process's SQLite
locks, including when the descriptor was opened only to tighten permissions
([SQLite locking guidance](https://www.sqlite.org/howtocorrupt.html#_posix_advisory_locks_canceled_by_a_separate_thread_doing_close_)).
If existing permissions need repair, Omnitron performs the no-follow,
owner-checked operation in a separate short-lived Node process and verifies the
result. Already-private databases need no subprocess. Symbolic links and
additional hard links are rejected. Equivalent Windows protection depends on
OS ACLs; POSIX modes do not establish Windows access control.
Use SQLite's online backup API for live databases. Do not copy, replace or
remove a live database's WAL/SHM files. A recovery status held in memory is not
sufficient evidence for activation: schema transitions require a current,
durably readable certificate identifying the running backend and owner.
`pnpm --filter @omnitron-dev/omnitron test:private-storage` checks cross-process
writer exclusion, repeated readers, committed visibility and file permissions
against the compiled runtime; the CI build runs it before accepting changes.
## Daemon scheduler — `daemon-scheduler.ts`
A small set of periodic jobs registered against `titan-scheduler`
and tied to the daemon's lifecycle. `registerDaemonJobs()` wires:
| Job | Cadence | Notes |
| -------------------- | ------------------------------------------------ | -------------------------------------------- |
| `metrics-collection` | `monitoring.metrics.interval` (default 5 s) | Polls app state, records via `titan-metrics` |
| `log-rotation` | 60 s | `LogManager.checkRotation` |
| `session-cleanup` | 5 min | Master only (requires the auth service / PG) |
| `alert-evaluation` | `monitoring.healthCheck.interval` (default 15 s) | Master only (requires PG) |
| `fleet-heartbeat` | `monitoring.healthCheck.interval` (default 15 s) | Master only (requires PG) |
The master-only jobs are skipped on a standalone single-node
setup where the auth / alert / fleet services aren't wired.
## Daemon configuration
The daemon's own config (not project / ecosystem config) lives in
the `DEFAULT_DAEMON_CONFIG` object — most operators never touch
it. Values:
| Field | Default |
| -------------------------------- | ----------------------------------- |
| `socketPath` | `~/.omnitron/daemon.sock` |
| `port` (TCP) | `9700` |
| `host` (TCP) | `127.0.0.1` |
| `httpPort` | `9800` |
| `pidFile` | `~/.omnitron/daemon.pid` |
| `stateFile` | `~/.omnitron/state.json` |
| `logDir` | `~/.omnitron/logs` |
| `role` | `master` |
| `cluster.enabled` | `false` |
| `cluster.discovery` | `postgres` |
| `cluster.leaseTtlMs` | `15 000 ms` |
| `cluster.heartbeatInterval` | `2 000 ms` |
| `secrets.provider` | `file` |
| `secrets.path` | `~/.omnitron/secrets.enc` |
| `healthMonitor.intervalMs` | `60 000` (1/min) |
| `healthMonitor.concurrency` | `20` |
| `healthMonitor.offlineTimeoutMs` | `90 000` |
| `healthMonitor.retentionDays` | `90` (uptime history) |
| `healthMonitor.uptimeIntervalMs` | `86 400 000` (24 h per bar segment) |
Override via CLI flags on `omnitron up` or via the daemon
configuration file (advanced — not part of the project ecosystem
config).
### Console request limits
The daemon shares one Netron limiter across HTTP and WebSocket. Configure `requestRateLimit` in the daemon settings. Its defaults are 3,000 requests per client and 6,000 total per 60-second window. Switching transport or opening another tab consumes the same allowance. The legacy `httpRateLimit` setting covers both listeners. If both settings exist, `requestRateLimit` takes precedence.
The Console gateway overwrites `X-Forwarded-For` and `X-Real-IP` on HTTP and WebSocket. With the default `trustProxy: false`, clients behind the same proxy share its socket-address allowance. Set `trustProxy: true` only when access to the upstream listeners is restricted to that trusted proxy. The local administrative Unix socket has its own OS access boundary. [Netron request budgets](/docs/titan/netron/transports#share-a-request-budget-across-http-and-websocket)
## RPC surface — `OmnitronDaemon` service
The daemon registers `@Service({ name: 'OmnitronDaemon' })` on
the Netron bus. Verified from
[`src/daemon/daemon.rpc-service.ts`](https://github.com/omnitron-dev/omni/tree/main/apps/omnitron/src/daemon/daemon.rpc-service.ts).
The daemon checks these roles for each request.
| Role | Members | Methods |
| ----------------------- | ----------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- |
| Control-plane read | `viewer`, `operator`, `admin`, authenticated `service_role` | `list`, `getApp`, `status`, `getMetrics`, `getHealth`, `getLogs`, `inspect`, `inspectInstalledArtifacts`, `getDependencyGraph`, `getWatchStatus` |
| Control-plane operation | `operator`, `admin`, authenticated `service_role` | `getEnv` (redacted values), `attestApplicationOwner` |
| `operator` | `operator`, `admin` | `startApp`, `startAll`, `stopApp`, `stopAll`, `restartApp`, `restartAfterChange`, `restartAll`, `reloadApp`, `scale`, `enableWatch`, `disableWatch` |
| `admin` | `admin` only | `shutdown`, `reloadConfig`, `setMetricsEnabled`, `exec`, `revealEnv` |
| anonymous | anyone with socket access | `ping` (allowAnonymous) |
### Methods, by intent
#### App lifecycle (`operator`)
| Method | Returns |
| -------------------------------------------------------------- | ---------------------- |
| `startApp({ name })` | `ProcessInfoDto` |
| `startAll()` | `ProcessInfoDto[]` |
| `stopApp({ name, force?, timeout? })` | `{ success: boolean }` |
| `stopAll({ force? })` | `{ count: number }` |
| `restartApp({ name })` | `ProcessInfoDto` |
| `restartAfterChange({ name, generation, definitionRevision })` | `SourceRefreshReceipt` |
| `restartAll()` | `ProcessInfoDto[]` |
| `reloadApp({ name })` | `ProcessInfoDto` |
| `scale({ name, process, instances })` | `ProcessInfoDto` |
`scale` requires an explicit declared pool process and a positive safe integer.
It returns the existing `ProcessInfoDto`; read `processes` for the selected pool's
observed `instances` and unchanged `declaredInstances`. Resizing lasts until
restart and leaves automatic scaling enabled. The owner enforces the pool's
configured bounds and must advertise `processPoolResize@1`; an older owner
refuses before mutation. This optional capability does not prevent a backend
update when the owner remains unchanged.
`restartAfterChange` is the automatic source-refresh operation used by SDK
watch mode. Supply the canonical application name, generation and declaration
revision observed in `getWatchStatus()`. The execution owner must advertise
`sourceRefresh@1`; it refuses stale bindings, disabled watching and pinned release
applications before restarting. The response is the owner's exact
`SourceRefreshReceipt`: `{ process, source }`. `process` confirms a new online
generation; `source` carries that same generation with its `name`, `directory` and
`definitionRevision`. A later inventory read must not be substituted for this
acknowledgement. A successful build alone does not prove that an application
loaded it. Application health probes remain separate from the refresh receipt.
#### Inspection (`viewer`)
| Method | Returns |
| -------------------------------------- | --------------------------------------------------- |
| `list()` | `ProcessInfoDto[]` |
| `getApp({ name })` | `ProcessInfoDto` |
| `status()` | `DaemonStatusDto` |
| `getMetrics({ name? })` | `AggregatedMetricsDto` |
| `getHealth({ name? })` | `AggregatedHealthDto` |
| `getLogs({ name?, lines? })` | `LogEntryDto[]` |
| `inspect({ name })` | `AppDiagnosticsDto` |
| `inspectInstalledArtifacts({ names })` | Installed artifact evidence per application |
| `getDependencyGraph({ name })` | Graph object |
| `getWatchStatus()` | `{ enabled, watching, apps: WatchedApplication[] }` |
#### Owner and environment inspection
`attestApplicationOwner({ owner: { id, pid, startedAt }, epoch })` returns the
clean build stamp of the observed owner. The node checks its active controller
lease. For a legacy owner, it verifies the supervised process and sealed runtime
before checking the lease again. Normal status polling performs no disk scan.
See [application release admission](/docs/omnitron/releases#build-and-inspect).
`getEnv({ name })` returns redacted environment values. `revealEnv({ name })`
requires an administrator and releases the clear values only after the audit database
confirms the write. A node without audit storage can still return redacted values.
#### Watch control (`operator`)
| Method | Returns |
| ------------------------ | ------------------------------------ |
| `enableWatch({ apps? })` | `{ watching: WatchedApplication[] }` |
| `disableWatch()` | `{ success: boolean }` |
#### Admin (`admin`)
| Method | Returns |
| --------------------------------------- | ---------------------- |
| `shutdown({ force? })` | `{ success: boolean }` |
| `reloadConfig()` | `{ success: boolean }` |
| `setMetricsEnabled({ name?, enabled })` | `{ success: boolean }` |
#### Heartbeat (anonymous)
| Method | Returns |
| -------- | -------------------------- |
| `ping()` | `{ uptime, version, pid }` |
#### RPC plumbing (`admin`)
| Method | Returns |
| --------------------------------------- | ---------------------------------------------- |
| `exec({ name, service, method, args })` | `unknown` (whatever the called method returns) |
`exec` is what `omnitron exec api users findById u_42` routes
through. It is **admin-gated** — invoking arbitrary methods on a
managed app is a privileged operation.
## Auth flow
Authentication is intentionally split between transports:
```mermaid
flowchart LR
CLI[CLI command]
Webapp[Webapp]
Remote[Remote daemon]
CLI -- Unix socket
OS-level identity --> AuthBypass[implicit admin]
Webapp -- HTTP + JWT --> JWT
Remote -- TCP + JWT --> JWT
JWT[Verify JWT
read roles]
JWT --> RoleCheck{Role enough
for method?}
RoleCheck -- yes --> Invoke[Run method]
RoleCheck -- no --> Reject[403 / FORBIDDEN]
```
The Unix socket auth-bypass is **deliberate**: file mode `0o600`
means only the daemon's UID can open it. If the OS trusts you to
write to the socket, the daemon trusts the request. TCP and HTTP
always require JWT.
## Daemon client — `daemon-client.ts`
The operator API exposes the typed local client through
`@omnitron-dev/omnitron/operations`. It connects to the daemon owner's Unix
socket and has the same authority as the local CLI. Keep it on the operator
host, not in browser code.
```typescript
import { createDaemonClient } from '@omnitron-dev/omnitron/operations';
const client = createDaemonClient();
try {
await client.ping();
await client.startApp({ name: 'api' });
const metrics = await client.getMetrics({ name: 'api' });
console.log(metrics);
} finally {
await client.disconnect();
}
```
Local and fleet TCP clients share connection lifetime handling. Concurrent
calls share one connection attempt; retained service handles reconnect after
backend replacement. Connect, reachability and teardown have separate
bounds. A lost in-flight reply rejects the call without replaying it. See
[long-lived RPC clients](/docs/omnitron/backend-handoff#long-lived-rpc-clients) before
upgrading an application that retains a client across backend replacements.
## Auto-restart and backoff
Applications inherit the ecosystem's `supervision` settings. Override them for
one app with `apps[].restartPolicy`. By default, up to five automatic restarts
may begin in a rolling 60-second window. The first retry waits 1 s. Later
retries in the same window use exponential backoff up to 30 s.
Set `maxRestarts: 0` or `enabled: false` on the app's restart policy to
disable automatic retries.
If a classic script uses its retries and fails again, the owner marks it
`crashed`, emits `app:escalated` and waits ten minutes before trying again
with a fresh budget. For Titan-supervised children, an exhausted budget
ends automatic recovery of that child. A critical child failure escalates
to its supervisor.
Use `omnitron stop ` during backoff or cool-down to cancel the pending
launch. The app stays stopped until a new start is requested. When ready, run
`omnitron start `. For a stopped stack, use
`omnitron stack start ` to restore its applications together.
## Failure scenarios
| Scenario | What the daemon does |
| ----------------------------------------- | ------------------------------------------------------------- |
| Classic app exits during a requested stop | Mark the app stopped |
| Classic app exits unexpectedly | Record the exit and apply the configured restart policy |
| Child process hangs (no heartbeat) | Health probe fails; restart after grace period |
| Daemon itself crashes | systemd / launchd respawns; persisted SQLite state rehydrates |
| Lock-file PID is alive but daemon hung | `omnitron kill` force-removes the lock |
| Two `omnitron up` race | Second loses the PID lock; exits with error |
| Storage failure persisting state | Logs at error; in-memory state continues; next flush retries |
| Disk full | State persistence + logs fail loudly |
| Cluster split (cluster.enabled) | Both sides may elect a leader; manual `step-down` resolves |
## Operating runbook
| Goal | Command |
| ----------------------------------- | ------------------------------------------------------------------------------------------------- |
| Is the daemon up? | `omnitron ping` |
| What's running right now? | `omnitron status` / `omnitron --json status` |
| Force shutdown | `omnitron down` (graceful) or `omnitron kill` (forceful) |
| Apply changed daemon startup policy | Prepare and review a [backend replacement](/docs/omnitron/backend-handoff), or schedule a restart |
| See the lock holder | `cat ~/.omnitron/daemon.pid` |
| Inspect replacement/recovery state | `omnitron backend status` |
Do not delete `daemon-state.db` to reset a process: it also holds project and node
registrations, execution intent and the backup index. Deleting `secrets.enc`
destroys the file-provider vault. Recover these stores only from a verified backup
with a reviewed data-loss and rollback plan. Use normal application and stack
lifecycle commands for process changes.
## Anti-patterns
- **Editing the daemon state DB by hand.** Persisted state lives
in a SQLite DB (`~/.omnitron/data/daemon-state.db`), not a
hand-editable JSON file. The schema may evolve; poke it directly
and you risk a broken next restart. Prefer using the CLI to
drive state changes.
- **Running two daemons against the same `~/.omnitron/`.** They
fight over the lock. Multiple daemons need different home
directories.
- **Exposing the TCP port without a JWT secret.** Anyone reaching
the port can inspect at minimum, and operate / admin with default
roles. Configure auth before opening port `9700`.
- **Treating `kill` as routine.** Force-kill skips graceful
shutdown — pending logs may be lost, child processes may be
reparented to init. Use `down` first.
- **Removing the lock file while the daemon is running.** Causes
inconsistent state where two daemons can subsequently start.
## See also
- [Architecture](/docs/omnitron/architecture) — where the daemon fits in
the broader picture
- [Orchestrator](/docs/omnitron/orchestrator) — what the daemon hands apps off to
- [Services reference](/docs/omnitron/services-reference) — all 19 RPC services
- [CLI](/docs/omnitron/cli) — every command that talks to this daemon
## Updating a source runtime
In coupled mode the daemon directly supervises application processes. Replacing
it requires a controlled maintenance window. A resident owner can retain running
applications across a backend update. Check the reported execution mode before
choosing the [handoff procedure](/docs/omnitron/backend-handoff).
1. Record the active build, enabled stacks, application states and remote node
process IDs. Save the daemon configuration, account database, project registry,
runtime dependencies and builds needed for rollback. Preserve persistent data
volumes and existing experiment registrations.
2. Build and validate the target revision separately. Use the repository's exact
`packageManager` version and test native dependencies with the Node executable
used by the installed service. Keep the package manager in a permanent path
available to that service.
3. Unload the OS supervisor before replacing a watched checkout. Wait for owned
application processes to stop. Fast-forward clean working copies and install
from the verified lockfiles; preserve other branches and local configuration.
4. Install each source application's adapter and Titan from its frozen lockfile.
They must resolve one physical Titan package. The daemon may use its own
SDK. Check the active owner's capabilities when upgrading an older
installation. Published deployments use their pinned framework artifacts.
5. Build the daemon and Console from the same revision, then restore OS supervision.
Check each enabled stack and its child processes against the saved baseline.
A responding daemon does not mean its applications have finished starting.
6. Reload the Console and verify [access controls](/docs/omnitron/access),
[SSH host identity and sessions](/docs/omnitron/terminal), application health and audit
records. Confirm remote node processes remain unchanged for a local upgrade.
Keep immutable experiment manifests intact when their recorded package trees are
unavailable. Recover the original environment or create a separately registered
experiment; changing old hashes would invalidate its reproducibility.
### Execution handoff: architecture boundary
In coupled mode the control daemon owns Titan PM supervisors, fork IPC
channels, worker threads and application routing. Fork workers terminate on parent disconnect; the parent-death watchdog
and cold-start janitor enforce that ownership. Simply disabling `stopAll` or
reusing PIDs would either lose in-process workers or create unmanaged processes.
In resident mode a separately supervised execution owner uses Titan PM to manage
application processes, threads, restart policy, log capture, metrics and the
child registry across backend replacements. Omnitron
retains project orchestration and RBAC, communicating through authenticated,
local Netron RPC. A private socket and a persistent host identity must be verified
before a controller can attach; PID existence alone is not identity evidence.
Ownership transfer requires a monotonic fencing epoch and one active writer.
Prepare the new controller, reconcile its desired configuration with the host's
actual registry, then atomically transfer the writer lease. Commands carry epoch
and operation identity; the host rejects old writers and deduplicates retries.
A controller outage leaves existing work supervised, while new mutations stop
until authorization and ownership can be verified. Telemetry has a bounded,
observable buffer and resumes with acknowledgements rather than silent loss.
Use the [backend handoff procedure](/docs/omnitron/backend-handoff) to inspect
capabilities, prepare the candidate, switch controllers and verify rollback. Migration of existing coupled workers needs one controlled restart;
only workers already owned by the execution host can survive later control-daemon
updates. Immutable application runtimes and experiment registrations must remain
separate from controller upgrades.