Daemon

The backend serves the CLI, Console and mesh. In the default coupled mode it also owns application processes. Opt-in resident execution keeps those processes in a separate owner so the backend can be replaced without restarting them. See backend replacement for the initial maintenance window, supported environments, artifact validation and recovery.

The lifecycle below describes coupled mode. In resident mode, backend retirement preserves the owner; an explicit shutdown still stops applications.

Verified against src/daemon/:

Lifecycle

sequenceDiagram
  participant Init as omnitron up
  participant PM as PID manager
  participant SS as State store
  participant N as Netron bus
  participant SVC as RPC services
  participant Orch as Orchestrator
  participant Apps

  Init->>PM: acquire ~/.omnitron/daemon.pid
  alt lock held by live process
    PM-->>Init: error — already running
    Init->>Init: exit 1
  end
  Init->>SS: open durable state and project registry
  Init->>Init: register cleanup before signal handlers
  Init->>N: bind Unix socket + [TCP / HTTP]
  Init->>SVC: register all services
  Init->>Orch: read registered projects and enabled stacks
  loop per enabled stack
    Orch->>Apps: resume selected release or development configuration
  end
  Init-->>Init: ready
  Note over Init: serve indefinitely
  Note over Init: SIGTERM
  Init->>Init: cancel remaining startup steps, join in-flight work
  Init->>Orch: stopAll (coupled mode)
  Init->>SS: persist final state
  Init->>PM: release lock
  Init->>Init: exit 0

Startup and node diagnostics

A master prepares its control-plane database before Titan eagerly initializes providers. Foreground CLI, detached CLI and native supervision share this path. A configured OMNITRON_DATABASE_URL uses the supplied database without creating a local PostgreSQL container. --no-infra skips Docker provisioning, but still initializes the master schema through the shared Kysera migration runner. A slave keeps its SQLite lifecycle and does not connect to PostgreSQL.

During backend handoff, this stage only verifies schema compatibility; it does not provision or migrate. Database or migration failures stop startup before claiming the resident owner or publishing listeners, preserving the concrete cause in the boot diagnostic. Prepare schema changes before replacing a backend.

omnitron up launches a detached daemon and releases both the child-process and IPC event-loop references. The CLI exits after its readiness checks while the daemon keeps running. Startup stderr is appended to the boot log reported by the command, including failures before RPC becomes available.

The Nodes API exposes getNodeDaemonStatus, getNodeDaemonHealth and getNodeDaemonMetrics for every registered node. Local requests use the same live daemon readers as the CLI; remote requests use the authenticated mesh. reachable: false includes the failure reason and no answer. It is not a health verdict, and a remote failure never falls back to the local node's data.

Shutdown during startup

The daemon registers cleanup before Titan installs its signal handlers. If SIGTERM or SIGINT arrives during startup, it stops beginning new startup phases and enabled stacks. Titan's first shutdown task waits for the current startup operation and any in-flight health-monitor spawn or restart before cleanup closes their resources. The existing Titan shutdown deadline still applies: an operation that cannot finish is reported as a shutdown failure, not successful cleanup.

An interrupted startup does not announce daemon.started or a replacement backend's readiness. The startup diagnostic remains an error if an actual operation failed while shutdown was pending. A resident owner's applications continue running when only its backend terminates; explicit omnitron down still requests application shutdown. Event-loop monitoring and the reload signal handler are disposed with the backend.

PID manager — pid-manager.ts

Owns the daemon lock file ~/.omnitron/daemon.pid. Responsibilities:

• Acquire on boot: write process.pid to the file with an exclusive open. If the file exists and the PID inside is still alive, abort.

• Signature verification. The file carries a process-start signature (PID + start time fingerprint). A stale PID with the same numeric value as a recycled new process is detected and treated as dead.

• Liveness sweep. A periodic check ensures the lock owner is still alive; reclaims if the holder has crashed without releasing.

• Release on SIGTERM / SIGINT / normal exit.

When omnitron up reports "daemon already running", this is the check that fired.

State store — state-store.ts

Persists daemon intent to a SQLite-backed key/value row (DaemonStateStore, state_kv key orchestrator:persisted-state) co-located with the daemon's other state surfaces (PID lock, project registry, node registry, backup index). SQLite WAL-mode transactions remove the torn-write risk that the legacy fsync + atomic-rename JSON recipe guarded against. Writes are serialised through an internal promise chain so back-to-back save() calls never interleave; callers use a synchronous API and the actual write is awaited in the background.

Each persisted app entry records name, pid, status, mode (classic | bootstrap), startedAt, restarts, and port:

Legacy migration. On the first boot of the SQLite-backed code, if an old ~/.omnitron/state.json exists it is read once, migrated into the SQLite row, and unlinked. After that the JSON file never reappears.

State is persisted on every status transition (start, stop, crash, restart). A crash of the daemon itself leaves the row in a consistent state — on next boot, the daemon rehydrates and relaunches whatever was running.

File permissions and SQLite locks

State, recovery certificates and the log journal use owner-only directories (0700) and files (0600) on POSIX systems. Permission checks on an open SQLite database inspect metadata without opening or closing another descriptor. Closing any descriptor for the same inode can cancel the process's SQLite locks, including when the descriptor was opened only to tighten permissions (SQLite locking guidance).

If existing permissions need repair, Omnitron performs the no-follow, owner-checked operation in a separate short-lived Node process and verifies the result. Already-private databases need no subprocess. Symbolic links and additional hard links are rejected. Equivalent Windows protection depends on OS ACLs; POSIX modes do not establish Windows access control.

Use SQLite's online backup API for live databases. Do not copy, replace or remove a live database's WAL/SHM files. A recovery status held in memory is not sufficient evidence for activation: schema transitions require a current, durably readable certificate identifying the running backend and owner. pnpm --filter @omnitron-dev/omnitron test:private-storage checks cross-process writer exclusion, repeated readers, committed visibility and file permissions against the compiled runtime; the CI build runs it before accepting changes.

Daemon scheduler — daemon-scheduler.ts

A small set of periodic jobs registered against titan-scheduler and tied to the daemon's lifecycle. registerDaemonJobs() wires:

The master-only jobs are skipped on a standalone single-node setup where the auth / alert / fleet services aren't wired.

Daemon configuration

The daemon's own config (not project / ecosystem config) lives in the DEFAULT_DAEMON_CONFIG object — most operators never touch it. Values:

Override via CLI flags on omnitron up or via the daemon configuration file (advanced — not part of the project ecosystem config).

Console request limits

The daemon shares one Netron limiter across HTTP and WebSocket. Configure requestRateLimit in the daemon settings. Its defaults are 3,000 requests per client and 6,000 total per 60-second window. Switching transport or opening another tab consumes the same allowance. The legacy httpRateLimit setting covers both listeners. If both settings exist, requestRateLimit takes precedence.

The Console gateway overwrites X-Forwarded-For and X-Real-IP on HTTP and WebSocket. With the default trustProxy: false, clients behind the same proxy share its socket-address allowance. Set trustProxy: true only when access to the upstream listeners is restricted to that trusted proxy. The local administrative Unix socket has its own OS access boundary. Netron request budgets

RPC surface — OmnitronDaemon service

The daemon registers @Service({ name: 'OmnitronDaemon' }) on the Netron bus. Verified from src/daemon/daemon.rpc-service.ts.

The daemon checks these roles for each request.

Methods, by intent

App lifecycle (operator)

scale requires an explicit declared pool process and a positive safe integer. It returns the existing ProcessInfoDto; read processes for the selected pool's observed instances and unchanged declaredInstances. Resizing lasts until restart and leaves automatic scaling enabled. The owner enforces the pool's configured bounds and must advertise processPoolResize@1; an older owner refuses before mutation. This optional capability does not prevent a backend update when the owner remains unchanged.

restartAfterChange is the automatic source-refresh operation used by SDK watch mode. Supply the canonical application name, generation and declaration revision observed in getWatchStatus(). The execution owner must advertise sourceRefresh@1; it refuses stale bindings, disabled watching and pinned release applications before restarting. The response is the owner's exact SourceRefreshReceipt: { process, source }. process confirms a new online generation; source carries that same generation with its name, directory and definitionRevision. A later inventory read must not be substituted for this acknowledgement. A successful build alone does not prove that an application loaded it. Application health probes remain separate from the refresh receipt.

Inspection (viewer)

Owner and environment inspection

attestApplicationOwner({ owner: { id, pid, startedAt }, epoch }) returns the clean build stamp of the observed owner. The node checks its active controller lease. For a legacy owner, it verifies the supervised process and sealed runtime before checking the lease again. Normal status polling performs no disk scan. See application release admission.

getEnv({ name }) returns redacted environment values. revealEnv({ name }) requires an administrator and releases the clear values only after the audit database confirms the write. A node without audit storage can still return redacted values.

Watch control (operator)

Admin (admin)

Heartbeat (anonymous)

RPC plumbing (admin)

exec is what omnitron exec api users findById u_42 routes through. It is admin-gated — invoking arbitrary methods on a managed app is a privileged operation.

Auth flow

Authentication is intentionally split between transports:

flowchart LR
  CLI[CLI command]
  Webapp[Webapp]
  Remote[Remote daemon]

  CLI -- Unix socket<br/>OS-level identity --> AuthBypass[implicit admin]
  Webapp -- HTTP + JWT --> JWT
  Remote -- TCP + JWT --> JWT

  JWT[Verify JWT<br/>read roles]
  JWT --> RoleCheck{Role enough<br/>for method?}
  RoleCheck -- yes --> Invoke[Run method]
  RoleCheck -- no --> Reject[403 / FORBIDDEN]

The Unix socket auth-bypass is deliberate: file mode 0o600 means only the daemon's UID can open it. If the OS trusts you to write to the socket, the daemon trusts the request. TCP and HTTP always require JWT.

Daemon client — daemon-client.ts

The operator API exposes the typed local client through @omnitron-dev/omnitron/operations. It connects to the daemon owner's Unix socket and has the same authority as the local CLI. Keep it on the operator host, not in browser code.

import { createDaemonClient } from '@omnitron-dev/omnitron/operations';

const client = createDaemonClient();
try {
  await client.ping();
  await client.startApp({ name: 'api' });
  const metrics = await client.getMetrics({ name: 'api' });
  console.log(metrics);
} finally {
  await client.disconnect();
}

Local and fleet TCP clients share connection lifetime handling. Concurrent calls share one connection attempt; retained service handles reconnect after backend replacement. Connect, reachability and teardown have separate bounds. A lost in-flight reply rejects the call without replaying it. See long-lived RPC clients before upgrading an application that retains a client across backend replacements.

Auto-restart and backoff

Applications inherit the ecosystem's supervision settings. Override them for one app with apps[].restartPolicy. By default, up to five automatic restarts may begin in a rolling 60-second window. The first retry waits 1 s. Later retries in the same window use exponential backoff up to 30 s. Set maxRestarts: 0 or enabled: false on the app's restart policy to disable automatic retries.

If a classic script uses its retries and fails again, the owner marks it crashed, emits app:escalated and waits ten minutes before trying again with a fresh budget. For Titan-supervised children, an exhausted budget ends automatic recovery of that child. A critical child failure escalates to its supervisor.

Use omnitron stop <app> during backoff or cool-down to cancel the pending launch. The app stays stopped until a new start is requested. When ready, run omnitron start <app>. For a stopped stack, use omnitron stack start <project> <stack> to restore its applications together.

Failure scenarios

Operating runbook

Do not delete daemon-state.db to reset a process: it also holds project and node registrations, execution intent and the backup index. Deleting secrets.enc destroys the file-provider vault. Recover these stores only from a verified backup with a reviewed data-loss and rollback plan. Use normal application and stack lifecycle commands for process changes.

Anti-patterns

• Editing the daemon state DB by hand. Persisted state lives in a SQLite DB (~/.omnitron/data/daemon-state.db), not a hand-editable JSON file. The schema may evolve; poke it directly and you risk a broken next restart. Prefer using the CLI to drive state changes.

• Running two daemons against the same ~/.omnitron/. They fight over the lock. Multiple daemons need different home directories.

• Exposing the TCP port without a JWT secret. Anyone reaching the port can inspect at minimum, and operate / admin with default roles. Configure auth before opening port 9700.

• Treating kill as routine. Force-kill skips graceful shutdown — pending logs may be lost, child processes may be reparented to init. Use down first.

• Removing the lock file while the daemon is running. Causes inconsistent state where two daemons can subsequently start.

See also

• Architecture — where the daemon fits in the broader picture

• Orchestrator — what the daemon hands apps off to

• Services reference — all 19 RPC services

• CLI — every command that talks to this daemon

Updating a source runtime

In coupled mode the daemon directly supervises application processes. Replacing it requires a controlled maintenance window. A resident owner can retain running applications across a backend update. Check the reported execution mode before choosing the handoff procedure.

1. Record the active build, enabled stacks, application states and remote node process IDs. Save the daemon configuration, account database, project registry, runtime dependencies and builds needed for rollback. Preserve persistent data volumes and existing experiment registrations.

2. Build and validate the target revision separately. Use the repository's exact packageManager version and test native dependencies with the Node executable used by the installed service. Keep the package manager in a permanent path available to that service.

3. Unload the OS supervisor before replacing a watched checkout. Wait for owned application processes to stop. Fast-forward clean working copies and install from the verified lockfiles; preserve other branches and local configuration.

4. Install each source application's adapter and Titan from its frozen lockfile. They must resolve one physical Titan package. The daemon may use its own SDK. Check the active owner's capabilities when upgrading an older installation. Published deployments use their pinned framework artifacts.

5. Build the daemon and Console from the same revision, then restore OS supervision. Check each enabled stack and its child processes against the saved baseline. A responding daemon does not mean its applications have finished starting.

6. Reload the Console and verify access controls, SSH host identity and sessions, application health and audit records. Confirm remote node processes remain unchanged for a local upgrade.

Keep immutable experiment manifests intact when their recorded package trees are unavailable. Recover the original environment or create a separately registered experiment; changing old hashes would invalidate its reproducibility.

Execution handoff: architecture boundary

In coupled mode the control daemon owns Titan PM supervisors, fork IPC channels, worker threads and application routing. Fork workers terminate on parent disconnect; the parent-death watchdog and cold-start janitor enforce that ownership. Simply disabling stopAll or reusing PIDs would either lose in-process workers or create unmanaged processes.

In resident mode a separately supervised execution owner uses Titan PM to manage application processes, threads, restart policy, log capture, metrics and the child registry across backend replacements. Omnitron retains project orchestration and RBAC, communicating through authenticated, local Netron RPC. A private socket and a persistent host identity must be verified before a controller can attach; PID existence alone is not identity evidence.

Ownership transfer requires a monotonic fencing epoch and one active writer. Prepare the new controller, reconcile its desired configuration with the host's actual registry, then atomically transfer the writer lease. Commands carry epoch and operation identity; the host rejects old writers and deduplicates retries. A controller outage leaves existing work supervised, while new mutations stop until authorization and ownership can be verified. Telemetry has a bounded, observable buffer and resumes with acknowledgements rather than silent loss.

Use the backend handoff procedure to inspect capabilities, prepare the candidate, switch controllers and verify rollback. Migration of existing coupled workers needs one controlled restart; only workers already owned by the execution host can survive later control-daemon updates. Immutable application runtimes and experiment registrations must remain separate from controller upgrades.