CLI reference

omnitron <command> is the single binary. Management commands use Netron RPC through ~/.omnitron/daemon.sock. Local build, configuration and migration commands execute in the CLI process; application migrations can run without a running daemon.

Verified against src/cli/omnitron.ts and src/commands/*.

Global options

Commands that support --json return one { "ok": true, "data": ... } result on stdout; errors use { "ok": false, "error": ... } on stderr. Machine output is an explicit action contract. Unsupported commands reject the flag before loading or executing their action, with exit 2, code JSON_OUTPUT_UNSUPPORTED and outcome: "not-started"; they do not perform the operation and then complain about its output. This also applies to OMNITRON_OUTPUT=json, nested commands and aliases.

A declared JSON command that returns without a result reports JSON_OUTPUT_CONTRACT_VIOLATION and outcome: "unknown". Its operation may have completed: inspect state before retrying. Finite commands validate their output and drain stdout and stderr before exiting, so large log exports remain complete when piped into another process. Check both the JSON result and the exit status.

Command index

Topology workspace

omnitron topology refresh <project> [stack] explicitly captures bootstrap dependency declarations in private SQLite without starting applications. It requires admin, uses bounded separate reader processes and returns per-application failures while retaining the last successful metadata. Graph reads remain available to viewers. Refresh after application or dependency rebuilds; inspect source/capture times and coverage warnings with project topology or topology show.

Use omnitron topology refresh <project> <stack> --release <id> to read the declarations of a prepared candidate for an independently owned local stack. The administrator can capture them while the previous release is running. See local release preparation.

omnitron topology show <project> <stack> reads configured, desired and observed placement. export writes portable JSON; draft <project> <stack> <file> saves a revision-protected draft; plan validates it; apply <project> <stack> <plan-id> accepts that target. Acceptance does not change executable configuration or run a migration. See topology review, permissions and limits.

Database migrations

omnitron migrate [status|verify|up|down] uses omnitron.migrations.ts by default. --config selects a configuration; --daemon explicitly selects the daemon database. up --dry-run previews, up --to NAME selects a prefix, and down --steps N rolls back a suffix. verify exits with 2 when integrity is unknown or mismatched. Direct up and down require a standalone installation. For a clustered controller schema change, use the reviewed backend schema workflow. See migration configuration and recovery.

Backend replacement

backend status --json reports whether the running daemon supports resident execution. backend upgrade <sealed-dist> --check validates a candidate and rollback artifact. Without --check it replaces only a resident-mode backend. backend abort <checkpoint> cancels preparation before retirement; backend recover <checkpoint> refuses to preempt an unrelated live backend; it can reconcile a verified successor when startup-selection publication was interrupted.

omnitron backend install <sealed-runtime> copies a verified release into permanent storage without activation. omnitron backend installation --json reports its startup selection and unfinished activation. omnitron backend select <sha256> selects an installed release only while the backend and execution owner are stopped. Running installations use backend upgrade; it commits the selection after readiness and preserves the rollback. The stable launcher is $OMNITRON_HOME/runtime/bin/omnitron.

Before scheduling an owner restart, install the candidate, then run its CLI against the instance.

omnitron --json backend select <installed-sha256> --check

The result identifies the current and proposed releases and the migration scope. You can run this check while applications are online. Selection repeats the checks under the installation lock after the backend and owner have stopped. An independently supervised worker with cluster coordination disabled compares its complete SQLite migration registry in both releases. Controllers, enrolled workers and installations without a proven managed role compare both registries. Changed worker migrations require a separate data transition. A successful cold-selection check does not establish live owner compatibility. Use backend upgrade --check for that decision.

For supported independent services, omnitron backend guardian prepare <sealed-runtime> records a private transition plan. omnitron backend guardian apply <plan> applies or resumes it while retaining the execution owner and application processes. The existing backend release is restored under the updated guardian.

omnitron backend schema prepare <sealed-runtime> reads the master database and saves a plan for explicitly reviewed compatible additions. omnitron backend schema apply <plan> applies or resumes it under the shared migration lock. After rollback, prepare <sealed-runtime> --from-plan <applied-plan> binds the same already-applied expansion to a corrected candidate, checking the original runtimes and exact database history. omnitron backend upgrade <sealed-dist> --schema-plan <plan> requires that exact applied plan for an upgrade or rollback between its two sealed releases. Legacy bridges, changed definitions, foreign databases and local node schema transitions are refused.

Use the candidate's CLI for backend upgrade --check. It checks the owner's declared lifecycle contracts before draining or delegating; an incompatible owner requires a planned owner transition. backend status --json includes execution.ownerCapabilities when the backend reports them.

Recover after an execution-owner crash

An unclean owner exit can leave application processes alive. A new owner cannot adopt their native process handles, and ordinary startup refuses a same-boot unclean receipt. Preserve the crash evidence and prepare a node-specific drain and rollback before stopping surviving applications.

On macOS with launchd, or Linux with systemd system services, omnitron backend owner-reconcile prepare <worker-pids> seals a private recovery plan. Supply every recorded pre-crash application, worker and descendant PID as a comma-separated list, including non-Node processes. The plan binds process lifetimes, the host boot, configured services, controller journal and unclean owner receipt. Linux also records the configured cgroups; macOS records the launchd domain, native boot session, observed descendants and their retained process groups. The command writes only private plan/seal records; it does not open SQLite, stop services or signal applications. Changed or legacy unsealed plans require a fresh preparation.

After the separately approved drain, omnitron backend owner-reconcile acknowledge <absolute-plan> requires the old owner identities to be dead, both configured services stopped, observed ownership groups empty, every recorded lifetime gone and execution sockets absent from the kernel. Unknown or incomplete observations, changed configuration, journal, boot session or plan refuse recovery. Only a proven stale owner socket may be removed. The original unclean receipt is archived; it is never changed to claim a clean shutdown.

On macOS, this proof covers the supplied identities, observed descendants and retained process groups. Ordinary launchd users have no cgroup-equivalent containment: an already detached and reparented process never recorded in that cohort cannot be discovered reliably. Include independently recorded descendants and audit the drain; do not treat an empty process-name search as evidence. Native ps, launchctl, sysctl and lsof observations must all succeed.

The macOS boot fence uses kern.bootsessionuuid, shared with process-lifetime inspection. Historical kern.boottime text can drift while the same kernel keeps running; a changed timestamp never proves a reboot. A dirty legacy receipt needs explicit reconciliation even when its text differs. A fresh plan binds the current UUID while preserving the complete original receipt and its hash. Old plans based on boot-time text must be prepared again. Genuinely clean legacy receipts remain eligible for normal startup. Legacy or malformed log-writer identities stay unconfirmed while their PID is alive; diagnostics explain the uncertainty and recovery proceeds only after native death is established.

A compatible replacement owner consumes the acknowledgment once under a kernel lock before starting applications. Linux uses flock; macOS uses its native lockf. Node retains the locked file descriptor after the helper exits; process death releases it without deleting a lock file or trusting a stale PID. A durable consume record also prevents replay when final acknowledgment publication fails. These private records protect against accidental races and interrupted recovery; they are not attestations against another process with the same filesystem UID.

A failed startup requires a fresh plan and fresh native evidence. An older runtime without this recovery contract can be restored only after a replacement has completed a genuinely clean shutdown. Do not promise an unconditional old-binary rollback after another owner crash. Automatic adoption of orphan processes is not implemented.

See requirements, first transition and recovery.

Local runtime updates

omnitron runtime owns native artifact verification. omnitron runtime update owns a reviewed transition of local supervision, Console and the recorded stacks.

From a clean Omnitron checkout, Xec prepares the same public operation:

export OMNITRON_DEV_CACHE=/path/to/private/development-cache
pnpm exec xec update local-sync

Review the returned scope, readiness conditions and recovery limitations before applying it. pnpm exec xec update local-sync --review <digest> performs the transition and may restart the listed applications. Inspect pnpm exec xec update local-sync --status after an interrupted response; do not repeat an uncertain operation. Recovery uses pnpm exec xec update local-sync --review <digest> --rollback. The cache directory must be private to the current user. Xec reuses a verified runtime for the same commit and native Node producer; a changed checkout requires a new preparation. In the development checkout, it also discovers the completed local release build for that exact commit, platform, architecture and Node version. The bytes are verified locally before use, without another build, archive copy or CI request. An invalid selected artifact fails verification instead of silently falling back to a different build. Preparation does not activate the runtime.

Preparation requires an idle controller and a current recovery checkpoint. omnitron backend status reports backup operations and other activity that defer it. Run preparation again after that work finishes.

The rollback backend must also support reviewed stack boot, unless every stack already has the exact saved infrastructure-preservation review. Installing a new candidate does not add this capability to the previous runtime. For an older installation, first check a backend-only transition with pnpm exec xec update local-check <sealed-runtime>/packages/0/dist. After that transition is reviewed and activated, prepare local-sync again. The backend-only step preserves the owner and applications; the subsequent local plan covers their restart and retains the upgraded backend for owner rollback. See backend handoff for its scope and recovery contract.

Programmatic verifiers import verifyRuntimeArtifact or inspectRuntimeArtifact from @omnitron-dev/omnitron/runtime-artifact. The latter returns an assertUnchanged() check for a verified filesystem snapshot. A changed snapshot invalidates the proof; it is not a durable authorization to execute or activate.

Daemon lifecycle

omnitron up

Start the daemon in the background. Auto-detects omnitron.config.ts at the current working directory or any ancestor.

Foreground startup returns control to the daemon lifecycle, not to the CLI exit handler. The process continues serving RPC after initialization; Titan handles its shutdown. If the daemon is already running, another up --foreground reports that state and returns without starting a second controller.

omnitron down

Stop the daemon — gracefully stops every project, stack, and infrastructure container.

omnitron release preflight

Inspect tool availability, the configured builder capacity, active and queued jobs, and support for shared-store publication. Use --json in a pipeline.

omnitron release builds

List controller jobs, optionally restricted to one project. Use --json in an agent or CI client.

omnitron release job

Read a single job by ID. Use --follow to observe phases until it settles.

omnitron release stop-build

Cancel the specified queued or executing job. Other projects retain their slots.

omnitron stack check-release

Check admission for a specified project, stack and release ID. This uses the same read-only method as the Console deployment dialog. The exit status is nonzero on refusal.

omnitron ping

Probe whether the daemon is alive. Exit code 0 = up, non-zero = down.

omnitron kill

Force-kill the daemon process. Use only when down doesn't respond.

omnitron service install|uninstall|status

Register the platform's processes with the operating system. In resident mode, the owner keeps applications running while the independent guardian supervises the backend. In coupled mode, the daemon also owns application processes, so its failure can interrupt them. OS supervision and application supervision have separate responsibilities.

Preparation defaults both roles to the selected sealed runtime. --owner-runtime selects another already installed, verified owner by its full SHA-256. Its native executable and declared capabilities must satisfy the selected backend. Install with the same owner hash, selected backend hash and service scope after the reviewed shutdown. Repeat service install --prepare while stopped to review different installed roles. The saved plan must still belong to the same instance, execution directory, scope and native services; its validated private environment is preserved exactly, including secrets absent from the current shell. Invalid saved records refuse preparation. Installation retains strict role matching under the runtime operation lock and requires stopped processes and quiescent native services; a missing PID alone is insufficient. It does not stop services, choose a fallback or upgrade an old owner's capabilities automatically. Backend-only updates use backend upgrade and keep the current owner.

Restarts are crash-only, so up and down keep meaning what they meant. A graceful omnitron down exits 0 and stays down; a crash or a SIGKILL exits abnormally and is respawned (KeepAlive={SuccessfulExit:false} on launchd, Restart=on-failure on systemd). Installing the service does not take down away from you.

In coupled mode, the supervisor runs the same daemon-entry.js that omnitron up forks — one daemon codepath with two launchers — and that entry point holds a single-instance pid guard, so the two can never both bind.

Development SDK

From Omnitron 0.4.9, starting or restarting a source application refreshes its existing esbuild contexts before launch. A repaired package installation is resolved again even when application source files have not changed. CLI and RPC report success only for confirmed online applications; inspect a failed or in-progress launch before retrying.

Each command accepts --project-dir <path>. The developer selection is machine-local. The watcher runs in the foreground of sdk dev; saving the selection does not create a background watcher. Keep that command running or resume it after exit.

From Omnitron 0.4.9, sdk status also verifies the current Turbo build inputs, native Node producer and output digests. It reports a current build only when they match the saved build checkpoint. Changed sources or dependencies, an incomplete build or an older checkpoint require sdk dev to rebuild; inspection never installs, builds or restarts applications.

SDK watching uses Xec Loader for filesystem observation. From Omnitron 0.4.9, the watched directories and shared files follow the Turbo task graph; Turbo inputs decide whether a change requires a build. Declare build inputs in Turbo, including generators and shared configuration. Unsupported or unbounded watch roots produce an explicit error. Atomic saves are detected, changes are coalesced, and edits made during a build are processed afterward. Generated output and installed dependencies do not trigger a rebuild loop. Cancellation closes the watchers and waits for the active build to settle. See local SDK development and registry deployments.

Project management

A project is one repository with many named stacks. Register the repository once; subdirectory configurations contribute stacks to that owner. See projects and stacks for composition and safe consolidation.

Aliases: omnitron proj works the same as omnitron project.

Stack management

A stack is a named environment within a project (typical values: dev, staging, prod). Stacks scope deployments and infrastructure to a subset of the project's apps.

An applications-only maintenance stop also disables automatic application resume and preserves the selected release. A backend or owner restart does not start it again; use an explicit stack start. Both stop modes share the same permissions, review, execution fence and per-node receipts. A legacy remote node that cannot confirm retention blocks preparation instead of receiving a destructive fallback. See confirmed stack stops.

Restart with preserved infrastructure

For a retained release, inspect the preservation start before execution:

omnitron stack start sample dev --release <release-id> \
  --infrastructure preserve --dry-run --json

After the reviewed stop, use the returned digest for that exact start:

omnitron stack start sample dev --release <release-id> \
  --infrastructure preserve --review <digest>

Preservation start rechecks release admission and native infrastructure evidence. It skips provisioning, setup, migrations, static uploads and environment provisioning; a mismatch refuses the start. A runtime or installed-path change may invalidate the earlier digest, requiring a fresh review.

stack stop --keep-infrastructure durably selects preservation policy. A plain start or autostart cannot silently replace it with reconciliation. Explicit --infrastructure reconcile is a separately reviewed provisioning, setup and migration operation. See infrastructure ownership for the responsibility boundary.

Before a first start or after changing an environment:

omnitron stack diagnose sample dev
omnitron --json stack diagnose sample dev

The same report appears under Stacks → Diagnose in Console. JSON includes checkedAt, durationMs, status, structured checks and explicit deferred coverage. Exit code 0 means prerequisite checks passed, possibly with planned creation/downloads; 1 means blockers were found; 2 means evidence is incomplete or the request failed. A successful prerequisite check does not certify database migrations or application startup. No provisioning or migration is performed.

Local Docker checks include declared container ownership, specification changes, health, image and named-volume availability, bind mounts and host port conflicts. Remote target-host checks remain explicitly incomplete; infra inspect and node check provide complementary evidence. Project configuration is the daemon's loaded configuration; reload it after editing project configuration files.

App lifecycle

These commands accept an [app] argument. With no app, they operate on all managed apps.

start, stop and restart use text output in 0.4.9. They reject --json and OMNITRON_OUTPUT=json before starting an action. Use omnitron list --json for a structured observation of the resulting application state.

Use a full application name such as omnitron start project/test/api or omnitron stop project/test/api to limit the request to one process group. An existing application uses its registered launch definition, including after a project reload. A short name shared by two stacks requires the operator to select the full name. When a registered handle is missing, start and restart accept a declaration from its matching project and stack. If a named start cannot reach the daemon, it exits with an error. Starting the daemon with omnitron up can also start its configured stacks. Use omnitron stop all to address every managed application. These operations require a responsive daemon. Start and stop report a connection failure with a nonzero exit code. In 0.4.9, restart can return exit code 0 after reporting an unavailable daemon; confirm the observed application state instead of relying only on that exit code. If the connection drops after sending the request, inspect omnitron list before retrying. Use omnitron down for an explicit daemon shutdown.

A named stop controls the current process group; it does not persist an application exclusion in an enabled stack. A later stack start or boot recovery can start that application again. Do not use a successful stop receipt as a promise that a port or workload identity stays available for a replacement. stack stop records durable intent for the whole stack; it is not a substitute for excluding one application while its siblings remain active. The scoped owner contract provides the lower-level fence, but the public per-application lifecycle does not yet expose that guarantee.

Information

Cancelling the overwrite prompt preserves the original file and exits with status 130. An existing file in a noninteractive session requires --force; a new file needs no confirmation.

Reading the child rows of omnitron list. Each indented row is one entry of the app's process topology, and two of its columns say something a single number cannot.

A pool entry — one declaring instances > 1 — runs several processes and the row prints one of their pids, so it also prints its live worker count: transform (worker) ×2. When that count has drifted from what the topology declares it is shown as ×1/2 in yellow; doctor reports the same drift as a finding.

RST shows -, not 0, when nothing counts that row's restarts. The supervisor keeps a restart count per child, and a pool is not a supervisor child, so for a pool row the honest answer is that it is not tracked. 0 would leave "never restarted" and "not counted" looking identical.

UPTIME is the uptime of the process whose pid the row prints — not the app's. A child that restarted an hour ago under an app that has been up for a week reads one hour.

omnitron system

omnitron system
omnitron --json system --node <registered-id>

The command uses the same bounded collectors as Console System. A named node is queried over the authenticated mesh. JSON includes the target, collection timing, completeness and the complete snapshot. Inspect snapshot.unanswered for per-collector failures. Exit code 1 indicates an unavailable, stale or partial reading, or a host clock too far ahead. The JSON still contains every successful measurement.

A stack start also exposes its outcome in JSON. outcome: "confirmed" requires a running stack, every reported application online and no missing infrastructure or nodes. A partial reply includes the affected resources and sets exit code 1. A transport timeout leaves the result unconfirmed. Inspect the stack and deployment history before repeating the request.

Monitoring

Resident mode uses independent owner and backend guardian services. See backend replacement for preparation, crash recovery, system scope and owner-failure boundaries.

omnitron logs [app]

Tail logs. Omit [app] to read the daemon's own log.

Use the full application handle for production diagnostics:

omnitron logs sample/production/mail --node worker-1 -n 100
omnitron logs sample/production/mail --node worker-1 --level unknown
omnitron --json logs sample/production/api --node worker-1 --grep 'auth\.login'

--node resolves a node name or unique ID prefix through the registry and uses the same OmnitronLogs.streamLogs contract as Console. --node and --file are mutually exclusive. Omit --node for the local machine. JSON output records the source and node; an empty result does not establish that no request occurred.

omnitron monit

Live TUI dashboard — terminal app showing per-process CPU / RSS / restart counts in real time. Quit with q.

omnitron health [app]

Composite health report from titan-health indicators.

omnitron metrics [app]

Snapshot of current CPU / memory / latency / RPC counters.

omnitron health-check [app]

Checks application-reported health, declared readiness and HTTP/TCP availability. A live process or HTTP 200 cannot override a failed application check. Without an app name, includes every local application and declared stack infrastructure.

Use omnitron --json health-check [app] for the same structured report. Both human and JSON modes exit with status 1 when any check is degraded or unhealthy; JSON output retains the individual checks and their reasons.

omnitron discover

Scan Docker + SSH on known hosts for Omnitron-managed processes not yet in the registry. Useful when migrating in an existing fleet.

Scaling

omnitron scale <app> <count> --process <name>

Resize one declared process pool in a running bootstrap application. The process name is required; classic applications and singleton processes cannot be resized. The count must be a positive safe integer. A fixed pool accepts 1 through its configured maxInstances (the declared size when no maximum is configured). An automatic pool retains its declared initial size as the minimum and uses its configured maximum, or four times its initial size when omitted.

This changes runtime membership until restart. It does not rewrite the application configuration; restart restores the declared size, and automatic scaling remains active. The result reports the selected pool's observed count. For agents, omnitron --json scale <app> <count> --process <name> returns the request, observation, and lifetime: "until-restart". If a dispatched request times out or its outcome is unknown, inspect that pool before retrying.

The owner must advertise processPoolResize@1. An older owner refuses the request without changing the application. Updating only the backend does not add this capability to a running owner.

Diagnostics

omnitron doctor

One pass over the whole installation, reporting problems with the evidence that identifies them and the command that fixes them.

It reads the database directly rather than through the daemon, because a daemon whose schema has gone missing still answers RPCs perfectly well.

What it examines. Every finding's id begins with the area it came from, so this table is the same list the output uses:

Exit code is 1 when any finding has severity error, so it works as a CI gate.

What "no problems found" means. A check that cannot run is reported separately from one that ran and found nothing:

[?] 5 area(s) not examined:
      db.pending-migrations — not checked: whether every migration
        has been applied — the database stopped answering
      ...

An unreachable database takes five checks down with it and a daemon that is not answering takes seven. Without that section a silent gap and a clean result print identically, and they point an operator in opposite directions.

In --json the same distinction is two fields:

{
  "ok": false,
  "complete": false,
  "worst": "error",
  "findings": [ { "id": "db.unreachable", "severity": "error", "title": "…",
                  "evidence": ["…"], "remedy": "…" } ],
  "skipped":  [ { "id": "logs.*", "reason": "not checked: … — the database stopped answering" } ]
}

ok is "nothing was found"; complete is "everything was looked at". A gate that reads only ok cannot tell a healthy installation from an unexamined one.

Findings carry a stable id — db.unreachable, app.port-unreachable, infra.detached, build.stale-sources, auth.anonymous-surface, logs.retry-loop, project.config-unloadable and so on. A skipped entry names the id its check would have reported under, or prefix.* where a check reports several.

project.config-unloadable is worth knowing about before you need it: the daemon refuses to start against a config it cannot load, rather than starting with no apps, so this finding is what a failed omnitron up in that project's directory would have told you. The check reads the files rather than asking the daemon, because a daemon holding a config it loaded hours ago cannot tell you the file has been edited since.

omnitron inspect <app>

Deep diagnostics for an app — services exposed, memory breakdown, pending requests.

omnitron exec <app> <service> <method> [args...]

Invoke an RPC method on a managed app:

omnitron exec api users findById u_42

Arguments are parsed as JSON when they look like JSON, otherwise strings.

omnitron env <app>

Show resolved environment variables for an app (after merging project / stack / per-app overrides).

Remote daemons

Register remote daemons as named aliases and address them via omnitron remote <alias>.

Fleet operations

Aggregate across all remotes (and the local daemon).

Cluster (leader election)

Cluster operations require cluster.enabled: true in the daemon config.

Secrets (encrypted at rest)

The daemon stores an encrypted envelope in its SQLite state store; the former ~/.omnitron/secrets.enc is a legacy migration source. A running daemon owns writes to the vault. A rejected RPC write never falls back to changing its files.

Omit the value to enter it without echo in a terminal, or feed it from a secure source on stdin. Input is limited to 64 KiB; one final LF or CRLF is removed. Empty input and conflicting positional/--stdin sources are refused. Positional values remain compatible, but emit a warning because shell history and process arguments can expose them. secret get deliberately prints the value; use it only when that output is intended.

Keep references in project configuration, for example settings.appEnv.api.API_KEY = { secret: 'service.api-key' }. References limit which application receives the value. Do not put credentials in source files, frontend bundles or command arguments. Encryption at rest does not remove the need to protect the vault passphrase, state backups and the host account.

Remote deployment sends resolved configuration and migration environment through Xec's SSH stdin channel, rather than embedding credentials in command arguments. The existing deployment lease still fences those operations. Node configuration is materialized in private files (0600) so applications can restart independently of the controller; an administrator of that host can read it. Migration output uses the common diagnostic redactor, but applications must still avoid printing secret values themselves. Failed or interrupted migrations are not automatically retried: inspect their ledger before deciding whether to run them again.

Deployment

Stack deployment and rollback

Deploy a built release with omnitron stack start <project> <stack> --release <id>. Returning to a previous release uses that same workflow with the reviewed previous ID. Admission, applicable transfer checks and readiness still apply. For a local stack, see Pin a local stack for preparation, controlled switching and rollback.

omnitron deploy app and omnitron rollback are retained refusals: they exit nonzero without changing applications. Their former per-app path could restart current code but could not install a requested version or restore an earlier artifact. The corresponding RPC mutations refuse before database or process changes as well. No rolling, blue-green or canary rollout is implied by accepting a strategy label in a historical record.

omnitron deploy build <app>

Build an application artifact without shipping or starting it. To build a release with source commits, manifests and gates, use omnitron release build.

Tor

omnitron tor

Show the Tor hidden-service onion addresses for the running stack. Top-level command; no flags.

Infrastructure

Manage infrastructure containers (Postgres / Redis / MinIO / custom) provisioned per app's omnitronConfig.infrastructure.

Direct infrastructure mutations require a standalone installation. For projects declaring named stacks, use the project and stack lifecycle to provision or remove containers. An enrolled node requires admission through the cluster lifecycle protocol. Direct container operations verify ownership and use the observed Docker ID before removal.

Pipelines (Xec automation)

All pipeline RPC methods require admin. Parameters are data, not source substitutions. Failed, cancelled and interrupted runs produce a nonzero status-command exit code. See Xec automation for definitions, limits and recovery behavior.

Backups

Use the explicit project and stack for resource backups. Infrastructure retained while application writers are stopped remains addressable. The daemon verifies container ownership and uses its immutable ID. The archive index records the project, stack and bucket; a restore refuses an unavailable origin instead of choosing another stack. Legacy archives without origin metadata require exactly one eligible local target. Legacy object archives refer to the storage bucket.

omnitron backup artifact storage-objects --project acme --stack prod --bucket uploads --json
omnitron backup artifact tor-keys --project acme --stack prod
omnitron backup list
omnitron backup restore <id>

Creation and restore require an administrator. CLI and Console clients share the OmnitronBackups Netron contract; createArtifactBackup accepts { project, stack, kind: 'storage-objects', bucket } or { project, stack, kind: 'tor-keys' }.

A full pass reports each target independently, including undeclared bucket inventories and failures. It covers provisioned resources on the receiving daemon; it does not automatically contact remote daemons. MinIO bucket inventory comes from the stack's declared buckets. Explicit artifact capture can name another bucket on that same provisioned service.

Object contents stream through the shared backup pipeline with bounded concurrency. Each operation owns a private credential directory; credentials do not appear in host command arguments. A snapshot is published only after transfer and archive validation. Inventory changes during the copy, unsupported object names and names that collide on a portable filesystem cause an explicit failure.

These object archives preserve current bytes, not S3 version history, policies, object metadata or tags. A full pass is not an atomic snapshot across services. Quiesce writers when application consistency is required. Restore overwrites the archived keys, leaves other keys intact and requires an existing bucket. If restore is interrupted, reconcile the destination before resuming writers.

Database dumps use a private staging file and one deadline for the local process, compression, file writes and publication. The default database transfer budget is 30 minutes per dump or restore. Configure it on each daemon, as described in database backup policy. The destination is created exclusively; an existing file is never overwritten. File contents and the containing directory are synchronized before a successful publication is acknowledged. If publication or indexing cannot be confirmed, a complete destination is retained and the error names the file and the uncertainty. Inspect that path and the backup index before repeating the operation.

Historical .meta.json records are removed only after the SQLite index durably confirms the same identity and metadata. A database error, conflicting record or failed file removal retains the source for a later retry. This protects the index; it does not verify that an archive can restore its application's data.

Failed requests, partial capture and an empty capture pass return a nonzero CLI exit status. Restore and schedule mutations require a positive acknowledgement. Read commands distinguish missing diagnostic data from an empty index and still show uncovered stacks when no backups or schedules exist. A previously failed scheduled pass remains a warning in an otherwise successful status query.

omnitron backup schedules reports the effective database transfer budget beside the schedules and their last outcomes; --json returns the same diagnostic data. An expired budget is a failed operation, never a successful partial backup. PostgreSQL restores ignore personal psql startup files and stop at the first SQL error. Neither a SQL error nor a timeout rolls back statements already applied to the target: the restore is not a single transaction. Keep writers stopped and inspect the destination before deciding whether to restore again or recover another way.

A local Docker client exiting does not prove that its in-container command has stopped. The pipeline reports unresolved cleanup instead of claiming termination. The database budget applies to each PostgreSQL dump or restore on this daemon; S3 transfers and Tor archives have separate limits. Per-database overrides and verified in-container exec termination are not implemented.

Schedules persist across daemon restarts; they do not run while the daemon is down. Active capture, restore and scheduled retention block backend handoff. Preparation pauses schedule admission, and an aborted handoff re-arms persisted schedules.

Omnitron 0.4.9 also fences explicit OmnitronBackups.deleteBackup operations. Archive removal is asynchronous, so large-file deletion does not stop RPC or heartbeats. Restore and deletion of one archive are serialized; independent archives remain independent. A deletion is acknowledged only after its files and index entry are removed. On failure, inspect the reported ID before retrying: files may already be absent while the index preserves unfinished cleanup. Listing never finishes that deletion implicitly.

Kubernetes

Nodes (infrastructure machines)

Register physical / virtual machines for fleet operations and remote deploys. Different from remote daemons — nodes are SSH targets that may or may not run a daemon.

Use omnitron --json node list for fleet automation. Each node includes its full ID, endpoints, runtime, tags and the most recent status. A null status means the controller has no observation yet. The same command works against older controllers.

For SSH key authentication, the node registry keeps a file reference or an encrypted private key. Reading or editing a node shows Keep stored key for an inline key. Changing a node's host, port or SSH account requires an explicit credential choice. Existing inline keys move to the vault during startup. If the vault is unavailable, the original credential is retained for recovery and the controller reports the pending migration.

node reconcile is read-only unless --restore is supplied. It compares retained source samples with PostgreSQL using the original timestamp, value and normalized labels. Matching samples are reused; missing samples are restored through Titan's durable storage transaction. The source accepts proof only after that commit.

omnitron --json node reconcile edge-1
omnitron --json node reconcile edge-1 --restore --pages 20

--pages defaults to 1 and accepts 1..2000; a page contains at most 256 entries and 2 MiB. JSON output includes counts, unresolved source rows and a continuation object. Resume its after, through and epoch using --after, --through and --epoch. After a failed acknowledgement, retry the same cursor. Repeating a completed repair makes no new samples.

The source must support the reconciliation protocol. An older node with no unverified entries answers as a no-op. For an older node with gaps, prepare a compatible update or a reviewed maintenance worker; the command does not silently execute SSH code. See historical reconciliation.

omnitron node add flags

With --ssh-secret-stdin, an interactive terminal displays a masked prompt: Enter submits the password or key passphrase, and Ctrl+C cancels without changing the node. A pipe is read until EOF; one final LF/CRLF is removed, while spaces are preserved. Input is limited to 64 KiB. Secrets are never command-line arguments.

omnitron node add --name production --host server.example.org \
  --ssh-key ~/.ssh/id_ed25519 --ssh-secret-stdin
omnitron node update production --ssh-secret-stdin

On node update, the secret belongs to the node's registered authentication method unless --ssh-auth explicitly changes it. Thus an existing key-authenticated node receives a key passphrase without needing to repeat --ssh-auth key.

Webapp (console UI)

All five commands support global --json. Build output includes distPath and durationMs; the builder's subprocess output is captured. Start reports observed running, healthy, port, url and autoStart; stop reports stopped and autoStart. A null autoStart means no saved daemon configuration existed to update. Open waits for the system browser launcher and reports browserOpened and whether it started the Console first.

Lifecycle errors distinguish outcome: "not-started" (for example, the daemon is absent), "unknown" (the container operation failed and may have made partial changes), and "applied" (the Console action completed, but a subsequent step, such as saving autostart preference or opening a browser, failed). An applied outcome is not a rollback. Inspect omnitron --json webapp status before retrying.

Knowledge base (MCP)

Exit codes

Combined with --json output, this is enough to drive any CI or agent pipeline.

Tips

• omnitron --json status | jq ... — pipe to jq for any scriptable check.

• omnitron logs my-app -f --grep error — narrow follow on a hot pattern.

• omnitron exec my-app users findById '"u_42"' — note the quoted JSON string for the arg.

• omnitron status before up — checking what's already running prevents accidental double-start.

• omnitron --json in agent workflows — never parse the styled output; it's not stable.

See also

• Architecture — what each component does

• Daemon — what up / down actually do

• Services reference — the RPC surface every CLI command targets

• Console — the GUI equivalent

Releases

omnitron release builds and verifies artifacts from project and runtime commits. See Releases and verification for build, attestation, artifact-store transfer and retention commands.

Use omnitron release status <project> to see current stack states, node reachability and application counts beside their last recorded deployments. --json carries the same evidence. Exit 2 signals an incomplete answer, including an unreported current release identity; it must not be treated as "nothing deployed". release list and show describe controller artifacts and audit history, not a claim that those artifacts are currently running.

omnitron release status <project> --inspect-installed adds an explicit read-only inspection of each stack's retained archives, bootstrap and reported framework build. It does not restart applications or change their launch evidence. Archive SHA-256 and byte count identify every matching catalog release; modifications, ambiguous matches and unavailable node backends stay visible. JSON keeps these results in installed, separate from verified launch receipts and audit history. A matching bootstrap is a partial check, not proof of all installed or loaded code. The operation is available to viewers and bounded by node, archive and time limits.

Additional operational commands

Account cleanup modes belong to the project's tool: --leftovers performs a census, --leftovers rehearse previews removal in rolled-back transactions, and --leftovers apply requests committed removal. Inspect the census and project policy before applying cleanup.

Fleet recovery and host ingress

omnitron fleet recover <node> <receipt> asks the selected node to recover a bundle activation from its receipt. Use the receipt produced by the relevant operation; recovery is not a fresh release or a command to deploy arbitrary files.

omnitron infra ingress manages host-owned ingress independently of application stacks. status reads configured and observed generation; recover recovers an interrupted publication. Manifest-driven operations use the expected generation for compare-and-swap checks, so a stale plan cannot overwrite a newer owner state. Consult omnitron infra ingress --help for the available operations and supply the JSON manifest belonging to the intended host and owner.

For a controlled maintenance window, preserve the owner's domains, TLS and reserved ports and use [{ "path": "/", "deny": 503 }] for each route's locations. Validate and apply before changing its services or static mounts. HTTPS returns 503, Retry-After: 60 and Cache-Control: no-store; other owners keep their routes. Retain the previous generation and maintenance owner revision for infra ingress restore. Open traffic only after private readiness checks pass. Restoring ingress changes routing, not application processes or data. Maintenance persists across daemon restarts until an explicit publication or restoration.

Console access and interactive SSH

omnitron access and omnitron terminal use the same services and policy as Console.

omnitron access show
omnitron access users
omnitron access create observer --role viewer --password-stdin
omnitron access role <id> operator --expect viewer
omnitron access revoke <id>
omnitron node ssh-key edge-1
omnitron node ssh edge-1
omnitron terminal list
omnitron terminal close <id>

See Console access for password input, account pagination and atomic role changes. See SSH workspace for independently verified host pins, session ownership and Ctrl+] disconnect.

File workspace

omnitron files shares the audited Console file service. Commands include roots, archive-formats, list, stat, read, save, mkdir, rename, trash, trash-list, restore, copy, transfers, pause, resume, cancel, upload, download, upload-status, upload-finish and upload-abort. All support --json; checked mutations use an observed --version. For one directory, copy --archive-format <format> --archive-output archive|directory --archive-name <name> chooses an archive file or restored folder explicitly; archive-formats --node <source> --to-node <destination> probes the node pair. See File workspace for flags, examples, ownership, limits and recovery.

Stack stop/status JSON includes a durable stopOperation with per-node confirmation. A lost reply is unconfirmed, and CLI exits unsuccessfully until every required node acknowledges the stop. See confirmed stack stops for retries, existing allocation bindings and upgrade prerequisites.