Orchestrator

The orchestrator is the daemon's per-app launch pipeline. The daemon decides when to start an app; the orchestrator handles how — locate the bootstrap, compile TypeScript if needed, resolve dependencies, fork the right child process(es), wire DI, attach a file watcher in dev, supervise crashes, and tear down cleanly.

Verified against src/orchestrator/:

Two launch modes

The orchestrator supports two distinct app-launch shapes. The mode is determined by the app's defineSystem(...) shape and the bootstrap.{js,ts} exports.

Classic mode

A single child process runs the app's bootstrap.{js,ts} end-to- end. The bootstrap is responsible for creating every Application, every @Module, and any internal sub-process management.

Module-worker mode (default for new apps)

One child process per IProcessEntry declared in defineSystem({processes: [...]}). Each child imports only that process's module file and runs Application.create(module) directly.

The orchestrator auto-selects the mode based on whether the bootstrap exports a defineSystem() result with processes: [...] (module-worker mode) or a classic single-app module (classic mode).

App handle

AppHandle is the per-app supervisor record kept inside the orchestrator. It carries:

class AppHandle {
  readonly entry:      IEcosystemAppEntry;
  readonly mode:       'classic' | 'bootstrap';
  get name():          string;        // entry.name
  // Classic mode only — raw ChildProcess
  childProcess:        ChildProcess | null;
  // Bootstrap mode — PM ProcessSupervisor + per-process workers
  supervisor:          ProcessSupervisor | null;
  // Timer for classic-mode crash restart backoff (cleared on explicit stop)
  crashRestartTimer:   ReturnType<typeof setTimeout> | null;
  // Restart accounting
  restarts:            number;
  // Persisted to the daemon state store
  status: 'stopped' | 'starting' | 'online' | 'stopping' | 'errored' | 'crashed';
}

One AppHandle per app. The handle survives restarts; the childProcess / supervisor reference is replaced on each relaunch.

Launch pipeline

sequenceDiagram
  participant D as Daemon
  participant Orch as Orchestrator
  participant BL as BootstrapLoader
  participant TSC as TS compiler
  participant DR as DepResolver
  participant CL as Launcher
  participant Child as Child process

  D->>Orch: launch(name)
  Orch->>BL: locate bootstrap path
  alt bootstrap is TypeScript (dev)
    BL->>TSC: compile to ESM
    TSC-->>BL: compiled artefact path
  end
  BL->>BL: import(bootstrap)
  BL-->>Orch: { mode, definition }
  alt mode = bootstrap (module-worker)
    Orch->>DR: resolve startup order
    DR-->>Orch: ordered processes
    loop per process
      Orch->>CL: fork module-worker-process.js
      CL-->>Child: spawn
      Child->>Child: init({bootstrapPath, processName})
      Child->>Child: Application.create(module)
      Child->>Child: onInit → onStart
      Child-->>Orch: ready handshake
    end
  else mode = classic
    Orch->>CL: fork bootstrap-process.js
    CL-->>Child: spawn
    Child->>Child: import(bootstrap)
    Child->>Child: run app
    Child-->>Orch: ready
  end
  Orch-->>D: ProcessInfoDto

The ready handshake is mandatory — the daemon does not consider an app "running" until at least one child reports ready.

Bootstrap loader — bootstrap-loader.ts

Locates and loads the app's bootstrap module. Resolution order:

1. Explicit path from IEcosystemAppEntry.bootstrap.

2. <cwd>/dist/bootstrap.js (production builds).

3. <cwd>/src/bootstrap.ts (dev mode — requires TS compiler).

When the source is TypeScript, it routes through the embedded TS compiler. When the source is already compiled JS, it imports directly.

TS compiler — ts-compiler.ts

The declaration reader compiles TypeScript bootstrap definitions with esbuild so plain Node can inspect fresh topology before starting workers. Declaration reads, worker builds and watched rebuilds share runtime-package-resolution.ts:

• TypeScript workspace dependencies are compiled with their relative imports, decorators and constructor parameter properties. ESM-only package exports work.

• Compiled SDK and registry packages stay external, preserving their installed module and DI identities. Each import retains the dependency selected from its original package, including a library's private dependency version.

• External paths are relative to the physical output directory. A registered project may be a directory symlink; Node resolves that symlink when importing the output. Using its lexical depth would produce an invalid dependency path. ESM paths also preserve spaces, # and % through URL encoding.

Qualify a development stack through its registered path, including any symlink, not only through the physical checkout. Verify an initial launch, a shared-library edit that triggers a watched rebuild, and a fresh manual restart after disabling watch. This checks the same compilation policy used by the resident owner.

This module is @deprecated — superseded by BuildService, which pre-bundles every entry point. It is retained for the bootstrap-loader fallback and classic-launcher compatibility.

Dependency resolver — dependency-resolver.ts

Reads the topology graph implied by each process's topology declarations (expose / access). Computes a startup order such that providers start before consumers:

flowchart LR
  Bootstrap[defineSystem processes]
  Bootstrap --> Build[Build topology graph]
  Build --> Sort[Topological sort]
  Sort --> Order[Startup order: providers → consumers]

A process declaring topology: { access: ['SomeService'] } waits for the process declaring topology: { expose: true } on SomeService to be ready. Cycles are detected and reported as configuration errors at boot.

Service router — service-router.ts

Routes RPC calls between the orchestrator and individual app processes. When the daemon's exec({name, service, method, args}) arrives, the router:

1. Looks up the AppHandle by app name.

2. Finds the process within the app that hosts the requested service (via the discovered service map).

3. Proxies the call through that process's Netron transport.

4. Returns the result.

For pool-mode (instances > 1), the router dispatches through the PM pool's execute(), which load-balances across instances using a power-of-two-choices (P2C) policy.

File watcher — file-watcher.ts

In dev mode (or when an app's watch is enabled), the orchestrator runs a per-app FileWatcher built on Node's native fs.watch({ recursive: true }) (not chokidar). On file change:

flowchart LR
  FS[fs.watch event]
  FS --> Debounce[debounce ~300 ms]
  Debounce --> Restart[orchestrator.restartApp]

Heuristics:

• Bootstrap-mode apps skip fs.watch entirely. Their import graph is watched precisely by BuildService (esbuild context.watch), which knows exactly which files contribute to the bundle — running fs.watch on the same tree on top of that produced duplicate, staggered restart triggers (a primary cause of dev-mode restart storms).

• Classic / non-bootstrap apps get the fs.watch watcher, which on a change debounces then calls restartApp.

• A missing watch directory is logged at error (hot reload is silently broken otherwise), not warn.

The default debounce is 300 ms, overridable per app via the watch config; editor saves often produce multiple events, so the watcher collapses a burst to a single restart per window.

Classic launcher — classic-launcher.ts

Forks bootstrap-process.js (a thin wrapper) and waits for ready handshake. Manages:

• stdio piping for log collection.

• Signal forwarding (SIGTERM → child).

• Crash detection via exit-code observation.

• Per-app crash-restart backoff timer.

Module-worker entrypoint — module-worker-process.ts

The generic @Process-compatible class the PM worker-runtime instantiates for each module-based topology entry (one per IProcessEntry). The orchestrator wires these through a PM ProcessSupervisor rather than forking this file directly. Each worker:

1. Receives its bootstrapPath + processName via init(bootstrapPath, processName).

2. Loads the bootstrap definition to find the matching IProcessEntry.

3. Imports the process's module file.

4. Discovers the @Service-decorated provider in the module's metadata.

5. Creates a Titan Application with the entry's module.

6. Runs lifecycle (onInit → onStart).

7. Exposes the service's methods over Netron RPC and reports ready.

A key invariant: the child imports the single module file declared in IProcessEntry.module — not the entire bootstrap — so an app with 10 processes loads 10 disjoint subsets of code.

Titan identity and installed application paths

The application and its bootstrap adapter must resolve the same physical Titan installation. Two copies of the same version export different Container classes. If dependency injection reports an identity mismatch, check both paths in the startup diagnostic and restore the application's frozen installation.

Both coupled and resident daemons resolve the adapter from the application's current dependencies. A fresh reader runs once per application generation, so single processes and pool workers share the chosen adapter. A damaged adapter or a different Titan installation prevents launch. When only Titan is installed, the application can use an owner adapter that resolves that same Titan.

Coupled mode also accepts a plain bootstrap with neither framework package installed or declared. In that case the daemon supplies its own adapter.

This behavior is advertised as the owner capability applicationRuntimeResolution: 1, reported in execution.ownerCapabilities by omnitron status --json and the shared daemon status API. Older owners can retain Node package resolutions across application restarts. Replacing node_modules under the same absolute bootstrap path can then launch the previous adapter against the new application. Updating only the backend does not update the owner's resolver. Restore the previous coherent application SDK or prepare a qualified owner transition with an application pause and rollback.

For a development SDK update, pause only the affected source watchers and applications, perform the normal frozen dependency installation, then restart those applications through Omnitron to rebuild their entry points. Preserve other projects' watcher selections. Verify readiness before restoring the original selection. Qualification must exercise upgrade and rollback under one resident owner; a successful import from a fresh shell does not cover its cached resolution history.

For an installed application, keep each code revision and its dependencies in a distinct physical directory. Preserve the project's registration and state paths. Verify the new launch through the existing owner, repeat the ordinary application start, and test the return to the previous installation. Check job commands separately when a scheduled job launches its own interpreter. An ordinary local development stack runs its working tree.

Process janitor — process-janitor.ts

Background sweep that reaps:

• Orphans — child processes whose AppHandle no longer exists (e.g., daemon was killed mid-spawn).

• Zombies — exited children whose exit status hasn't been collected.

Runs periodically; never aggressive enough to kill legitimate children. The janitor's existence means a daemon restart doesn't leave a leak of orphaned child PIDs.

Build service — build-service.ts

Pre-bundles each process entry point (via esbuild) into the app's .omnitron-build/ directory so the daemon imports a single compiled bootstrap.js instead of recompiling source per launch. This is the runtime bundling path; it also drives the esbuild import-graph watcher used for bootstrap-mode hot reload.

The deployment tarball for omnitron deploy build <app> is a separate path: it goes through ArtifactBuilder (src/project/artifact-builder.ts), which runs the app's pnpm build, resolves workspace deps, and writes <projectRoot>/.omnitron/artifacts/<app>-<version>.tar.gz. The CLI reports the artifact path, version, size, and checksum.

Reload semantics

omnitron reload <app> triggers a graceful restart:

• In module-worker mode: workers cycle one-by-one; pool size is maintained throughout; no request loss.

• In classic mode: a new bootstrap is forked side-by-side; the old one is signalled to drain (SIGTERM), the new one takes over once it reports ready.

Reload differs from restart: restart stops everything first, incurring a downtime window; reload keeps capacity online throughout.

Watch control (operator API)

The OmnitronDaemon service exposes:

The CLI surfaces these implicitly through the --no-watch flag on omnitron up — at runtime, you can toggle without restart.

Per-app observability

Each launched app reports:

• stdio logs → ingested by the daemon's log-collector service.

• Process metrics are sampled by the daemon. Worker samples reach the controller through the authenticated replication outbox.

• Health probes → pulled by the daemon's health scheduler.

• Ready / shutdown → IPC handshake on the fork channel.

The orchestrator instruments every fork so these signals are automatic — apps don't have to wire anything explicitly beyond loading titan-health / titan-metrics modules.

Failure recovery

Anti-patterns

• Single huge bootstrap with many @Modules in classic mode. Loads everything in one process. Prefer module-worker mode with one process per concern.

• Poking the daemon state DB to remove a crashed app. Use omnitron stop project/stack/app to address the affected application.

• File watcher in production. Wastes CPU and risks accidentally hot-reloading prod on a config push. Production ecosystems should set watch.enabled: false.

• Cyclic topology declarations. The dependency resolver detects cycles and refuses to start. Refactor to break the cycle (often: introduce a shared interface module that both sides import).

• Mixing classic and worker mode in one app. Pick one. The orchestrator can run both modes side-by-side, but only across apps — within one app, the mode is global.

See also

• Daemon — what invokes the orchestrator

• Architecture — where the orchestrator fits

• titan-pm — the process manager the orchestrator builds on

• Services reference — OmnitronDaemon.exec routes through the service router

• CLI — start, stop, reload, restart, scale

Stop acknowledgements

An application is marked stopped only after its supervisor, classic process group and topology pools confirm termination. Failed pools retain their handles for recovery. The classic launcher uses Titan's process termination helper; a sent signal or expired wait cannot become a successful stop.

Infrastructure teardown reports failed containers and retains stack ownership and port reservations until removal succeeds. The Xec Docker adapter resolves a name to an immutable container ID and verifies absence after removal. A replaced name, unavailable daemon, or denied request is an explicit failure.

Omnitron 0.4.9 uses Xec 0.12.0 from the registry, with the exact dependency graph recorded in its lockfile. The Docker adapter preserves the requested stop grace period and allows time for the transport acknowledgement. An expired command deadline does not prove that the container stopped; inspect its state before retrying or escalating.

RPC service withdrawal is part of the acknowledgement. Failed withdrawal retains its router for retry while process termination still runs. Destroyed pools reject late background spawns; destruction is terminal.

The owner stores a FULL-durability SQLite fence per project/stack. It binds the stop to exact handle generations and an operation identity. Automatic resume cannot clear a deliberate stop. See confirmed stack stops for controller/node receipts, rollout prerequisites and remaining allocation limits.