---
id: "omnitron/orchestrator"
title: "Orchestrator"
description: "The per-app launch pipeline — bootstrap, fork, watch, hand-off."
---

# Orchestrator

The orchestrator is the daemon's per-app launch pipeline. The
daemon decides **when** to start an app; the orchestrator handles
**how** — locate the bootstrap, compile TypeScript if needed,
resolve dependencies, fork the right child process(es), wire DI,
attach a file watcher in dev, supervise crashes, and tear down
cleanly.

Verified against `src/orchestrator/`:

**orchestrator/**

- `orchestrator.service.ts` — top-level service
- `app-handle.ts` — per-app supervisor state
- `bootstrap-loader.ts` — locates + loads bootstrap.{js,ts}
- `bootstrap-process.ts` — per-app bootstrap entrypoint (classic)
- `module-worker-process.ts` — per-process entrypoint (worker mode)
- `classic-launcher.ts` — fork classic-mode bootstrap
- `ts-compiler.ts` — JIT TypeScript compile
- `file-watcher.ts` — hot reload in dev
- `dependency-resolver.ts` — startup ordering by topology
- `service-router.ts` — RPC routing to apps
- `process-janitor.ts` — orphan + zombie reap
- `build-service.ts` — production bundle / tarball

## Two launch modes

The orchestrator supports two distinct app-launch shapes. The
mode is determined by the app's `defineSystem(...)` shape and the
`bootstrap.{js,ts}` exports.

### Classic mode

A single child process runs the app's `bootstrap.{js,ts}` end-to-
end. The bootstrap is responsible for creating every
`Application`, every `@Module`, and any internal sub-process
management.

| Trait           | Behaviour                                          |
| --------------- | -------------------------------------------------- |
| Forks           | 1 per app                                          |
| Memory          | All Titan modules in one process                   |
| Boot time       | Higher — loads everything                          |
| Use case        | Legacy apps, monolith-style services               |
| Crash semantics | App-wide; whole bootstrap dies, daemon restarts it |

### Module-worker mode (default for new apps)

One child process per `IProcessEntry` declared in
`defineSystem({processes: [...]})`. Each child imports **only**
that process's module file and runs `Application.create(module)`
directly.

| Trait           | Behaviour                                                       |
| --------------- | --------------------------------------------------------------- |
| Forks           | N — one per `IProcessEntry` (× `instances`)                     |
| Memory          | Per-process minimum — only what that module uses                |
| Boot time       | Lower per process; total may be similar but parallelisable      |
| Use case        | Modern multi-process apps, worker pools, isolation requirements |
| Crash semantics | Per-process; daemon restarts just the failed process            |

The orchestrator auto-selects the mode based on whether the
bootstrap exports a `defineSystem()` result with `processes: [...]`
(module-worker mode) or a classic single-app module (classic mode).

## App handle

`AppHandle` is the per-app supervisor record kept inside the
orchestrator. It carries:

```typescript
class AppHandle {
  readonly entry:      IEcosystemAppEntry;
  readonly mode:       'classic' | 'bootstrap';
  get name():          string;        // entry.name
  // Classic mode only — raw ChildProcess
  childProcess:        ChildProcess | null;
  // Bootstrap mode — PM ProcessSupervisor + per-process workers
  supervisor:          ProcessSupervisor | null;
  // Timer for classic-mode crash restart backoff (cleared on explicit stop)
  crashRestartTimer:   ReturnType<typeof setTimeout> | null;
  // Restart accounting
  restarts:            number;
  // Persisted to the daemon state store
  status: 'stopped' | 'starting' | 'online' | 'stopping' | 'errored' | 'crashed';
}
```

One `AppHandle` per app. The handle survives restarts; the
`childProcess` / `supervisor` reference is replaced on each
relaunch.

## Launch pipeline

```mermaid
sequenceDiagram
  participant D as Daemon
  participant Orch as Orchestrator
  participant BL as BootstrapLoader
  participant TSC as TS compiler
  participant DR as DepResolver
  participant CL as Launcher
  participant Child as Child process

  D->>Orch: launch(name)
  Orch->>BL: locate bootstrap path
  alt bootstrap is TypeScript (dev)
    BL->>TSC: compile to ESM
    TSC-->>BL: compiled artefact path
  end
  BL->>BL: import(bootstrap)
  BL-->>Orch: { mode, definition }
  alt mode = bootstrap (module-worker)
    Orch->>DR: resolve startup order
    DR-->>Orch: ordered processes
    loop per process
      Orch->>CL: fork module-worker-process.js
      CL-->>Child: spawn
      Child->>Child: init({bootstrapPath, processName})
      Child->>Child: Application.create(module)
      Child->>Child: onInit → onStart
      Child-->>Orch: ready handshake
    end
  else mode = classic
    Orch->>CL: fork bootstrap-process.js
    CL-->>Child: spawn
    Child->>Child: import(bootstrap)
    Child->>Child: run app
    Child-->>Orch: ready
  end
  Orch-->>D: ProcessInfoDto
```

The ready handshake is mandatory — the daemon does not consider
an app "running" until at least one child reports ready.

## Bootstrap loader — `bootstrap-loader.ts`

Locates and loads the app's bootstrap module. Resolution order:

1. Explicit path from `IEcosystemAppEntry.bootstrap`.
2. `<cwd>/dist/bootstrap.js` (production builds).
3. `<cwd>/src/bootstrap.ts` (dev mode — requires TS compiler).

When the source is TypeScript, it routes through the embedded
TS compiler. When the source is already compiled JS, it `import`s
directly.

## TS compiler — `ts-compiler.ts`

The declaration reader compiles TypeScript bootstrap definitions with esbuild so
plain Node can inspect fresh topology before starting workers. Declaration reads,
worker builds and watched rebuilds share `runtime-package-resolution.ts`:

- TypeScript workspace dependencies are compiled with their relative imports,
  decorators and constructor parameter properties. ESM-only package exports work.
- Compiled SDK and registry packages stay external, preserving their installed
  module and DI identities. Each import retains the dependency selected from its
  original package, including a library's private dependency version.
- External paths are relative to the **physical output directory**. A registered
  project may be a directory symlink; Node resolves that symlink when importing
  the output. Using its lexical depth would produce an invalid dependency path.
  ESM paths also preserve spaces, `#` and `%` through URL encoding.

Qualify a development stack through its registered path, including any symlink,
not only through the physical checkout. Verify an initial launch, a shared-library
edit that triggers a watched rebuild, and a fresh manual restart after disabling
watch. This checks the same compilation policy used by the resident owner.

> This module is `@deprecated` — superseded by `BuildService`,
> which pre-bundles every entry point. It is retained for the
> bootstrap-loader fallback and classic-launcher compatibility.

| Behaviour            | When                                                                                                                              |
| -------------------- | --------------------------------------------------------------------------------------------------------------------------------- |
| Fresh compile        | Cold start; cache miss (by source mtime)                                                                                          |
| Cached output reused | Source `mtime` unchanged since last compile                                                                                       |
| Compile errors       | Thrown — surfaced to the caller                                                                                                   |
| Source maps          | Not emitted (`sourcemap: false`)                                                                                                  |
| Output location      | Next to the source as `<name>.omnitron-compiled.mjs` (gitignored), unlinked after import in the daemon path — no global cache dir |

## Dependency resolver — `dependency-resolver.ts`

Reads the topology graph implied by each process's `topology`
declarations (`expose` / `access`). Computes a startup order such
that **providers** start before **consumers**:

```mermaid
flowchart LR
  Bootstrap[defineSystem processes]
  Bootstrap --> Build[Build topology graph]
  Build --> Sort[Topological sort]
  Sort --> Order[Startup order: providers → consumers]
```

A process declaring `topology: { access: ['SomeService'] }`
waits for the process declaring `topology: { expose: true }` on
`SomeService` to be ready. Cycles are detected and reported as
configuration errors at boot.

## Service router — `service-router.ts`

Routes RPC calls between the orchestrator and individual app
processes. When the daemon's `exec({name, service, method,
args})` arrives, the router:

1. Looks up the `AppHandle` by app name.
2. Finds the process within the app that hosts the requested
   service (via the discovered service map).
3. Proxies the call through that process's Netron transport.
4. Returns the result.

For pool-mode (`instances > 1`), the router dispatches through
the PM pool's `execute()`, which load-balances across instances
using a power-of-two-choices (P2C) policy.

## File watcher — `file-watcher.ts`

In dev mode (or when an app's `watch` is enabled), the
orchestrator runs a per-app `FileWatcher` built on Node's native
`fs.watch({ recursive: true })` (not chokidar). On file change:

```mermaid
flowchart LR
  FS[fs.watch event]
  FS --> Debounce[debounce ~300 ms]
  Debounce --> Restart[orchestrator.restartApp]
```

Heuristics:

- **Bootstrap-mode apps skip `fs.watch` entirely.** Their import
  graph is watched precisely by `BuildService` (esbuild
  `context.watch`), which knows exactly which files contribute to
  the bundle — running `fs.watch` on the same tree on top of that
  produced duplicate, staggered restart triggers (a primary cause
  of dev-mode restart storms).
- **Classic / non-bootstrap apps** get the `fs.watch` watcher,
  which on a change debounces then calls `restartApp`.
- A missing watch directory is logged at **error** (hot reload is
  silently broken otherwise), not warn.

The default debounce is **300 ms**, overridable per app via the
`watch` config; editor saves often produce multiple events, so
the watcher collapses a burst to a single restart per window.

## Classic launcher — `classic-launcher.ts`

Forks `bootstrap-process.js` (a thin wrapper) and waits for ready
handshake. Manages:

- stdio piping for log collection.
- Signal forwarding (`SIGTERM` → child).
- Crash detection via exit-code observation.
- Per-app crash-restart backoff timer.

## Module-worker entrypoint — `module-worker-process.ts`

The generic `@Process`-compatible class the PM worker-runtime
instantiates for each module-based topology entry (one per
`IProcessEntry`). The orchestrator wires these through a PM
`ProcessSupervisor` rather than forking this file directly. Each
worker:

1. Receives its `bootstrapPath` + `processName` via
   `init(bootstrapPath, processName)`.
2. Loads the bootstrap definition to find the matching
   `IProcessEntry`.
3. Imports the process's `module` file.
4. Discovers the `@Service`-decorated provider in the module's
   metadata.
5. Creates a Titan `Application` with the entry's module.
6. Runs lifecycle (`onInit` → `onStart`).
7. Exposes the service's methods over Netron RPC and reports
   ready.

A key invariant: the child imports the **single module file**
declared in `IProcessEntry.module` — not the entire bootstrap —
so an app with 10 processes loads 10 disjoint subsets of code.

## Titan identity and installed application paths

The application and its bootstrap adapter must resolve the same physical Titan installation. Two copies of the same version export different `Container` classes. If dependency injection reports an identity mismatch, check both paths in the startup diagnostic and restore the application's frozen installation.

Both coupled and resident daemons resolve the adapter from the application's current dependencies. A fresh reader runs once per application generation, so single processes and pool workers share the chosen adapter. A damaged adapter or a different Titan installation prevents launch. When only Titan is installed, the application can use an owner adapter that resolves that same Titan.

Coupled mode also accepts a plain bootstrap with neither framework package installed or declared. In that case the daemon supplies its own adapter.

This behavior is advertised as the owner capability `applicationRuntimeResolution: 1`, reported in `execution.ownerCapabilities` by `omnitron status --json` and the shared daemon status API. Older owners can retain Node package resolutions across application restarts. Replacing `node_modules` under the same absolute bootstrap path can then launch the previous adapter against the new application. Updating only the backend does not update the owner's resolver. Restore the previous coherent application SDK or prepare a qualified owner transition with an application pause and rollback.

For a development SDK update, pause only the affected source watchers and applications, perform the normal frozen dependency installation, then restart those applications through Omnitron to rebuild their entry points. Preserve other projects' watcher selections. Verify readiness before restoring the original selection. Qualification must exercise upgrade and rollback under one resident owner; a successful import from a fresh shell does not cover its cached resolution history.

For an installed application, keep each code revision and its dependencies in a distinct physical directory. Preserve the project's registration and state paths. Verify the new launch through the existing owner, repeat the ordinary application start, and test the return to the previous installation. Check job commands separately when a scheduled job launches its own interpreter. An ordinary local development stack runs its working tree.

## Process janitor — `process-janitor.ts`

Background sweep that reaps:

- **Orphans** — child processes whose `AppHandle` no longer
  exists (e.g., daemon was killed mid-spawn).
- **Zombies** — exited children whose exit status hasn't been
  collected.

Runs periodically; never aggressive enough to kill legitimate
children. The janitor's existence means a daemon restart doesn't
leave a leak of orphaned child PIDs.

## Build service — `build-service.ts`

Pre-bundles each process entry point (via esbuild) into the
app's `.omnitron-build/` directory so the daemon imports a single
compiled `bootstrap.js` instead of recompiling source per launch.
This is the runtime bundling path; it also drives the esbuild
import-graph watcher used for bootstrap-mode hot reload.

> The deployment **tarball** for `omnitron deploy build <app>` is
> a separate path: it goes through `ArtifactBuilder`
> (`src/project/artifact-builder.ts`), which runs the app's
> `pnpm build`, resolves workspace deps, and writes
> `<projectRoot>/.omnitron/artifacts/<app>-<version>.tar.gz`. The
> CLI reports the artifact path, version, size, and checksum.

## Reload semantics

`omnitron reload <app>` triggers a **graceful** restart:

- In **module-worker mode**: workers cycle one-by-one; pool size
  is maintained throughout; no request loss.
- In **classic mode**: a new bootstrap is forked side-by-side; the
  old one is signalled to drain (`SIGTERM`), the new one takes
  over once it reports ready.

Reload differs from restart: restart stops everything first,
incurring a downtime window; reload keeps capacity online
throughout.

## Watch control (operator API)

The `OmnitronDaemon` service exposes:

| Method                 | Effect                                      |
| ---------------------- | ------------------------------------------- |
| `enableWatch({apps?})` | Start watching listed apps (all if omitted) |
| `disableWatch()`       | Stop all file watchers                      |
| `getWatchStatus()`     | `{ enabled, apps: [{name, directory}] }`    |

The CLI surfaces these implicitly through the `--no-watch` flag
on `omnitron up` — at runtime, you can toggle without restart.

## Per-app observability

Each launched app reports:

- **stdio logs** → ingested by the daemon's log-collector service.
- **Process metrics** are sampled by the daemon. Worker samples reach the controller through the authenticated replication outbox.
- **Health probes** → pulled by the daemon's health scheduler.
- **Ready / shutdown** → IPC handshake on the fork channel.

The orchestrator instruments every fork so these signals are
automatic — apps don't have to wire anything explicitly beyond
loading `titan-health` / `titan-metrics` modules.

## Failure recovery

| Symptom                            | Orchestrator response                                            |
| ---------------------------------- | ---------------------------------------------------------------- |
| Child exits non-zero               | Apply restart policy; backoff; persist `restarts++`              |
| Child times out on ready handshake | Treat as start failure; backoff; retry up to `maxRestarts`       |
| Container identity mismatch        | Surface error; mark app `crashed`; no auto-retry                 |
| TS compile error                   | Surface error; if dev mode, watcher will re-attempt on next save |
| Bootstrap module not found         | Mark `crashed`; surface error from the loader                    |
| Janitor finds orphan               | `SIGKILL` the orphan; log                                        |

## Anti-patterns

- **Single huge bootstrap with many `@Module`s in classic mode.**
  Loads everything in one process. Prefer module-worker mode with
  one process per concern.
- **Poking the daemon state DB to remove a crashed app.** Use
  `omnitron stop project/stack/app` to address the affected application.
- **File watcher in production.** Wastes CPU and risks
  accidentally hot-reloading prod on a config push. Production
  ecosystems should set `watch.enabled: false`.
- **Cyclic topology declarations.** The dependency resolver
  detects cycles and refuses to start. Refactor to break the
  cycle (often: introduce a shared interface module that both
  sides import).
- **Mixing classic and worker mode in one app.** Pick one. The
  orchestrator can run both modes side-by-side, but only across
  apps — within one app, the mode is global.

## See also

- [Daemon](/docs/omnitron/daemon) — what invokes the orchestrator
- [Architecture](/docs/omnitron/architecture) — where the orchestrator fits
- [titan-pm](/docs/titan/modules/pm) — the process manager the
  orchestrator builds on
- [Services reference](/docs/omnitron/services-reference) — `OmnitronDaemon.exec`
  routes through the service router
- [CLI](/docs/omnitron/cli) — `start`, `stop`, `reload`, `restart`, `scale`

## Stop acknowledgements

An application is marked `stopped` only after its supervisor, classic process group
and topology pools confirm termination. Failed pools retain their handles for
recovery. The classic launcher uses Titan's process termination helper; a sent
signal or expired wait cannot become a successful stop.

Infrastructure teardown reports failed containers and retains stack ownership and
port reservations until removal succeeds. The Xec Docker adapter resolves a name
to an immutable container ID and verifies absence after removal. A replaced name,
unavailable daemon, or denied request is an explicit failure.

Omnitron 0.4.9 uses Xec 0.12.0 from the registry, with the exact dependency graph
recorded in its lockfile. The Docker adapter preserves the requested stop grace
period and allows time for the
transport acknowledgement. An expired command deadline does not prove that the
container stopped; inspect its state before retrying or escalating.

RPC service withdrawal is part of the acknowledgement. Failed withdrawal retains
its router for retry while process termination still runs. Destroyed pools reject
late background spawns; destruction is terminal.

The owner stores a FULL-durability SQLite fence per project/stack. It binds the stop
to exact handle generations and an operation identity. Automatic resume cannot
clear a deliberate stop. See [confirmed stack stops](/docs/omnitron/projects#confirmed-stack-stops)
for controller/node receipts, rollout prerequisites and remaining allocation limits.