Kubernetes

For production at scale. The pattern: one Deployment per Titan app, one Statefulset for the daemon (with its persistent state), managed Postgres + Redis as separate clusters.

Topology

flowchart TB
  Ingress[Ingress / Gateway]
  Ingress --> APIsvc[Service: api]
  Ingress --> WebappSvc[Service: webapp]

  subgraph apps["Apps (one Deployment each)"]
    APIsvc --> ApiDeploy[Deployment: api<br/>replicas: 3]
    WebappSvc --> WebappDeploy[Deployment: webapp]
  end

  subgraph daemon["Omnitron daemon (StatefulSet)"]
    DaemonSts[Daemon Pod<br/>PVC: ~/.omnitron]
  end

  subgraph infra["Managed infrastructure"]
    PG[(Postgres<br/>managed)]
    Redis[(Redis<br/>managed)]
    S3[(S3<br/>managed)]
  end

  ApiDeploy --> PG
  ApiDeploy --> Redis
  ApiDeploy --> S3
  ApiDeploy -. report status .-> DaemonSts
  WebappDeploy --> DaemonSts

In Kubernetes you have two valid patterns:

Pattern	When
Daemon-per-pod (Omnitron supervises the apps in the pod)	Small clusters; one or two services per pod
K8s-supervises-apps (Omnitron is the control plane only)	Large clusters; one Deployment per Titan app; Omnitron daemon as a separate StatefulSet for control/inspection

This page covers pattern 2 — the production default.

Per-app Deployment

# k8s/api/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
  namespace: platform
spec:
  replicas: 3
  selector:
    matchLabels: { app: api }
  template:
    metadata:
      labels: { app: api }
    spec:
      containers:
        - name: api
          image: registry.example.com/platform-api:v1.4.2
          ports:
            - { name: http, containerPort: 3001 }
          env:
            - { name: NODE_ENV, value: production }
            - name: DATABASE_URL
              valueFrom: { secretKeyRef: { name: api-secrets, key: database-url } }
            - name: REDIS_URL
              valueFrom: { secretKeyRef: { name: api-secrets, key: redis-url } }
            - name: JWT_SECRET
              valueFrom: { secretKeyRef: { name: api-secrets, key: jwt-secret } }
          resources:
            requests: { cpu: '500m', memory: '512Mi' }
            limits:   { cpu: '2000m', memory: '2Gi' }
          livenessProbe:
            httpGet: { path: /health, port: http }
            initialDelaySeconds: 30
            periodSeconds:       10
            timeoutSeconds:      5
            failureThreshold:    3
          readinessProbe:
            httpGet: { path: /health, port: http }
            initialDelaySeconds: 5
            periodSeconds:       3
            timeoutSeconds:      3
            failureThreshold:    2
          startupProbe:
            httpGet: { path: /health, port: http }
            initialDelaySeconds: 0
            periodSeconds:       2
            timeoutSeconds:      2
            failureThreshold:    60      # 2 min grace for cold start
          lifecycle:
            preStop:
              exec:
                command: ['sh', '-c', 'sleep 10']    # let LB drain
      terminationGracePeriodSeconds: 45
---
apiVersion: v1
kind: Service
metadata:
  name: api
  namespace: platform
spec:
  selector: { app: api }
  ports:
    - { name: http, port: 80, targetPort: http }

Key points:

Three probes: liveness, readiness, startup. The Titan Netron HTTP transport serves /health on each process's HTTP port — it returns 200 with {"status":"online"} once the transport is up, 503 otherwise. That single endpoint backs all three probes here.
- Liveness → /health. Kill if it fails persistently.
- Readiness → /health. Stop sending traffic when failing; resume when passing.
- Startup → /health with longer failureThreshold — masks liveness during cold-start (especially when eager-loading heavy services).
⚠️ NEEDS REWRITE — there are no /healthz or /readyz HTTP routes. Richer health (the live() / ready() probes from titan-health, dependency checks) is exposed over Netron RPC, not HTTP, so it isn't directly reachable by a k8s httpGet probe. If you need a true readiness signal that reflects DB/Redis health, add a customRoutes HTTP handler in the process's transports.http config that calls the health service, and point readinessProbe at it.
preStop sleep 10 s — lets the load balancer's endpoint list update before SIGTERM. Avoids 502s during pod rotation.
terminationGracePeriodSeconds: 45 — > app's shutdown.timeout. Otherwise k8s SIGKILLs mid-drain.

Omnitron daemon as StatefulSet

The daemon needs persistent state (~/.omnitron/) and a stable identity:

# k8s/daemon/statefulset.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: omnitron
  namespace: platform
spec:
  serviceName: omnitron
  replicas: 1
  selector:
    matchLabels: { app: omnitron }
  template:
    metadata:
      labels: { app: omnitron }
    spec:
      containers:
        - name: omnitron
          image: registry.example.com/platform-omnitron:v1.4.2
          ports:
            - { name: tcp,     containerPort: 9700 }   # fleet RPC
            - { name: http,    containerPort: 9801 }   # Netron HTTP RPC
            - { name: ws,      containerPort: 9802 }   # Netron WebSocket
            - { name: metrics, containerPort: 9803 }   # /metrics + /healthz
          env:
            # Daemon state lives under $HOME/.omnitron — point HOME at the PVC.
            - { name: HOME,     value: /var/lib/omnitron }
            - { name: NODE_ENV, value: production }
          volumeMounts:
            - { name: state, mountPath: /var/lib/omnitron }
          resources:
            requests: { cpu: '200m', memory: '256Mi' }
            limits:   { cpu: '1000m', memory: '1Gi' }
  volumeClaimTemplates:
    - metadata: { name: state }
      spec:
        accessModes: ['ReadWriteOnce']
        resources: { requests: { storage: 5Gi } }
        storageClassName: 'fast-ssd'
---
apiVersion: v1
kind: Service
metadata:
  name: omnitron
  namespace: platform
spec:
  selector: { app: omnitron }
  ports:
    - { name: tcp,     port: 9700, targetPort: tcp }
    - { name: http,    port: 9801, targetPort: http }
    - { name: ws,      port: 9802, targetPort: ws }
    - { name: metrics, port: 9803, targetPort: metrics }

The daemon does not serve the Console UI itself — that's the separate webapp Deployment (an nginx container that proxies to the daemon's 9801/9802). To reach the daemon's RPC directly, operators can kubectl port-forward svc/omnitron 9801; to open the Console, port-forward the webapp service instead.

Ingress

# k8s/ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: platform
  namespace: platform
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt
    nginx.ingress.kubernetes.io/proxy-body-size: '50m'
spec:
  ingressClassName: nginx
  tls:
    - hosts: [api.example.com, app.example.com]
      secretName: platform-tls
  rules:
    - host: api.example.com
      http:
        paths:
          - { path: /, pathType: Prefix, backend: { service: { name: api, port: { number: 80 } } } }
    - host: app.example.com
      http:
        paths:
          - { path: /, pathType: Prefix, backend: { service: { name: webapp, port: { number: 80 } } } }

Two hosts — one for the API (Titan apps), one for the Console webapp (the nginx-served webapp Deployment, not the daemon).

Secrets

# k8s/secrets.yaml — do NOT commit. Use sealed-secrets or external-secrets in real life.
apiVersion: v1
kind: Secret
metadata:
  name: api-secrets
  namespace: platform
type: Opaque
stringData:
  database-url: postgres://platform@platform-pg.svc.cluster.local:5432/platform
  redis-url:    redis://platform-redis.svc.cluster.local:6379
  jwt-secret:   '<set via secret manager>'

Production options:

Sealed Secrets (Bitnami): commit encrypted secrets to git.
External Secrets Operator: sync from AWS Secrets Manager / GCP Secret Manager / Vault.
Pod-level secret manager mount (e.g., AWS Secrets Store CSI Driver).

HorizontalPodAutoscaler

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: api
  namespace: platform
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind:       Deployment
    name:       api
  minReplicas: 2
  maxReplicas: 10
  metrics:
    - type: Resource
      resource: { name: cpu,    target: { type: Utilization, averageUtilization: 70 } }
    - type: Resource
      resource: { name: memory, target: { type: Utilization, averageUtilization: 80 } }

titan-pm's own autoscaler operates at the worker-pool level within a process; HPA operates at the pod level. Use both — HPA for cross-pod scale, titan-pm for cross-worker.

PodDisruptionBudget

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: api
  namespace: platform
spec:
  minAvailable: 2
  selector:
    matchLabels: { app: api }

Keeps at least 2 pods up during voluntary disruptions (node drains, cluster upgrades).

NetworkPolicy

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: api-ingress
  namespace: platform
spec:
  podSelector:
    matchLabels: { app: api }
  policyTypes: [Ingress, Egress]
  ingress:
    - from:
        - namespaceSelector: { matchLabels: { name: ingress-nginx } }
      ports:
        - { protocol: TCP, port: 3001 }
  egress:
    - to:
        - namespaceSelector: { matchLabels: { kubernetes.io/metadata.name: platform } }
    - to:
        - namespaceSelector: { matchLabels: { kubernetes.io/metadata.name: kube-system } }
      ports:
        - { protocol: UDP, port: 53 }

Allow ingress only from the nginx ingress controller; restrict egress to in-namespace + DNS.

Rolling updates

spec:
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge:       1
      maxUnavailable: 0

maxUnavailable: 0 keeps capacity at 100% throughout the rollout. maxSurge: 1 adds one new pod, waits for it to become ready, then terminates an old one.

For high-throughput apps, set maxSurge: 25% for faster rollouts.

Database migrations

Run as an initContainer or pre-deploy Job:

# Pre-deploy Job, runs to completion before pods rotate
apiVersion: batch/v1
kind: Job
metadata:
  name: migrate-{{ .Values.deployVersion }}
  namespace: platform
spec:
  backoffLimit: 1
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: migrate
          image: registry.example.com/platform-api:v1.4.2
          command: ['pnpm', 'omnitron', 'infra', 'migrate']
          env:
            - { name: DATABASE_URL, valueFrom: { secretKeyRef: { name: api-secrets, key: database-url } } }

Wire as a Helm pre-install / pre-upgrade hook.

Webapp deployment

Static assets behind an nginx container:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: webapp
spec:
  replicas: 2
  template:
    spec:
      containers:
        - name: webapp
          image: registry.example.com/platform-webapp:v1.4.2
          ports: [{ name: http, containerPort: 80 }]
          # Built static; nginx serves it

This mirrors what omnitron webapp does on a single host: an nginx:alpine container serving the built Console SPA from /usr/share/nginx/html and proxying /netron/, /ws, /api/health, /api/metrics to the daemon. In k8s, point its upstream at the omnitron Service (9801 HTTP, 9802 WS) instead of host.docker.internal.

Observability

Logs → stdout → cluster log aggregator (Loki / Datadog / ELK).
Metrics → scrape the daemon's Prometheus endpoint on port 9803 (/metrics), which aggregates per-app gauges:
```
# On the omnitron StatefulSet pod template:
annotations:
  prometheus.io/scrape: 'true'
  prometheus.io/port:   '9803'
  prometheus.io/path:   '/metrics'
```
The per-app HTTP transport port (e.g. 3001) also has a /metrics route, but it returns JSON transport stats and is bearer-token gated — it is not Prometheus exposition text, so don't point a scrape config at it. Bind the daemon metrics port to 0.0.0.0 and set a bearer token if Prometheus runs off-host.
Traces → OTel collector as DaemonSet; apps export to it (via titan-telemetry-relay).
Alerts → OmnitronAlerts + PrometheusRules.

Disaster recovery

Backup priorities:

Postgres — managed snapshots + WAL archiving.
Omnitron daemon state (omnitron-data PVC) — snapshot regularly; survival of the daemon is important for the uptime-bar history.
Secrets — backed up via the secret manager itself.

The Postgres data is the only catastrophic loss. Everything else can be reconstructed.

Cost-savings tips

Spot / preemptible for worker pools; on-demand for the daemon.
Memory requests > limits, CPU limits > requests: gives headroom under burst while bin-packing efficiently.
One daemon per cluster, not per app.
titan-cache L1 only for read-heavy apps to avoid Redis hops.