Operations

Running, updating, auditing, and troubleshooting the gateway.

Private maintainer document. This file is intentionally excluded from the lean public source export. Never put credentials, personal data, or release secret values in it.

This runbook has two independent scopes:

1. operation and recovery of the public native release channels; and
2. backup, restore, and incident response for a separately operated self-hosted Compose deployment.

The public native pipeline does not deploy or upgrade the Compose stack. A successful native release is not evidence that a self-hosted deployment was backed up, upgraded, or tested.

Public release-channel operations

The detailed release contract and one-time secret/variable inventory are in RELEASE.md. This section is the operating procedure after that configuration exists.

Authorization boundary

• Do not push, merge, tag, deploy, publish, or change a public repository without Srinath’s explicit authorization for that action.
• A website/legal merge to main is verification-only and cannot deploy skilakmesh.com. The owner-authorized release tag is the deployment decision.
• An authorized vMAJOR.MINOR.PATCH tag push can publish every artifact channel without another approval. Treat the tag push itself as the public release decision.
• Never repair a failed run by locally rebuilding, hand-signing, replacing an object, moving a tag, or attaching an unsealed file.

Once an authorized tag is pushed, normal operation is unattended. The private-beta-signing and public-release GitHub environments must not require human reviewers if that no-intervention behavior is desired. Missing variables, credentials, runners, DNS, or repository permissions fail closed.

Control-plane map

SkilakLabs/releases remains private and outside this map.

Candidate operations

A successful same-repository main CI completion starts .github/workflows/release-candidate.yml. It binds the exact commit and version, calls .github/workflows/native-release.yml, builds portable outputs from those verified native bytes, and seals one manifest.

Check the candidate in this order:

1. Bind candidate identity names the intended main SHA and version.
2. Linux x86-64 final evidence reports checksum, containment, credential-free install, automatic startup, lifecycle, and public release channel.
3. Windows x86-64 final evidence reports valid Authenticode identity, timestamping, checksum, credential-free install, automatic startup, lifecycle, and public release channel.
4. macOS arm64 and x86-64 evidence reports the expected Developer ID team, accepted notarization, stapling, credential-free install, automatic startup, lifecycle, and public release channel.
5. Portable output contains wheel, sdist, install.sh, install.ps1, uninstall.ps1, the public Windows checksum lane, smesh.rb, and collision-free checksum manifests. It does not contact Fulcio or Rekor and contains no public Sigstore bundle.
6. Fresh hosted Linux, macOS arm64, macOS Intel, and Windows jobs execute the exact final portable installers against the accepted native files, prove start/restart health and automatic startup, then prove owned uninstall residue is gone. These jobs run before a candidate can be sealed.
7. Fresh macOS arm64 and Intel jobs install the unchanged Homebrew formula from Homebrew’s own preseeded cache with downloads disabled, and a fresh Linux job installs the exact wheel plus a byte-identical wheel rebuilt offline from the exact sdist. Both routes must pass before sealing.
8. Seal one complete release candidate verifies the complete inventory and uploads the single 90-day candidate artifact.

Do not infer readiness from an intermediate artifact. Unsigned Windows bytes, Mac signing outputs, and isolated platform upload success are not promotable.

The candidate workflow has no manual or branch trigger because its native jobs inherit protected signing credentials. The first live event-chain proof therefore requires an owner-authorized merge to main; if GitHub does not create a candidate for that merge’s CI completion, use one additional reviewed main commit rather than weakening candidate selection.

Tag-promotion operations

After owner authorization, the tag workflow runs these phases:

1. Match tag, version, commit, and one successful main candidate.
2. Before any public mutation, require the core repository to remain private, the market source and tap repositories to remain public, all three default branches to remain main, and the tag commit to remain the exact private main tip.
3. Re-download and re-verify the private seal without rebuilding package bytes. Create and cryptographically verify Linux and macOS checksum provenance against .github/workflows/release.yml@refs/tags/vVERSION, then extend the aggregate checksum file and deterministically create a tag-bound seal. On a rerun, download and re-verify the first tag-provenance-handoff-SHA against the original run ID and attempt recorded in its tag seal instead of creating different Sigstore bundles.
4. Before any R2 write, reject a newer or conflicting latest.json, and byte-check every already-present object under the intended version prefix.
5. Publish immutable versioned R2 keys with manifest.json last.
6. Install and uninstall those public URLs on hosted Linux, Windows, macOS arm64, and macOS Intel runners; install the sealed Homebrew formula on both Mac architectures.
7. Fast-forward the lean public source and stage/compare the exact GitHub draft.
8. Create/compare the Homebrew formula and version tag, then verify the real public tap without GitHub authentication on both Mac architectures.
9. Deploy the exact tag-bound website source through the public-release environment.
10. Re-verify and publish the GitHub draft unchanged.
11. Advance stable installer aliases and latest.json, then compare the direct and skilakmesh.com routes with the sealed files.

All releases share the public-release-promotion concurrency group. Do not cancel an older in-progress promotion to force a newer tag through it. Diagnose the older run first; interleaved writes to latest are deliberately prevented.

Channel health checks

Run these checks without GitHub authentication. Substitute an actually published version; never test a placeholder URL as if it were evidence.

curl --fail --location --proto '=https' --tlsv1.2 \
  https://downloads.skilakmesh.com/releases/vVERSION/manifest.json \
  --output manifest.json

curl --fail --location --proto '=https' --tlsv1.2 \
  https://downloads.skilakmesh.com/latest/install.sh \
  --output install.sh

curl --fail --location --proto '=https' --tlsv1.2 \
  https://downloads.skilakmesh.com/latest/install.ps1 \
  --output install.ps1

curl --fail --location --proto '=https' --tlsv1.2 \
  https://downloads.skilakmesh.com/latest.json \
  --output latest.json

Also require:

• https://skilakmesh.com/download, /terms, /privacy, and /license load;
• a browser on https://skilakmesh.com/download can read the stable manifest under the bucket’s narrow CORS rule and enables only version-specific, immutable download links;
• https://skilakmesh.com/install.sh and /install.ps1 resolve to the same bytes as the direct stable routes;
• the SkilakLabs/smesh source tag and GitHub release exist and agree with the sealed manifest;
• SkilakLabs/homebrew-tap/Formula/smesh.rb names the versioned R2 URLs and sealed architecture hashes; and
• a clean Mac can run brew install skilaklabs/tap/smesh without GitHub auth.

Do not print or paste the entire manifest into an issue by default. Record the version, source SHA, candidate run ID/attempt, manifest SHA-256, channel result, and timestamp.

R2 invariants and recovery

The versioned prefix is immutable. The publisher:

• lists the prefix before writing;
• refuses unexpected keys;
• creates missing objects conditionally;
• byte-compares any existing object and its SHA-256 metadata; and
• creates manifest.json only after the artifact set is complete.

If publication stops midway, rerun the same tag and same sealed candidate. The publisher may complete missing keys. Stop and investigate if an existing key differs, metadata is absent/wrong, or an unexpected object is present. Do not delete or overwrite a conflicting public object as an automated recovery.

latest.json cannot move to an older semantic version. Stable installer aliases can change only through the r2-latest phase after the complete version prefix and downstream public channels have passed. If that phase fails, the immutable version remains usable while the stable alias stays on the prior good release.

Public GitHub and tap recovery

The public-source exporter creates a clean allowlisted tree, not a branch or fork of the private repository. It may fast-forward public main; it must not force-push private history. Existing public tags/releases and tap tags must either match the candidate exactly or cause a failure.

If the GitHub App cannot publish:

1. confirm its installation still covers both public repositories;
2. confirm Contents write and branch/ruleset bypass are still allowed;
3. confirm the configured client ID and private key belong to that installation;
4. rerun the same tag only after the permission issue is corrected; and
5. never substitute a personal token in a checked-in workflow.

Website operations

.github/workflows/website.yml is reusable-only. It has no push, workflow_run, or manual-dispatch trigger. The tag promoter calls it only after the real public Homebrew tap passes, supplies the exact source SHA already bound to the sealed candidate, and waits for the Pages deployment before finalizing the already-staged GitHub draft or advancing R2 latest.

Before an owner-authorized release tag containing website or legal changes:

• run npm test --prefix website;
• run npm run build --prefix website;
• verify the download commands and legal copy against the checked-in source;
• confirm the current BUSL-versus-BSL wording has owner/counsel approval; and
• verify the candidate source SHA is the exact commit the owner intends to tag.

If the Pages deployment fails, the previous site remains the public state. Fix the configuration/source in a reviewed change and produce a new candidate; do not upload an unverified local dist directory manually. The failed deployment prevents both GitHub release finalization and R2 latest promotion.

Runner stewardship

• The Apple Silicon build, signing, and verification roles must remain separate. Only the signing account can access the Developer ID keychain and notary profile. Always let the workflow cleanup and relock steps finish.
• Intel Mac build and final verification use GitHub-hosted macos-15-intel; the protected Apple Silicon signing role signs both architecture handoffs without executing Intel output.
• Windows signing and final verification use clean GitHub-hosted windows-2025; a configured release runner may be used for the unsigned build only.
• The physical Windows VM is an owner acceptance environment, not a signing secret store. Remove temporary autologon, shut it down, and deallocate it when acceptance is complete.
• Linux hosted proof uses Ubuntu 24.04/glibc 2.39. The physical Linux machine is still required for owner launch acceptance.

Release incident response

If a published release may be corrupt, wrongly signed, unexpectedly routed, or legally incorrect:

1. stop promotion work and preserve the workflow run, candidate identity, and public object metadata;
2. do not move/delete the tag or overwrite the immutable version;
3. determine whether latest.json advanced and which public channels completed;
4. if credentials may be exposed, revoke/rotate them at the issuer before any rerun;
5. publish customer mitigation separately if required; and
6. fix the private core source and ship a higher patch version through the full candidate and tag flow.

A release can be yanked from marketing or marked affected, but its immutable evidence must remain available unless legal/security counsel directs a separate documented removal.

Self-hosted runtime scope and posture

The remainder of this runbook covers the self-hosted stack in docker-compose.release.yml. Before production use, execute the backup/restore drill below and record its evidence.

Operate it as a fail-closed product: if scanning, audit writes, attestation, or required health checks fail, stop and fix the system before allowing traffic.

The audit trail is payload-free by design. It records metadata, hashes, verdicts, and bounded response metadata; it does not retain prompts, raw files, or provider payloads. Treat the audit and attestation stores as sensitive anyway.

Service names, ports, and persistent state in the release stack:

• ingress publishes ${SKILAK_BIND_ADDR:-127.0.0.1}:${SKILAK_BIND_PORT:-51737}:4000.
• gateway serves http://localhost:51737/healthz inside the container. The liveness route is deliberately outside the /p/<alias>/ prefix; probing /health/liveliness reaches the catch-all, which resolves the alias first and answers 404 unknown_provider_alias, so the container never goes healthy.
• dashboard publishes 127.0.0.1:${SKILAK_DASHBOARD_PORT:-8080}:8080 and serves http://localhost:8080/health inside the container.
• exporter has no compose health check.
• skilak_data stores /app/data/audit.jsonl and /app/data/file-attestations.sqlite3.
• skilak_export_state stores /app/state/offsets.json when the enterprise profile enables exporter.

Docker Compose may materialize the named volumes with a project prefix. Resolve the actual runtime volume names before backup or restore:

docker volume ls --format '{{.Name}}' | grep '_skilak_data$'
docker volume ls --format '{{.Name}}' | grep '_skilak_export_state$'

If your deployment uses a non-default Compose project name, record the exact volume names used in that environment with the drill evidence.

Run every command in this runbook under bash. zsh — the macOS default shell — does not word-split unquoted parameter expansions, so $COMPOSE_ARGS below reaches Docker as a single argument and every compose command fails with unknown shorthand flag: 'f'.

Resolve the compose entrypoint. Some hosts have only the standalone docker-compose binary even when the plugin is installed, so detect it the same way Makefile does:

COMPOSE="$(docker compose version >/dev/null 2>&1 && echo "docker compose" || echo "docker-compose")"

Set the compose arguments for the deployment you are actually operating:

export COMPOSE_ARGS="-f docker-compose.release.yml"
# If the deployment enables the dashboard and exporter:
# export COMPOSE_ARGS="-f docker-compose.release.yml --profile enterprise"

Every Compose invocation below uses $COMPOSE $COMPOSE_ARGS ….

Backup

Back up skilak_data every time. Back up skilak_export_state whenever the enterprise profile is enabled and exporter is in use.

Prepare the backup directory and record the exact volume names:

export AUDIT_STATE_VOLUME="<actual volume name for skilak_data>"
export EXPORT_STATE_VOLUME="<actual volume name for skilak_export_state>"
export BACKUP_ROOT="<backup directory>"
export STAMP="$(date -u +%Y%m%dT%H%M%SZ)"
mkdir -p "$BACKUP_ROOT/$STAMP"

$BACKUP_ROOT must be writable by the invoking user. Archives written by the root utility container will be root-owned on the host; chown them afterward if required.

Stop the stack before taking a filesystem-level backup:

$COMPOSE $COMPOSE_ARGS down

Archive the audit and attestation state from skilak_data:

docker run --rm --user 0:0 \
  -v "$AUDIT_STATE_VOLUME":/src:ro \
  -v "$BACKUP_ROOT/$STAMP":/dst \
  "$SKILAK_GATEWAY_IMAGE" \
  sh -lc 'cd /src && tar -czf /dst/skilak_data.tgz .'

If exporter is enabled, archive the exporter checkpoint state from skilak_export_state:

docker run --rm --user 0:0 \
  -v "$EXPORT_STATE_VOLUME":/src:ro \
  -v "$BACKUP_ROOT/$STAMP":/dst \
  "$SKILAK_GATEWAY_IMAGE" \
  sh -lc 'cd /src && tar -czf /dst/skilak_export_state.tgz .'

Capture the non-secret deployment inputs selected by the release compose file:

mkdir -p "$BACKUP_ROOT/$STAMP/config"
cp docker-compose.release.yml "$BACKUP_ROOT/$STAMP/config/"
cp "${SKILAK_POLICY_FILE:-./config/skilak.yaml}" "$BACKUP_ROOT/$STAMP/config/"
cp "${SKILAK_EXPORT_FILE:-./config/exports.json}" "$BACKUP_ROOT/$STAMP/config/"
cp DEPLOY.md RELEASE.md docs/OPERATIONS.md "$BACKUP_ROOT/$STAMP/config/"

Record the pinned image digest set exactly as deployed:

cat > "$BACKUP_ROOT/$STAMP/image-digests.txt" <<EOF
SKILAK_INGRESS_IMAGE=$SKILAK_INGRESS_IMAGE
SKILAK_GATEWAY_IMAGE=$SKILAK_GATEWAY_IMAGE
SKILAK_ENTERPRISE_IMAGE=$SKILAK_ENTERPRISE_IMAGE
EOF

Record an environment-variable inventory without values:

cat > "$BACKUP_ROOT/$STAMP/env-inventory.txt" <<'EOF'
SKILAK_INGRESS_IMAGE
SKILAK_GATEWAY_IMAGE
SKILAK_ENTERPRISE_IMAGE
ANTHROPIC_API_KEY
OPENAI_API_KEY
SKILAK_HASH_SALT
SKILAK_EDGE_ATTEST_KEY
SKILAK_POLICY_FILE
SKILAK_EXPORT_FILE
SKILAK_EXPORT_WEBHOOK_TOKEN
SKILAK_DASHBOARD_TOKEN
SKILAK_DASHBOARD_PORT
SKILAK_DENY_ALL_FILES
SKILAK_DENY_ALL_IMAGES
EOF

Restart the stack after the backup:

$COMPOSE $COMPOSE_ARGS up -d --wait --no-build

Frequency:

• Take a fresh backup before every upgrade, rollback, key rotation, or incident response action.
• Run a scheduled backup at least daily while production traffic is present.
• Increase frequency if the acceptable audit data loss window is less than 24 hours.

Protection requirements:

• Encrypt backups at rest. The audit is payload-free, but it still contains sensitive metadata and keyed correlations.
• Store at least one offline or offsite copy outside the primary Docker host.
• Protect backup access with the same or stricter operator controls as the live host.

Restore

Restore only into a stopped stack. The runtime services run as UID/GID 10001:10001; restore the files, then enforce that ownership before start.

Prepare the target host:

export AUDIT_STATE_VOLUME="<actual volume name for skilak_data>"
export EXPORT_STATE_VOLUME="<actual volume name for skilak_export_state>"
export BACKUP_ROOT="<backup directory>"
export STAMP="<backup timestamp>"
$COMPOSE $COMPOSE_ARGS down
docker volume create "$AUDIT_STATE_VOLUME"
docker volume create "$EXPORT_STATE_VOLUME"

Restore skilak_data:

docker run --rm --user 0:0 \
  -v "$AUDIT_STATE_VOLUME":/dst \
  -v "$BACKUP_ROOT/$STAMP":/backup:ro \
  "$SKILAK_GATEWAY_IMAGE" \
  sh -lc 'cd /dst && tar -xzf /backup/skilak_data.tgz'

Restore skilak_export_state if the deployment uses exporter:

docker run --rm --user 0:0 \
  -v "$EXPORT_STATE_VOLUME":/dst \
  -v "$BACKUP_ROOT/$STAMP":/backup:ro \
  "$SKILAK_GATEWAY_IMAGE" \
  sh -lc 'cd /dst && tar -xzf /backup/skilak_export_state.tgz'

Enforce ownership:

docker run --rm --user 0:0 -v "$AUDIT_STATE_VOLUME":/dst "$SKILAK_GATEWAY_IMAGE" sh -lc 'chown -R 10001:10001 /dst'
docker run --rm --user 0:0 -v "$EXPORT_STATE_VOLUME":/dst "$SKILAK_GATEWAY_IMAGE" sh -lc 'chown -R 10001:10001 /dst'

Start the restored stack:

$COMPOSE $COMPOSE_ARGS up -d --wait --no-build

Restore verification drill

Run the drill on a fresh environment after restoring the volumes and before declaring recovery complete.

1. Verify the containers start and the compose wait gate succeeds.

   $COMPOSE $COMPOSE_ARGS up -d --wait --no-build

   PASS: the command exits 0.

2. Verify the health-checked services are healthy.

   docker inspect -f '{{.State.Health.Status}}' "$($COMPOSE $COMPOSE_ARGS ps -q ingress)"
   docker inspect -f '{{.State.Health.Status}}' "$($COMPOSE $COMPOSE_ARGS ps -q gateway)"

   PASS: each command prints healthy.

   If the deployment enables dashboard, also run:

   docker inspect -f '{{.State.Health.Status}}' "$($COMPOSE $COMPOSE_ARGS ps -q dashboard)"

   PASS: the command prints healthy.

3. Verify the non-health-checked exporter container is running when the enterprise profile is enabled.

   docker inspect -f '{{.State.Status}}' "$($COMPOSE $COMPOSE_ARGS ps -q exporter)"

   PASS: the command prints running.

4. Verify the restored audit file is present, readable, and non-empty.

   docker run must be given -i here. Without it the container gets no stdin, python - reads EOF, executes nothing, prints nothing, and still exits 0 — the check would pass against an empty volume.

   docker run --rm -i -v "$AUDIT_STATE_VOLUME":/data:ro "$SKILAK_GATEWAY_IMAGE" python - <<'PY'

import os audit = “/data/audit.jsonl” with open(audit, “rb”) as fh: line_count = sum(1 for _ in fh) size = os.path.getsize(audit) if line_count < 1 or size < 1: raise SystemExit(“FAIL audit.jsonl empty”) print(f“PASS audit.jsonl lines={line_count} bytes={size}”) PY

PASS: the command prints `PASS audit.jsonl lines=<n> bytes=<n>`.

5. Verify the restored file-attestation database is intact.

Provider files are enabled by default, and successful uploads mint current
file-ID approvals. A deployment that has never completed one may still have
no database; absence is not a restore failure, but it must be reported.

```sh
docker run --rm -i -v "$AUDIT_STATE_VOLUME":/data:ro "$SKILAK_GATEWAY_IMAGE" python - <<'PY'
import os
import sqlite3
attest = "/data/file-attestations.sqlite3"
if not os.path.exists(attest):
 print("SKIP file-attestations.sqlite3 absent — no file approvals minted")
 raise SystemExit(0)
if os.path.getsize(attest) < 1:
 raise SystemExit("FAIL file-attestations.sqlite3 empty")
conn = sqlite3.connect(f"file:{attest}?mode=ro", uri=True)
row = conn.execute("PRAGMA integrity_check;").fetchone()
if not row or row[0] != "ok":
 raise SystemExit(f"FAIL integrity_check={row}")
print("PASS file-attestations.sqlite3 integrity_check=ok")
PY

PASS: the command prints PASS file-attestations.sqlite3 integrity_check=ok, or SKIP … when no retained state exists. Record which result you got with the drill evidence.

6. Verify the exporter checkpoint file is present if exporter is enabled.

   docker run --rm -i -v "$EXPORT_STATE_VOLUME":/state:ro "$SKILAK_GATEWAY_IMAGE" python - <<'PY'

import os path = “/state/offsets.json” if not os.path.exists(path): raise SystemExit(“FAIL offsets.json missing”) print(f“PASS offsets.json bytes={os.path.getsize(path)}”) PY

PASS: the command prints `PASS offsets.json bytes=<n>`.

Do not stop at "non-empty". Before the backup, record the size and SHA-256 of
`audit.jsonl`; after the restore, confirm those bytes are a prefix of the restored
file. Non-emptiness cannot distinguish a correct restore from a truncated or
stale one.

Record the commands, outputs, restored backup timestamp, actual volume names, and operator identity as the drill evidence.

This runbook was last exercised end to end on **2026-07-24** against the release
stack, including a simulated total volume loss. Backup and restore were
byte-exact; that run also found and fixed four defects in the procedure below,
including verification steps that silently passed against an empty volume.

## Retention

The repository does not specify mandatory retention durations. Set and record them under your legal, compliance, and incident-response requirements; do not guess them per host.

Apply these minimum rules:

- Keep live audit and attestation state in `skilak_data` for at least `<AUDIT_RETENTION_PERIOD>`.
- Keep backup archives for at least `<BACKUP_RETENTION_PERIOD>`.
- Keep the previous known-good backup set until the next backup and restore drill has passed.
- Do not delete `audit.jsonl` while a lagging `exporter` still needs events; the exporter is at-least-once and cannot recover records removed before checkpoint catch-up.

For deletion:

- Delete expired backup objects from encrypted storage according to the storage platform's retention controls.
- Securely delete any temporary local backup copies with your host-approved secure-deletion procedure.
- If the platform relies on volume or object-store encryption for media sanitization, document that crypto-erase path in the drill evidence.

## Key rotation

Use the same operator pattern every time: stage the new value, stop new traffic, restart only the services that require the new value, verify health and access, then revoke the old value.

Common stop-new-traffic step:

```sh
$COMPOSE $COMPOSE_ARGS stop ingress

SKILAK_HASH_SALT

Affected service: gateway.

Rotation impact:

• future audit hashes and model/route/content pseudonyms no longer correlate with historical entries;
• existing stored file attestations in file-attestations.sqlite3 no longer match the new keyed state; and
• previously attested provider file IDs stop being accepted. The mesh cannot reapprove an old ID without the upload event; re-upload the inspected content through the enabled Files API to mint a new provider ID.

Procedure:

$COMPOSE $COMPOSE_ARGS up -d --wait --no-build gateway
$COMPOSE $COMPOSE_ARGS up -d --wait --no-build ingress

Verify:

• gateway health prints healthy.
• new traffic is accepted.
• a fully enumerable POST /files succeeds and its returned ID is accepted by a later request; earlier IDs remain refused.

Revoke the old salt from the secret manager after verification.

SKILAK_EDGE_ATTEST_KEY

Affected service: gateway.

Rotation impact:

• the gateway loads the new key after restart; and
• rotation does not change provider_storage policy or file-ID approvals. The sole-edge request path uses no cross-process scan-attestation header.

Procedure:

$COMPOSE $COMPOSE_ARGS up -d --wait --no-build gateway
$COMPOSE $COMPOSE_ARGS up -d --wait --no-build ingress

Verify:

• gateway health prints healthy.
• a normal fully enumerable model request succeeds.
• /files follows its enabled default, while /batches and fine-tuning remain disabled unless their validated per-prefix switches are explicitly enabled.

Revoke the old attestation key from the secret manager after verification.

SKILAK_DASHBOARD_TOKEN

Affected service: dashboard.

Procedure:

$COMPOSE $COMPOSE_ARGS --profile enterprise up -d --wait --no-build dashboard

Verify:

• dashboard health prints healthy.
• requests with the new token succeed.
• requests with the old token fail.

Revoke the old dashboard token from the secret manager after verification.

Provider credentials

Affected service: gateway.

Procedure:

$COMPOSE $COMPOSE_ARGS up -d --wait --no-build gateway
$COMPOSE $COMPOSE_ARGS up -d --wait --no-build ingress

Verify:

• gateway health prints healthy.
• the enabled routes using the new provider credential succeed.
• the old provider credential no longer works after revocation at the provider.

Revoke the old provider credential only after successful verification with the new one.

Monitoring & alerting

Monitor only the signals the release stack actually exposes.

Routes outside the admitted set

Every forwarded unknown route writes a payload-free decision marker. smesh status prefers the gateway’s /app/data audit-volume view, because the release named volume is not the CLI’s host data directory, and falls back to a local audit only for portable layouts. Review the rolling seven-day number each week. Exact wording means readable retained history covers the whole window; retention loss, corrupt records, missing numeric generations, shorter history, or the bounded scan limit yields an At least N lower bound. unavailable is not evidence of zero traffic.

Foreground run sessions

smesh run <command> registers a bounded in-memory session in the gateway, then injects /s/<opaque-id>/p/<alias>/ base URLs into the foreground child. The outer session observer counts the request before the in-flight limiter can shed it, strips the prefix, and leaves the existing alias and policy pipeline unchanged. The bodyless start/finish controls require a custom local protocol header, reject Origin and query strings, and never contact a provider or write provider-decision audit events.

The release and native launch commands must keep one gateway worker while this counter is in memory. A restart or finish failure makes the CLI report observation unavailable. Active sessions expire after 24 hours; completed counts remain briefly for an idempotent finish retry. The registry refuses new sessions at capacity and never evicts an active one. The child runs without a shell, inherits its terminal streams, and retains its exit status. Local mode-0600 records are payload-free and per-session; they must never acquire argv, environment, cwd, credentials, prompts, response content, or command output.

Health checks:

• Alert if ingress or gateway is not healthy for more than 2 minutes.
• Alert if dashboard is enabled and not healthy for more than 2 minutes.
• Alert if exporter is enabled and its container state is not running.

Restart counts:

• Alert on any restart of ingress, gateway, exporter, or dashboard.
• Escalate immediately if restart counts increase more than once in 15 minutes for gateway.

Bounded logs:

• ingress log cap: 1m x 2 files.
• gateway log cap: 10m x 2 files.
• exporter log cap: 2m x 2 files.
• dashboard log cap: 2m x 2 files.
• Alert if a container is restarting and its bounded logs are rotating continuously; capture them before they age out.

Exporter destination liveness:

• The checked-in config/exports.json has "enabled": false and no sinks. There is nothing to alert on until the operator supplies a real SKILAK_EXPORT_FILE.
• When enabled, alert if offsets.json stops changing while exporter is running and the configured poll_interval_seconds and max_attempts window has been exceeded.
• Alert on repeated delivery failures to any configured sink and on any unexpected change to the configured sink list.

Volume disk usage:

• Alert before the Docker host runs out of space for the volumes backing skilak_data and skilak_export_state.
• Treat low free space on skilak_data as a release-blocking condition because audit write failures block traffic by design.

Block-rate anomalies:

• Watch for a sudden spike in block verdicts or audit write failures.
• Treat a sustained fail-closed spike as a possible outage, attack, policy regression, or storage failure until proved otherwise.

Incident response

If you suspect a leak, bypass, unexpected direct egress, or tampering, stop the stack before further triage.

Stop egress immediately:

$COMPOSE $COMPOSE_ARGS stop

Preserve evidence before any cleanup or restart. Snapshot the audit and attestation volume first:

export AUDIT_STATE_VOLUME="<actual volume name for skilak_data>"
export BACKUP_ROOT="<evidence directory>"
export STAMP="$(date -u +%Y%m%dT%H%M%SZ)"
mkdir -p "$BACKUP_ROOT/$STAMP"
docker run --rm --user 0:0 \
  -v "$AUDIT_STATE_VOLUME":/src:ro \
  -v "$BACKUP_ROOT/$STAMP":/dst \
  "$SKILAK_GATEWAY_IMAGE" \
  sh -lc 'cd /src && tar -czf /dst/skilak_data-incident.tgz .'

What the payload-free audit can tell you afterward:

• whether the system allowed, warned, or blocked;
• which rule IDs fired;
• bounded response status_code and latency_ms;
• stable keyed pseudonyms and hashes for correlation; and
• whether audit, scan, or attestation failures occurred.

What it cannot tell you afterward:

• the raw prompt or response payload;
• the contents of uploaded or downloaded files;
• the cleartext model or route names; or
• a reconstructed copy of exfiltrated data.

Escalate and notify:

• notify the security incident owner and the operator responsible for the host;
• preserve the backup set, active image digest set, and selected config files;
• record the time window, affected services, and any external destinations involved.

Post-incident:

• patch or roll back the fault before re-opening traffic;
• rotate SKILAK_HASH_SALT, SKILAK_EDGE_ATTEST_KEY, SKILAK_DASHBOARD_TOKEN, and any provider credentials that may be exposed;
• re-run the independent adversarial review before calling the release production-ready again;
• execute the restore verification drill again after recovery changes.

Upgrade

This section applies only to a separately operated Compose deployment. The current public native release pipeline does not build, sign, or publish the three container images, so a public Skilak Mesh tag is not provenance for them.

Use only image digests obtained and verified through that deployment’s approved container delivery process. Keep the previous verified digest set on hand before changing anything.

Record the currently deployed digest set:

cat > "<upgrade working directory>/previous-image-digests.txt" <<EOF
SKILAK_INGRESS_IMAGE=$SKILAK_INGRESS_IMAGE
SKILAK_GATEWAY_IMAGE=$SKILAK_GATEWAY_IMAGE
SKILAK_ENTERPRISE_IMAGE=$SKILAK_ENTERPRISE_IMAGE
EOF

Verify the digests and signatures using the exact identity documented by the independent container delivery process, then pull and replace without building on the deployment host. Do not reuse the native candidate or tag-workflow identity as a substitute.

$COMPOSE $COMPOSE_ARGS pull
$COMPOSE $COMPOSE_ARGS up -d --wait --no-build

Health gate:

• require ingress and gateway to report healthy;
• require dashboard to report healthy if enabled;
• require exporter to remain running if enabled; and
• if any gate fails, stop and roll back to the previous verified digest set.

The Stage-2 attestation migration invalidates legacy alias-less file approvals. Plan for affected provider file IDs to be re-uploaded through the upgraded mesh; do not interpret the resulting fail-closed file_reference_unattested refusals as provider downtime.

Rollback

Roll back to the previous verified digest set, not to a mutable tag.

Restore the previous image references in the operator environment, verify them again through their independent container delivery process, then redeploy:

export SKILAK_INGRESS_IMAGE="<previous verified ingress image ref>"
export SKILAK_GATEWAY_IMAGE="<previous verified gateway image ref>"
export SKILAK_ENTERPRISE_IMAGE="<previous verified enterprise image ref>"

$COMPOSE $COMPOSE_ARGS pull
$COMPOSE $COMPOSE_ARGS up -d --wait --no-build

Restore the volumes from the pre-upgrade backup as part of the rollback if any of these are true:

• the failed release wrote corrupt or unreadable audit.jsonl, file-attestations.sqlite3, or offsets.json;
• the older image set cannot read the newer on-disk state; or
• the rollback starts but the restore verification drill fails.

If a volume restore is required, follow the restore procedure in this runbook before restarting the rolled-back stack.

Disaster recovery

Use this sequence to rebuild on a new host from backups.

Prerequisites:

• Docker with Compose support on the new host;
• the verified values for SKILAK_INGRESS_IMAGE, SKILAK_GATEWAY_IMAGE, SKILAK_ENTERPRISE_IMAGE;
• the operator-provided secrets for SKILAK_HASH_SALT, SKILAK_EDGE_ATTEST_KEY, optional provider credentials, optional SKILAK_DASHBOARD_TOKEN, and optional SKILAK_EXPORT_WEBHOOK_TOKEN;
• the selected non-secret config files: docker-compose.release.yml, SKILAK_POLICY_FILE and SKILAK_EXPORT_FILE; and
• the backup archives for skilak_data and, if used, skilak_export_state.

Order of operations:

1. Provision the new host and copy the non-secret config files into place.
2. Set the image refs and required secrets in the operator-controlled environment.
3. Verify the three image signatures with cosign verify.
4. Create the target volumes.
5. Restore skilak_data and, if used, skilak_export_state.
6. Enforce UID/GID 10001:10001 ownership on both volumes.
7. Start the stack with $COMPOSE $COMPOSE_ARGS up -d --wait --no-build.
8. Run the full restore verification drill.

Final verification:

• ingress and gateway are healthy;
• dashboard is healthy if enabled;
• exporter is running if enabled;
• audit.jsonl is present and readable;
• file-attestations.sqlite3 passes PRAGMA integrity_check;; and
• the exact backup timestamp and runtime volume names are recorded in the recovery evidence.

Source file: docs/OPERATIONS.md in SkilakLabs/mesh.