AlphaSwarm Runbooks

This index provides a centralized location for all operational and developer runbooks across the AlphaSwarm platform.

Deployment Runbooks

• Cluster Deployment: Guide for local and remote Kubernetes deployment.
• Alpha Testing Deployment: Current guide for the MacBook client/local-server to Ubuntu-hosted platform topology.
• Onboarding Integrations: Adding and managing external API connections.
• CI/CD Deploy: How the automated pipeline works.

Operational Runbooks

• Debugging and Troubleshooting: Systematic guide for fixing issues.
• DataOps Airbyte and Dagster: Triage sync failures, Dagster daemon/schedule/sensor incidents, data lag, and dbt failures.
• Local agent-first data-infra ops: PowerShell cookbook for alphaswarm-cli data infra, native Dagster MCP reads/writes, and orchestration-operator diagnosis (no UI, no dg, no Terraform apply).
• Portable Dagster and Prefect orchestration: Deploy, shadow-verify, cut over, reconcile, and roll back one-engine-owned portable definitions.
• CLI Operator Guide: Comprehensive CLI usage.
• Disaster Recovery: Restore: Recovering from data loss.
• Multi-Account Rollout: Expanding to new cloud accounts.

Game-Day Runbooks

Repeatable game-day exercises for Phase 4.6 of the Unified Infrastructure Control Plane Plan (the plan itself was relocated to the internal alphaswarm_internal repo on 2026-07-18; see ADR 026). Each proves a deployment-safety property with measurable success criteria, an abort/rollback path, and evidence archived to the WORM audit ledger.

• Game-Day: Zero-Orphaned-Order Drain: Prove a trading-hours pod termination leaves zero open orders (fail-closed drain finalizer, PreStop flatten, PDB, Karpenter do-not-disrupt, partitioned rollout).
• Game-Day: Kill-Switch Fleet Halt + Resume: Prove the Redis-backed, fail-closed, cluster-wide halt blocks every mutating op (a gated apply is refused) and resumes cleanly.
• Game-Day: DR Replay from Encrypted State and WORM Ledger: Restore a cell from OpenTofu-encrypted state with its per-cell KMS key and verify the hash-chained WORM ledger with audit_verify.
• Game-Day: BYOC Tenant Offboard with Crypto-Shred Verification: Run the offboarding saga to its terminal crypto-shred (destroy the tenant CMK, retain the Transit key) and prove the data is unrecoverable.

Maintenance Runbooks

• QuestDB WAL Stall: Resolving write-ahead log issues.
• Quota Exhaustion: Managing cloud resource limits.
• Snapshot Deadlock: Resolving database snapshotting issues.

Further Guidance

• For architecture, see the Architecture Index.
• For a project-wide map, see the Documentation Index.