Observability stack
AlphaSwarm's centralized telemetry plane lives in the
alphaswarm-observability namespace. The canonical, CRD-free data flow
(see observability/README.md)
is VictoriaMetrics-based: vmagent is the only Prometheus-style scraper,
VictoriaMetrics is the sole metrics store, vmalert owns recording/alert
rules, and a single raw OTel gateway (no DaemonSet agent) fans traces and
logs out to their destinations. The older kube-prometheus-stack/,
opentelemetry-operator/, and opentelemetry-collector-gateway/ folders
still exist but are explicitly unselected rollback/reference assets only.
Components
| Component | Folder | Notes |
|---|---|---|
| VictoriaMetrics + vmagent + vmalert | observability/victoriametrics/, observability/vmagent/, observability/vmalert/ | canonical metrics authority — replaces Prometheus/kube-prometheus-stack |
| OTel gateway | observability/otel-gateway/ | canonical single OTLP ingress (no DaemonSet agent); otel-collector Service name kept as a compat alias |
| Jaeger + OpenSearch | observability/jaeger/, observability/opensearch/ | trace storage/query (traces also land in Phoenix for AI spans) |
| Loki | observability/loki/ | log storage |
| Phoenix | observability/phoenix/ | AI/agent trace explorer |
| kube-prometheus-stack, OpenTelemetry Operator, opentelemetry-collector-gateway | (legacy folders, same parent dir) | unselected rollback/reference assets only |
Routing rule (gateway)
The tail_sampling processor in
otel-gateway/configmap.yaml
inspects every span and keeps it when any policy matches:
errors—status_code == ERROR.ai_traces—attributes["openinference.span.kind"]is set.llm_traces—attributes["llm.model_name"]is set.agent_traces—attributes["agent.name"]is set.probabilistic_baseline— a flat 1% of everything else.
There's a single traces pipeline (not split into traces/ai /
traces/infra): every span that survives tail sampling is exported to
both otlp/jaeger and otlp/phoenix.
DataMCP tools
| Tool | Surface |
|---|---|
data.observability.prometheus.query | Instant PromQL. |
data.observability.prometheus.query_range | Range PromQL. |
data.observability.prometheus.list_alerts | Active alerts. |
data.observability.grafana.list_dashboards | Dashboard catalog. |
data.observability.grafana.export_dashboard | Dashboard JSON. |
data.observability.phoenix.list_projects | Phoenix projects. |
data.observability.phoenix.get_trace | LLM / agent trace. |
data.observability.phoenix.annotate_span | Write evaluator verdict. |
Frontend
- /observability
in
alphaswarm_client(the active Vite operator UI) — observability overview page.