Saltar al contenido principal

Connector Control Plane & Data Mesh

The Connector Control Plane expands the AlphaSwarm data fabric into a distributed data mesh: customers bring their own data sources and credentials (BYOB / BYODB / BYOK), AlphaSwarm ships curated source templates and a public-data catalog, and a federated query layer lets analytics span the lakehouse, customer sources, and public datasets through one SQL surface. It is governed by hard rules 61-63 and .cursor/rules/connector-control-plane.mdc (moved from the alphaswarm repo to the alphaswarm_internal SSoT on 2026-07-19; alphaswarm/.cursor/rules/pointer.mdc is the stub left behind).

Capabilities​

  • BYOB (Bring-Your-Own-Bucket) - read a customer-owned S3 / GCS / Azure Blob / S3-compatible bucket via the byob_object dataset kind.
  • BYODB (Bring-Your-Own-Database) - federated read of a customer Postgres / MySQL / SQL Server / Snowflake / Redshift via the byodb_sql dataset kind (no copy).
  • BYOK (Bring-Your-Own-Key) - per-tenant AWS KMS CMK wraps the DEKs that encrypt connector credentials; disabling / deleting the CMK crypto-shreds the tenant's data (the kill-switch).
  • Source templates + public data - the alphaswarm_ingest marketplace catalog (AWS Open Data, SEC EDGAR, FRED, Hugging Face / Croissant, World Bank, IMF SDMX 3.0, Nasdaq) with license + attribution.
  • Federated query - Apache Gravitino (catalog-of-catalogs, Metalake = tenant boundary) + Trino + Trino Gateway + OPA.
  • Outbound zero-copy sharing - OpenFGA check + signed share tokens (read-only). The design canon calls for a standalone alphaswarm_data_federation broker, but that repo does not exist yet; the live implementation today is alphaswarm.authz.rebac (ReBACClient, mint_share_token) plus alphaswarm.data.metering inside the monolith, which the module docstrings describe as lifting the same OpenFGA-check-then-share-token pattern (see alphaswarm_kb_federation for the analogous, also-not-yet-live, KB-side broker).

Architecture​

Registering a customer source​

POST /me/data-sources (step-up MFA). For a vaulted credential the secret payload is accepted once, envelope-encrypted, and dropped. For cross-account AWS, the platform derives an ExternalId; deploy the returned trust policy (GET /me/data-sources/{id}/trust-policy) in your account, then the registration validator runs the AssumeRole-without-ExternalId negative test. A validated source appears in the discovery browser and is selectable as a hosted dataset.

Security​

  • Cross-account: sts:AssumeRole with a platform-generated, non-guessable ExternalId (confused-deputy guard); short-lived STS creds are never persisted.
  • BYOK: per-tenant CMK provisioned through TerraformRuntime (tenant_byok_cmk module) with kms:ViaService / kms:CallerAccount conditions + rotation; crypto-shred = disable / schedule-delete.
  • SSRF: every customer-supplied URL is validated (scheme allow-list, DNS re-resolution per redirect, link-local / 169.254.169.254 blocked); IMDSv2 is enforced fleet-wide as defense-in-depth.
  • Tenancy: customer_data_sources is RLS-protected by workspace_id and envelope-encrypted on top; secrets never appear in any response / log / audit.

Federated query​

data.federation.query runs read-only SQL through Trino. Authz is enforced server-side by the OPA plugin (catalog/schema/table + row filters scoped to the caller's workspace); the client never makes a local allow/deny decision. Gravitino federates the Iceberg lakehouse, customer sources, and public datasets behind one Iceberg REST surface.

Outbound sharing​

Cross-tenant zero-copy sharing follows the pattern design-canon calls the standalone alphaswarm_data_federation gateway, but that gateway repo doesn't exist yet: today the flow runs in-monolith via alphaswarm.authz.rebac — OpenFGA check -> mint an aud-bound share token (RS256 when a private key is configured, an HS256 dev fallback otherwise; jti; TTL <= 600s; never a bearer passthrough). Shares record a SHARED_WITH ownership-graph edge (alphaswarm.data.metering). The read-only design intent still holds.