Task 15 Execution Packet: Messaging Federation + Clustering (Aligned to 2026-01-25 Plan)

This packet replaces the old Task 15 notes. It is aligned to Phase 14 / Task 15 in docs/plans/2026-01-25-end-state.md and to the current repo layout (infra/nats/*, infra/docker/*, apps/sea-mq-worker/*).

Use this as the exact execution guide for an agent implementing Task 15.

Objective

Deliver messaging that scales single instance → cluster → mesh, with deterministic topology, JetStream durability, and federation across clusters.

Minimum outcomes (per plan):

• Multi-node JetStream clustering
• Multi-cluster federation (leafnodes or supercluster)
• Deterministic subject topology + replay semantics
• Env-only configuration
• Durable consumers survive node + cluster loss
• E2E tests prove federation works

Non-negotiables

• Env-config only (no hardcoded topology in code).
• Backwards compatible defaults (single-node still works).
• Deterministic subject naming and replay semantics.
• No generated edits.
• At least one real federation E2E test (skip-safe by env).

Step 0 — Ground current code reality (must confirm before editing)

Verify these in-repo surfaces:

• infra/nats/nats.conf (cluster-only config)
• infra/nats/nats.mesh.conf (if present)
• infra/nats/setup-streams.sh (stream provisioning)
• infra/docker/docker-compose.*.yml (NATS wiring)
• apps/sea-mq-worker/src/main.rs (env config)
• apps/sea-mq-worker/tests/e2e_messaging_federation_test.rs (existing or placeholder)

If any of these differ, adjust this packet before coding.

Step 1 — Federation mechanism (resolved)

Use NATS supercluster gateways (resolved default).

• Supercluster gateways allow cross-cluster pub/sub.
• JetStream remains durable per cluster domain.
• Stream replication across clusters can be layered later via mirror/source.

Step 2 — NATS config: cluster + federation + JetStream domain

Required file edits

• infra/nats/nats.conf (keep cluster-only path for local HA)
• infra/nats/nats.mesh.conf (federation config for mesh profile)

Required env vars

Cluster:

• NATS_CLUSTER_NAME
• NATS_CLUSTER_PORT (default 6222)
• NATS_ROUTE_1/2/3

Federation (supercluster gateways):

• NATS_GATEWAY_NAME
• NATS_GATEWAY_PORT (default 7522)
• NATS_PEER_GATEWAY_NAME
• NATS_PEER_GATEWAY_URL

JetStream:

• JETSTREAM_DOMAIN

Required config blocks (mesh file)

• cluster { ... } with routes list
• gateway { ... } with peer gateway URLs
• jetstream { store_dir: /data, domain: $JETSTREAM_DOMAIN }

Success signal: nats --domain <domain> stream ls works on each cluster domain.

Step 3 — Compose wiring for mesh profile

Add a mesh profile with two 3-node clusters (sea-a, sea-b).

Files

• Modify infra/docker/docker-compose.dev.yml or
• Create infra/docker/docker-compose.mesh.yml (recommended for clarity)

Requirements

• Cluster A nodes use JETSTREAM_DOMAIN=sea-a.
• Cluster B nodes use JETSTREAM_DOMAIN=sea-b.
• Gateway peers are cross-linked.
• Per-node volumes are unique.
• Use consistent container names for mesh nodes:
  • sea-nats-a-1, sea-nats-a-2, sea-nats-a-3
  • sea-nats-b-1, sea-nats-b-2, sea-nats-b-3
• Recommended host port mappings (avoid clashing with dev NATS):
  • Cluster A client: 4422/4423/4424
  • Cluster B client: 5422/5423/5424
  • Cluster A monitoring: 8822/8823/8824
  • Cluster B monitoring: 9822/9823/9824

Success signal: Gateway connections appear in NATS monitoring endpoints, and cross-cluster pub/sub works.

Step 4 — Stream provisioning (deterministic + env-only)

Modify infra/nats/setup-streams.sh:

Add env controls

• STREAM_REPLICAS (default 1)
• JETSTREAM_DOMAIN (optional)
• Keep NATS_URL

Requirements

• Use --replicas=$STREAM_REPLICAS when creating streams/consumers.
• If JETSTREAM_DOMAIN is set, pass --domain to nats stream add and nats consumer add.
• Ensure subject topology is deterministic and matches ADR-032 (no per-run randomness).

Step 5 — Worker env support (only if missing)

apps/sea-mq-worker/src/main.rs must support env-only configuration for:

• cluster name
• JetStream domain
• replication factor
• federation peers (gateway URLs)

Add only what is missing; preserve defaults and backwards compatibility.

Step 6 — E2E test: federation actually works

Create/replace:

• apps/sea-mq-worker/tests/e2e_messaging_federation_test.rs

Test contract (minimum)

• Skip-safe if env vars are missing.
• Connect to Cluster A and publish on a deterministic subject (e.g., sea.event.federation_probe.v1).
• Connect to Cluster B and subscribe to the same subject.
• Assert the message arrives within a bounded timeout.

Required env vars (skip-safe)

• E2E_NATS_A_URL (e.g., nats://localhost:4422)
• E2E_NATS_B_URL (e.g., nats://localhost:5422)

Minimum additional tests (per plan)

• Cluster leader loss → durable consumer resumes without duplication.
• Cross-cluster publish → remote consumer receives event.
• DLQ replay works across federation boundary.

These can be separate tests or a combined suite, but must be deterministic and skip-safe if the mesh isn’t running.

Step 7 — Verification

Bring up mesh (example):

1
docker compose -f infra/docker/docker-compose.mesh.yml --profile nats-mesh up -d

Run E2E test:

1
2
3
export E2E_NATS_A_URL="nats://localhost:4422"
export E2E_NATS_B_URL="nats://localhost:5422"
cargo test -p sea-mq-worker e2e_federation_pubsub_across_clusters -- --nocapture

Only mark Task 15 complete after tests pass.

Acceptance criteria (from plan)

• At-least-once delivery with idempotency guarantees.
• Ordering guarantees are explicit and tested.
• Federation behavior is documented and observable.