Task 23 Execution Packet: Chaos Testing Suite (Aligned to 2026-01-25 Plan)

This packet replaces the old Task 23 notes. It is aligned to Phase 16 / Task 23 in docs/plans/2026-01-25-end-state.md and the current repo infrastructure.

Use this as the exact execution guide for an agent implementing Task 23.

Objective

Add a chaos test suite that validates platform invariants under failure:

1. NATS partition across clusters (mesh)
2. OPA restart during active policy-gateway traffic
3. OpenObserve ingest stall (via otel-collector)
4. Postgres restart with outbox recovery

Non-negotiables

• Tests are repeatable and deterministic (bounded polling, no unbounded sleeps).
• Chaos runs are opt-in (not part of default unit/CI).
• Failures produce actionable logs (container logs + probe output).
• No generated-zone edits.

Step 0 — Ground current infra reality (resolved)

Based on repo state:

• OpenObserve container exists in infra/docker/docker-compose.dev.yml (sea-openobserve).
• OpenTelemetry collector exists in infra/docker/docker-compose.dev.yml (sea-otel-collector).
• OPA exists in infra/docker/docker-compose.skeleton.yml (sea-opa).
• Outbox worker is apps/sea-mq-worker (no existing docker-compose service). We will add it to a chaos compose file.
• NATS mesh is expected from Task 15 (infra/docker/docker-compose.mesh.yml).
  • Mesh container names should follow Task 15: sea-nats-a-1 … sea-nats-b-3.
  • Mesh host ports should follow Task 15: A client 4422, B client 5422, gateway port 7522.

Default ports for chaos stack (match dev unless mesh is used):

• Postgres: 5432
• NATS single-node: 4222 (monitor 8222)
• OPA: 8181
• Policy Gateway: 8080 (entrypoint default)
• OpenObserve: 5080
• OTEL collector: 4317/4318/8888/13133

Step 1 — Add a dedicated chaos compose file

Create:

• infra/docker/docker-compose.chaos.yml

Include services (with profiles):

• postgres (profile chaos-outbox)
• nats (single-node JetStream, profile chaos-outbox)
• sea-mq-worker (profile chaos-outbox)
• opa (profile chaos-opa)
• policy-gateway (profile chaos-opa)
• openobserve (profile chaos-observe)
• otel-collector (profile chaos-observe)

Rule: services must be profile-gated so scenarios run only what they need.

Port consistency: use the default ports listed in Step 0 unless a scenario explicitly requires alternate ports.

Step 2 — Harness structure

Create:

1
2
3
4
5
6
7
8
9
10
11
12
tests/chaos/
  run_chaos.py
  scenarios/
    nats_partition_mesh.py
    opa_restart.py
    openobserve_stall.py
    postgres_restart_outbox.py
  probes/
    http_probe.py
    nats_probe.py
    jetstream_probe.py
    db_probe.py

Add a manual just chaos recipe (do not add to just ci).

Step 3 — Scenario definitions (explicit, deterministic)

A) nats_partition_mesh

Compose file: infra/docker/docker-compose.mesh.yml (from Task 15)

Fault injection: block gateway traffic inside one node container (e.g., sea-nats-a-1) by dropping port 7522.

Invariant:

• During partition, replication may pause.
• After heal, mirror replication resumes and new messages appear in mirror stream within timeout.

Probe:

• Publish to domain A stream SEA_EVENTS subject sea.event.chaos_probe.v1.
• Poll mirror stream in domain B (SEA_EVENTS_MIRROR).

Defaults:

• CHAOS_NATS_A_URL=nats://localhost:4422
• CHAOS_NATS_B_URL=nats://localhost:5422
• CHAOS_DOMAIN_A=sea-a
• CHAOS_DOMAIN_B=sea-b

B) opa_restart

Compose file: infra/docker/docker-compose.chaos.yml with profile chaos-opa

Fault injection: docker restart sea-opa.

Invariant:

• policy-gateway remains responsive (health endpoint OK)
• decisions remain deterministic (allow/deny per fixture)
• no sustained 5xxs

C) openobserve_stall

Compose file: infra/docker/docker-compose.chaos.yml with profile chaos-observe

Fault injection: stop sea-openobserve container.

Invariant (collector-based, resolved):

• otel-collector continues running and reports export failure
• application endpoints remain responsive
• after heal, collector resumes exports

D) postgres_restart_outbox

Compose file: infra/docker/docker-compose.chaos.yml with profile chaos-outbox

Fault injection: docker restart <postgres>.

Invariant:

• outbox publisher reconnects
• unpublished outbox rows are eventually published
• idempotency prevents double-apply

Probe path (resolved):

• Insert an outbox row directly via SQL (schema in infra/db/migrations/001_outbox_events.sql).
• Observe publish via NATS subject.

Step 4 — Probe implementation (low LoE)

Prefer CLI-based probes for determinism:

• NATS CLI for pub/sub and JetStream reads
• psql or docker exec for DB inserts
• curl for HTTP health and policy evaluation

If Python clients are used, keep deps minimal and pinned in tests/chaos/requirements.txt.

Step 5 — Deterministic pass/fail criteria

Each scenario must:

• inject_fault() completes within 10s
• assert_invariants() completes within 60–180s
• heal() runs even on failure
• On failure, dump:
  • last 200 lines of relevant container logs
  • probe output
  • executed commands

Step 6 — Commands

Add a manual recipe (not in just ci):

chaos scenario="nats_partition_mesh":
  CHAOS_SCENARIO= python tests/chaos/run_chaos.py

Acceptance criteria (from plan)

• No silent loss beyond declared semantics
• Recovery behavior observable and deterministic
• Chaos suite is opt-in only