Implementation Plan: Messaging Substrate

Unified reliability layer for cross-context messaging: NATS JetStream with outbox/inbox patterns, canonical event envelopes, per-context streams, deduplication, DLQ handling, and external integration transports.

Consolidation Note: This plan merges P006 (Integration Gateway) and the previous P017 (Messaging Infrastructure) per ADR-032, which defines outbox/inbox + JetStream as a unified reliability pattern.

Provenance & Traceability

Architectural Decisions (ADRs)

Product Requirements (PRDs)

Software Design Specifications (SDS)

Provenance Chain

graph TD
  ADR32[ADR-032: JetStream + Outbox/Inbox] --> PRD22[PRD-022: Cross-Context Messaging]
  ADR11[ADR-011: IFL] --> PRD3[PRD-003: IFL + Identity Tokens]
  PRD22 --> SDS47[SDS-047: Messaging Infrastructure]
  PRD3 --> SDS30[SDS-030: Observability Envelope]
  SDS47 --> Impl[Implementation]
  SDS30 --> Impl

Architecture and Design

Design Principles Applied

• Isomorphism: Outbox/Inbox entities map 1:1 to schema and event definitions. External integrations observe events as canonical envelopes; internal models remain generated and stable.
• Invariants: Versioned event subjects (.v1, .v2) and immutable meaning (ADR-032). Per-context streams are mandatory.
• Idempotency: Outbox publish dedupe via Nats-Msg-Id; inbox processing dedupe via inbox PK.

High-Level Architecture

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
┌─────────────────────────────────────────────────────────────────────────┐
│                        Messaging Substrate                               │
├─────────────────────────────────────────────────────────────────────────┤
│                                                                          │
│  ┌──────────────────────────────────────────────────────────────────┐   │
│  │                    Internal Messaging Layer                       │   │
│  │  ┌─────────────┐        ┌─────────────┐        ┌─────────────┐   │   │
│  │  │   Outbox    │───────▶│  JetStream  │◀───────│    Inbox    │   │   │
│  │  │  Publisher  │        │   Streams   │        │   Consumer  │   │   │
│  │  └──────┬──────┘        └──────┬──────┘        └──────┬──────┘   │   │
│  │         │                      │                      │          │   │
│  │         ▼                      ▼                      ▼          │   │
│  │  ┌───────────────────────────────────────────────────────────┐   │   │
│  │  │                 PostgreSQL (per-context DB)                │   │   │
│  │  │         outbox_events  │  inbox_messages                   │   │   │
│  │  └───────────────────────────────────────────────────────────┘   │   │
│  └──────────────────────────────────────────────────────────────────┘   │
│                                                                          │
│  ┌──────────────────────────────────────────────────────────────────┐   │
│  │                    Integration Layer (Shell)                      │   │
│  │  ┌─────────────────┐                    ┌─────────────────────┐   │   │
│  │  │ Outbound Edge   │                    │  Inbound Edge       │   │   │
│  │  │ (JetStream →    │                    │  (Partner Signal →  │   │   │
│  │  │  Webhooks/A2A)  │                    │   Kernel Command)   │   │   │
│  │  └─────────────────┘                    └─────────────────────┘   │   │
│  └──────────────────────────────────────────────────────────────────┘   │
│                                                                          │
└─────────────────────────────────────────────────────────────────────────┘

Dependency Justification

NATS JetStream

Outbox/Inbox Pattern

Rust Worker (sea-mq-worker)

Installation Commands

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
# Node.js dependencies
pnpm add nats ts-event-bus uuid

# Python dependencies
pip install nats-py uuid

# Docker (add to infra/docker-compose.yml)
# nats:
#   image: nats:2.10-alpine
#   command: ["--jetstream", "--store_dir=/data"]
#   ports: ["4222:4222"]

# Rust dependencies (in Cargo.toml)
# async-nats = "0.34"
# sqlx = { version = "0.8", features = ["postgres", "runtime-tokio"] }
# tokio = { version = "1", features = ["full"] }
# reqwest = { version = "0.12", features = ["json"] }

Integration Surface (Conceptual)

Outbound (Internal → External):

• Domain events in JetStream → External transports (A2A/ACP/OASF/Webhooks)
• Canonical envelope format for all transports

Inbound (External → Internal):

• Partner signals validated, normalized, and translated into kernel commands
• Correlation IDs propagated end-to-end for observability

Expected Filetree

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
/
├── schemas/events/
│   ├── envelope-v1.schema.json          # Canonical event envelope
│   └── messaging/                        # Outbox/inbox schemas
│       ├── outbox-event.schema.json
│       └── inbox-message.schema.json
├── docs/specs/shared/
│   ├── adr/032-nats-jetstream-messaging.md
│   └── sds/047-messaging-infrastructure.md
├── libs/shared/messaging/               # Messaging library (TS)
│   └── src/gen/**                       # Generated adapters/publishers
└── apps/sea-mq-worker/                  # Rust worker
    └── src/
        ├── outbox_publisher.rs
        ├── inbox_consumer.rs
        └── main.rs

Proposed Cycles

Task Breakdown

Wave 1: Core Schemas & Conventions (Parallel)

• C1A: Formalize outbox/inbox entities and event envelope schemas
  • Implements: REQ-050/051/054 (outbox/inbox/dedupe)
  • Satisfies: PRD-022 acceptance criteria, SDS-047 §2
  • Files: docs/specs/shared/sds/047-messaging-infrastructure.md, schemas/events/messaging/*
• C1B: Lock stream + subject conventions and versioning rules
  • Implements: ADR-032 constraints (per-context streams, versioned subjects)
  • Satisfies: PRD-022 REQ-055, INV-023
  • Files: docs/specs/shared/adr/032-nats-jetstream-messaging.md

Wave 2: Core Workers (Depends on Wave 1)

• C2A: Implement outbox publisher (Rust worker)
  • Implements: Outbound event publishing with Nats-Msg-Id dedupe
  • Satisfies: PRD-022 REQ-050/054, SDS-047 §4 CMD-050
  • Files: apps/sea-mq-worker/src/outbox_publisher.rs
• C2B: Implement inbox consumer (Rust worker)
  • Implements: Inbound processing with inbox PK dedupe
  • Satisfies: PRD-022 REQ-051/052, SDS-047 §4 CMD-051
  • Files: apps/sea-mq-worker/src/inbox_consumer.rs

Wave 3: Error Handling & Envelope (Depends on Wave 2)

• C3A: Define DLQ handling and replay policies
  • Implements: Poison message routing and backpressure behavior
  • Satisfies: PRD-022 REQ-052/053, SDS-047 §4 CMD-052
  • Files: docs/specs/shared/sds/047-messaging-infrastructure.md
• C3B: Standardize and validate the canonical event envelope for all integration transports
  • Implements: Envelope contract used by outbound and inbound edges
  • Satisfies: PRD-022, PRD-003 (attestable artifacts)
  • Files: schemas/events/envelope-v1.schema.json, schemas/events/integration/*

Wave 4: Integration Adapters (Depends on Wave 3)

• C4A: Generate inbound adapters that validate and translate partner signals into kernel requests
  • Implements: Validation + normalization + correlation ID propagation
  • Satisfies: ADR-033 “thin shells”, SDS-030 envelope
  • Files: docs/specs/shared/sds/030-semantic-observability.md, generated **/src/gen/**

Validation & Verification

Spec Validation

• Event subject names conform to {context}.event.{name}.v{n} (ADR-032)
• Outbox/inbox schemas validate against declared JSON schemas
• Published events conform to schemas/events/envelope-v1.schema.json

Implementation Validation

• Outbox writes are atomic with state changes (REQ-050, INV-022)
• Outbox publish is idempotent under retries (Nats-Msg-Id header set)
• Inbox dedupe prevents double-processing (REQ-051, INV-021)
• Inbox processing is exactly-once (inbox_messages PK dedupe)
• Correlation IDs propagate end-to-end for observability (SDS-030)
• DLQ routing works for 422 responses

System Invariants Enforced

Open Questions (Resolved)

Risks & Mitigation

Rollback Strategy

1. Partial: Disable external transport adapters while keeping internal broker publishing intact.
2. Full: Disable inbound consumers (keep outbox publishing) while fixing inbox/dedupe issues.

Related Documents

Linked Specifications

Superseded Plans

• P006 (Integration Gateway) v0.2.0 — Merged into this plan
• P017 (Messaging Infrastructure) v0.2.0 — Replaced by this consolidated version