ADR-029: Observability Stack Architecture

Status: Accepted
Version: 1.0
Date: 2025-12-27
Supersedes: N/A
Related ADRs: ADR-028, ADR-022
Related PRDs: PRD-010

Context

Enterprise AI systems require comprehensive observability across metrics, logs, and traces. Traditional monitoring stacks (Prometheus + Grafana + Jaeger/ELK) create operational overhead through:

1. Multiple data stores — Prometheus for metrics, Elasticsearch for logs, Jaeger for traces
2. Configuration fragmentation — Separate pipelines for each signal type
3. Compliance gaps — Manual auditing and evidence collection
4. Cost complexity — Multiple SaaS subscriptions or self-hosted infrastructure

SEA-Forge™ requires:

• Unified telemetry ingestion from all services
• Semantic context preservation (domain, concept, regime)
• Privacy-aware data handling (PII scrubbing, provenance hashing)
• Compliance automation (Vanta SOC2/ISO integration)
• Python-first developer experience (Logfire)

Decision

Adopt a unified observability stack based on OpenTelemetry as the instrumentation standard:

Stack Components

Reference Architecture

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
┌─────────────────────────────────────────────────────────────────┐
│  SEA-Forge™ Observability Architecture                           │
├─────────────────────────────────────────────────────────────────┤
│                                                                 │
│  ┌──────────────────┐   ┌──────────────────┐                    │
│  │ SEA™ Services     │   │ Policy Gateway   │                    │
│  │ (Python/Rust)    │   │ (Sidecar)        │                    │
│  └────────┬─────────┘   └────────┬─────────┘                    │
│           │                      │                               │
│           ▼                      ▼                               │
│  ┌──────────────────────────────────────────┐                   │
│  │     OpenTelemetry SDK (Auto + Manual)    │                   │
│  │     • Traces: HTTP, gRPC spans           │                   │
│  │     • Metrics: Counters, Gauges          │                   │
│  │     • Logs: Structured via Logfire       │                   │
│  └────────┬─────────────────────────────────┘                   │
│           │ OTLP (gRPC/HTTP)                                    │
│           ▼                                                     │
│  ┌──────────────────────────────────────────┐                   │
│  │     OTel Collector                       │                   │
│  │     • Receivers: OTLP, Prometheus        │                   │
│  │     • Processors: PII scrubbing, batch   │                   │
│  │     • Exporters: Multi-destination       │                   │
│  └────────┬──────────┬──────────┬───────────┘                   │
│           │          │          │                               │
│           ▼          ▼          ▼                               │
│  ┌────────────┐ ┌─────────┐ ┌─────────┐                         │
│  │ OpenObserve│ │  Vanta  │ │(Archive)│                         │
│  │ Metrics    │ │ Evidence│ │ S3/Blob │                         │
│  │ Logs       │ │ Auto    │ └─────────┘                         │
│  │ Traces     │ └─────────┘                                     │
│  │ Dashboards │                                                 │
│  └────────────┘                                                 │
│                                                                 │
└─────────────────────────────────────────────────────────────────┘

Rationale

1. OpenTelemetry is the CNCF standard for telemetry instrumentation, ensuring vendor neutrality
2. OTel Collector provides processing pipelines (PII scrubbing, semantic enrichment) before export
3. OpenObserve unifies all three signals into one backend, reducing operational complexity
4. Vanta automates compliance evidence collection directly from telemetry
5. Logfire provides Python developers with structured logging that auto-correlates with traces

Why Not Traditional Stack?

Constraints (MUST/MUST NOT)

Critical for generator choices. These constraints flow directly into manifests and SEA-DSL.

• MUST use OpenTelemetry SDKs for all instrumentation
• MUST route telemetry through OTel Collector (no direct-to-backend)
• MUST apply PII scrubbing processor in Collector pipeline
• MUST preserve semantic context (sea.domain, sea.concept, sea.regime_id) as resource attributes
• MUST export governance metrics via OpenTelemetry metrics (not custom endpoints)
• MUST integrate Vanta agent for compliance evidence collection
• MUST use Logfire for Python service structured logging
• MUST NOT store PII in observability data without scrubbing
• MUST NOT bypass OTel Collector for telemetry export

Isomorphic Guarantees

Defines structure-preserving mappings from this ADR to implementation.

System Invariants

Non-negotiable truths that must hold across the system.

Configuration Example

OTel Collector Pipeline

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
# otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 1s
    send_batch_size: 1024

  # PII Scrubbing
  attributes:
    actions:
      - key: user.email
        action: hash
      - key: user.ip_address
        action: delete

  # Semantic Context Enrichment
  resource:
    attributes:
      - key: sea.platform
        value: sea-forge
        action: insert

exporters:
  otlp/openobserve:
    endpoint: https://api.openobserve.ai
    headers:
      Authorization: ${OPENOBSERVE_TOKEN}

  otlp/vanta:
    endpoint: https://evidence.vanta.com
    headers:
      Authorization: ${VANTA_API_KEY}

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch, attributes, resource]
      exporters: [otlp/openobserve]

    metrics:
      receivers: [otlp]
      processors: [batch, resource]
      exporters: [otlp/openobserve, otlp/vanta]

    logs:
      receivers: [otlp]
      processors: [batch, attributes, resource]
      exporters: [otlp/openobserve]

Logfire Python Integration

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
# observability.py
import logfire

# Auto-correlate with OpenTelemetry traces
logfire.configure(
    token="${LOGFIRE_TOKEN}",
    service_name="sea-forge",
    send_to_logfire=True,
    console=logfire.ConsoleOptions(verbose=True),
)

# Structured logging with semantic context
logfire.info(
    "Policy evaluated",
    sea_domain="governance",
    sea_concept="PolicyEvaluation",
    decision="allowed",
    latency_ms=12.5,
)

Bounded Contexts Impacted

• semantic-core — Semantic context propagation
• architectural-governance — Compliance evidence export
• cognitive-extension — AI agent telemetry
• developer-tooling — Local observability setup

Consequences

Benefits

• Unified platform — Single pane of glass for all observability signals
• Vendor neutral — OpenTelemetry allows backend swapping without re-instrumentation
• Compliance automation — Vanta integration reduces audit preparation time by 80%
• Python DX — Logfire provides idiomatic logging with trace correlation
• Cost efficiency — OpenObserve open-source reduces infrastructure costs

Trade-offs

• Migration effort — Existing Prometheus instrumentations need OTel migration
• New tooling — Teams need to learn OpenObserve UI instead of Grafana
• Dependency on SaaS — Vanta is a SaaS dependency for compliance automation

Related Specs

• ADR-028: GovernedSpeed™ LLMOps Architecture (governance metrics)
• SDS-030: Semantic Observability Envelope (semantic context schema)
• SDS-047: Risk & Evidence Service (evidence collection)
• PRD-010: AI Governance Runtime (metrics requirements)