Behavior Correlation with OpenObserve

Runbook for Runtime Behavior Correlation OpenObserve Integration

Last Updated: 2026-01-25 Related Specs: ADR-029 (Observability Stack), SDS-030 (Semantic Observability Envelope), P3.3 (Runtime Behavior Correlation)

Overview

This runbook covers the integration between the SEA Behavior Correlation system and OpenObserve, including dashboard setup, alert configuration, and deep linking from the Workbench UI.

Architecture

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
┌─────────────────────────────────────────────────────────────────┐
│  Behavior Correlation → OpenObserve Integration                 │
├─────────────────────────────────────────────────────────────────┤
│                                                                 │
│  ┌──────────────────┐    OTLP    ┌──────────────────────┐      │
│  │  SEA Services    │ ────────>  │  OTel Collector      │      │
│  │  (traces/logs)   │            │  (PII scrubbing)     │      │
│  └──────────────────┘            └──────────┬───────────┘      │
│                                             │                    │
│                                             ▼                    │
│  ┌──────────────────────────────────────────────────────────┐  │
│  │              OpenObserve (sea-openobserve:5080)          │  │
│  │  ┌──────────────────────────────────────────────────┐   │  │
│  │  │  Traces │ Logs │ Metrics │ Dashboards │ Alerts   │   │  │
│  │  └──────────────────────────────────────────────────┘   │  │
│  └──────────────────────────┬───────────────────────────────┘  │
│                             │                                   │
│                             │ Deep Links                        │
│                             ▼                                   │
│  ┌──────────────────────────────────────────────────────────┐  │
│  │  Workbench UI (Runtime Correlation Dashboard)            │  │
│  │  - View traces in OpenObserve                            │  │
│  │  - View logs in OpenObserve                              │  │
│  │  - View metrics in OpenObserve                           │  │
│  └──────────────────────────────────────────────────────────┘  │
│                                                                 │
└─────────────────────────────────────────────────────────────────┘

Environment Configuration

Required Environment Variables

1
2
3
4
5
6
7
# OpenObserve Connection
OPENOBSERVE_URL=http://localhost:5080          # OpenObserve base URL
OPENOBSERVE_ORG=default                        # Organization name
OPENOBSERVE_AUTH=Basic <credentials>           # Authentication (optional)

# Alert Webhooks (optional)
ALERT_WEBHOOK_URL=https://hooks.slack.com/...  # Webhook for alerts

Docker Compose Setup

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
# infra/docker/docker-compose.dev.yml
services:
  sea-openobserve:
    image: public.ecr.aws/zinclabs/openobserve:latest
    ports:
      - "5080:5080"
    environment:
      ZO_ROOT_USER_EMAIL: admin@sea-forge.io
      ZO_ROOT_USER_PASSWORD: ${OPENOBSERVE_PASSWORD:-ComplexPass123!}
    volumes:
      - openobserve-data:/data

  otel-collector:
    image: otel/opentelemetry-collector-contrib:latest
    volumes:
      - ./infra/otel/otel-collector-config.yaml:/etc/otelcol-contrib/config.yaml
    command: ["--config=/etc/otelcol-contrib/config.yaml"]
    depends_on:
      - sea-openobserve
    environment:
      OPENOBSERVE_AUTH: ${OPENOBSERVE_AUTH}

OTel Collector Configuration

The OTel Collector exports telemetry to OpenObserve via the otlp/openobserve exporter:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
# infra/otel/otel-collector-config.yaml
exporters:
  otlp/openobserve:
    endpoint: "sea-openobserve:5080"
    tls:
      insecure: true
    headers:
      "Authorization": "Basic ${env:OPENOBSERVE_AUTH}"
      "organization": "default"

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, attributes/pii, batch]
      exporters: [debug, otlp/openobserve]

    metrics:
      receivers: [otlp]
      processors: [memory_limiter, resource/sea, batch]
      exporters: [debug, otlp/openobserve]

    logs:
      receivers: [otlp]
      processors: [memory_limiter, attributes/pii, batch]
      exporters: [debug, otlp/openobserve]

Key Features:

• PII Scrubbing: attributes/pii processor hashes emails and deletes IPs
• Semantic Enrichment: resource/sea processor adds sea.platform attribute
• Batch Processing: Reduces network overhead with batch processor

Dashboard Setup

Import the Behavior Drift Dashboard

1. Navigate to OpenObserve:
   • Open http://localhost:5080 in your browser
   • Login with credentials (admin@sea-forge.io / configured password)
2. Import Dashboard:
   • Go to Dashboard → New Dashboard → Import from JSON
   • Select infra/otel/dashboards/behavior-drift-dashboard.json
   • Click Import
3. Verify Dashboard Panels:
   • Drift Level Distribution (pie chart)
   • Average Drift Score Trend (line chart)
   • High Drift Nodes (table)
   • Drift by Context (bar chart)
   • Recent Drift Events Timeline (table)
   • Evidence Velocity (gauge)
   • Alert Rule Violations (table)

Dashboard Variables

The dashboard includes variables for filtering:

• context: Filter by bounded context (multi-select)
• time_range: Time window for queries (default: last 1 hour)

Alert Configuration

Built-in Alerts

The dashboard includes three pre-configured alerts:

Configure Alert Actions

1. Set Webhook URL:

   1
   export ALERT_WEBHOOK_URL="https://hooks.slack.com/services/YOUR/WEBHOOK/URL"
   

2. Enable Alerts in OpenObserve:
   • Go to Alerts → New Alert
   • Select from the predefined alert templates
   • Configure the webhook action
   • Set the alert to Enabled
3. Test Alerts:
   • Use the synthetic telemetry fixtures to trigger alerts:

     1
     just test-behavior-drift-gate --fail
     

Deep Linking from Workbench UI

The Workbench UI provides deep links to OpenObserve for traces, logs, and metrics.

API Endpoint

1
GET /api/behavior/openobserve-config

Response:

1
2
3
4
5
{
  "traceUrl": "http://localhost:5080/web/default/traces",
  "logUrl": "http://localhost:5080/web/default/logs",
  "metricUrl": "http://localhost:5080/web/default/metrics"
}

Client-Side Link Builders

The behavior-api.ts module provides safe deep link builders:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
import {
  buildOpenObserveTraceLink,
  buildOpenObserveLogLink,
  buildOpenObserveMetricLink,
  getOpenObserveConfig
} from '@/lib/behavior-api';

// Get configuration
const config = await getOpenObserveConfig();

// Build trace link
const traceLink = buildOpenObserveTraceLink(config.traceUrl, traceId, {
  startTime: '2026-01-25T10:00:00Z',
  endTime: '2026-01-25T11:00:00Z',
  service: 'workbench-bff'
});

// Build log link
const logLink = buildOpenObserveLogLink(config.logUrl, traceId, {
  startTime: '2026-01-25T10:00:00Z'
});

// Build metric link
const metricLink = buildOpenObserveMetricLink(config.metricUrl, 'workbench-bff', {
  metricName: 'sea_behavior_drift_score'
});

UI Integration

The BehaviorDriftCard component displays deep links in the evidence details:

1
2
3
4
// When expanded, each evidence item shows:
- Trace ID → OpenObserve Traces (opens in new tab)
- Log ID → OpenObserve Logs (opens in new tab)
- Metric Name → OpenObserve Metrics (opens in new tab)

Troubleshooting

Issue: No Telemetry in OpenObserve

Symptoms:

• Dashboard shows no data
• Traces/logs/metrics not appearing

Diagnosis:

1
2
3
4
5
6
7
8
# Check OTel Collector logs
docker logs otel-collector

# Check OpenObserve logs
docker logs sea-openobserve

# Verify OTLP endpoint is reachable
curl -v http://localhost:4317

Solutions:

1. Ensure OTel Collector is running and connected to OpenObserve
2. Verify OPENOBSERVE_AUTH is set correctly
3. Check that services are exporting OTLP telemetry
4. Verify network connectivity between collector and OpenObserve

Issue: Deep Links Don’t Work

Symptoms:

• Clicking trace links shows 404 or empty pages
• OpenObserve links are incorrect

Diagnosis:

1
2
3
4
5
# Check environment variables
env | grep OPENOBSERVE

# Test API endpoint
curl http://localhost:8010/api/behavior/openobserve-config

Solutions:

1. Set OPENOBSERVE_URL to the correct base URL (include port)
2. Set OPENOBSERVE_ORG to your organization name (default: “default”)
3. Restart the Workbench BFF service after changing environment variables

Issue: Alerts Not Firing

Symptoms:

• Drift exceeds threshold but no alert triggered
• Alert shows “Not Triggered” status

Diagnosis:

1. Check alert query syntax in OpenObserve
2. Verify alert is enabled
3. Check webhook URL is configured

Solutions:

1. Use Test Alert button in OpenObserve to verify query
2. Ensure ALERT_WEBHOOK_URL is set and accessible
3. Check alert condition threshold is appropriate

Validation Checklist

• OpenObserve is accessible at configured URL
• OTel Collector is exporting to OpenObserve
• Dashboard is imported and showing data
• Alerts are configured with webhook URLs
• Deep links work from Workbench UI
• Synthetic telemetry triggers alerts correctly

References

• ADR-029: Observability Stack Architecture
• SDS-030: Semantic Observability Envelope
• P3.3 Plan: Runtime Behavior Correlation Implementation Plan
• Dashboard JSON: infra/otel/dashboards/behavior-drift-dashboard.json
• OTel Collector Config: infra/otel/otel-collector-config.yaml