Implementation Plan: LLM Provider Abstraction

Phase 9 Artifact — This plan enables vertical slice delivery per ENGINEERING.SOP.md.

Purpose

Provide a unified LLM provider interface via LiteLLM, enabling SEA™ services to interact with OpenAI, Anthropic, Ollama, and OpenRouter through a single abstraction layer with Policy Gateway integration.

Upstream/Downstream Dependencies

Critical ordering — This plan must be implemented after its dependencies and before services that consume it.

Pre-Flight Validation

STOP. Before implementing, validate all input specifications.

ADR Validation

PRD Validation

SDS Validation

Provenance Chain

Complete traceability from ADR → PRD → SDS → Implementation.

graph TD
    ADR35[ADR-035: LLM Provider Abstraction] --> PRD10[PRD-010: AI Governance Runtime]
    ADR28[ADR-028: GovernedSpeed™ LLMOps] --> PRD10
    PRD10 --> SDS49[SDS-049: LLM Provider Service]
    SDS49 --> SDS47[SDS-047: GovernedSpeed™ Runtime]

    SDS49 --> C1A[C1A: Domain + Ports]
    SDS49 --> C1B[C1B: LiteLLM Adapter]
    SDS49 --> C2A[C2A: Fake Adapter + Tests]

    style ADR35 fill:#e1f5ff
    style ADR28 fill:#e1f5ff
    style PRD10 fill:#fff4e1
    style SDS49 fill:#e8f5e9
    style SDS47 fill:#e8f5e9

Implementation Order

Per ENGINEERING.SOP.md Phase 9.

1. Domain model + invariants — ProviderConfig, ModelSpec, ChatMessage, Embedding
2. Ports — LlmProviderPort interface + FakeLlmAdapter
3. Adapters — LiteLLMAdapter wrapping LiteLLM Python
4. Wiring — HTTP client for TypeScript consumers
5. Observability — OpenTelemetry spans on all LLM calls
6. Tests — Unit tests with FakeLlmAdapter
7. Feature flag — llm.provider.enabled for rollout

Proposed Cycles (Worktree-First)

Wave 1 (Parallel)

• C1A: Domain model and port interface
  • Worktree: ../SEA-p31-c1A
  • Implements: ProviderConfig, ModelSpec, ChatMessage, Embedding, LlmProviderPort
  • Satisfies: SDS-049 §2, §7
  • Generator: just generator-bc llm-provider
  • Files:
    • libs/llm-provider/domain/src/lib/provider-config.ts
    • libs/llm-provider/domain/src/lib/chat-message.ts
    • libs/llm-provider/domain/src/lib/embedding.ts
    • libs/llm-provider/ports/src/lib/llm-provider.port.ts
• C1B: LiteLLM Python adapter
  • Worktree: ../SEA-p31-c1B
  • Implements: LiteLLMAdapter with Policy Gateway routing
  • Satisfies: SDS-049 §8, INV-LLM-001
  • Files:
    • services/llm-provider/src/adapters/litellm_adapter.py
    • services/llm-provider/src/api/routes.py
    • services/llm-provider/Dockerfile

Wave 2 (Depends on Wave 1)

• C2A: Fake adapter and test suite
  • Worktree: ../SEA-p31-c2A
  • Implements: FakeLlmAdapter with deterministic responses
  • Satisfies: SDS-049 §9
  • Depends on: C1A
  • Files:
    • libs/llm-provider/adapters/src/lib/fake.adapter.ts
    • libs/llm-provider/adapters/src/lib/fake.adapter.spec.ts

Wave 3 (Depends on Wave 2)

• C3A: TypeScript HTTP client for LiteLLM service
  • Worktree: ../SEA-p31-c3A
  • Implements: HTTP client that calls LiteLLM service
  • Satisfies: SDS-049 §11
  • Depends on: C1B
  • Files:
    • libs/llm-provider/adapters/src/lib/litellm.adapter.ts
    • libs/llm-provider/adapters/src/lib/litellm.adapter.spec.ts

Nx Generators to Use

Expected Filetree After Implementation

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
libs/llm-provider/
├── domain/
│   └── src/
│       ├── index.ts
│       └── lib/
│           ├── provider-config.ts       # [SDS-049: Entity ProviderConfig]
│           ├── model-spec.ts            # [SDS-049: Entity ModelSpec]
│           ├── chat-message.ts          # [SDS-049: VO ChatMessage]
│           └── embedding.ts             # [SDS-049: VO Embedding]
├── ports/
│   └── src/
│       └── lib/
│           ├── llm-provider.port.ts     # [SDS-049: PORT-LLM-001]
│           └── policy-gateway.port.ts   # [SDS-049: PORT-LLM-002]
├── adapters/
│   └── src/
│       └── lib/
│           ├── litellm.adapter.ts       # HTTP client to Python service
│           ├── litellm.adapter.spec.ts
│           ├── fake.adapter.ts          # Deterministic test double
│           └── fake.adapter.spec.ts
└── application/
    └── src/
        └── lib/
            ├── complete-chat.handler.ts  # [SDS-049: CMD-LLM-001]
            └── generate-embedding.handler.ts  # [SDS-049: CMD-LLM-002]

services/llm-provider/                    # Python FastAPI service
├── src/
│   ├── adapters/
│   │   └── litellm_adapter.py           # LiteLLM wrapper
│   ├── api/
│   │   └── routes.py                    # OpenAI-compatible API
│   └── main.py
├── pyproject.toml
└── Dockerfile

Dependencies Introduced

Validation & Verification

Pre-Merge Checks

• just spec-guard passes
• just test passes
• just nx-run workspace check passes
• git diff --exit-code (repo clean after generation)

Post-Implementation Verification

• Ollama responds to chat completion via LiteLLM adapter
• OpenAI/Anthropic fallback works when Ollama unavailable
• Policy Gateway filters requests (PII detection tested)
• OpenTelemetry spans emitted with llm.model, llm.provider attributes
• FakeLlmAdapter returns deterministic responses for tests

Verification Commands

1
2
3
4
5
6
7
8
9
10
# Start Ollama locally
just dev-up

# Test LLM service
curl -X POST http://localhost:8001/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "ollama/llama3.2", "messages": [{"role": "user", "content": "Hello"}]}'

# Run adapter tests
just test libs/llm-provider

Risks & Mitigations

Open Questions

1. Should LiteLLM run as a sidecar or dedicated service? Dedicated service
2. How to handle streaming responses through Policy Gateway? Chunk-level filtering

References

Note: SDS-042 (Policy Gateway Service) has been superseded by SDS-047 per ADR-031.