Drift Remediation with GitHub PRs

This guide covers configuring and using automatic drift remediation with GitHub pull requests.

Overview

The drift remediation system can automatically create pull requests to fix detected drift. This is controlled by the GITHUB_REMEDIATION_ENABLED environment variable and integrates with the GitHubPRCreator adapter.

Prerequisites

GitHub Token Requirements

To create remediation PRs, you need a GitHub personal access token (PAT) with the following scopes:

Note: The repo scope includes both pull_requests:write and contents:write. If you want to be more restrictive, you can use fine-grained tokens with only the specific scopes needed.

Creating a GitHub Token

1. Go to GitHub Settings → Developer settings → Personal access tokens → Tokens (classic)
2. Click “Generate new token” → “Generate new token (classic)”
3. Set the following scopes:
   • repo (for full repository access)
   • Or individually: pull_requests:write, contents:write
4. Generate and copy the token

Configuration

Environment Variables

*Required when GITHUB_REMEDIATION_ENABLED=true

Environment-Based Behavior

The remediation engine behaves differently based on the ENVIRONMENT variable:

Example Configuration

1
2
3
4
5
6
7
8
9
10
11
12
13
# Enable remediation with real GitHub PRs (production)
export GITHUB_REMEDIATION_ENABLED=true
export ENVIRONMENT=production
export GITHUB_TOKEN=ghp_your_token_here
export GITHUB_REPOSITORY=your-org/your-repo

# Enable remediation with mock PRs (development/staging)
export GITHUB_REMEDIATION_ENABLED=true
export ENVIRONMENT=development
# No token needed for mock mode

# Disable remediation entirely
export GITHUB_REMEDIATION_ENABLED=false

Security Best Practices

Token Storage

NEVER commit GitHub tokens to your repository. Use one of these secure storage methods:

1. Secrets Manager (Recommended for production):

   1
   2
   3
   4
   5
   # AWS Secrets Manager
   aws secretsmanager get-secret-value --secret-id drift-remediation-token
   
   # HashiCorp Vault
   vault kv get -field=token secret/drift-remediation
   

2. Environment Variables (Development):

   1
   2
   # Load from .env file (never commit .env)
   export GITHUB_TOKEN=$(cat .env.local | grep GITHUB_TOKEN | cut -d= -f2)
   

3. CI/CD Secrets:
   • GitHub Actions: Repository secrets
   • GitLab CI: Masked variables
   • Jenkins: Credentials binding

Token Rotation

Regularly rotate your GitHub tokens:

1. Recommended cadence: Every 90 days
2. Automation: Use a token provider function

   1
   2
   3
   4
   5
   6
   from src.adapters.pr_creator import GitHubPRCreator
   
   def token_provider():
       return fetch_token_from_vault()
   
   creator = GitHubPRCreator(token_provider=token_provider)
   

3. Revocation: Immediately revoke compromised tokens

Principle of Least Privilege

• Use fine-grained tokens instead of classic tokens when possible
• Limit token scope to specific repositories
• Set token expiration dates
• Monitor token usage in GitHub audit logs

PR Creation Behavior

Draft PRs

All remediation PRs are created as draft PRs by default. This requires:

• Manual review before merging
• Explicit approval from a team member
• Verification that the remediation is correct

PR Labels

Remediation PRs are automatically labeled with:

• drift-remediation: Identifies the PR as auto-generated for drift
• auto-generated: Indicates it was created by the system

Branch Naming

PR branches follow this pattern:

1
drift-remediation/{context}-{random-8-chars}

Example: drift-remediation/workbench-bff-a1b2c3d4

PR Title and Body

Title Format:

1
fix(drift): {action} {context}

Body Template:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
## Drift Remediation

**Action**: {action}
**Context**: {context}
**Node**: {node_id}

### Description
{suggestion_description}

### Labels
- drift-remediation
- auto-generated

---
*This PR was automatically generated by the SEA-Forge drift remediation system.*

Error Handling

Rate Limiting

The GitHubPRCreator includes automatic retry logic with exponential backoff:

If all retries fail, the remediation is marked as failed but logged in the audit trail.

Authentication Failures

On 401/bad credentials errors:

• If a token_provider is configured, the token is refreshed
• Retries continue with the new token
• If no provider is set, the remediation fails

Common Errors

Idempotency Guarantees

The remediation system provides idempotency through:

1. Audit Trail: Every remediation action is logged with a unique audit_id
2. Idempotent Regeneration: Running just pipeline <context> multiple times produces the same result
3. Branch Uniqueness: Random suffix prevents branch name collisions
4. No Duplicate PRs: The system checks for existing PRs before creating new ones

Checking Audit History

1
2
3
4
5
# View recent remediation actions
curl http://localhost:8000/drift/history?limit=10

# Or read the audit log directly
cat artifacts/workbench/drift-remediation-audit.jsonl | jq .

Audit Trail Format

Each remediation action is logged as a JSON line:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
{
  "audit_id": "550e8400-e29b-41d4-a716-446655440000",
  "node_id": "workbench-bff:api/user_routes",
  "suggestion_id": "regenerate-from-spec",
  "action": "regenerate",
  "status": "success",
  "pr_url": "https://github.com/org/repo/pull/123",
  "error_message": null,
  "timestamp": "2024-01-15T10:30:00Z",
  "metadata": {
    "branch": "drift-remediation-workbench-bff-a1b2c3d4",
    "dry_run": false
  }
}

Status Values

Testing

Local Development

Use MockPRCreator for local testing:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
from src.adapters.pr_creator import MockPRCreator
from src.adapters.remediation_engine import RemediationEngine

mock_creator = MockPRCreator()
engine = RemediationEngine(pr_creator=mock_creator)

result = engine.execute(
    node_id="test-node",
    suggestion_id="regenerate-001",
    create_pr=True,
    dry_run=False,
)

# Verify mock behavior
assert len(mock_creator.created_prs) == 1
print(mock_creator.created_prs[0]["pr_url"])
# Output: https://github.com/mock/repo/pull/1

Dry Run Mode

Test remediation without creating PRs:

1
2
3
4
5
6
7
8
9
result = engine.execute(
    node_id="test-node",
    suggestion_id="regenerate-001",
    create_pr=False,  # Skip PR creation
    dry_run=True,     # Simulate only
)

assert result.metadata["dry_run"] is True
assert result.status.value == "success"

Running Tests

1
2
3
4
5
6
7
8
# Run remediation PR tests
pytest services/workbench-bff/tests/test_remediation_pr.py -v

# Run all drift-related tests
pytest services/workbench-bff/tests/test_drift*.py -v

# Run with coverage
pytest services/workbench-bff/tests/test_remediation_pr.py --cov=src.adapters.pr_creator --cov=src.adapters.remediation_engine

Troubleshooting

PRs Not Being Created

1. Check environment variables:

   1
   2
   3
   echo $GITHUB_REMEDIATION_ENABLED  # Should be "true"
   echo $ENVIRONMENT  # Should be "production" for real PRs
   echo $GITHUB_REPOSITORY  # Should be "owner/repo"
   

2. Check logs for errors:

   1
   tail -f artifacts/workbench/drift-remediation-audit.jsonl | jq 'select(.status=="failed")'
   

3. Verify token permissions:

   1
   2
   curl -H "Authorization: token $GITHUB_TOKEN" \
     https://api.github.com/user/repos
   

Rate Limit Issues

If you encounter rate limiting:

1. Increase backoff time in GitHubPRCreator.INITIAL_BACKOFF_SECONDS
2. Reduce concurrent remediation requests
3. Use a GitHub App with higher rate limits (instead of PAT)

Branch Already Exists

If the branch name collides:

1. The system uses a random 8-character suffix to avoid collisions
2. If collision occurs, the PR creation fails with a clear error
3. Retry the remediation to generate a new branch

Advanced Usage

Custom Token Provider

Implement dynamic token refresh:

1
2
3
4
5
6
7
8
9
10
from src.adapters.pr_creator import GitHubPRCreator

def fetch_token():
    # Custom token fetching logic
    return secrets_manager.get("drift-remediation-token")

creator = GitHubPRCreator(
    repo="org/repo",
    token_provider=fetch_token
)

Custom Base Branch

Specify a different target branch:

1
2
3
4
5
6
7
8
pr_result = creator.create_remediation_pr(
    title="Custom PR",
    body="Description",
    branch="feature-branch",
    changes=[...],
    base_branch="main",  # Default is "dev"
    draft=True,
)

Max PR Per Run

Limit the number of PRs created in a single batch:

1
2
3
4
5
6
7
from src.adapters.remediation_engine import RemediationConfig

config = RemediationConfig(
    max_pr_per_run=3,  # Maximum 3 PRs per batch
)

engine = RemediationEngine(config=config, pr_creator=creator)

References

• Spec Source: docs/workdocs/auto-drift-healing-plan.md (P3.2 Automatic Drift Remediation)
• Implementation: services/workbench-bff/src/adapters/pr_creator.py
• API Routes: services/workbench-bff/src/api/drift_routes.py
• Tests: services/workbench-bff/tests/test_remediation_pr.py

Support

For issues or questions:

1. Check the audit trail for error details
2. Review GitHub token permissions
3. Consult the drift remediation plan document
4. Contact the infrastructure team for token/vault issues