Milestone: M7 — Reclaim Protocol | SOW Reference: FR4 | Requirement Clarity: ⚠️ Depends on M7-01 sign-off | Dev Status: ❌ Not started Moved from docs/requirements/tasks/POC/external-data-source-technical-specifications.md. Note this covers a broader scope (HMRC + Open Banking + generic "Reclaim Protocol SDK") than the SOW's FR4, which only requires HMRC for the MVP, extensible later — treat the Open Banking sections as future-proofing reference, not in-scope for M7. Unmodified below.

M7-02 External Data Source Integration - Technical Specifications¶

REC Verifiable Credentialing Platform - Reclaim Protocol Integration¶

Executive Summary¶

This document defines the technical specifications for integrating external data sources (HMRC, Open Banking, etc.) through Reclaim Protocol to enable individual credential claiming capabilities.

1. System Architecture Overview¶

Current Architecture (Phase 1)¶

Organization Admin → Employee Data → VNF Credential → Email → Employee Claims

Target Architecture (Phase 2)¶

Individual User → External Data Sources → Reclaim Protocol → 
Data Validation → VNF Credential → Personal Wallet

Hybrid Architecture (Final State)¶

┌─ Organization Admin → Employee Data ─┐
│                                      ├─→ VNF Credential → Distribution
└─ Individual User → External Data ────┘

2. External Data Source Specifications¶

2.1 HMRC (HM Revenue & Customs) Integration¶

API Endpoints:¶

interface HMRCApiEndpoints {
  // Employment History
  employmentHistory: '/individuals/employment/paye/{nino}/{taxYear}';
  // P60 Data
  p60Data: '/individuals/employment/p60/{nino}/{taxYear}';
  // P45 Data  
  p45Data: '/individuals/employment/p45/{nino}';
  // Real Time Information
  rtiData: '/individuals/employment/rti/{nino}/{taxYear}';
  // Self Assessment
  selfAssessment: '/individuals/self-assessment/{utr}/{taxYear}';
}

Authentication:¶

interface HMRCAuthConfig {
  clientId: string;
  clientSecret: string;
  scope: string[];
  redirectUri: string;
  authUrl: 'https://test-api.service.hmrc.gov.uk/oauth/authorize';
  tokenUrl: 'https://test-api.service.hmrc.gov.uk/oauth/token';
}

Data Schema:¶

interface HMRCEmploymentData {
  nino: string;
  taxYear: string;
  employments: Array<{
    employerName: string;
    employerRef: string;
    startDate: string;
    endDate?: string;
    payrollId: string;
    totalPay: number;
    totalTax: number;
    jobTitle?: string;
    workingHours?: number;
  }>;
  p60Data?: {
    totalPay: number;
    totalTax: number;
    employerName: string;
    employerRef: string;
  };
}

2.2 Open Banking Integration¶

Supported Banks:¶

Barclays, HSBC, Lloyds, NatWest, Santander, TSB
Challenger banks: Monzo, Starling, Revolut
Building societies: Nationwide, Halifax

API Endpoints:¶

interface OpenBankingEndpoints {
  // Account Information
  accounts: '/open-banking/v3.1/aisp/accounts';
  accountDetails: '/open-banking/v3.1/aisp/accounts/{accountId}';
  // Transaction History
  transactions: '/open-banking/v3.1/aisp/accounts/{accountId}/transactions';
  // Standing Orders (for salary verification)
  standingOrders: '/open-banking/v3.1/aisp/accounts/{accountId}/standing-orders';
  // Direct Debits
  directDebits: '/open-banking/v3.1/aisp/accounts/{accountId}/direct-debits';
}

Data Schema:¶

interface OpenBankingData {
  accountId: string;
  accountType: 'Personal' | 'Business';
  currency: string;
  accountSubType: 'CurrentAccount' | 'Savings';
  transactions: Array<{
    transactionId: string;
    amount: number;
    currency: string;
    creditDebitIndicator: 'Credit' | 'Debit';
    status: 'Booked' | 'Pending';
    bookingDateTime: string;
    valueDateTime: string;
    transactionInformation: string;
    merchantDetails?: {
      merchantName: string;
      merchantCategoryCode: string;
    };
  }>;
  salaryPayments?: Array<{
    employerName: string;
    amount: number;
    frequency: 'Monthly' | 'Weekly' | 'Fortnightly';
    lastPaymentDate: string;
  }>;
}

2.3 Other Government Data Sources¶

DVLA (Driver and Vehicle Licensing Agency):¶

interface DVLAData {
  licenceNumber: string;
  licenceType: string;
  validFrom: string;
  validTo: string;
  categories: string[];
  endorsements?: Array<{
    code: string;
    description: string;
    dateOfOffence: string;
  }>;
}

Companies House:¶

interface CompaniesHouseData {
  companyNumber: string;
  companyName: string;
  companyStatus: string;
  incorporationDate: string;
  officers: Array<{
    name: string;
    role: string;
    appointedOn: string;
    resignedOn?: string;
  }>;
}

3. Reclaim Protocol Integration¶

3.1 Reclaim Protocol SDK¶

Installation:¶

npm install @reclaimprotocol/js-sdk

Configuration:¶

interface ReclaimConfig {
  apiKey: string;
  baseUrl: string;
  version: 'v1';
  timeout: number;
  retryAttempts: number;
}

const reclaimConfig: ReclaimConfig = {
  apiKey: process.env.RECLAIM_API_KEY,
  baseUrl: 'https://api.reclaimprotocol.org',
  version: 'v1',
  timeout: 30000,
  retryAttempts: 3,
};

3.2 Data Source Registration¶

Register External Data Sources:¶

interface DataSourceConfig {
  id: string;
  name: string;
  type: 'government' | 'financial' | 'employment';
  authMethod: 'oauth2' | 'api_key' | 'certificate';
  endpoints: Record<string, string>;
  credentials: Record<string, string>;
  rateLimit: {
    requestsPerMinute: number;
    requestsPerDay: number;
  };
}

const hmrcDataSource: DataSourceConfig = {
  id: 'hmrc-uk',
  name: 'HM Revenue & Customs',
  type: 'government',
  authMethod: 'oauth2',
  endpoints: {
    auth: 'https://test-api.service.hmrc.gov.uk/oauth/authorize',
    token: 'https://test-api.service.hmrc.gov.uk/oauth/token',
    employment: 'https://test-api.service.hmrc.gov.uk/individuals/employment',
  },
  credentials: {
    clientId: process.env.HMRC_CLIENT_ID,
    clientSecret: process.env.HMRC_CLIENT_SECRET,
  },
  rateLimit: {
    requestsPerMinute: 10,
    requestsPerDay: 1000,
  },
};

3.3 Data Retrieval Workflow¶

Step 1: User Authentication¶

async function initiateDataSourceAuth(
  userId: string,
  dataSourceId: string,
): Promise<AuthSession> {
  const authUrl = await reclaimClient.generateAuthUrl({
    userId,
    dataSourceId,
    scopes: ['employment:read', 'personal:read'],
    redirectUri: `${process.env.BASE_URL}/auth/callback/${dataSourceId}`,
  });

  return {
    sessionId: generateSessionId(),
    authUrl,
    expiresAt: new Date(Date.now() + 15 * 60 * 1000), // 15 minutes
    status: 'pending',
  };
}

Step 2: Data Extraction¶

async function extractUserData(
  sessionId: string,
  authCode: string,
): Promise<ExternalDataResult> {
  try {
    const accessToken = await reclaimClient.exchangeAuthCode({
      sessionId,
      authCode,
    });

    const userData = await reclaimClient.extractData({
      accessToken,
      dataTypes: ['employment', 'income', 'identity'],
      dateRange: {
        from: '2020-01-01',
        to: new Date().toISOString().split('T')[0],
      },
    });

    return {
      success: true,
      data: userData,
      extractedAt: new Date(),
      dataQuality: calculateDataQuality(userData),
    };
  } catch (error) {
    return {
      success: false,
      error: error.message,
      retryable: isRetryableError(error),
    };
  }
}

4. Data Validation & Quality Assurance¶

4.1 Data Validation Rules¶

Employment Data Validation:¶

interface ValidationRule {
  field: string;
  type: 'required' | 'format' | 'range' | 'custom';
  rule: string | RegExp | Function;
  message: string;
}

const employmentValidationRules: ValidationRule[] = [
  {
    field: 'employerName',
    type: 'required',
    rule: 'notEmpty',
    message: 'Employer name is required',
  },
  {
    field: 'startDate',
    type: 'format',
    rule: /^\d{4}-\d{2}-\d{2}$/,
    message: 'Start date must be in YYYY-MM-DD format',
  },
  {
    field: 'salary',
    type: 'range',
    rule: (value: number) => value >= 0 && value <= 1000000,
    message: 'Salary must be between 0 and 1,000,000',
  },
  {
    field: 'nino',
    type: 'format',
    rule: /^[A-CEGHJ-PR-TW-Z]{1}[A-CEGHJ-NPR-TW-Z]{1}[0-9]{6}[A-D]{1}$/,
    message: 'Invalid National Insurance Number format',
  },
];

Data Quality Scoring:¶

interface DataQualityMetrics {
  completeness: number; // 0-100%
  accuracy: number; // 0-100%
  consistency: number; // 0-100%
  timeliness: number; // 0-100%
  overall: number; // 0-100%
}

function calculateDataQuality(data: ExternalData): DataQualityMetrics {
  const completeness = calculateCompleteness(data);
  const accuracy = validateDataAccuracy(data);
  const consistency = checkDataConsistency(data);
  const timeliness = assessDataTimeliness(data);

  return {
    completeness,
    accuracy,
    consistency,
    timeliness,
    overall: (completeness + accuracy + consistency + timeliness) / 4,
  };
}

4.2 Fraud Detection¶

Anomaly Detection:¶

interface FraudDetectionRule {
  name: string;
  description: string;
  severity: 'low' | 'medium' | 'high';
  check: (data: ExternalData) => boolean;
}

const fraudDetectionRules: FraudDetectionRule[] = [
  {
    name: 'salary_anomaly',
    description: 'Salary significantly higher than industry average',
    severity: 'medium',
    check: (data) => data.salary > getIndustryAverage(data.jobTitle) * 3,
  },
  {
    name: 'employment_gap',
    description: 'Unexplained employment gaps',
    severity: 'low',
    check: (data) => hasUnexplainedGaps(data.employmentHistory),
  },
  {
    name: 'data_inconsistency',
    description: 'Inconsistent data across sources',
    severity: 'high',
    check: (data) => hasInconsistentData(data),
  },
];

5. Database Schema Extensions¶

5.1 New Entities¶

External Data Sources:¶

CREATE TABLE external_data_sources (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    name VARCHAR(255) NOT NULL,
    type VARCHAR(50) NOT NULL, -- 'government', 'financial', 'employment'
    api_base_url VARCHAR(500) NOT NULL,
    auth_method VARCHAR(50) NOT NULL, -- 'oauth2', 'api_key', 'certificate'
    is_active BOOLEAN DEFAULT true,
    rate_limit_per_minute INTEGER DEFAULT 10,
    rate_limit_per_day INTEGER DEFAULT 1000,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

User Consents:¶

CREATE TABLE user_consents (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    user_id UUID NOT NULL REFERENCES users(id),
    data_source_id UUID NOT NULL REFERENCES external_data_sources(id),
    consent_given_at TIMESTAMP NOT NULL,
    consent_expires_at TIMESTAMP,
    scopes JSONB NOT NULL, -- ['employment:read', 'income:read']
    is_active BOOLEAN DEFAULT true,
    withdrawn_at TIMESTAMP,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

Data Claims:¶

CREATE TABLE data_claims (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    user_id UUID NOT NULL REFERENCES users(id),
    data_source_id UUID NOT NULL REFERENCES external_data_sources(id),
    consent_id UUID NOT NULL REFERENCES user_consents(id),
    status VARCHAR(50) NOT NULL, -- 'pending', 'processing', 'completed', 'failed'
    raw_data JSONB,
    processed_data JSONB,
    data_quality_score DECIMAL(5,2),
    extracted_at TIMESTAMP,
    error_message TEXT,
    retry_count INTEGER DEFAULT 0,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

External Credentials:¶

CREATE TABLE external_credentials (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    user_id UUID NOT NULL REFERENCES users(id),
    data_claim_id UUID NOT NULL REFERENCES data_claims(id),
    credential_type VARCHAR(100) NOT NULL,
    vnf_credential_id VARCHAR(255),
    credential_data JSONB NOT NULL,
    status VARCHAR(50) NOT NULL, -- 'draft', 'issued', 'claimed', 'revoked'
    issued_at TIMESTAMP,
    claimed_at TIMESTAMP,
    expires_at TIMESTAMP,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

5.2 Indexes and Constraints¶

-- Performance indexes
CREATE INDEX idx_user_consents_user_id ON user_consents(user_id);
CREATE INDEX idx_user_consents_data_source ON user_consents(data_source_id);
CREATE INDEX idx_data_claims_user_id ON data_claims(user_id);
CREATE INDEX idx_data_claims_status ON data_claims(status);
CREATE INDEX idx_external_credentials_user_id ON external_credentials(user_id);

-- Unique constraints
ALTER TABLE user_consents ADD CONSTRAINT unique_active_consent 
    UNIQUE (user_id, data_source_id) WHERE is_active = true;

6. API Specifications¶

6.1 External Data Source Management¶

List Available Data Sources:¶

GET /api/v1/external-data-sources

Response: {
  dataSources: Array<{
    id: string;
    name: string;
    type: 'government' | 'financial' | 'employment';
    description: string;
    supportedCredentialTypes: string[];
    authMethod: string;
    isAvailable: boolean;
  }>;
}

Initiate Data Source Connection:¶

POST /api/v1/external-data-sources/{id}/connect

Request: {
  scopes: string[];
  redirectUri?: string;
}

Response: {
  sessionId: string;
  authUrl: string;
  expiresAt: string;
}

6.2 Data Extraction & Processing¶

Process Authentication Callback:¶

POST /api/v1/external-data-sources/{id}/callback

Request: {
  sessionId: string;
  authCode: string;
  state?: string;
}

Response: {
  claimId: string;
  status: 'processing' | 'completed' | 'failed';
  estimatedCompletionTime?: string;
}

Get Data Extraction Status:¶

GET /api/v1/data-claims/{claimId}

Response: {
  id: string;
  status: 'pending' | 'processing' | 'completed' | 'failed';
  progress: number; // 0-100
  dataQuality?: {
    completeness: number;
    accuracy: number;
    overall: number;
  };
  extractedData?: any;
  error?: string;
}

6.3 Credential Generation from External Data¶

Preview Credential from External Data:¶

POST /api/v1/credentials/preview-from-external-data

Request: {
  claimId: string;
  credentialType: string;
  customData?: Record<string, any>;
}

Response: {
  credentialPreview: any;
  dataMapping: Record<string, string>;
  qualityWarnings: string[];
}

Generate Credential from External Data:¶

POST /api/v1/credentials/generate-from-external-data

Request: {
  claimId: string;
  credentialType: string;
  customData?: Record<string, any>;
  confirmQualityWarnings: boolean;
}

Response: {
  credentialId: string;
  vnfCredentialId: string;
  status: 'issued';
  claimUrl: string;
  expiresAt: string;
}

7. Security Specifications¶

7.1 Data Encryption¶

Encryption at Rest:¶

interface EncryptionConfig {
  algorithm: 'AES-256-GCM';
  keyRotationPeriod: '90 days';
  keyManagement: 'AWS KMS' | 'Azure Key Vault' | 'HashiCorp Vault';
}

// Encrypt sensitive external data
function encryptExternalData(data: any, keyId: string): EncryptedData {
  const cipher = crypto.createCipher('aes-256-gcm', getEncryptionKey(keyId));
  const encrypted = cipher.update(JSON.stringify(data), 'utf8', 'hex');
  return {
    data: encrypted + cipher.final('hex'),
    keyId,
    algorithm: 'AES-256-GCM',
    iv: cipher.getAuthTag().toString('hex'),
  };
}

Encryption in Transit:¶

// All external API calls must use TLS 1.3
const httpsAgent = new https.Agent({
  secureProtocol: 'TLSv1_3_method',
  ciphers: 'ECDHE-RSA-AES256-GCM-SHA384',
  honorCipherOrder: true,
});

7.2 Access Control¶

Role-Based Access Control:¶

enum ExternalDataPermission {
  VIEW_DATA_SOURCES = 'external_data:view_sources',
  CONNECT_DATA_SOURCE = 'external_data:connect',
  VIEW_OWN_DATA = 'external_data:view_own',
  GENERATE_CREDENTIALS = 'external_data:generate_credentials',
  ADMIN_ALL_DATA = 'external_data:admin_all',
}

interface UserRole {
  name: string;
  permissions: ExternalDataPermission[];
}

const individualUserRole: UserRole = {
  name: 'individual_user',
  permissions: [
    ExternalDataPermission.VIEW_DATA_SOURCES,
    ExternalDataPermission.CONNECT_DATA_SOURCE,
    ExternalDataPermission.VIEW_OWN_DATA,
    ExternalDataPermission.GENERATE_CREDENTIALS,
  ],
};

7.3 Audit Logging¶

Audit Event Types:¶

enum AuditEventType {
  DATA_SOURCE_CONNECTED = 'data_source_connected',
  DATA_EXTRACTED = 'data_extracted',
  CREDENTIAL_GENERATED = 'credential_generated',
  CONSENT_GIVEN = 'consent_given',
  CONSENT_WITHDRAWN = 'consent_withdrawn',
  DATA_ACCESSED = 'data_accessed',
  FRAUD_DETECTED = 'fraud_detected',
}

interface AuditEvent {
  id: string;
  userId: string;
  eventType: AuditEventType;
  dataSourceId?: string;
  claimId?: string;
  credentialId?: string;
  ipAddress: string;
  userAgent: string;
  metadata: Record<string, any>;
  timestamp: Date;
}

8. Performance Specifications¶

8.1 Response Time Requirements¶

Operation	Target Response Time	Maximum Response Time
List data sources	<500ms	1s
Initiate connection	<1s	3s
Data extraction	<30s	2min
Credential generation	<5s	15s
Status check	<200ms	500ms

8.2 Throughput Requirements¶

Metric	Target	Maximum
Concurrent users	100	500
Data extractions/hour	1,000	5,000
Credentials generated/hour	500	2,000
API calls/minute	10,000	50,000

8.3 Caching Strategy¶

Redis Caching Configuration:¶

interface CacheConfig {
  dataSources: {
    ttl: 3600; // 1 hour
    key: 'external_data_sources:list';
  };
  userConsents: {
    ttl: 1800; // 30 minutes
    key: 'user_consents:{userId}';
  };
  extractedData: {
    ttl: 86400; // 24 hours
    key: 'extracted_data:{claimId}';
  };
}

9. Error Handling & Resilience¶

9.1 Error Classification¶

Error Types:¶

enum ExternalDataErrorType {
  // Authentication errors
  AUTH_FAILED = 'AUTH_FAILED',
  TOKEN_EXPIRED = 'TOKEN_EXPIRED',
  INSUFFICIENT_PERMISSIONS = 'INSUFFICIENT_PERMISSIONS',

  // Data source errors
  DATA_SOURCE_UNAVAILABLE = 'DATA_SOURCE_UNAVAILABLE',
  RATE_LIMIT_EXCEEDED = 'RATE_LIMIT_EXCEEDED',
  INVALID_REQUEST = 'INVALID_REQUEST',

  // Data quality errors
  INCOMPLETE_DATA = 'INCOMPLETE_DATA',
  INVALID_DATA_FORMAT = 'INVALID_DATA_FORMAT',
  DATA_QUALITY_TOO_LOW = 'DATA_QUALITY_TOO_LOW',

  // System errors
  NETWORK_ERROR = 'NETWORK_ERROR',
  TIMEOUT = 'TIMEOUT',
  INTERNAL_ERROR = 'INTERNAL_ERROR',
}

Retry Strategy:¶

interface RetryConfig {
  maxAttempts: number;
  baseDelay: number; // milliseconds
  maxDelay: number; // milliseconds
  backoffMultiplier: number;
  retryableErrors: ExternalDataErrorType[];
}

const defaultRetryConfig: RetryConfig = {
  maxAttempts: 3,
  baseDelay: 1000,
  maxDelay: 30000,
  backoffMultiplier: 2,
  retryableErrors: [
    ExternalDataErrorType.DATA_SOURCE_UNAVAILABLE,
    ExternalDataErrorType.RATE_LIMIT_EXCEEDED,
    ExternalDataErrorType.NETWORK_ERROR,
    ExternalDataErrorType.TIMEOUT,
  ],
};

9.2 Circuit Breaker Pattern¶

interface CircuitBreakerConfig {
  failureThreshold: number;
  recoveryTimeout: number; // milliseconds
  monitoringPeriod: number; // milliseconds
}

class ExternalDataCircuitBreaker {
  private state: 'CLOSED' | 'OPEN' | 'HALF_OPEN' = 'CLOSED';
  private failureCount = 0;
  private lastFailureTime = 0;

  async execute<T>(operation: () => Promise<T>): Promise<T> {
    if (this.state === 'OPEN') {
      if (Date.now() - this.lastFailureTime > this.config.recoveryTimeout) {
        this.state = 'HALF_OPEN';
      } else {
        throw new Error('Circuit breaker is OPEN');
      }
    }

    try {
      const result = await operation();
      this.onSuccess();
      return result;
    } catch (error) {
      this.onFailure();
      throw error;
    }
  }
}

10. Testing Specifications¶

10.1 Unit Testing¶

Test Coverage Requirements:¶

External data service functions: >90%
Data validation logic: >95%
Error handling: >85%
Security functions: >95%

Mock External APIs:¶

// Mock HMRC API responses
const mockHMRCResponses = {
  employmentHistory: {
    nino: 'AB123456C',
    taxYear: '2023-24',
    employments: [
      {
        employerName: 'Test Company Ltd',
        employerRef: '123/AB12345',
        startDate: '2023-04-06',
        endDate: '2024-04-05',
        totalPay: 35000,
        totalTax: 7000,
      },
    ],
  },
};

10.2 Integration Testing¶

External API Testing:¶

describe('HMRC Integration', () => {
  it('should successfully authenticate with HMRC sandbox', async () => {
    const authUrl = await hmrcService.generateAuthUrl(testUserId);
    expect(authUrl).toContain('test-api.service.hmrc.gov.uk');
  });

  it('should extract employment data from HMRC', async () => {
    const data = await hmrcService.extractEmploymentData(testAuthCode);
    expect(data).toHaveProperty('employments');
    expect(data.employments).toBeInstanceOf(Array);
  });
});

10.3 End-to-End Testing¶

User Journey Testing:¶

describe('External Data Credential Generation E2E', () => {
  it('should complete full journey from data source to credential', async () => {
    // 1. User selects data source
    const dataSources = await request(app).get('/api/v1/external-data-sources');

    // 2. User initiates connection
    const connection = await request(app)
      .post(`/api/v1/external-data-sources/${hmrcId}/connect`)
      .send({ scopes: ['employment:read'] });

    // 3. Simulate auth callback
    const callback = await request(app)
      .post(`/api/v1/external-data-sources/${hmrcId}/callback`)
      .send({ sessionId: connection.body.sessionId, authCode: 'test-code' });

    // 4. Wait for data extraction
    await waitForDataExtraction(callback.body.claimId);

    // 5. Generate credential
    const credential = await request(app)
      .post('/api/v1/credentials/generate-from-external-data')
      .send({ claimId: callback.body.claimId, credentialType: 'EmploymentPastV1.1' });

    expect(credential.status).toBe(200);
    expect(credential.body).toHaveProperty('credentialId');
  });
});

11. Monitoring & Observability¶

11.1 Metrics Collection¶

Key Metrics:¶

interface ExternalDataMetrics {
  // Performance metrics
  dataExtractionDuration: Histogram;
  credentialGenerationDuration: Histogram;
  apiResponseTime: Histogram;

  // Success/failure metrics
  dataExtractionSuccessRate: Counter;
  credentialGenerationSuccessRate: Counter;
  apiErrorRate: Counter;

  // Business metrics
  activeDataSources: Gauge;
  dailyDataExtractions: Counter;
  userConsentRate: Counter;
  dataQualityScore: Histogram;
}

Alerting Rules:¶

# Prometheus alerting rules
groups:
  - name: external_data_alerts
    rules:
      - alert: HighDataExtractionFailureRate
        expr: rate(data_extraction_failures[5m]) > 0.1
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: "High data extraction failure rate detected"

      - alert: ExternalAPIDown
        expr: up{job="external-api"} == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "External API is down"

11.2 Logging Strategy¶

Structured Logging:¶

interface ExternalDataLogEvent {
  timestamp: string;
  level: 'info' | 'warn' | 'error';
  service: 'external-data-service';
  operation: string;
  userId?: string;
  dataSourceId?: string;
  claimId?: string;
  duration?: number;
  error?: string;
  metadata: Record<string, any>;
}

// Example log entry
const logEvent: ExternalDataLogEvent = {
  timestamp: '2026-03-10T08:45:00.000Z',
  level: 'info',
  service: 'external-data-service',
  operation: 'extract_hmrc_data',
  userId: 'user-123',
  dataSourceId: 'hmrc-uk',
  claimId: 'claim-456',
  duration: 15000,
  metadata: {
    dataQuality: 0.95,
    recordsExtracted: 24,
  },
};

12. Deployment Specifications¶

12.1 Infrastructure Requirements¶

Additional Services:¶

```yaml

docker-compose.external-data.yml¶

version: '3.8' services: external-data-service: build: ./external-data-service environment: - RECLAIM_API_KEY=${RECLAIM_API_KEY} - HMRC_CLIENT_ID=${HMRC_CLIENT_ID} - HMRC_CLIENT_SECRET=${HMRC_CLIENT_SECRET} depends