Milestone: M7 — Reclaim Protocol | SOW Reference: FR4 | Requirement Clarity: ⚠️ Depends on M7-01 | Dev Status: ❌ Not started Moved from docs/requirements/tasks/POC/updated-architecture-diagram-with-reclaim.md. Companion diagram set to M7-02-hmrc-integration-technical-design.md. Unmodified below.

M7-02b Updated POC Architecture Diagram - Including Reclaim Protocol Integration¶

REC Verifiable Credentialing Platform¶

Current POC Architecture (Phase 1)¶

High-Level Process Flow - Current State¶

flowchart TD
    Start([Organization Admin Login]) --> Upload[Upload Employee CSV]
    Upload --> Validate{Validate Data}
    Validate -->|Invalid| Error[Show Validation Errors]
    Error --> Upload
    Validate -->|Valid| Store[Store Employee Records]

    Store --> Queue[Queue Credential Creation]
    Queue --> Batch[Batch Process Records]
    Batch --> VNF[Generate VNF Credentials]
    VNF --> Success{Credential Created?}

    Success -->|No| Retry[Retry Logic]
    Retry --> VNF
    Success -->|Yes| Email[Send Email Notifications]

    Email --> Track[Track Email Status]
    Track --> Dashboard[Update Dashboard]
    Dashboard --> Monitor[Monitor Claim Status]

    Monitor --> Complete{All Processed?}
    Complete -->|No| Batch
    Complete -->|Yes| Report[Generate Reports]
    Report --> End([Process Complete])

    %% Parallel processes
    Dashboard --> Export[Export Data]
    Monitor --> Alerts[Send Alerts if Needed]

    %% Styling
    classDef startEnd fill:#4caf50,stroke:#2e7d32,stroke-width:2px,color:#fff
    classDef process fill:#2196f3,stroke:#1565c0,stroke-width:2px,color:#fff
    classDef decision fill:#ff9800,stroke:#ef6c00,stroke-width:2px,color:#fff
    classDef error fill:#f44336,stroke:#c62828,stroke-width:2px,color:#fff

    class Start,End startEnd
    class Upload,Store,Queue,Batch,VNF,Email,Track,Dashboard,Monitor,Report,Export,Alerts process
    class Validate,Success,Complete decision
    class Error,Retry error

Future Architecture with Reclaim Protocol (Phase 2)¶

Dual Workflow Process Flow - Future State¶

flowchart TD
    %% B2B Workflow (Current)
    StartB2B([Organization Admin Login]) --> UploadB2B[Upload Employee CSV]
    UploadB2B --> ValidateB2B{Validate Data}
    ValidateB2B -->|Valid| StoreB2B[Store Employee Records]
    StoreB2B --> QueueB2B[Queue Credential Creation]

    %% B2C Workflow (New - Reclaim Protocol)
    StartB2C([Individual User Login]) --> SelectSource[Select Data Sources]
    SelectSource --> ConnectHMRC{Connect to HMRC?}
    ConnectHMRC -->|Yes| AuthHMRC[HMRC OAuth Flow]
    ConnectHMRC -->|No| ConnectBank{Connect to Bank?}
    ConnectBank -->|Yes| AuthBank[Open Banking OAuth]
    ConnectBank -->|No| ConnectOther[Other Data Sources]

    AuthHMRC --> ReclaimHMRC[Reclaim Protocol - Extract HMRC Data]
    AuthBank --> ReclaimBank[Reclaim Protocol - Extract Bank Data]
    ConnectOther --> ReclaimOther[Reclaim Protocol - Extract Other Data]

    ReclaimHMRC --> ValidateExternal{Validate External Data}
    ReclaimBank --> ValidateExternal
    ReclaimOther --> ValidateExternal

    ValidateExternal -->|Invalid| DataError[Show Data Quality Issues]
    DataError --> SelectSource
    ValidateExternal -->|Valid| TransformData[Transform to VNF Format]
    TransformData --> QueueB2C[Queue Individual Credential]

    %% Unified Processing
    QueueB2B --> UnifiedBatch[Unified Batch Processor]
    QueueB2C --> UnifiedBatch

    UnifiedBatch --> VNFProcess[Generate VNF Credentials]
    VNFProcess --> SuccessCheck{Credential Created?}

    SuccessCheck -->|No| RetryLogic[Retry Logic]
    RetryLogic --> VNFProcess
    SuccessCheck -->|Yes| RouteNotification{Route Notification}

    RouteNotification -->|B2B| EmailB2B[Send Email to Employee]
    RouteNotification -->|B2C| WalletB2C[Send to Personal Wallet]

    EmailB2B --> TrackEmail[Track Email Status]
    WalletB2C --> TrackWallet[Track Wallet Status]

    TrackEmail --> UnifiedDashboard[Unified Dashboard]
    TrackWallet --> UnifiedDashboard

    UnifiedDashboard --> MonitorAll[Monitor All Credentials]
    MonitorAll --> ReportAll[Generate Unified Reports]
    ReportAll --> EndProcess([Process Complete])

    %% Styling
    classDef startEnd fill:#4caf50,stroke:#2e7d32,stroke-width:2px,color:#fff
    classDef b2b fill:#2196f3,stroke:#1565c0,stroke-width:2px,color:#fff
    classDef b2c fill:#9c27b0,stroke:#6a1b9a,stroke-width:2px,color:#fff
    classDef unified fill:#ff9800,stroke:#ef6c00,stroke-width:2px,color:#fff
    classDef decision fill:#ffc107,stroke:#ff8f00,stroke-width:2px,color:#000
    classDef error fill:#f44336,stroke:#c62828,stroke-width:2px,color:#fff
    classDef external fill:#00bcd4,stroke:#0097a7,stroke-width:2px,color:#fff

    class StartB2B,StartB2C,EndProcess startEnd
    class UploadB2B,ValidateB2B,StoreB2B,QueueB2B,EmailB2B,TrackEmail b2b
    class SelectSource,AuthHMRC,AuthBank,ConnectOther,ReclaimHMRC,ReclaimBank,ReclaimOther,TransformData,QueueB2C,WalletB2C,TrackWallet b2c
    class UnifiedBatch,VNFProcess,UnifiedDashboard,MonitorAll,ReportAll unified
    class ConnectHMRC,ConnectBank,ValidateExternal,SuccessCheck,RouteNotification decision
    class DataError,RetryLogic error

Updated System Architecture with Reclaim Protocol¶

graph TB
    %% External Systems - Expanded
    subgraph "External Systems"
        VNF[Velocity Network Foundation]
        SMTP[SMTP Email Service]
        Auth0[Auth0 Authentication]

        %% New External Systems for Reclaim Protocol
        subgraph "Reclaim Protocol Integration"
            ReclaimAPI[Reclaim Protocol API]
            HMRC[HMRC APIs]
            OpenBanking[Open Banking APIs]
            GovAPIs[Other Government APIs]
        end
    end

    %% Frontend Layer - Enhanced
    subgraph "Frontend Layer (Next.js)"
        %% Existing Components
        UI[User Interface]
        Upload[CSV Upload Component]
        Dashboard[Status Dashboard]
        EmailEditor[Email Template Editor]
        Auth[Authentication Pages]

        %% New Components for Individual Users
        subgraph "Individual User Components"
            DataSourceSelector[Data Source Selection]
            ConsentManager[Consent Management]
            DataPreview[External Data Preview]
            PersonalWallet[Personal Credential Wallet]
            SelfService[Self-Service Portal]
        end
    end

    %% Backend Layer - Significantly Expanded
    subgraph "Backend Layer (NestJS)"
        API[API Gateway]

        subgraph "Existing Controllers"
            UploadCtrl[Upload Controller]
            CredCtrl[Credential Controller]
            EmailCtrl[Email Controller]
            UserCtrl[User Controller]
        end

        subgraph "New Controllers for External Data"
            ExternalDataCtrl[External Data Controller]
            ConsentCtrl[Consent Controller]
            DataSourceCtrl[Data Source Controller]
            IndividualCredCtrl[Individual Credential Controller]
        end

        subgraph "Existing Services"
            UploadSvc[Upload Service]
            CredSvc[Credential Service]
            EmailSvc[Email Service]
            VNFSvc[VNF Integration Service]
        end

        subgraph "New Services for Reclaim Protocol"
            ReclaimSvc[Reclaim Protocol Service]
            ExternalDataSvc[External Data Service]
            DataValidationSvc[Data Validation Service]
            ConsentMgmtSvc[Consent Management Service]
            DataTransformSvc[Data Transformation Service]
            FraudDetectionSvc[Fraud Detection Service]
        end

        subgraph "Enhanced Background Jobs"
            BatchProcessor[Unified Batch Processor]
            EmailQueue[Email Queue]
            StatusUpdater[Status Updater]
            ExternalDataProcessor[External Data Processor]
            DataQualityChecker[Data Quality Checker]
        end
    end

    %% Data Layer - Extended
    subgraph "Data Layer"
        DB[(PostgreSQL Database)]
        FileStore[File Storage]
        Cache[Redis Cache]

        %% New Data Stores
        subgraph "External Data Storage"
            ExternalDataStore[Encrypted External Data Store]
            ConsentStore[Consent Records]
            AuditStore[Audit Trail Store]
        end
    end

    %% Data Flow - B2B (Existing)
    UI --> API
    Upload --> UploadCtrl
    Dashboard --> CredCtrl
    EmailEditor --> EmailCtrl
    Auth --> UserCtrl

    UploadCtrl --> UploadSvc
    CredCtrl --> CredSvc
    EmailCtrl --> EmailSvc
    UserCtrl --> Auth0

    UploadSvc --> FileStore
    UploadSvc --> BatchProcessor
    CredSvc --> VNFSvc
    EmailSvc --> EmailQueue

    VNFSvc --> VNF
    EmailQueue --> SMTP
    BatchProcessor --> DB
    StatusUpdater --> DB

    %% Data Flow - B2C (New)
    DataSourceSelector --> ExternalDataCtrl
    ConsentManager --> ConsentCtrl
    DataPreview --> DataSourceCtrl
    PersonalWallet --> IndividualCredCtrl
    SelfService --> ExternalDataCtrl

    ExternalDataCtrl --> ReclaimSvc
    ConsentCtrl --> ConsentMgmtSvc
    DataSourceCtrl --> ExternalDataSvc
    IndividualCredCtrl --> DataTransformSvc

    ReclaimSvc --> ReclaimAPI
    ReclaimAPI --> HMRC
    ReclaimAPI --> OpenBanking
    ReclaimAPI --> GovAPIs

    ExternalDataSvc --> DataValidationSvc
    DataValidationSvc --> FraudDetectionSvc
    DataTransformSvc --> BatchProcessor

    ExternalDataProcessor --> ExternalDataStore
    ConsentMgmtSvc --> ConsentStore
    DataQualityChecker --> AuditStore

    API --> Cache
    API --> DB

    %% Styling
    classDef external fill:#ffeb3b,stroke:#f57f17,stroke-width:2px
    classDef frontend fill:#4caf50,stroke:#2e7d32,stroke-width:2px
    classDef backend fill:#2196f3,stroke:#1565c0,stroke-width:2px
    classDef data fill:#ff9800,stroke:#ef6c00,stroke-width:2px
    classDef new fill:#9c27b0,stroke:#6a1b9a,stroke-width:2px
    classDef reclaim fill:#00bcd4,stroke:#0097a7,stroke-width:2px

    class VNF,SMTP,Auth0 external
    class ReclaimAPI,HMRC,OpenBanking,GovAPIs reclaim
    class UI,Upload,Dashboard,EmailEditor,Auth frontend
    class DataSourceSelector,ConsentManager,DataPreview,PersonalWallet,SelfService new
    class API,UploadCtrl,CredCtrl,EmailCtrl,UserCtrl,UploadSvc,CredSvc,EmailSvc,VNFSvc,BatchProcessor,EmailQueue,StatusUpdater backend
    class ExternalDataCtrl,ConsentCtrl,DataSourceCtrl,IndividualCredCtrl,ReclaimSvc,ExternalDataSvc,DataValidationSvc,ConsentMgmtSvc,DataTransformSvc,FraudDetectionSvc,ExternalDataProcessor,DataQualityChecker new
    class DB,FileStore,Cache data
    class ExternalDataStore,ConsentStore,AuditStore new

Enhanced Data Architecture with External Data¶

erDiagram
    %% Existing Entities
    ORGANIZATION ||--o{ USER : has
    ORGANIZATION ||--o{ CANDIDATE : manages
    CANDIDATE ||--o{ CREDENTIAL_OFFER : receives
    CREDENTIAL_OFFER ||--o{ EMAIL_LOG : tracks
    USER ||--o{ UPLOAD_SESSION : creates
    UPLOAD_SESSION ||--o{ CANDIDATE : contains

    %% New Entities for External Data Integration
    EXTERNAL_DATA_SOURCE ||--o{ USER_CONSENT : supports
    USER ||--o{ USER_CONSENT : gives
    USER_CONSENT ||--o{ DATA_CLAIM : enables
    DATA_CLAIM ||--o{ EXTERNAL_CREDENTIAL : generates
    EXTERNAL_DATA_SOURCE ||--o{ DATA_CLAIM : processes
    USER ||--o{ EXTERNAL_CREDENTIAL : owns

    %% Existing Entities (unchanged structure)
    ORGANIZATION {
        uuid id PK
        string name
        string did
        string vnf_org_id
        timestamp created_at
        timestamp updated_at
    }

    USER {
        uuid id PK
        uuid organization_id FK
        string email
        string name
        string auth0_id
        string user_type "organization_user|individual_user"
        timestamp created_at
    }

    CANDIDATE {
        uuid id PK
        uuid organization_id FK
        uuid upload_session_id FK
        string first_name
        string last_name
        string email
        string phone
        json employment_details
        string status
        timestamp created_at
    }

    CREDENTIAL_OFFER {
        uuid id PK
        uuid candidate_id FK
        string vnf_offer_id
        string credential_type
        json credential_data
        string status
        string claim_url
        timestamp expires_at
        timestamp created_at
        timestamp claimed_at
    }

    EMAIL_LOG {
        uuid id PK
        uuid credential_offer_id FK
        string recipient_email
        string subject
        string template_used
        string status
        timestamp sent_at
        timestamp delivered_at
        timestamp opened_at
    }

    UPLOAD_SESSION {
        uuid id PK
        uuid user_id FK
        string filename
        integer total_records
        integer processed_records
        integer failed_records
        string status
        json error_log
        timestamp created_at
        timestamp completed_at
    }

    %% New Entities for External Data
    EXTERNAL_DATA_SOURCE {
        uuid id PK
        string name
        string type "government|financial|employment"
        string api_base_url
        string auth_method "oauth2|api_key|certificate"
        boolean is_active
        integer rate_limit_per_minute
        integer rate_limit_per_day
        json configuration
        timestamp created_at
        timestamp updated_at
    }

    USER_CONSENT {
        uuid id PK
        uuid user_id FK
        uuid data_source_id FK
        timestamp consent_given_at
        timestamp consent_expires_at
        json scopes "employment:read,income:read,etc"
        boolean is_active
        timestamp withdrawn_at
        timestamp created_at
    }

    DATA_CLAIM {
        uuid id PK
        uuid user_id FK
        uuid data_source_id FK
        uuid consent_id FK
        string status "pending|processing|completed|failed"
        json raw_data
        json processed_data
        decimal data_quality_score
        timestamp extracted_at
        text error_message
        integer retry_count
        timestamp created_at
        timestamp updated_at
    }

    EXTERNAL_CREDENTIAL {
        uuid id PK
        uuid user_id FK
        uuid data_claim_id FK
        string credential_type
        string vnf_credential_id
        json credential_data
        string status "draft|issued|claimed|revoked"
        timestamp issued_at
        timestamp claimed_at
        timestamp expires_at
        timestamp created_at
        timestamp updated_at
    }

Updated Component Flow - Dual Workflow¶

sequenceDiagram
    participant IndividualUser
    participant OrgAdmin
    participant Frontend
    participant API
    participant ReclaimSvc
    participant ExternalDataSvc
    participant VNFSvc
    participant ReclaimAPI
    participant HMRC
    participant VNF
    participant DB

    %% B2C Flow (Individual User - New)
    Note over IndividualUser,DB: Individual User Workflow (B2C)
    IndividualUser->>Frontend: Select HMRC as data source
    Frontend->>API: POST /external-data-sources/hmrc/connect
    API->>ReclaimSvc: Initiate HMRC connection
    ReclaimSvc->>ReclaimAPI: Request HMRC auth URL
    ReclaimAPI->>HMRC: OAuth2 authorization
    HMRC-->>IndividualUser: Redirect to consent page
    IndividualUser->>HMRC: Grant consent
    HMRC-->>ReclaimAPI: Return auth code
    ReclaimAPI->>ReclaimSvc: Process auth callback
    ReclaimSvc->>ExternalDataSvc: Extract employment data
    ExternalDataSvc->>DB: Store external data claim
    ExternalDataSvc->>VNFSvc: Generate credential from external data
    VNFSvc->>VNF: Create individual credential
    VNF-->>VNFSvc: Return credential offer
    VNFSvc->>DB: Store external credential
    DB-->>Frontend: Update individual dashboard
    Frontend-->>IndividualUser: Show credential ready

    %% B2B Flow (Organization Admin - Existing)
    Note over OrgAdmin,DB: Organization Admin Workflow (B2B)
    OrgAdmin->>Frontend: Upload employee CSV
    Frontend->>API: POST /upload/employees
    API->>ExternalDataSvc: Process CSV (existing flow)
    ExternalDataSvc->>VNFSvc: Create batch credentials
    VNFSvc->>VNF: Generate organization credentials
    VNF-->>VNFSvc: Return credential offers
    VNFSvc->>DB: Store credential offers
    DB-->>Frontend: Update organization dashboard
    Frontend-->>OrgAdmin: Show batch progress

Updated Technology Stack with External Integrations¶

graph LR
    subgraph "Frontend Stack"
        NextJS[Next.js 14]
        React[React 18]
        Redux[Redux Toolkit]
        TailwindCSS[Tailwind CSS]
        TypeScript1[TypeScript]
    end

    subgraph "Backend Stack"
        NestJS[NestJS]
        TypeORM[TypeORM]
        TypeScript2[TypeScript]
        JWT[JWT/NextAuth]
        Bull[Bull Queue]
        %% New additions
        ReclaimSDK[Reclaim Protocol SDK]
        ValidationLib[Data Validation Library]
    end

    subgraph "Database & Storage"
        PostgreSQL[PostgreSQL]
        Redis[Redis]
        S3[File Storage]
        %% New additions
        EncryptedStorage[Encrypted External Data Storage]
        AuditDB[Audit Trail Database]
    end

    subgraph "External Services"
        VNF_API[VNF Node.js SDK]
        Auth0_Service[Auth0]
        SMTP_Service[SMTP Provider]
        %% New external services
        ReclaimProtocol[Reclaim Protocol API]
        HMRC_API[HMRC Government APIs]
        OpenBanking_API[Open Banking APIs]
        OtherGovAPIs[Other Government APIs]
    end

    NextJS --> NestJS
    NestJS --> PostgreSQL
    NestJS --> Redis
    NestJS --> VNF_API
    NextJS --> Auth0_Service
    NestJS --> SMTP_Service
    %% New connections
    NestJS --> ReclaimProtocol
    ReclaimProtocol --> HMRC_API
    ReclaimProtocol --> OpenBanking_API
    ReclaimProtocol --> OtherGovAPIs
    NestJS --> EncryptedStorage
    NestJS --> AuditDB

Updated Security Architecture with External Data Protection¶

graph TB
    subgraph "Enhanced Security Layers"
        subgraph "Authentication"
            Auth0_JWT[Auth0 JWT Tokens]
            Session[Session Management]
            MFA[Multi-Factor Auth]
            %% New
            ExternalAuth[External OAuth Flows]
        end

        subgraph "Authorization"
            RBAC[Role-Based Access]
            OrgScope[Organization Scoping]
            APIKeys[API Key Management]
            %% New
            ConsentMgmt[Consent Management]
            DataScope[Data Source Scoping]
        end

        subgraph "Data Protection"
            Encryption[Data Encryption]
            TLS[TLS/HTTPS]
            Validation[Input Validation]
            %% New
            ExternalDataEncryption[External Data Encryption]
            PIIProtection[PII Data Protection]
            DataMinimization[Data Minimization]
        end

        subgraph "Monitoring & Compliance"
            AuditLog[Audit Logging]
            RateLimit[Rate Limiting]
            Monitoring[Security Monitoring]
            %% New
            ConsentAudit[Consent Audit Trail]
            DataAccessLog[External Data Access Logging]
            ComplianceMonitor[GDPR Compliance Monitoring]
        end
    end

    Auth0_JWT --> RBAC
    Session --> OrgScope
    ExternalAuth --> ConsentMgmt
    RBAC --> Encryption
    ConsentMgmt --> DataScope
    Encryption --> ExternalDataEncryption
    DataScope --> PIIProtection
    ExternalDataEncryption --> AuditLog
    PIIProtection --> ConsentAudit
    AuditLog --> ComplianceMonitor

Implementation Phases with Reclaim Protocol¶

gantt
    title Updated Implementation Timeline - POC + Reclaim Protocol
    dateFormat  YYYY-MM-DD

    section Phase 1: POC Completion (CRITICAL)
    VNF Sandbox Integration    :crit, vnf-real, 2024-03-11, 7d
    Replace Mock APIs          :crit, mock-replace, after vnf-real, 5d
    Enhanced Batch Processing  :active, batch-enhance, after mock-replace, 7d
    Production Deployment      :prod-deploy, after batch-enhance, 5d

    section Phase 2A: Reclaim Planning (6 weeks)
    Architecture Design        :arch-design, after prod-deploy, 14d
    Reclaim Protocol Research  :reclaim-research, after prod-deploy, 10d
    Business Case Development  :business-case, after reclaim-research, 7d
    Stakeholder Approval       :milestone, stakeholder-approval, after business-case, 1d

    section Phase 2B: Core Integration (12 weeks)
    Reclaim Protocol SDK       :reclaim-sdk, after stakeholder-approval, 14d
    HMRC API Integration       :hmrc-api, after reclaim-sdk, 14d
    Open Banking Integration   :banking-api, after hmrc-api, 14d
    Data Validation Services   :data-validation, after banking-api, 14d
    Individual User Interface  :individual-ui, after data-validation, 14d
    Consent Management System  :consent-mgmt, after individual-ui, 14d

    section Phase 2C: Testing & Deployment (12 weeks)
    Integration Testing        :integration-test, after consent-mgmt, 21d
    Security Testing          :security-test, after integration-test, 14d
    Performance Testing       :perf-test, after security-test, 7d
    Production Deployment     :final-deploy, after perf-test, 14d

    section Milestones
    POC Complete              :milestone, poc-complete, after prod-deploy, 1d
    Phase 2 Go/No-Go         :milestone, phase2-decision, after stakeholder-approval, 1d
    Beta Launch              :milestone, beta-launch, after final-deploy, 1d

Key Architectural Changes Summary¶

Phase 1 (Current POC) - No Changes¶

Existing B2B workflow remains unchanged
Focus on replacing mock VNF APIs with real integration
Production-ready organizational credential issuance

Phase 2 (Reclaim Protocol Integration) - Major Changes¶

New Components Added:¶

External Data Integration Layer
Reclaim Protocol Service
External Data Source Manager
Data Validation & Quality Services
Fraud Detection Services
Individual User Interface
Data Source Selection Components
Consent Management Interface
Personal Credential Wallet
Self-Service Portal
Enhanced Security & Compliance
Consent Management System
External Data Encryption
GDPR Compliance Monitoring
Enhanced Audit Trails
Extended Database Schema
4 new entities for external data management
Encrypted storage for sensitive external data
Comprehensive audit trail storage

Integration Points:¶

Reclaim Protocol API: Core external data extraction
HMRC APIs: Government employment and tax data
Open Banking APIs: Financial institution data
Other Government APIs: DVLA, Companies House, etc.

Dual Workflow Support:¶

B2B Workflow: Organization admins issuing credentials to employees (existing)
B2C Workflow: Individuals claiming credentials from external data sources (new)
Unified Processing: Both workflows converge at VNF credential generation

Critical Dependencies & Blockers¶

Phase 1 (Immediate)¶

⚠️ VNF Sandbox Access: Customer must provide sandbox account
⚠️ Real API Integration: Replace mock implementation

Phase 2 (Future)¶

🔒 Reclaim Protocol Partnership: API access and integration support
🏛️ HMRC API Access: Government approval process
🏦 Open Banking Certification: Financial services compliance
⚖️ Legal & Compliance Review: GDPR, data protection regulations

This updated architecture demonstrates the evolution from a simple B2B POC to a comprehensive dual-workflow platform supporting both organizational and individual credential issuance through external data integration.