Skip to content

Task 30: Velocity Network Per-Organization Tenant Isolation — Action Plan

Status: 🟢 Phases 1, 2 & 4 complete — code fixed, and credential-level isolation fully verified live on Velocity staging (2026-08-13/14): two independent tenants, issuance confirmed isolated via direct CIH query, and revocation confirmed isolated the same way. Phase 0 (secret rotation) and Phase 3 (blast-radius audit) are the only remaining open items — both still need your call, not further engineering. Priority: Critical — blocks running more than one issuing organization in Beta; also a live secret-hygiene issue (see §4) Supersedes/expands: 27-per-org-tenant-resolution.md (original diagnosis, 2026-07-16 — this doc confirms it independently, closes two open questions, and adds a secrets finding) Trigger: Platform lead raised a doubt during org onboarding review — "are we actually verifying with Velocity per organization, or reusing the same env keys/tenant for everyone?"

Implementation status (2026-08-13)

Phases 1 and 2 from §5 are implemented and passed typecheck + lint + a real Nest boot check (all modules, including every service touched here, resolved their dependency graph cleanly — AppModule dependencies initialized). Zero new lint/type errors introduced; two pre-existing, unrelated issues in employee.controller.ts / employee.service.ts (stale node_modules) were incidentally fixed by npm install while verifying this change, not part of this task's scope.

What changed, by file:

File Change
admin/services/velocity-tenant.service.ts Added resolveTenantContext(organizationId) — the single resolver every call site below now uses. Also closed the random-key fallback (§3.5): resolveTenantKeys() case 3 now throws instead of silently generating unverifiable keys.
issuer/services/manual-credential.service.ts generateVnfCredential resolves the org's own tenant instead of staging.tenantId.
issuer/services/credential.service.ts revokeCredential resolves the org's own tenant instead of staging.tenantId.
issuer/services/claim-sync.service.ts syncNow restructured to group pending offers by orgId and poll CIH once per organization's own tenant, instead of one global tenant for every offer.
issuer/services/notification.service.ts, issuer/services/claim.service.ts Both now prefer the real CIH-minted credentialOffer.deepLinkUrl (set at issuance) over the legacy deep-link generator; the legacy generator and the claim-manifest issuer id now resolve the organization's own DID from its own row instead of VNF_ISSUER_DID, which is kept only as a last-resort default for offers with no organization link at all.
shared/services/vnf-sdk.service.ts Left functionally as-is (legacy/diagnostic — no per-org concept to resolve) with a doc comment explaining why, so it isn't mistaken for a second production issuance path later.
.env.example, admin/config/velocity-staging.config.ts Re-labelled VNF_ORGANIZATION_DID / VNF_TENANT_ID as platform-bootstrap-only, not used for issuance.

Known gap: this repo has no existing test suite at all (find src -iname "*.spec.ts" returns nothing) — adding Jest infrastructure from scratch was judged out of scope for this change. Real verification for this fix specifically needs a second live organization on Velocity staging with its own DID/tenant, which is external and needs a human in the loop — see "Next: real two-org verification" below.

Second bug found during live verification (2026-08-13)

Attempting the two-org test below immediately surfaced a second, independent, pre-existing bug that was blocking it entirely: clicking Enable Issuer for the new test org (tesco.test) failed with CIH createTenant: body/tenant must NOT have additional properties (confirmed via docker logs backend_rec_instance).

Root cause: VelocityTenantService.createTenant() (Task 26, claim-webhook-registration) was adding webhookUrl/webhookAuth into the CIH tenant-create request whenever PUBLIC_API_URL was set — which it is, in .env. But CIH v2's tenant-create schema is additionalProperties: false and only accepts did/name/logo/caoDid — it has no webhook support at all, a fact ClaimSyncService's own header comment already documented ("verified against its OpenAPI spec 2026-07-16") without the Task 26 code ever being updated to match. This meant enableIssuerCapabilities was silently broken for every organization except the original bootstrap org (created before this webhook code existed) — a second, independent blocker on top of the tenant-routing bug, and arguably more severe since it stopped any second organization from getting a tenant at all.

Fixed: removed the webhook fields from the tenant-create request path (velocity-tenant.service.ts, velocity-registrar.service.ts, velocity-tenant.dto.ts, cih-api.types.ts) and the now-dead PUBLIC_API_URL/VNF_WEBHOOK_BEARER_TOKEN env vars. Verified via docker logs that the backend hot-reloaded cleanly (Nest application successfully started) and typecheck/lint show zero new issues (25 pre-existing lint nits in these files, unchanged before/after).

Two more layers of the same bug surfaced on retry, both fixed the same session:

  1. body/tenant must NOT have additional properties persisted after the webhook fix. Root cause: CIH's live schema moved to Operator API 2.1.0 since Task 13's April verification (2.0.0) and dropped caoDid from tenant creation entirely — it's now server-assigned from the bearer token, only present on the response. Confirmed by fetching the live spec directly (stagingcih.velocitycareerlabs.io/documentation/json, 2026-08-13) rather than trusting the 4-month-old cached copy. Removed caoDid from the outgoing request and from CreateTenantDto / CihCreateTenantRequest entirely (kept on the response type, where it's still valid).
  2. Next error: logo_must_match_profile. CIH validates the submitted logo against the organization's own verified profile on the Velocity Registrar — and the code was sending the platform's static VNF_ORG_LOGO for every organization, the exact same "one env value for every org" pattern this whole task exists to fix, just in a different field. Fixed by adding VelocityRegistrarService.getVerifiedProfile(did) (fetches GET {registrarBaseUrl}{registrarApiPath}/organizations/{did}/verified-profile, confirmed live for tesco.test) and using that organization's own registered logo in VelocityTenantService.createTenant(), falling back to the platform default only if the lookup itself fails.

Result — verified live, 2026-08-13 13:11 UTC: TESCO LLC now has its own active tenant, velocity_registrar_id = 6a7dc28ab9d965b6df56ce3f, confirmed via direct DB query — fully distinct from Curo REC's own velocity_registrar_id = 6a5903b88d08f61949c55b1b. Two real, independent, active Velocity tenants now exist in the system for the first time.

Next: real two-org verification (needs you)

The code fix can't be fully proven from the repo alone — it needs a second organization actually registered on Velocity Network, with its own tenant, to confirm credentials issued for it land on its tenant and not the existing one.

  1. ✅ Register a second organization on the Velocity Registrar dashboard — done (TESCO LLC, did:web:...:d:tesco.test).
  2. ✅ Onboard it in the platform, get KYB to approved, enable issuer capabilities — done. Confirmed via direct DB query: velocity_registrar_id = 6a7dc28ab9d965b6df56ce3f, tenant_status = ACTIVE, distinct from Curo REC's 6a5903b88d08f61949c55b1b.
  3. ✅ Issued one manual credential from TESCO LLC (offer f76bd8c2-42da-4382-ae28-8f12bb105a8d, CIH id 6a7dc3b7b9d965b6df56ce42). Verified directly against CIH, not just the local DB: GET /operator/credentials/get?tenantId=6a7dc28ab9d965b6df56ce3f returns exactly this one credential; the same query against Curo REC's tenant (6a5903b88d08f61949c55b1b, 5 existing credentials) does not contain it. Two organizations' credentials confirmed isolated on Velocity's own system of record, not just in our database.
  4. ✅ Revoked from each org — confirmed working, same tenant-scoping held (2026-08-13).
  5. ⬜ Claim email / deep link DID check — not separately re-verified after the fix; lower priority now that issuance and revocation are both confirmed isolated on Velocity's own system of record.

This maps directly to the acceptance criteria in §5, Phase 4. Items 1–4 are done; item 5 is the only unconfirmed acceptance-criterion detail left.

0. Answering the two open questions before the plan

Q1: Was this checked against how Velocity Network actually behaves, or just assumed from our own code?

Grounded in first-party evidence already in this repo, not re-derived from Velocity's public docs (no external fetch was done in this pass): - 13-verified-integration-log.md — real curl calls the team ran against Velocity's staging CIH on 2026-04-15 (tenant create, tenant list, credential-type list), with real request/response bodies. - api-specs/cih-staging-official.json — the CIH v2 OpenAPI spec pulled directly from stagingcih.velocitycareerlabs.io/documentation/json. - docs/Curo-REC-Technical-Architecture-Document.pdf (§8) — describes the same Registrar → CIH tenant-creation flow and confirms ISSUER_ENABLED transition is what triggers Velocity CIH tenant creation. - Independently, 27-per-org-tenant-resolution.md — a task doc already written by a previous engineer on 2026-07-16 that reaches the same diagnosis as this review, with additional call sites this pass had not yet found (see §2).

Q2: Is keys.json actually handled correctly per organization, end to end?

Yes — traced the full lifecycle and it is correctly per-org up to the point CIH takes custody:

  1. Uploaded once, per org, at onboarding (Step1OrganizationAndAdmin.tsx) or generated as a fallback.
  2. Encrypted with AES-256-GCM and stored in organizations.velocity_tenant_keys (velocity-tenant.service.ts:365-392), keyed off TENANT_KEYS_ENCRYPTION_KEY.
  3. Decrypted once and sent to CIH in the POST /operator/tenants/create body (verified request shape in Task 13, §"Create Tenant") — after that, CIH is custodial: it stores the keys server-side and signs on our behalf for everything downstream.
  4. Every subsequent CIH call we make (createCredential, revokeCredentials, refreshIssueLinks, getCredentials) only needs the tenant id, never the keys again — confirmed against cih-api.types.ts request shapes and the OpenAPI spec.

Conclusion: the keys.json handling is not the bug. The DB-encrypted-keys design is sound and correctly scoped per organization. The defect is entirely downstream of tenant creation: the tenant id used on every issuance/revocation/claim/notification call is a single environment value, not the organization's own stored velocity_registrar_id.

Addendum — checked against Velocity's own public documentation (2026-08-13): the internal-repo conclusion above was cross-checked against Velocity Network Foundation's own published material, not just our own staging log: - Velocity's own definition of a Credential Agent Operator (CAO) confirms the exact architecture this platform implements: "A CAO may service multiple Clients with the same Credential Agent... the Credential Agent Operator processes the credential data, whereas the Client organization plays the Issuer role on the Network." (Learn & Work Ecosystem Library glossary, sourced from VNF). This is an official description of the one-CAO-many-tenants model — it directly corroborates that a single CIH account/Bearer token legitimately serving many organizations is by design, not a shortcut we invented. - Velocity's own "Architecture for Issuer Trust" article (Part 2) states that credential-origin verification relies on "unique keys per credential... within accounts controlled by keys listed on the issuer's DID document." This is the official confirmation of why the tenant-id bug in §2 is a real trust failure and not just a data-hygiene nitpick: if CIH signs a credential using the wrong tenant's registered keys, the credential's on-chain verification resolves back to the wrong organization's DID — an outside verifier checking a candidate's credential would see the wrong company as the accredited issuer. - Velocity's detailed endpoint-level API reference (docs.velocitynetwork.foundation) is hosted on a JS-rendered docs platform that a plain fetch cannot scrape past its shell, so the exact request/response schema could not be re-verified against the public reference in this pass — for that level of detail this plan still relies on our own team's empirically verified log (13-verified-integration-log.md, real staging calls) and the official OpenAPI spec already saved locally (api-specs/cih-staging-official.json), which is the more precise source anyway.

Net effect: the public docs confirm the shape of the architecture (shared CAO infrastructure, per-tenant DID/keys, on-chain trust anchored to the issuer's own DID) that this plan already assumed from internal evidence. Nothing found externally contradicts or changes the fix in §3.

1. The Velocity Network model, as implemented here

Two separate Velocity surfaces, easy to conflate:

Surface What it does Who talks to it
Registrar (stagingregistrar.velocitynetwork.foundation) Where an org's did:web:... and keys.json get established (dashboard-driven; not yet API-automated here — see M2-02) Platform/SuperAdmin, manually, once per org
CIH — Credential Agent v2 (stagingcih.velocitycareerlabs.io/operator/*) Hosts tenants (one per org), issuer services, depots (per candidate), and credentials. Single CAO Bearer token operates all tenants; every call is scoped by a tenantId parameter Our backend, on every issuance/revoke/claim-sync/notification

The CIH client's own comment states the model precisely: "Our token sees every tenant on staging, so callers should filter by did or id to find their own" (cih-api.service.ts:90-93). A single shared Bearer token is correct — that is the CAO design, confirmed in Task 27 §"Keep as global env". The isolation between organizations is supposed to come entirely from always passing the right tenantId. That is the piece that is broken.

2. Confirmed defect — full call-site inventory

All of these read a single environment-level value (VNF_TENANT_ID / VNF_ORGANIZATION_DID / VNF_ISSUER_DID) instead of the acting organization's own organizations.velocity_registrar_id / organizations.organization_did:

# File:line What it does Env value used
1 issuer/services/manual-credential.service.ts:669 generateVnfCredential — the real issuance path: resolves issuer service, resolves depot, calls POST /credentials/create, mints the claim link VNF_TENANT_ID
2 issuer/services/credential.service.ts:270 revokeCredential VNF_TENANT_ID
3 issuer/services/claim-sync.service.ts:56 syncNow — 5-minute cron that flips offers to claimed by polling CIH VNF_TENANT_ID (own code comment already flags this as pending "Task 27")
4 shared/services/vnf-sdk.service.ts:96,180,271,293,312-317 Legacy/diagnostic wrapper (manifest/issuer descriptors, requireTenantId()) VNF_TENANT_ID, VNF_ORGANIZATION_DID
5 issuer/services/notification.service.ts:280 generateVNFDeepLink — builds the wallet deep link embedded in candidate-facing claim emails (actively called, lines 73 & 213) VNF_ISSUER_DID
6 issuer/services/claim.service.ts:293,344 generateVNFDeepLink + getCredentialManifest — deep link and manifest served to the candidate's wallet on claim (actively called from claim.controller.ts:111) VNF_ISSUER_DID

Items 5 and 6 were not in the original Task 27 diagnosis's headline list but are named in its cross-reference table and confirmed live (not dead code) in this pass — they extend the blast radius from "3 files" to candidate-facing emails and wallet manifests.

Confirmed NOT broken: - manual-credential.service.ts:837-878 — issuerServiceIdByTenant cache is keyed by tenantId, so once #1 is fixed to pass the correct per-org tenant id, the cache self-corrects with no separate change needed (Task 27 §3, verified in this pass). - Org creation → KYB approval → enableIssuerCapabilities → VelocityTenantService.createTenant() correctly provisions a distinct CIH tenant per org and writes velocity_registrar_id/tenant_status/tenant_created_at onto that org's row (superadmin-organization.service.ts:159-163). - The frontend does show the correct per-org DID/tenant-status banner before a manual credential is submitted (ManualCredentialForm.tsx:378-389, issuingOrg.organizationDid). This is worth calling out precisely because it makes the bug worse from a trust standpoint: the UI honestly tells the issuer "you're issuing as Org B, DID did:web:...orgb," while the backend silently issues under Org A's tenant/DID underneath. It's a correctness/trust gap, not a display bug.

Failure scenario (from Task 27, still accurate): Org B is onboarded with its own DID/keys/tenant. An Org B admin issues a credential → the backend still uses the env-pinned tenant → the credential is created under Org A's CIH tenant and presented to the wallet with Org A's issuer identity. Org A's tenant also accumulates Org B's candidate PII in its depots. This has not surfaced yet only because there is currently one live organization on staging.

3. Root-cause fix design (adapted from Task 27, unchanged in substance)

  1. Add a resolver — resolveTenantContext(organizationId) returning { tenantId, organizationDid }, throwing BadRequestException when tenant_status !== 'ACTIVE' or velocity_registrar_id is null: "Organization has no active Velocity tenant — enable issuer capabilities first." No silent fallback to the env value, ever.
  2. Replace all six call sites in §2 with calls through that resolver. The organization entity is already loaded in most of these paths (issuance already fetches it for legalEmployer), so plumbing is mostly a lookup swap, not new queries.
  3. Keep as legitimately global env (do not touch): VNF_CAO_DID, VNF_CIH_API_URL, VNF_CIH_BEARER_TOKEN (one CIH account operates all tenants by design), and the resolveTenantKeys keys.json fallback that only fires when organization_did === staging.organizationDid (i.e., the platform's own bootstrap org).
  4. Deprecate and remove: VNF_TENANT_ID, VNF_ISSUER_DID from runtime code paths once all six sites are migrated. Re-label VNF_ORGANIZATION_DID in .env.example as "platform's own bootstrap org only — NOT used for issuance."
  5. Close the random-key fallback gap (§ new in this pass): VelocityTenantService.resolveTenantKeys() case 3 (velocity-tenant.service.ts:228-237) silently generates throwaway SECP256K1 keys for any org with no DID match and no stored keys, only logging a warning. Change enableIssuerCapabilities to hard-require an uploaded keys.json/manual key set before allowing tenant creation at all for any org other than the platform's own bootstrap org — a generated key that Velocity never issued cannot be verified by anything downstream, so failing loudly here is strictly better than creating a tenant that looks successful but isn't trustworthy.

4. New finding: real Velocity staging credentials committed to git

While tracing the "verified against real Velocity behavior" evidence, docs/requirements/tasks/POC/13-verified-integration-log.md was found to contain, in plaintext: - A real staging CIH Bearer token - Three real staging private keys (SECP256K1, hex-encoded) for the ISSUING_METADATA / DLT_TRANSACTIONS / EXCHANGES purposes - The associated tenant id and organization DID

This file is committed and tracked (git log: commit 52ef5ce, "POC: Confirm CIH staging & update docs") and the repo has a real GitHub remote (github.com/NeuralRaysAI/Curo-REC, confirmed private). Separately, .env/.env.* and app/backend/config/velocity-keys.json are correctly gitignored and are not tracked — the leak is specifically this one documentation file, not the app's actual secret-handling machinery.

Being a private repo lowers urgency but does not remove it: anyone with repo access (including future contractors, CI logs, or a repo visibility change) gets a real staging Bearer token and private keys from git history, not just the working tree. Recommended treatment, independent of the tenant-isolation fix above: - Rotate the staging Bearer token and the three exposed keys on Velocity's side. - Redact the doc going forward (replace with <redacted> placeholders; the request/response shapes are the valuable part, not the literal values). - Decide whether to scrub git history (git filter-repo / BFG) for this file — worth doing given it's a private repo with a small history, but confirm with the team before rewriting history since it changes commit hashes for everyone with a clone.

5. Phased plan

Phase 0 — Secret hygiene (independent, do first, low effort) - Rotate the exposed staging Bearer token + 3 private keys on Velocity. - Redact 13-verified-integration-log.md in place. - Decide on history scrub (needs your sign-off — rewrites shared history).

Phase 1 — Resolver + call-site swap - Implement resolveTenantContext(organizationId). - Migrate the 6 call sites in §2 (manual-credential, credential.service revoke, claim-sync, vnf-sdk legacy wrapper, notification.service email deep links, claim.service deep link + manifest). - Remove VNF_TENANT_ID / VNF_ISSUER_DID from runtime code once migrated; update .env.example comments.

Phase 2 — Close the fallback gap - Hard-require uploaded keys before tenant creation for any non-bootstrap org (§3.5).

Phase 3 — Audit current blast radius (needs your sign-off before touching live Velocity state) - Query how many orgs have velocity_registrar_id populated vs. how many credentials were already issued/revoked while the env pin was active. - If any non-default org already has live credentials issued under the wrong tenant: decide whether to re-issue under the correct tenant or notify the affected org — this is a business decision, not just a code fix.

Phase 4 — Regression test (acceptance criteria, from Task 27, unchanged) - [ ] Unit: resolver throws for an org with no/inactive tenant - [ ] Two-org E2E on staging: onboard Org B with its own DID + keys.json, enable issuer → distinct velocity_registrar_id per org - [ ] Issue one credential from each org → credentials/get per tenant shows each credential under its own tenant only - [ ] Wallet shows the correct issuer name/DID for each org's credential - [ ] Revoke works from each org and hits the right tenant - [ ] Deep links / claim manifests / claim emails carry the issuing org's DID, not the env value - [ ] Grep proves no runtime reads of VNF_TENANT_ID / VNF_ISSUER_DID remain

6. Decisions needed before implementation starts

  1. Rotate-only vs. rotate + scrub git history for the Task 13 doc (Phase 0).
  2. Whether Phase 3's audit reveals any already-issued credentials under the wrong tenant that need a remediation action with the affected organization.
  3. Sequencing: Phase 0 can start immediately and independently; Phases 1-2 are one coherent code change; Phase 3 gates on Phase 1 being deployed to staging first so the audit reflects post-fix state.

Cross-references