pfm/Architecture.md

# PostFundManagement (PFM) Platform — Architectural Specification

> **Document Version**: 2.0  
> **Target Runtimes**: .NET 10.0 (C# 13) | React 19 / TypeScript 5.8 / Vite  
> **Scope**: System Architecture, Security Topology, RBAC Enforcement, and Maker-Checker Pattern

---

## 1. System Topology & Architectural Layers

The PFM platform is constructed using **Clean Architecture** and **Domain-Driven Design (DDD)** principles to guarantee strict decoupling of business rules, persistence concerns, and user interfaces.

```
┌────────────────────────────────────────────────────────────────────────┐
│                          PRESENTATION TIER                             │
│  PostFundManagement.ConsoleUI      │  PostFundManagement.PartnerUI     │
│  (Internal Staff & Governance)     │  (External Recipient Portal)      │
│  React 19 + TypeScript + Vite      │  React 19 + WebGL (OGL) + Vite    │
└───────────────────────────────────┬────────────────────────────────────┘
                                    │ HTTPS / REST (v1/api/*) + OIDC PKCE
┌───────────────────────────────────▼────────────────────────────────────┐
│                             API GATEWAY                                │
│  PostFundManagement.Api (ASP.NET Core Minimal APIs / Carter)           │
│  Authentication: JWT Bearer + OpenTelemetry Observability              │
│  Quartz Client: Scheduler-Only (ZeroSizeThreadPool / Enqueues Jobs)    │
└───────────────────────────────────┬────────────────────────────────────┘
                                    │
┌───────────────────────────────────▼────────────────────────────────────┐
│                          APPLICATION LAYER                             │
│  PostFundManagement.Application                                        │
│  CQRS (Mediator), Business Validators, Domain Services, Event Handlers │
└──────────────────┬─────────────────────────────────┬───────────────────┘
                   │                                 │
┌──────────────────▼──────────────────┐   ┌──────────▼───────────────────┐
│            DOMAIN LAYER             │   │    INFRASTRUCTURE LAYER      │
│  PostFundManagement.Domain          │   │  PostFundManagement.Infrastr.│
│  Entities, Value Objects, Enums,    │   │  PostgreSQL (EF Core 10),    │
│  Domain Invariants, Audit Models    │   │  Skoruba STS, S3 Garage,     │
└─────────────────────────────────────┘   │  Redis, Quartz (pfm-scheduler)│
                                          └──────────────┬───────────────┘
                                                         │
┌────────────────────────────────────────────────────────▼───────────────┐
│                     BACKGROUND JOBS WORKER HOST                        │
│  PostFundManagement.Jobs (.NET 10 Background Host / Quartz Runner)     │
│  Recurring Job Registration (JobsWorker) & Mediated Execution Engine   │
│  Quartz Runner: Full Execution (DefaultThreadPool MaxConcurrency = 5)  │
└────────────────────────────────────────────────────────────────────────┘
```

---

## 2. Security & Identity Architecture

Authentication and authorization are separated cleanly across system boundaries:

* **Security Token Service (STS)**: Duende IdentityServer / Skoruba Administration (`https://sts.security.khongisa.co.za`) issues cryptographically signed JWT access tokens using OpenID Connect (OIDC) with Proof Key for Code Exchange (PKCE).
* **Audience & Scopes**: Web clients request standard OIDC scopes (`openid`, `profile`, `roles`, `email`, `address`) along with the target API resource scope (`pfm-api-dev`).
* **Tenant Isolation**: Every authenticated principal carries a `tenant_id` claim. Multitenant query filters in EF Core enforce cross-tenant partition boundaries at the data access layer.
* **Partner Isolation**: Grantees and external recipients possess the `partner` role. The `ConsoleUI` enforces immediate lockout for `partner` roles, restricting external users exclusively to `PartnerUI`.

---

## 3. Role-Based Access Control (RBAC) Architecture

PFM implements a fine-grained, claim-based RBAC model where system permissions are decomposed into **15 operational domains** and **5 progressive action capabilities**.

### 3.1 Role Claims Inventory

| Claim Key | Operational Domain | Functional Scope |
| :--- | :--- | :--- |
| `dashboard` | Executive Dashboard | Macro portfolio KPIs, disbursed capital, and ISO 31000 risk heatmap |
| `governance` | Governance & Structure | Holding entities, funder portfolios, programmes, rules, and instruments |
| `projects_milestones` | Projects & Milestones | Work breakdown structure (WBS), contractual milestones, and activities |
| `grant_financials` | Grants & Financials | Statutory funding awards, budget lines, contracts, and disbursement tranches |
| `evidence_risks` | M&E, Evidence & Risks | Deliverable evidence verification queue, indicators, and risk registers |
| `beneficiaries` | Beneficiaries & Impact | Demographic targets, POPIA compliance items, and outcome deliverables |
| `change_management` | Change Management | Project issues, contractual change requests, and budget variations |
| `site_visits` | Site Visits | Physical field inspections, checklist observations, and inspector findings |
| `settings_invites` | Settings Invites | Onboarding invitations and candidate notifications dispatch |
| `settings_users` | Settings Users | User provisioning, role assignments, and account lockouts |
| `settings_auditlog` | Settings Audit Log | Immutable audit ledger and system telemetry records |
| `settings_notifications`| Settings Notifications | SMTP engines, notification channels, and dispatch telemetry |
| `tenant_self_service` | Tenant Self-Service | Tenant profile configuration and delegated sub-user management |
| `tenant_management` | Tenant Management | Multi-tenant directory, partition allocation, and solution ownership |
| `unrestricted` | System Administration | Root administrative bypass across all system partitions |

### 3.2 Claim Actions & Capability Hierarchy

Action capabilities within a claim domain follow a strict subset hierarchy:

$$\text{view} \subset \text{add} \subset \text{update} \quad \text{and} \quad \text{view} \subset \text{export}$$

* **`view`**: Read-only access to entity lists, detailed records, metrics, and progress scores.
* **`add`**: Can create new records, edit draft records, submit milestone evidence, and change workflow statuses *other than approval* (includes `view`).
* **`update`**: Authoritative governance capability. Can execute **approval** status transitions (`Approved`, `Active` contracts) on records (includes `view` and `add`).
* **`export`**: Can generate and download structured spreadsheets (`.xlsx`) and formal PDF reports (includes `view`).
* **`all`**: Comprehensive authority (`view`, `add`, `update`, `export`) over the claim domain.

### 3.3 Authoritative Role-to-Claims Matrix

| Role | Assigned Claim Domains | Standard Capabilities | Record Approval Authority (`update`) |
| :--- | :--- | :--- | :---: |
| **`Admin`** | `unrestricted` | `all` | **Yes (System-wide)** |
| **`Executive`** | `dashboard`, `governance`, `projects_milestones`, `grant_financials`, `evidence_risks`, `tenant_self_service` | `all` | **Yes** |
| **`Manager`** | `dashboard`, `projects_milestones`, `grant_financials`, `evidence_risks`, `beneficiaries`, `change_management`, `site_visits`, `settings_invites`, `settings_users`, `settings_auditlog`, `settings_notifications`, `tenant_self_service` | `all` | **Yes** |
| **`Supervisor`** | `dashboard`, `projects_milestones`, `grant_financials`, `evidence_risks`, `beneficiaries`, `change_management`, `site_visits`, `settings_invites`, `settings_users` | `view, add, update, export`<br>*(Dashboard: `view, export`)* | **Yes** |
| **`Contributor`** | `projects_milestones`, `grant_financials`, `evidence_risks`, `beneficiaries`, `change_management`, `site_visits`, `settings_invites` | **`view, add`** | ❌ **Strictly Prohibited** |

---

## 4. Maker-Checker Pattern Architecture

To eliminate fraud risk, self-dealing, and conflicts of interest, PFM enforces the **Maker-Checker Pattern** (Segregation of Duties / Dual Authorization).

### 4.1 Core Principle
> **The user who creates or drafts a record (the "Maker") is strictly prohibited from verifying or approving that record (the "Checker"), regardless of their administrative role or RBAC claim privileges.**

Even if an `Executive`, `Manager`, or `Supervisor` possesses the `update` claim action on a domain, they cannot approve a record if they are identified as the record's creator.

```mermaid
flowchart LR
    A["Maker (User A)<br/>Creates Record"] --> B["Record Persisted<br/>with createdBy = User A"]
    B --> C{"Checker Evaluation<br/>(Approver Login)"}
    C -->|"Checker is User A"| D["Approval BLOCKED: Maker-Checker Violation"]
    C -->|"Checker is User B with update claim"| E["Approval GRANTED: Status Approved"]
```

### 4.2 Record Attribution Model
Every operational domain entity immutably captures the creator identity:
* `createdBy`: Cryptographic user identifier (Guid / Subject ID) populated from the verified token claim `context.GetUserId()` upon creation.
* `approvedBy`: User identifier of the independent verifier populated upon successful transition to `Approved`.

### 4.3 Client-Side Enforcement Hooks & Helpers
In `PostFundManagement.ConsoleUI`, the Maker-Checker rule is evaluated dynamically via [`useRbac()`](file:///c:/Users/Admin/source/repos/PostFundManagement/PostFundManagement.ConsoleUI/src/auth/rbac.ts):

* **`isMaker(createdBy)`**: Compares `createdBy` against current user `user.id`, `sub`, and `user.email`.
* **`canApproveRecord(claim, createdBy)`**: Evaluates:
  $$\text{canApproveRecord} = \text{canApprove}(\text{claim}) \land \neg\text{isMaker}(\text{createdBy})$$
* **`getApprovalRestrictionReason(claim, createdBy)`**: Surfaces explanatory feedback:
  - `"Requires update/approval permission"` if the claim lacks `update`.
  - `"Maker-Checker policy: Record creators cannot approve their own records"` if `isMaker` is true.

### 4.4 UI Guard Implementations
1. **Status Dropdowns**: The `Approved` option in status `<select>` controls is disabled with contextual labels:
   ```tsx
   <option value={ApprovalStatus.Approved} disabled={!canApproveRecord('projects_milestones', record.createdBy)}>
     Approved {isMaker(record.createdBy) ? '(Maker-Checker: Cannot approve own record)' : ''}
   </option>
   ```
2. **Action Buttons**: Fast-track "Approve" buttons (e.g. Budget Approval, Disbursement Authorisation, Evidence Review) are hidden or accompanied by a Maker-Checker indicator badge when viewed by the creator.
3. **Export Controls**: `<ExportButton>` enforces domain claim checks explicitly (`claim="grant_financials"`), rejecting unauthenticated exports.

---

## 5. Caching Strategy, Mutation Versioning & Performance Optimization

To achieve sub-50ms page transitions while guaranteeing data freshness across mutations in a distributed architecture, PFM employs a multi-tiered caching topology.

```
┌────────────────────────────────────────────────────────────────────────┐
│ REACT QUERY CACHE (In-Memory Browser Tier)                             │
│ staleTime: 2m (dynamic) | 5m (taxonomy) | gcTime: 10m                  │
└───────────────────────────────────┬────────────────────────────────────┘
                                    │ Cache Miss / Background Refetch
┌───────────────────────────────────▼────────────────────────────────────┐
│ AXIOS CLIENT INTERCEPTOR & MUTATION VERSIONING (_v)                     │
│ Reads/Browsing: attaches active `_v` epoch token                       │
│ Mutations (POST/PUT/PATCH/DELETE): auto-bumps epoch -> new `_v` token  │
└───────────────────────────────────┬────────────────────────────────────┘
                                    │ HTTP GET /v1/api/*?_v=<epoch>
┌───────────────────────────────────▼────────────────────────────────────┐
│ ASP.NET CORE OUTPUTCACHE (Redis / Memory Tier)                         │
│ Base Policy: 2-minute expiration partitioned by URI + QueryString       │
│ Hits same `_v` -> 2ms response | New `_v` -> DB query & cache refresh  │
└────────────────────────────────────────────────────────────────────────┘
```

### 5.1 The OutputCache Stale Data Challenge
ASP.NET Core Minimal APIs in `PostFundManagement.Api` attach `.CacheOutput()` to GET endpoints, using a 2-minute base sliding expiration backed by Redis / Memory cache. 

Because OutputCache keys are generated from the HTTP Method, Host, Path, and **Query String**, a naive client refetch after a mutation (e.g. `queryClient.invalidateQueries({ queryKey: ['projects'] })`) calls `GET /v1/api/projects`. OutputCache intercepts this identical URI, returning the pre-mutation response cached earlier. The updated project status or approved milestone does not reflect in the UI until the 2-minute cache expires.

### 5.2 Client-Side Mutation Epoch Versioning
PFM solves this without modifying backend endpoints via **Client-Side Mutation Versioning**:
1. **Epoch Tracking (`activeMutationEpoch`)**: `httpClient.ts` tracks an active mutation timestamp initialized from `sessionStorage` or `Date.now()`.
2. **Automatic Mutation Interceptor**: Every successful mutating HTTP response (`POST`, `PUT`, `PATCH`, `DELETE` with HTTP 2xx) triggers `bumpMutationEpoch()`.
3. **Query Parameter Injection**: Outgoing `GET` requests to internal APIs automatically append `_v: getMutationEpoch()`.
4. **Instant Invalidation**: When a mutation succeeds, the epoch is bumped. Subsequent React Query refetches send the new `_v`, producing an immediate **cache miss** in ASP.NET Core OutputCache. The backend queries PostgreSQL, returns fresh data, and caches the new state under the new `_v` key.
5. **Cache Hit Snappiness**: Navigations between pages while no mutations have occurred reuse the stable `_v`, yielding sub-5ms responses from Redis/Memory OutputCache and 0ms from React Query in-memory cache.
6. **Explicit Freshness (`_t`)**: When `Cache-Control: no-cache` is specified or during explicit refresh operations, a unique timestamp `_t` is attached to force a guaranteed fresh fetch.

### 5.3 RBAC Startup Pipeline Acceleration
Following the introduction of fine-grained RBAC and Maker-Checker checks, startup performance was streamlined to eliminate sequential network waterfalls:
* **Cache-First Role Claims**: `fetchAndCacheSystemRoles` and `fetchAndCacheRoleClaims` check in-memory and `localStorage` caches *before* dispatching network calls.
* **Parallel User Resolution**: `getUserRolesFromApi` and `getUserClaimsFromApi` are resolved concurrently using `Promise.allSettled`.
* **Parallel Claim Resolution**: Claims across all assigned roles are retrieved concurrently rather than in serial loops.
* **Immediate Non-Blocking Session Render**: If valid session tokens and user profiles exist in `localStorage`, `AuthProvider` immediately unblocks UI rendering (`isLoading: false`) and runs tenant lockout checks, account synchronization, and claim revalidation asynchronously in the background.
* **In-Memory User Sync & Lockout Caching**: `isUserSyncedInBrowser` and `verifyTenantLockout` maintain in-memory references to eliminate synchronous `JSON.parse` operations from the request pipeline.

---

## 6. Distributed Background Job Processing Architecture (PostFundManagement.Jobs)

The PFM platform decouples synchronous API request handling from asynchronous, long-running, and scheduled workloads using a distributed **Quartz.NET (v4.6+)** clustering topology backed by PostgreSQL (`pfm-scheduler`).

### 6.1 Split-Node Topology: Scheduler Client vs. Execution Host

To maintain high API throughput, low memory footprint, and complete process isolation, PFM implements a split-node architecture between `PostFundManagement.Api` and `PostFundManagement.Jobs`:

```
┌──────────────────────────────────────────────┐
│          PostFundManagement.Api              │
│       (API Gateway / Scheduler Client)       │
│                                              │
│  * AddQuartzSchedulerClient()                │
│  * ZeroSizeThreadPool (0 execution threads)  │
│  * SchedulerStatus.Created (Standby mode)    │
│  * Strictly schedules / enqueues events      │
│  * Never acquires triggers or executes jobs  │
└──────────────────────┬───────────────────────┘
                       │ Schedules IEvent jobs via IJobOrchestrator
                       ▼
┌──────────────────────────────────────────────┐
│           PostgreSQL JobStore                │
│             (pfm-scheduler)                  │
│                                              │
│  * Clustered tables (QRTZ_TRIGGERS, etc.)    │
│  * System.Text.Json serialized IEvent payload│
│  * Distributed locking & check-in heartbeat  │
└──────────────────────▲───────────────────────┘
                       │
                       │ Acquires triggers & executes (Polling loop)
┌──────────────────────┴───────────────────────┐
│         PostFundManagement.Jobs              │
│        (Dedicated Worker Service)            │
│                                              │
│  * AddQuartzScheduler()                      │
│  * DefaultThreadPool (MaxConcurrency = 5)    │
│  * SchedulerStatus.Running (Started loop)    │
│  * JobsWorker background service:            │
│    - Registers recurring cron jobs on start  │
│    - Enqueues one-time jobs when needed      │
│  * Executes MediatorJob<TNotification>       │
│  * Deserializes and publishes via IMediator  │
└──────────────────────────────────────────────┘
```

| Characteristic | `PostFundManagement.Api` | `PostFundManagement.Jobs` |
| :--- | :--- | :--- |
| **Architectural Role** | API Gateway & Event Producer | Dedicated Execution Daemon & Consumer |
| **Registration Extension** | `AddQuartzSchedulerClient(...)` | `AddQuartzScheduler(...)` |
| **Thread Pool** | `ZeroSizeThreadPool` (0 threads) | `DefaultThreadPool` (`MaxConcurrency = 5`) |
| **Hosted Service** | None (`AddQuartzHostedService` omitted) | `AddQuartzHostedService(WaitForJobsToComplete = true)` |
| **Lifecycle State** | `SchedulerStatus.Created` (Standby) | `SchedulerStatus.Running` (Active execution loop) |
| **Readiness Check** | Verifies DB connectivity without requiring `Running` | Verifies active execution engine & DB connectivity |
| **Trigger Acquisition** | ❌ **Disabled** (Cannot acquire triggers or run jobs) | ✅ **Active** (Polls DB, acquires triggers, runs jobs) |

### 6.2 Event-Driven Mediated Orchestration (`IEvent` & `MediatorJob`)

Background tasks are modeled as strongly-typed domain events implementing `IEvent : INotification` (`PostFundManagement.Domain.Abstractions`).

```mermaid
sequenceDiagram
    autonumber
    participant API as PostFundManagement.Api
    participant Orchestrator as IJobOrchestrator
    participant DB as PostgreSQL pfm-scheduler
    participant Host as PostFundManagement.Jobs
    participant QuartzThread as Quartz Execution Engine
    participant Mediator as IMediator MediatorJob
    participant Handler as Domain Event Handlers

    Note over API,Orchestrator: 1. Event Scheduling (API Node)
    API->>Orchestrator: SendAsync(notification) / ScheduleAsync(notification, cron)
    Orchestrator->>DB: Stores JobDetail and Trigger with serialized Payload JSON
    Note over DB,Host: 2. Distributed Acquisition (Jobs Node)
    QuartzThread->>DB: Polls and acquires due trigger with row-level locks
    QuartzThread->>Host: Dispatches to local thread pool
    Host->>Mediator: Instantiates MediatorJob[TNotification]
    Mediator->>Mediator: Deserializes payload from MergedJobDataMap
    Mediator->>Handler: Publish(notification, cancellationToken)
    Handler-->>Host: Execution complete and updates business data in pfm DB
    Host->>DB: Releases trigger and commits execution state
```

1. **Scheduling via `IJobOrchestrator`**:
   - `SendAsync<TNotification>(notification)`: Schedules a durable, one-time execution in the `onetime-jobs` group with `StartNow()` and `DisallowConcurrentExecution()`.
   - `ScheduleAsync<TNotification>(notification, cronExpression)`: Configures a durable, cron-driven recurring job in the `scheduled-jobs` group with automatic misfire handling (`CronTriggerMisfireInstruction.IgnoreMisfires`).
   - `InterruptAsync(eventName, correlationId)`: Allows runtime cancellation of active job executions across the cluster.
2. **Transparent Deserialization & Dispatch (`MediatorJob<TNotification>`)**:
   `MediatorJob<TNotification>` resolves `IMediator` from DI, deserializes the JSON payload, initiates OpenTelemetry distributed tracing (`MediatorTelemetry.Source`), and publishes the event to all registered in-process handlers.

### 6.3 Recurring vs. One-Time Job Patterns

* **Recurring Jobs**: Scheduled by `JobsWorker` (`BackgroundService` in `PostFundManagement.Jobs`) during application startup. These jobs govern periodic health sweeps, SLA tracking, notification batching, and risk evaluations on predictable cron intervals.
* **One-Time Jobs**: Primarily scheduled by `PostFundManagement.Api` as immediate or deferred side-effects of HTTP requests and lifecycle mutations (e.g. user invites, award registration triggers). Furthermore, a running recurring job in `PostFundManagement.Jobs` can dispatch a one-time job (`jobOrchestrator.SendAsync`) when handling fan-out tasks or invoking decoupled activities that cannot be triggered directly by the originating job.

### 6.4 Core Background Processing Capabilities

The background job runner powers four essential operational workloads:

1. **Automated Project Monitoring & Reminders**:
   - Sweeps active projects, contractual deliverables, and milestone deadlines.
   - Computes planned vs. actual progress variance and escalates overdue milestones (`FR-M04-017`, `FR-M07-009`).
   - Dispatches automated reminders to Grantees/Partners prior to milestone due dates and submission cut-offs.
2. **Risk Monitoring & Automated Notifications**:
   - Evaluates project risk registers, Key Risk Indicators (KRIs), and issue escalation thresholds (`FR-M09-001` - `011`).
   - Detects risk score creep (Likelihood $\times$ Impact) and triggers automated alerts when risk tolerance thresholds are breached.
   - Monitors overdue mitigation action plans and escalates unresolved high-severity risks to Programme Managers.
3. **Email Notification Processing**:
   - Continuously consumes and batches pending notification records (`Platform.Email` / `NotificationPlatform.Email`).
   - Dispatches transactional emails via MailKit / MimeKit SMTP with exponential backoff and retry policies.
   - Updates delivery status (`Sent`, `Failed`, retry count) and logs dispatch telemetry for audit compliance (`FR-M25-001`).
4. **Platform Extensibility**:
   - New background jobs and recurring routines can be introduced by creating an `IEvent` model in `PostFundManagement.Application` and scheduling it via `JobsWorker` or `IJobOrchestrator`, with zero impact on the API surface.