Updated documentation

Architecture.md

@@ -22,12 +22,13 @@ The PFM platform is constructed using **Clean Architecture** and **Domain-Driven
 │                             API GATEWAY                                │
 │  PostFundManagement.Api (ASP.NET Core Minimal APIs / Carter)           │
 │  Authentication: JWT Bearer + OpenTelemetry Observability              │
+│  Quartz Client: Scheduler-Only (ZeroSizeThreadPool / Enqueues Jobs)    │
 └───────────────────────────────────┬────────────────────────────────────┘
                                     │
 ┌───────────────────────────────────▼────────────────────────────────────┐
 │                          APPLICATION LAYER                             │
 │  PostFundManagement.Application                                        │
-│  CQRS (Mediator), Business Validators, Domain Services, Quartz Jobs   │
+│  CQRS (Mediator), Business Validators, Domain Services, Event Handlers │
 └──────────────────┬─────────────────────────────────┬───────────────────┘
                    │                                 │
 ┌──────────────────▼──────────────────┐   ┌──────────▼───────────────────┐
@@ -35,8 +36,15 @@ The PFM platform is constructed using **Clean Architecture** and **Domain-Driven
 │  PostFundManagement.Domain          │   │  PostFundManagement.Infrastr.│
 │  Entities, Value Objects, Enums,    │   │  PostgreSQL (EF Core 10),    │
 │  Domain Invariants, Audit Models    │   │  Skoruba STS, S3 Garage,     │
-└─────────────────────────────────────┘   │  Redis HybridCache           │
+└─────────────────────────────────────┘   │  Redis, Quartz (pfm-scheduler)│
-                                          └──────────────────────────────┘
+                                          └──────────────┬───────────────┘
+                                                         │
+┌────────────────────────────────────────────────────────▼───────────────┐
+│                     BACKGROUND JOBS WORKER HOST                        │
+│  PostFundManagement.Jobs (.NET 10 Background Host / Quartz Runner)     │
+│  Recurring Job Registration (JobsWorker) & Mediated Execution Engine   │
+│  Quartz Runner: Full Execution (DefaultThreadPool MaxConcurrency = 5)  │
+└────────────────────────────────────────────────────────────────────────┘
 ```
 
 ---
@@ -113,8 +121,8 @@ Even if an `Executive`, `Manager`, or `Supervisor` possesses the `update` claim
 flowchart LR
     A["Maker (User A)<br/>Creates Record"] --> B["Record Persisted<br/>with createdBy = User A"]
     B --> C{"Checker Evaluation<br/>(Approver Login)"}
-    C -->|Checker == User A| D["Approval BLOCKED<br/>Maker-Checker Violation"]
+    C -->|"Checker is User A"| D["Approval BLOCKED: Maker-Checker Violation"]
-    C -->|Checker == User B<br/>with 'update' claim| E["Approval GRANTED<br/>Status -> Approved"]
+    C -->|"Checker is User B with update claim"| E["Approval GRANTED: Status Approved"]
 ```
 
 ### 4.2 Record Attribution Model
@@ -189,4 +197,122 @@ Following the introduction of fine-grained RBAC and Maker-Checker checks, startu
 * **Immediate Non-Blocking Session Render**: If valid session tokens and user profiles exist in `localStorage`, `AuthProvider` immediately unblocks UI rendering (`isLoading: false`) and runs tenant lockout checks, account synchronization, and claim revalidation asynchronously in the background.
 * **In-Memory User Sync & Lockout Caching**: `isUserSyncedInBrowser` and `verifyTenantLockout` maintain in-memory references to eliminate synchronous `JSON.parse` operations from the request pipeline.
 
+---
+
+## 6. Distributed Background Job Processing Architecture (PostFundManagement.Jobs)
+
+The PFM platform decouples synchronous API request handling from asynchronous, long-running, and scheduled workloads using a distributed **Quartz.NET (v4.6+)** clustering topology backed by PostgreSQL (`pfm-scheduler`).
+
+### 6.1 Split-Node Topology: Scheduler Client vs. Execution Host
+
+To maintain high API throughput, low memory footprint, and complete process isolation, PFM implements a split-node architecture between `PostFundManagement.Api` and `PostFundManagement.Jobs`:
+
+```
+┌──────────────────────────────────────────────┐
+│          PostFundManagement.Api              │
+│       (API Gateway / Scheduler Client)       │
+│                                              │
+│  * AddQuartzSchedulerClient()                │
+│  * ZeroSizeThreadPool (0 execution threads)  │
+│  * SchedulerStatus.Created (Standby mode)    │
+│  * Strictly schedules / enqueues events      │
+│  * Never acquires triggers or executes jobs  │
+└──────────────────────┬───────────────────────┘
+                       │ Schedules IEvent jobs via IJobOrchestrator
+                       ▼
+┌──────────────────────────────────────────────┐
+│           PostgreSQL JobStore                │
+│             (pfm-scheduler)                  │
+│                                              │
+│  * Clustered tables (QRTZ_TRIGGERS, etc.)    │
+│  * System.Text.Json serialized IEvent payload│
+│  * Distributed locking & check-in heartbeat  │
+└──────────────────────▲───────────────────────┘
+                       │
+                       │ Acquires triggers & executes (Polling loop)
+┌──────────────────────┴───────────────────────┐
+│         PostFundManagement.Jobs              │
+│        (Dedicated Worker Service)            │
+│                                              │
+│  * AddQuartzScheduler()                      │
+│  * DefaultThreadPool (MaxConcurrency = 5)    │
+│  * SchedulerStatus.Running (Started loop)    │
+│  * JobsWorker background service:            │
+│    - Registers recurring cron jobs on start  │
+│    - Enqueues one-time jobs when needed      │
+│  * Executes MediatorJob<TNotification>       │
+│  * Deserializes and publishes via IMediator  │
+└──────────────────────────────────────────────┘
+```
+
+| Characteristic | `PostFundManagement.Api` | `PostFundManagement.Jobs` |
+| :--- | :--- | :--- |
+| **Architectural Role** | API Gateway & Event Producer | Dedicated Execution Daemon & Consumer |
+| **Registration Extension** | `AddQuartzSchedulerClient(...)` | `AddQuartzScheduler(...)` |
+| **Thread Pool** | `ZeroSizeThreadPool` (0 threads) | `DefaultThreadPool` (`MaxConcurrency = 5`) |
+| **Hosted Service** | None (`AddQuartzHostedService` omitted) | `AddQuartzHostedService(WaitForJobsToComplete = true)` |
+| **Lifecycle State** | `SchedulerStatus.Created` (Standby) | `SchedulerStatus.Running` (Active execution loop) |
+| **Readiness Check** | Verifies DB connectivity without requiring `Running` | Verifies active execution engine & DB connectivity |
+| **Trigger Acquisition** | ❌ **Disabled** (Cannot acquire triggers or run jobs) | ✅ **Active** (Polls DB, acquires triggers, runs jobs) |
+
+### 6.2 Event-Driven Mediated Orchestration (`IEvent` & `MediatorJob`)
+
+Background tasks are modeled as strongly-typed domain events implementing `IEvent : INotification` (`PostFundManagement.Domain.Abstractions`).
+
+```mermaid
+sequenceDiagram
+    autonumber
+    participant API as PostFundManagement.Api
+    participant Orchestrator as IJobOrchestrator
+    participant DB as PostgreSQL pfm-scheduler
+    participant Host as PostFundManagement.Jobs
+    participant QuartzThread as Quartz Execution Engine
+    participant Mediator as IMediator MediatorJob
+    participant Handler as Domain Event Handlers
+
+    Note over API,Orchestrator: 1. Event Scheduling (API Node)
+    API->>Orchestrator: SendAsync(notification) / ScheduleAsync(notification, cron)
+    Orchestrator->>DB: Stores JobDetail and Trigger with serialized Payload JSON
+    Note over DB,Host: 2. Distributed Acquisition (Jobs Node)
+    QuartzThread->>DB: Polls and acquires due trigger with row-level locks
+    QuartzThread->>Host: Dispatches to local thread pool
+    Host->>Mediator: Instantiates MediatorJob[TNotification]
+    Mediator->>Mediator: Deserializes payload from MergedJobDataMap
+    Mediator->>Handler: Publish(notification, cancellationToken)
+    Handler-->>Host: Execution complete and updates business data in pfm DB
+    Host->>DB: Releases trigger and commits execution state
+```
+
+1. **Scheduling via `IJobOrchestrator`**:
+   - `SendAsync<TNotification>(notification)`: Schedules a durable, one-time execution in the `onetime-jobs` group with `StartNow()` and `DisallowConcurrentExecution()`.
+   - `ScheduleAsync<TNotification>(notification, cronExpression)`: Configures a durable, cron-driven recurring job in the `scheduled-jobs` group with automatic misfire handling (`CronTriggerMisfireInstruction.IgnoreMisfires`).
+   - `InterruptAsync(eventName, correlationId)`: Allows runtime cancellation of active job executions across the cluster.
+2. **Transparent Deserialization & Dispatch (`MediatorJob<TNotification>`)**:
+   `MediatorJob<TNotification>` resolves `IMediator` from DI, deserializes the JSON payload, initiates OpenTelemetry distributed tracing (`MediatorTelemetry.Source`), and publishes the event to all registered in-process handlers.
+
+### 6.3 Recurring vs. One-Time Job Patterns
+
+* **Recurring Jobs**: Scheduled by `JobsWorker` (`BackgroundService` in `PostFundManagement.Jobs`) during application startup. These jobs govern periodic health sweeps, SLA tracking, notification batching, and risk evaluations on predictable cron intervals.
+* **One-Time Jobs**: Primarily scheduled by `PostFundManagement.Api` as immediate or deferred side-effects of HTTP requests and lifecycle mutations (e.g. user invites, award registration triggers). Furthermore, a running recurring job in `PostFundManagement.Jobs` can dispatch a one-time job (`jobOrchestrator.SendAsync`) when handling fan-out tasks or invoking decoupled activities that cannot be triggered directly by the originating job.
+
+### 6.4 Core Background Processing Capabilities
+
+The background job runner powers four essential operational workloads:
+
+1. **Automated Project Monitoring & Reminders**:
+   - Sweeps active projects, contractual deliverables, and milestone deadlines.
+   - Computes planned vs. actual progress variance and escalates overdue milestones (`FR-M04-017`, `FR-M07-009`).
+   - Dispatches automated reminders to Grantees/Partners prior to milestone due dates and submission cut-offs.
+2. **Risk Monitoring & Automated Notifications**:
+   - Evaluates project risk registers, Key Risk Indicators (KRIs), and issue escalation thresholds (`FR-M09-001` - `011`).
+   - Detects risk score creep (Likelihood $\times$ Impact) and triggers automated alerts when risk tolerance thresholds are breached.
+   - Monitors overdue mitigation action plans and escalates unresolved high-severity risks to Programme Managers.
+3. **Email Notification Processing**:
+   - Continuously consumes and batches pending notification records (`Platform.Email` / `NotificationPlatform.Email`).
+   - Dispatches transactional emails via MailKit / MimeKit SMTP with exponential backoff and retry policies.
+   - Updates delivery status (`Sent`, `Failed`, retry count) and logs dispatch telemetry for audit compliance (`FR-M25-001`).
+4. **Platform Extensibility**:
+   - New background jobs and recurring routines can be introduced by creating an `IEvent` model in `PostFundManagement.Application` and scheduling it via `JobsWorker` or `IJobOrchestrator`, with zero impact on the API surface.
+
+
 

GEMINI.md

@@ -71,6 +71,7 @@ PostFundManagement/
 ├── PostFundManagement.Infrastructure/  # EF Core 10, PostgreSQL (Npgsql), Migrations, Data Protection
 ├── PostFundManagement.Application/     # CQRS Handlers, Mediator, DTOs, Validations, S3, Email, Quartz
 ├── PostFundManagement.Api/             # Minimal APIs / Carter endpoints, Auth, Swagger/Scalar, Versioning
+├── PostFundManagement.Jobs/            # Dedicated Quartz.NET Background Worker Host (Recurring & One-Time Execution)
 ├── PostFundManagement.Tests/           # Unit & Integration test suite (xUnit, Coverlet)
 ├── PostFundManagement.ConsoleUI/       # Internal Staff & Admin Web Portal (React 19 + Vite 6 + Tailwind 4)
 ├── PostFundManagement.PartnerUI/       # External Grantee / Partner Portal (React 19 + Vite 8 + Tailwind 4)
@@ -85,6 +86,7 @@ PostFundManagement/
 | **`PostFundManagement.Infrastructure`** | `net10.0` | `Microsoft.EntityFrameworkCore` 10.0.12, `Npgsql.EntityFrameworkCore.PostgreSQL` 10.0.3, `Duende.AccessTokenManagement` 4.2.0, EF Core Tools & Design | Houses `PostFundManagementDbContext`, EF Core fluent configurations, entity-to-table mappings, PostgreSQL schema definitions, multitenancy filters, and migrations. |
 | **`PostFundManagement.Application`** | `net10.0` | `Mediator.SourceGenerator` / `Mediator.Abstractions` 3.0.2, `FluentResults` 4.0.0, `Microsoft.Extensions.Caching.Hybrid` 10.10.0, `Microsoft.Extensions.Caching.StackExchangeRedis` 10.0.12, `Refit` 15.2.0, `Duende.AccessTokenManagement.OpenIdConnect` 4.2.0, `Quartz` 4.1.0, `AWSSDK.S3` | Implements CQRS command/query handlers, domain validators, S3 object storage clients (Garage S3), background jobs (Quartz), notification dispatchers, and business logic. |
 | **`PostFundManagement.Api`** | `net10.0` | `Microsoft.AspNetCore.OpenApi` 10.0.12, `Scalar.AspNetCore` 2.17.4, `Asp.Versioning.Http` 10.2.3, `IdentityModel.AspNetCore` 4.3.0, `Microsoft.AspNetCore.Authentication.JwtBearer` 10.0.12, `AspNetCore.HealthChecks.NpgSql` 9.0.0, `AspNetCore.HealthChecks.UI` | Minimal API endpoint definitions divided by business slices (`Grants`, `Projects`, `Execution`, `Governance`, `Evidence`, `Communications`, `Users`). Handles routing, auth policies, error problem details, and OpenAPI metadata. |
+| **`PostFundManagement.Jobs`** | `net10.0` | `Quartz` 4.6.2, `Quartz.Extensions.DependencyInjection` 4.6.2, `Quartz.Serialization.SystemTextJson` 4.6.2, `Mediator.Abstractions` 3.0.2 | Dedicated background execution host. Executes recurring Quartz jobs (registered on startup by `JobsWorker`), processes asynchronous domain events (`IEvent`), and executes jobs concurrently (`MaxConcurrency = 5`). |
 | **`PostFundManagement.Tests`** | `net10.0` | `xunit` 2.9.3, `xunit.runner.visualstudio` 3.1.4, `Microsoft.NET.Test.Sdk` 17.14.1, `coverlet.collector` 6.0.4 | Unit and integration test coverage for domain rules, CQRS handlers, financial calculation models, and gate condition validations. |
 
 ### 2.2 Frontend Projects (React 19 / TypeScript)
@@ -120,6 +122,22 @@ PostFundManagement/
 * **Authentication / STS**: Duende IdentityServer / Skoruba Administration (`https://sts.security.khongisa.co.za`).
 * **Observability**: OpenTelemetry exporters streaming traces, metrics, and logs to Aspire Dashboard / OTLP collectors.
 
+### 2.4 Distributed Background Processing Architecture (Quartz.NET & PostFundManagement.Jobs)
+* **Dedicated Execution Daemon**: `PostFundManagement.Jobs` is a standalone background host responsible for recurring job scheduling, lifecycle sweeps, and asynchronous event execution.
+* **API Isolation (Scheduler Client)**: `PostFundManagement.Api` is strictly configured as a scheduler client via `AddQuartzSchedulerClient(...)` using `ZeroSizeThreadPool` (0 worker threads) and running in standby (`SchedulerStatus.Created`). The API enqueues jobs and triggers into the clustered PostgreSQL database (`pfm-scheduler`) but never acquires triggers or executes jobs locally.
+* **Worker Execution Engine**: `PostFundManagement.Jobs` configures full Quartz execution via `AddQuartzScheduler(...)` using `DefaultThreadPool` (`MaxConcurrency = 5`) and `AddQuartzHostedService(options => options.WaitForJobsToComplete = true)`.
+* **Event-Driven Orchestration (`IEvent` & `MediatorJob`)**:
+  - The API acts as the primary source of one-time events triggered by user actions and lifecycle transitions (implementing `IEvent : INotification`).
+  - Events are enqueued via `IJobOrchestrator.SendAsync` or `ScheduleAsync` and persisted in `pfm-scheduler` as `MediatorJob<TNotification>` with serialized JSON payloads.
+  - When acquired by `PostFundManagement.Jobs`, `MediatorJob` deserializes the event payload, propagates OpenTelemetry distributed traces, and publishes the event via `IMediator.Publish(...)`.
+* **Startup Recurring Registration**: On daemon startup, `JobsWorker : BackgroundService` initializes and registers recurring cron jobs across tenant portfolios.
+* **Cascading One-Time Execution**: If a recurring job identifies complex follow-up items that cannot be processed in the recurring execution window, it dispatches decoupled one-time jobs (`jobOrchestrator.SendAsync`) across worker threads.
+* **Core Job Workloads**:
+  1. *Automated Project Monitoring & Reminders*: Sweeps milestones, checks contractual SLA deadlines, sets overdue flags (`FR-M07-009`), and sends reminder notices to Grantees/Partners.
+  2. *Risk Monitoring & Automated Notifications*: Evaluates risk score creep ($\text{Likelihood} \times \text{Impact}$), flags tolerance breaches, and sends escalation alerts for overdue mitigation actions.
+  3. *Email Notification Processing*: Batches and dispatches pending notification entries (`Platform.Email` / `NotificationPlatform.Email`) via MailKit/MimeKit SMTP and records delivery telemetry (`FR-M25-001`).
+  4. *Future Jobs*: Seamlessly supports future scheduled workloads (e.g. drawdown reconciliations, compliance expiries, portfolio aggregations) simply by adding `IEvent` handlers.
+
 ---
 
 ## 3. Master Business Rules (BR-001 to BR-010)
@@ -848,7 +866,7 @@ The platform uses **Hashids** to obfuscate internal sequential database primary
 * **Implementation**: Managed by `HashService` (`PostFundManagement.Application/Services/Shared/HashService.cs`) using the `Hashids.net` (v1.7.0) library.
 * **Shared Configuration (`HasherSettings`)**:
   * **Salt**: Sourced from configuration (`HasherSettings:Salt` or `HASHER_SALT`), defaulting to:
-    `"TGlnaHQgQ2hhcm1zLCBNaWRyYW5kQm9va3MgbWFrZSBsb3RzIG9mIG1vbmV5IGFuZCBhcmUgYSBzdWNjZXNzZnVsIHZpcmFsIHN0b3J5IGluIFNvdXRoIEFmcmljYQ=="`
+    `"salt-random-string"`
   * **Min Hash Length**: `HasherSettings:MinHashLength` or `HASHER_MIN_HASH_LENGTH` (default: `11`).
 * **Core Service Methods**:
   * `HashEncodeLongId(long id)` / `HashEncodeIntId(int id)`: Encodes integers into obfuscated HashID strings (e.g. `1` $\to$ `kY5aB9m...`).

Processes.md

@@ -88,6 +88,16 @@ The PFM platform governs the lifecycle of grant, development, innovation, and im
   4. Resolution: `Approved & Enacted` or `Rejected`.
   5. **Maker-Checker Rule**: Change request initiator cannot approve or enact the change.
 
+### 2.7 Process 7: Automated Background Workflows & Scheduled Jobs (PostFundManagement.Jobs)
+* **Pre-condition**: Background jobs host (`PostFundManagement.Jobs`) active and connected to `pfm-scheduler`.
+* **Actors**: System Engine, `JobsWorker` Daemon, Domain Event Handlers.
+* **Workflow**:
+  1. **Event Triggering**: API operations emit strongly typed events (`IEvent`) and enqueue them into `pfm-scheduler` via `IJobOrchestrator`.
+  2. **Cron Schedule Execution**: `JobsWorker` initializes recurring schedules on daemon startup for periodic monitoring, risk assessments, and communication batches.
+  3. **Job Execution**: Quartz cluster dispatches acquired triggers to `MediatorJob<TNotification>` within `PostFundManagement.Jobs` (concurrency limit = 5).
+  4. **Mediated Processing**: Event payload is deserialized and published via `IMediator` to execute domain logic (project monitoring, risk threshold checks, email dispatch).
+  5. **Cascading One-Time Jobs**: When a recurring job identifies an action item requiring separate asynchronous execution, it triggers a decoupled one-time job (`jobOrchestrator.SendAsync`).
+
 ---
 
 ## 3. The Maker-Checker Operational Policy
@@ -139,20 +149,20 @@ To ensure that caching never impedes record lifecycle updates or displays stale
 sequenceDiagram
     autonumber
     actor User as User / Maker
-    participant UI as ConsoleUI (React Query)
+    participant UI as ConsoleUI React Query
     participant Client as Axios HttpClient
-    participant OutputCache as Backend OutputCache (Redis/Memory)
+    participant OutputCache as Backend OutputCache Redis/Memory
     participant DB as PostgreSQL Database
 
-    User->>UI: Submits Record Mutation (Create / Update / Approve)
+    User->>UI: Submits Record Mutation Create/Update/Approve
     UI->>Client: Dispatches POST / PUT / PATCH / DELETE
     Client->>DB: Executes Mutation in Database
     DB-->>Client: Returns 200 OK / 201 Created
     Client->>Client: Interceptor detects mutation: bumps activeMutationEpoch
-    UI->>UI: onSuccess triggers queryClient.invalidateQueries()
+    UI->>UI: onSuccess triggers invalidateQueries
     UI->>Client: Refetches active queries with _v = activeMutationEpoch
-    Client->>OutputCache: GET /v1/api/*?_v=<newEpoch>
+    Client->>OutputCache: GET /v1/api/* with new _v epoch
-    Note over OutputCache: Cache Miss (new _v key never seen before)
+    Note over OutputCache: Cache Miss: new _v key never seen before
     OutputCache->>DB: Queries PostgreSQL for fresh data
     DB-->>OutputCache: Returns updated entities
     OutputCache-->>UI: Caches fresh data under _v and returns to UI
@@ -164,4 +174,74 @@ sequenceDiagram
 2. **Deterministic Invalidation**: Calling standard `queryClient.invalidateQueries({ queryKey: [...] })` or `invalidateAndBustCache(queryClient, filters)` guarantees that the subsequent refetch bypasses stale ASP.NET Core OutputCache entries.
 3. **Optimized Read Cache**: When users navigate across unchanged views or switch tabs, identical `_v` tokens are reused, allowing backend OutputCache and React Query in-memory caches to fulfill requests in < 5ms without unnecessary database hits.
 
+---
+
+## 6. Background Job Lifecycle & Event Execution Workflows
+
+The dedicated worker host (`PostFundManagement.Jobs`) operates as a decoupled daemon executing recurring maintenance routines and asynchronous domain events dispatched from the API or internally from other jobs.
+
+### 6.1 Event Ingestion & Job Persistence Flow
+
+```mermaid
+flowchart TD
+    subgraph Producer ["PostFundManagement.Api (Event Source)"]
+        A["User Action / Lifecycle Transition"] --> B["Domain Command / Mutation"]
+        B --> C["Instantiate IEvent (e.g. SyncUserEvent, ProjectReminderEvent)"]
+        C --> D["IJobOrchestrator.SendAsync() or ScheduleAsync()"]
+    end
+
+    subgraph Store ["PostgreSQL Clustered Store (pfm-scheduler)"]
+        D --> E[("QRTZ_JOB_DETAILS and QRTZ_TRIGGERS")]
+    end
+
+    subgraph Consumer ["PostFundManagement.Jobs (Execution Host)"]
+        E -->|"Acquire Trigger Lock"| F["Quartz Scheduler Engine (MaxConcurrency = 5)"]
+        F --> G["MediatorJob[TNotification]"]
+        G --> H["Deserialize JSON Payload from JobDataMap"]
+        H --> I["IMediator.Publish(notification)"]
+        I --> J["Domain Event Handler: Business Logic in pfm DB"]
+        J -.->|"Cascading Sub-Task Needed"| K["IJobOrchestrator.SendAsync: Dispatch One-Time Job"]
+        K -.-> E
+    end
+```
+
+### 6.2 Standard Background Processes
+
+#### 1. Automated Project Monitoring & Reminders Process
+* **Trigger**: Scheduled cron job registered on `JobsWorker` startup.
+* **Operational Cycle**:
+  1. Job queries active projects and non-completed milestones across tenant partitions.
+  2. Compares milestone contractual due dates against current UTC time:
+     - **T - 14 / T - 7 / T - 1 Days**: Emits reminder notification events to Grantee/Partner project managers.
+     - **T + 0 Days (Overdue)**: Automatically flags the milestone as `Overdue` (`FR-M07-009`) and updates project RAG status to `At Risk` (`FR-M04-017`).
+  3. Escalates overdue notifications to internal Programme Managers for intervention.
+
+#### 2. Risk Monitoring & Automated Escalation Process
+* **Trigger**: Periodic recurring schedule.
+* **Operational Cycle**:
+  1. Queries all active project risk registers and Key Risk Indicators (KRIs) (`FR-M09-001` - `011`).
+  2. Evaluates current severity score ($\text{Risk Score} = \text{Likelihood} \times \text{Impact}$).
+  3. If risk score breaches the configured programme tolerance threshold:
+     - Automatically logs a risk escalation event.
+     - Dispatches alerts to Risk Managers and Portfolio Executives.
+  4. Sweeps assigned risk mitigation actions; identifies overdue mitigation plans and emits reminders to responsible assignees.
+
+#### 3. Email Notification Processing Workflow (`Platform.Email`)
+* **Trigger**: High-frequency recurring polling job or deferred one-time dispatch.
+* **Operational Cycle**:
+  1. Queries pending notifications from the transactional database where `Platform == NotificationPlatform.Email` and `Status == NotificationStatus.Pending` or `Queued`.
+  2. Resolves tenant context using `HeadlessTenantContextService`.
+  3. Formats email templates with recipient credentials and branding.
+  4. Transmits messages via MailKit/MimeKit SMTP engine with rate-limiting and connection pooling.
+  5. On successful delivery: Updates record status to `Sent`, stamps `DeliveredAt` timestamp, and logs dispatch telemetry (`FR-M25-001`).
+  6. On failure: Increments retry counter; if retry limit exceeded, marks as `Failed` with diagnostic error trace.
+
+### 6.3 Recurring-to-One-Time Job Cascading Pattern
+
+When a recurring job evaluates a broad batch of records (e.g., scanning thousands of project indicators or verifying compliance expirations), it must not execute long-running operations synchronously in the recurring job thread:
+1. The recurring job identifies items requiring heavy processing (e.g. generating an executive PDF digest or calling external webhooks).
+2. For each identified item, it enqueues an isolated one-time job via `IJobOrchestrator.SendAsync(specificEvent)`.
+3. This releases the recurring job quickly, distributes execution across the 5 available worker threads in `PostFundManagement.Jobs`, and guarantees that a single failed sub-task can be retried independently without stalling the entire recurring cycle.
+
+
 

README.md

@@ -41,6 +41,16 @@ The application follows a modular, hierarchical architecture where high-level fu
 ### **5. Notifications & Platform Services**
 * Multi-channel delivery engine (`NotificationPlatform`) tracking candidate invitations, reminders, and status updates (`NotificationStatus`) across Email, SMS, WhatsApp, In-App, and Webhooks.
 
+### **6. Background Processing & Distributed Jobs Engine**
+* **`PostFundManagement.Jobs`**: Dedicated background execution host powered by **Quartz.NET (v4.6+)** with a PostgreSQL clustered JobStore (`pfm-scheduler`).
+* **Decoupled Architecture**: `PostFundManagement.Api` serves strictly as a scheduler client (`ZeroSizeThreadPool`), dispatching events into the cluster while leaving execution exclusively to `PostFundManagement.Jobs` (`MaxConcurrency = 5`).
+* **Event-Driven Execution**: Domain events implementing `IEvent : INotification` are mediated and serialized into `MediatorJob<TNotification>`.
+* **Core Automated Operations**:
+  - **Automated Project Monitoring & Reminders**: Tracks milestones, evaluates planned vs actual deadlines, flags overdue statuses, and emits reminders to Grantees/Partners.
+  - **Risk Monitoring & Automated Notifications**: Evaluates risk registers, tracks Key Risk Indicators (KRIs), flags score creep, and escalates overdue mitigation actions.
+  - **Email Processing**: Asynchronously processes `NotificationPlatform.Email` records via MailKit/MimeKit SMTP.
+  - **Recurring & Cascading Jobs**: `JobsWorker` registers recurring cron schedules on startup and can trigger isolated one-time jobs when sub-activities cannot be executed directly by another job.
+
 ---
 
 ## 🧩 Key System Enums
@@ -68,6 +78,7 @@ The application follows a modular, hierarchical architecture where high-level fu
 * **Object Storage**: S3-compatible storage (Garage S3 / AWS S3)
 * **Caching & OutputCache Invalidation**: Redis StackExchange + ASP.NET Core `OutputCaching` (2-minute base policy) coupled with TanStack Query v5 and Axios client-side mutation versioning (`_v` parameter) for instantaneous reads and guaranteed zero-staleness on mutations
 * **Telemetry**: OpenTelemetry Metrics, Traces & Logs with Aspire / OTLP
+* **Distributed Background Jobs**: Quartz.NET 4.6+ with clustered PostgreSQL JobStore (`pfm-scheduler`), separating the API (Scheduler Client with `ZeroSizeThreadPool`) from `PostFundManagement.Jobs` (Worker Execution Daemon with `MaxConcurrency = 5`)
 
 ---
 
@@ -105,16 +116,18 @@ For in-depth architectural and operational specifications, see:
 
 ```text
 PostFundManagement/
- ├── PostFundManagement.Domain/          # Core Domain Entities, Enums & Interfaces
+ ├── PostFundManagement.Domain/          # Core Domain Entities, Enums, Value Objects & Abstractions (IEvent)
- ├── PostFundManagement.Infrastructure/  # EF Core 10, PostgreSQL, Skoruba STS, Migrations
+ ├── PostFundManagement.Infrastructure/  # EF Core 10, PostgreSQL (Npgsql), Migrations, Data Protection
- ├── PostFundManagement.Application/     # CQRS Handlers, DTOs, Business Rules & Validations
+ ├── PostFundManagement.Application/     # CQRS Handlers, Mediator, DTOs, Validations, S3, Email, Job Orchestrator
- ├── PostFundManagement.Api/             # REST Endpoints, Middleware, Auth & Swagger/Scalar
+ ├── PostFundManagement.Api/             # ASP.NET Core Minimal APIs / Carter, Auth (OIDC), Swagger/Scalar, Versioning
- ├── PostFundManagement.Tests/           # Unit & Integration Test Suite
+ ├── PostFundManagement.Jobs/            # Dedicated Quartz.NET Background Worker Host (Recurring & Event-Driven Jobs)
- ├── PostFundManagement.ConsoleUI/       # Internal Staff & Admin Web Portal (React 19 + Vite)
+ ├── PostFundManagement.Tests/           # Unit & Integration Test Suite (xUnit, Coverlet)
- ├── PostFundManagement.PartnerUI/       # External Grantee / Recipient Portal (React 19 + Vite)
+ ├── PostFundManagement.ConsoleUI/       # Internal Staff & Admin Web Portal (React 19 + Vite + Tailwind 4)
- ├── Architecture.md                     # System Architecture, RBAC & Maker-Checker Spec
+ ├── PostFundManagement.PartnerUI/       # External Grantee / Recipient Portal (React 19 + Vite + WebGL OGL)
- ├── Processes.md                        # Operational Grant Lifecycle & Process Flows
+ ├── Architecture.md                     # System Architecture, Security, RBAC, Caching & Quartz Spec
- └── docker-compose.yml                  # Container orchestration
+ ├── Processes.md                        # Operational Grant Lifecycle, Maker-Checker & Background Workflows
+ ├── GEMINI.md                           # Master Technical & Architecture Roadmap (SRS v2.0 Alignment)
+ └── docker-compose.yml                  # Container orchestration for API and Frontends
 ```
 
 ## 🚀 Getting Started