diff --git a/Architecture.md b/Architecture.md index 14b8374..56decaa 100644 --- a/Architecture.md +++ b/Architecture.md @@ -22,12 +22,13 @@ The PFM platform is constructed using **Clean Architecture** and **Domain-Driven │ API GATEWAY │ │ PostFundManagement.Api (ASP.NET Core Minimal APIs / Carter) │ │ Authentication: JWT Bearer + OpenTelemetry Observability │ +│ Quartz Client: Scheduler-Only (ZeroSizeThreadPool / Enqueues Jobs) │ └───────────────────────────────────┬────────────────────────────────────┘ │ ┌───────────────────────────────────▼────────────────────────────────────┐ │ APPLICATION LAYER │ │ PostFundManagement.Application │ -│ CQRS (Mediator), Business Validators, Domain Services, Quartz Jobs │ +│ CQRS (Mediator), Business Validators, Domain Services, Event Handlers │ └──────────────────┬─────────────────────────────────┬───────────────────┘ │ │ ┌──────────────────▼──────────────────┐ ┌──────────▼───────────────────┐ @@ -35,8 +36,15 @@ The PFM platform is constructed using **Clean Architecture** and **Domain-Driven │ PostFundManagement.Domain │ │ PostFundManagement.Infrastr.│ │ Entities, Value Objects, Enums, │ │ PostgreSQL (EF Core 10), │ │ Domain Invariants, Audit Models │ │ Skoruba STS, S3 Garage, │ -└─────────────────────────────────────┘ │ Redis HybridCache │ - └──────────────────────────────┘ +└─────────────────────────────────────┘ │ Redis, Quartz (pfm-scheduler)│ + └──────────────┬───────────────┘ + │ +┌────────────────────────────────────────────────────────▼───────────────┐ +│ BACKGROUND JOBS WORKER HOST │ +│ PostFundManagement.Jobs (.NET 10 Background Host / Quartz Runner) │ +│ Recurring Job Registration (JobsWorker) & Mediated Execution Engine │ +│ Quartz Runner: Full Execution (DefaultThreadPool MaxConcurrency = 5) │ +└────────────────────────────────────────────────────────────────────────┘ ``` --- @@ -113,8 +121,8 @@ Even if an `Executive`, `Manager`, or `Supervisor` possesses the `update` claim flowchart LR A["Maker (User A)
Creates Record"] --> B["Record Persisted
with createdBy = User A"] B --> C{"Checker Evaluation
(Approver Login)"} - C -->|Checker == User A| D["Approval BLOCKED
Maker-Checker Violation"] - C -->|Checker == User B
with 'update' claim| E["Approval GRANTED
Status -> Approved"] + C -->|"Checker is User A"| D["Approval BLOCKED: Maker-Checker Violation"] + C -->|"Checker is User B with update claim"| E["Approval GRANTED: Status Approved"] ``` ### 4.2 Record Attribution Model @@ -189,4 +197,122 @@ Following the introduction of fine-grained RBAC and Maker-Checker checks, startu * **Immediate Non-Blocking Session Render**: If valid session tokens and user profiles exist in `localStorage`, `AuthProvider` immediately unblocks UI rendering (`isLoading: false`) and runs tenant lockout checks, account synchronization, and claim revalidation asynchronously in the background. * **In-Memory User Sync & Lockout Caching**: `isUserSyncedInBrowser` and `verifyTenantLockout` maintain in-memory references to eliminate synchronous `JSON.parse` operations from the request pipeline. +--- + +## 6. Distributed Background Job Processing Architecture (PostFundManagement.Jobs) + +The PFM platform decouples synchronous API request handling from asynchronous, long-running, and scheduled workloads using a distributed **Quartz.NET (v4.6+)** clustering topology backed by PostgreSQL (`pfm-scheduler`). + +### 6.1 Split-Node Topology: Scheduler Client vs. Execution Host + +To maintain high API throughput, low memory footprint, and complete process isolation, PFM implements a split-node architecture between `PostFundManagement.Api` and `PostFundManagement.Jobs`: + +``` +┌──────────────────────────────────────────────┐ +│ PostFundManagement.Api │ +│ (API Gateway / Scheduler Client) │ +│ │ +│ * AddQuartzSchedulerClient() │ +│ * ZeroSizeThreadPool (0 execution threads) │ +│ * SchedulerStatus.Created (Standby mode) │ +│ * Strictly schedules / enqueues events │ +│ * Never acquires triggers or executes jobs │ +└──────────────────────┬───────────────────────┘ + │ Schedules IEvent jobs via IJobOrchestrator + ▼ +┌──────────────────────────────────────────────┐ +│ PostgreSQL JobStore │ +│ (pfm-scheduler) │ +│ │ +│ * Clustered tables (QRTZ_TRIGGERS, etc.) │ +│ * System.Text.Json serialized IEvent payload│ +│ * Distributed locking & check-in heartbeat │ +└──────────────────────▲───────────────────────┘ + │ + │ Acquires triggers & executes (Polling loop) +┌──────────────────────┴───────────────────────┐ +│ PostFundManagement.Jobs │ +│ (Dedicated Worker Service) │ +│ │ +│ * AddQuartzScheduler() │ +│ * DefaultThreadPool (MaxConcurrency = 5) │ +│ * SchedulerStatus.Running (Started loop) │ +│ * JobsWorker background service: │ +│ - Registers recurring cron jobs on start │ +│ - Enqueues one-time jobs when needed │ +│ * Executes MediatorJob │ +│ * Deserializes and publishes via IMediator │ +└──────────────────────────────────────────────┘ +``` + +| Characteristic | `PostFundManagement.Api` | `PostFundManagement.Jobs` | +| :--- | :--- | :--- | +| **Architectural Role** | API Gateway & Event Producer | Dedicated Execution Daemon & Consumer | +| **Registration Extension** | `AddQuartzSchedulerClient(...)` | `AddQuartzScheduler(...)` | +| **Thread Pool** | `ZeroSizeThreadPool` (0 threads) | `DefaultThreadPool` (`MaxConcurrency = 5`) | +| **Hosted Service** | None (`AddQuartzHostedService` omitted) | `AddQuartzHostedService(WaitForJobsToComplete = true)` | +| **Lifecycle State** | `SchedulerStatus.Created` (Standby) | `SchedulerStatus.Running` (Active execution loop) | +| **Readiness Check** | Verifies DB connectivity without requiring `Running` | Verifies active execution engine & DB connectivity | +| **Trigger Acquisition** | ❌ **Disabled** (Cannot acquire triggers or run jobs) | ✅ **Active** (Polls DB, acquires triggers, runs jobs) | + +### 6.2 Event-Driven Mediated Orchestration (`IEvent` & `MediatorJob`) + +Background tasks are modeled as strongly-typed domain events implementing `IEvent : INotification` (`PostFundManagement.Domain.Abstractions`). + +```mermaid +sequenceDiagram + autonumber + participant API as PostFundManagement.Api + participant Orchestrator as IJobOrchestrator + participant DB as PostgreSQL pfm-scheduler + participant Host as PostFundManagement.Jobs + participant QuartzThread as Quartz Execution Engine + participant Mediator as IMediator MediatorJob + participant Handler as Domain Event Handlers + + Note over API,Orchestrator: 1. Event Scheduling (API Node) + API->>Orchestrator: SendAsync(notification) / ScheduleAsync(notification, cron) + Orchestrator->>DB: Stores JobDetail and Trigger with serialized Payload JSON + Note over DB,Host: 2. Distributed Acquisition (Jobs Node) + QuartzThread->>DB: Polls and acquires due trigger with row-level locks + QuartzThread->>Host: Dispatches to local thread pool + Host->>Mediator: Instantiates MediatorJob[TNotification] + Mediator->>Mediator: Deserializes payload from MergedJobDataMap + Mediator->>Handler: Publish(notification, cancellationToken) + Handler-->>Host: Execution complete and updates business data in pfm DB + Host->>DB: Releases trigger and commits execution state +``` + +1. **Scheduling via `IJobOrchestrator`**: + - `SendAsync(notification)`: Schedules a durable, one-time execution in the `onetime-jobs` group with `StartNow()` and `DisallowConcurrentExecution()`. + - `ScheduleAsync(notification, cronExpression)`: Configures a durable, cron-driven recurring job in the `scheduled-jobs` group with automatic misfire handling (`CronTriggerMisfireInstruction.IgnoreMisfires`). + - `InterruptAsync(eventName, correlationId)`: Allows runtime cancellation of active job executions across the cluster. +2. **Transparent Deserialization & Dispatch (`MediatorJob`)**: + `MediatorJob` resolves `IMediator` from DI, deserializes the JSON payload, initiates OpenTelemetry distributed tracing (`MediatorTelemetry.Source`), and publishes the event to all registered in-process handlers. + +### 6.3 Recurring vs. One-Time Job Patterns + +* **Recurring Jobs**: Scheduled by `JobsWorker` (`BackgroundService` in `PostFundManagement.Jobs`) during application startup. These jobs govern periodic health sweeps, SLA tracking, notification batching, and risk evaluations on predictable cron intervals. +* **One-Time Jobs**: Primarily scheduled by `PostFundManagement.Api` as immediate or deferred side-effects of HTTP requests and lifecycle mutations (e.g. user invites, award registration triggers). Furthermore, a running recurring job in `PostFundManagement.Jobs` can dispatch a one-time job (`jobOrchestrator.SendAsync`) when handling fan-out tasks or invoking decoupled activities that cannot be triggered directly by the originating job. + +### 6.4 Core Background Processing Capabilities + +The background job runner powers four essential operational workloads: + +1. **Automated Project Monitoring & Reminders**: + - Sweeps active projects, contractual deliverables, and milestone deadlines. + - Computes planned vs. actual progress variance and escalates overdue milestones (`FR-M04-017`, `FR-M07-009`). + - Dispatches automated reminders to Grantees/Partners prior to milestone due dates and submission cut-offs. +2. **Risk Monitoring & Automated Notifications**: + - Evaluates project risk registers, Key Risk Indicators (KRIs), and issue escalation thresholds (`FR-M09-001` - `011`). + - Detects risk score creep (Likelihood $\times$ Impact) and triggers automated alerts when risk tolerance thresholds are breached. + - Monitors overdue mitigation action plans and escalates unresolved high-severity risks to Programme Managers. +3. **Email Notification Processing**: + - Continuously consumes and batches pending notification records (`Platform.Email` / `NotificationPlatform.Email`). + - Dispatches transactional emails via MailKit / MimeKit SMTP with exponential backoff and retry policies. + - Updates delivery status (`Sent`, `Failed`, retry count) and logs dispatch telemetry for audit compliance (`FR-M25-001`). +4. **Platform Extensibility**: + - New background jobs and recurring routines can be introduced by creating an `IEvent` model in `PostFundManagement.Application` and scheduling it via `JobsWorker` or `IJobOrchestrator`, with zero impact on the API surface. + + diff --git a/GEMINI.md b/GEMINI.md index cd26b55..c6fc956 100644 --- a/GEMINI.md +++ b/GEMINI.md @@ -71,6 +71,7 @@ PostFundManagement/ ├── PostFundManagement.Infrastructure/ # EF Core 10, PostgreSQL (Npgsql), Migrations, Data Protection ├── PostFundManagement.Application/ # CQRS Handlers, Mediator, DTOs, Validations, S3, Email, Quartz ├── PostFundManagement.Api/ # Minimal APIs / Carter endpoints, Auth, Swagger/Scalar, Versioning +├── PostFundManagement.Jobs/ # Dedicated Quartz.NET Background Worker Host (Recurring & One-Time Execution) ├── PostFundManagement.Tests/ # Unit & Integration test suite (xUnit, Coverlet) ├── PostFundManagement.ConsoleUI/ # Internal Staff & Admin Web Portal (React 19 + Vite 6 + Tailwind 4) ├── PostFundManagement.PartnerUI/ # External Grantee / Partner Portal (React 19 + Vite 8 + Tailwind 4) @@ -85,6 +86,7 @@ PostFundManagement/ | **`PostFundManagement.Infrastructure`** | `net10.0` | `Microsoft.EntityFrameworkCore` 10.0.12, `Npgsql.EntityFrameworkCore.PostgreSQL` 10.0.3, `Duende.AccessTokenManagement` 4.2.0, EF Core Tools & Design | Houses `PostFundManagementDbContext`, EF Core fluent configurations, entity-to-table mappings, PostgreSQL schema definitions, multitenancy filters, and migrations. | | **`PostFundManagement.Application`** | `net10.0` | `Mediator.SourceGenerator` / `Mediator.Abstractions` 3.0.2, `FluentResults` 4.0.0, `Microsoft.Extensions.Caching.Hybrid` 10.10.0, `Microsoft.Extensions.Caching.StackExchangeRedis` 10.0.12, `Refit` 15.2.0, `Duende.AccessTokenManagement.OpenIdConnect` 4.2.0, `Quartz` 4.1.0, `AWSSDK.S3` | Implements CQRS command/query handlers, domain validators, S3 object storage clients (Garage S3), background jobs (Quartz), notification dispatchers, and business logic. | | **`PostFundManagement.Api`** | `net10.0` | `Microsoft.AspNetCore.OpenApi` 10.0.12, `Scalar.AspNetCore` 2.17.4, `Asp.Versioning.Http` 10.2.3, `IdentityModel.AspNetCore` 4.3.0, `Microsoft.AspNetCore.Authentication.JwtBearer` 10.0.12, `AspNetCore.HealthChecks.NpgSql` 9.0.0, `AspNetCore.HealthChecks.UI` | Minimal API endpoint definitions divided by business slices (`Grants`, `Projects`, `Execution`, `Governance`, `Evidence`, `Communications`, `Users`). Handles routing, auth policies, error problem details, and OpenAPI metadata. | +| **`PostFundManagement.Jobs`** | `net10.0` | `Quartz` 4.6.2, `Quartz.Extensions.DependencyInjection` 4.6.2, `Quartz.Serialization.SystemTextJson` 4.6.2, `Mediator.Abstractions` 3.0.2 | Dedicated background execution host. Executes recurring Quartz jobs (registered on startup by `JobsWorker`), processes asynchronous domain events (`IEvent`), and executes jobs concurrently (`MaxConcurrency = 5`). | | **`PostFundManagement.Tests`** | `net10.0` | `xunit` 2.9.3, `xunit.runner.visualstudio` 3.1.4, `Microsoft.NET.Test.Sdk` 17.14.1, `coverlet.collector` 6.0.4 | Unit and integration test coverage for domain rules, CQRS handlers, financial calculation models, and gate condition validations. | ### 2.2 Frontend Projects (React 19 / TypeScript) @@ -120,6 +122,22 @@ PostFundManagement/ * **Authentication / STS**: Duende IdentityServer / Skoruba Administration (`https://sts.security.khongisa.co.za`). * **Observability**: OpenTelemetry exporters streaming traces, metrics, and logs to Aspire Dashboard / OTLP collectors. +### 2.4 Distributed Background Processing Architecture (Quartz.NET & PostFundManagement.Jobs) +* **Dedicated Execution Daemon**: `PostFundManagement.Jobs` is a standalone background host responsible for recurring job scheduling, lifecycle sweeps, and asynchronous event execution. +* **API Isolation (Scheduler Client)**: `PostFundManagement.Api` is strictly configured as a scheduler client via `AddQuartzSchedulerClient(...)` using `ZeroSizeThreadPool` (0 worker threads) and running in standby (`SchedulerStatus.Created`). The API enqueues jobs and triggers into the clustered PostgreSQL database (`pfm-scheduler`) but never acquires triggers or executes jobs locally. +* **Worker Execution Engine**: `PostFundManagement.Jobs` configures full Quartz execution via `AddQuartzScheduler(...)` using `DefaultThreadPool` (`MaxConcurrency = 5`) and `AddQuartzHostedService(options => options.WaitForJobsToComplete = true)`. +* **Event-Driven Orchestration (`IEvent` & `MediatorJob`)**: + - The API acts as the primary source of one-time events triggered by user actions and lifecycle transitions (implementing `IEvent : INotification`). + - Events are enqueued via `IJobOrchestrator.SendAsync` or `ScheduleAsync` and persisted in `pfm-scheduler` as `MediatorJob` with serialized JSON payloads. + - When acquired by `PostFundManagement.Jobs`, `MediatorJob` deserializes the event payload, propagates OpenTelemetry distributed traces, and publishes the event via `IMediator.Publish(...)`. +* **Startup Recurring Registration**: On daemon startup, `JobsWorker : BackgroundService` initializes and registers recurring cron jobs across tenant portfolios. +* **Cascading One-Time Execution**: If a recurring job identifies complex follow-up items that cannot be processed in the recurring execution window, it dispatches decoupled one-time jobs (`jobOrchestrator.SendAsync`) across worker threads. +* **Core Job Workloads**: + 1. *Automated Project Monitoring & Reminders*: Sweeps milestones, checks contractual SLA deadlines, sets overdue flags (`FR-M07-009`), and sends reminder notices to Grantees/Partners. + 2. *Risk Monitoring & Automated Notifications*: Evaluates risk score creep ($\text{Likelihood} \times \text{Impact}$), flags tolerance breaches, and sends escalation alerts for overdue mitigation actions. + 3. *Email Notification Processing*: Batches and dispatches pending notification entries (`Platform.Email` / `NotificationPlatform.Email`) via MailKit/MimeKit SMTP and records delivery telemetry (`FR-M25-001`). + 4. *Future Jobs*: Seamlessly supports future scheduled workloads (e.g. drawdown reconciliations, compliance expiries, portfolio aggregations) simply by adding `IEvent` handlers. + --- ## 3. Master Business Rules (BR-001 to BR-010) @@ -848,7 +866,7 @@ The platform uses **Hashids** to obfuscate internal sequential database primary * **Implementation**: Managed by `HashService` (`PostFundManagement.Application/Services/Shared/HashService.cs`) using the `Hashids.net` (v1.7.0) library. * **Shared Configuration (`HasherSettings`)**: * **Salt**: Sourced from configuration (`HasherSettings:Salt` or `HASHER_SALT`), defaulting to: - `"TGlnaHQgQ2hhcm1zLCBNaWRyYW5kQm9va3MgbWFrZSBsb3RzIG9mIG1vbmV5IGFuZCBhcmUgYSBzdWNjZXNzZnVsIHZpcmFsIHN0b3J5IGluIFNvdXRoIEFmcmljYQ=="` + `"salt-random-string"` * **Min Hash Length**: `HasherSettings:MinHashLength` or `HASHER_MIN_HASH_LENGTH` (default: `11`). * **Core Service Methods**: * `HashEncodeLongId(long id)` / `HashEncodeIntId(int id)`: Encodes integers into obfuscated HashID strings (e.g. `1` $\to$ `kY5aB9m...`). diff --git a/Processes.md b/Processes.md index 7f1ffb1..856c03c 100644 --- a/Processes.md +++ b/Processes.md @@ -12,7 +12,7 @@ The PFM platform governs the lifecycle of grant, development, innovation, and im ┌────────────────────────────────────────────────────────────────────────────────────────┐ │ POST-FUND LIFECYCLE │ │ │ -│ [1. Award Intake] ──> [2. Contracting] ──> [3. Onboarding] │ +│ [1. Award Intake] ──> [2. Contracting] ──> [3. Onboarding] │ │ External Approval Draft Agreement KYC & Banking Gate │ │ Recorded (BR-001) Auto-created (BR-002) Required for Disb. (BR-003) │ │ │ │ @@ -88,6 +88,16 @@ The PFM platform governs the lifecycle of grant, development, innovation, and im 4. Resolution: `Approved & Enacted` or `Rejected`. 5. **Maker-Checker Rule**: Change request initiator cannot approve or enact the change. +### 2.7 Process 7: Automated Background Workflows & Scheduled Jobs (PostFundManagement.Jobs) +* **Pre-condition**: Background jobs host (`PostFundManagement.Jobs`) active and connected to `pfm-scheduler`. +* **Actors**: System Engine, `JobsWorker` Daemon, Domain Event Handlers. +* **Workflow**: + 1. **Event Triggering**: API operations emit strongly typed events (`IEvent`) and enqueue them into `pfm-scheduler` via `IJobOrchestrator`. + 2. **Cron Schedule Execution**: `JobsWorker` initializes recurring schedules on daemon startup for periodic monitoring, risk assessments, and communication batches. + 3. **Job Execution**: Quartz cluster dispatches acquired triggers to `MediatorJob` within `PostFundManagement.Jobs` (concurrency limit = 5). + 4. **Mediated Processing**: Event payload is deserialized and published via `IMediator` to execute domain logic (project monitoring, risk threshold checks, email dispatch). + 5. **Cascading One-Time Jobs**: When a recurring job identifies an action item requiring separate asynchronous execution, it triggers a decoupled one-time job (`jobOrchestrator.SendAsync`). + --- ## 3. The Maker-Checker Operational Policy @@ -139,20 +149,20 @@ To ensure that caching never impedes record lifecycle updates or displays stale sequenceDiagram autonumber actor User as User / Maker - participant UI as ConsoleUI (React Query) + participant UI as ConsoleUI React Query participant Client as Axios HttpClient - participant OutputCache as Backend OutputCache (Redis/Memory) + participant OutputCache as Backend OutputCache Redis/Memory participant DB as PostgreSQL Database - User->>UI: Submits Record Mutation (Create / Update / Approve) + User->>UI: Submits Record Mutation Create/Update/Approve UI->>Client: Dispatches POST / PUT / PATCH / DELETE Client->>DB: Executes Mutation in Database DB-->>Client: Returns 200 OK / 201 Created Client->>Client: Interceptor detects mutation: bumps activeMutationEpoch - UI->>UI: onSuccess triggers queryClient.invalidateQueries() + UI->>UI: onSuccess triggers invalidateQueries UI->>Client: Refetches active queries with _v = activeMutationEpoch - Client->>OutputCache: GET /v1/api/*?_v= - Note over OutputCache: Cache Miss (new _v key never seen before) + Client->>OutputCache: GET /v1/api/* with new _v epoch + Note over OutputCache: Cache Miss: new _v key never seen before OutputCache->>DB: Queries PostgreSQL for fresh data DB-->>OutputCache: Returns updated entities OutputCache-->>UI: Caches fresh data under _v and returns to UI @@ -164,4 +174,74 @@ sequenceDiagram 2. **Deterministic Invalidation**: Calling standard `queryClient.invalidateQueries({ queryKey: [...] })` or `invalidateAndBustCache(queryClient, filters)` guarantees that the subsequent refetch bypasses stale ASP.NET Core OutputCache entries. 3. **Optimized Read Cache**: When users navigate across unchanged views or switch tabs, identical `_v` tokens are reused, allowing backend OutputCache and React Query in-memory caches to fulfill requests in < 5ms without unnecessary database hits. +--- + +## 6. Background Job Lifecycle & Event Execution Workflows + +The dedicated worker host (`PostFundManagement.Jobs`) operates as a decoupled daemon executing recurring maintenance routines and asynchronous domain events dispatched from the API or internally from other jobs. + +### 6.1 Event Ingestion & Job Persistence Flow + +```mermaid +flowchart TD + subgraph Producer ["PostFundManagement.Api (Event Source)"] + A["User Action / Lifecycle Transition"] --> B["Domain Command / Mutation"] + B --> C["Instantiate IEvent (e.g. SyncUserEvent, ProjectReminderEvent)"] + C --> D["IJobOrchestrator.SendAsync() or ScheduleAsync()"] + end + + subgraph Store ["PostgreSQL Clustered Store (pfm-scheduler)"] + D --> E[("QRTZ_JOB_DETAILS and QRTZ_TRIGGERS")] + end + + subgraph Consumer ["PostFundManagement.Jobs (Execution Host)"] + E -->|"Acquire Trigger Lock"| F["Quartz Scheduler Engine (MaxConcurrency = 5)"] + F --> G["MediatorJob[TNotification]"] + G --> H["Deserialize JSON Payload from JobDataMap"] + H --> I["IMediator.Publish(notification)"] + I --> J["Domain Event Handler: Business Logic in pfm DB"] + J -.->|"Cascading Sub-Task Needed"| K["IJobOrchestrator.SendAsync: Dispatch One-Time Job"] + K -.-> E + end +``` + +### 6.2 Standard Background Processes + +#### 1. Automated Project Monitoring & Reminders Process +* **Trigger**: Scheduled cron job registered on `JobsWorker` startup. +* **Operational Cycle**: + 1. Job queries active projects and non-completed milestones across tenant partitions. + 2. Compares milestone contractual due dates against current UTC time: + - **T - 14 / T - 7 / T - 1 Days**: Emits reminder notification events to Grantee/Partner project managers. + - **T + 0 Days (Overdue)**: Automatically flags the milestone as `Overdue` (`FR-M07-009`) and updates project RAG status to `At Risk` (`FR-M04-017`). + 3. Escalates overdue notifications to internal Programme Managers for intervention. + +#### 2. Risk Monitoring & Automated Escalation Process +* **Trigger**: Periodic recurring schedule. +* **Operational Cycle**: + 1. Queries all active project risk registers and Key Risk Indicators (KRIs) (`FR-M09-001` - `011`). + 2. Evaluates current severity score ($\text{Risk Score} = \text{Likelihood} \times \text{Impact}$). + 3. If risk score breaches the configured programme tolerance threshold: + - Automatically logs a risk escalation event. + - Dispatches alerts to Risk Managers and Portfolio Executives. + 4. Sweeps assigned risk mitigation actions; identifies overdue mitigation plans and emits reminders to responsible assignees. + +#### 3. Email Notification Processing Workflow (`Platform.Email`) +* **Trigger**: High-frequency recurring polling job or deferred one-time dispatch. +* **Operational Cycle**: + 1. Queries pending notifications from the transactional database where `Platform == NotificationPlatform.Email` and `Status == NotificationStatus.Pending` or `Queued`. + 2. Resolves tenant context using `HeadlessTenantContextService`. + 3. Formats email templates with recipient credentials and branding. + 4. Transmits messages via MailKit/MimeKit SMTP engine with rate-limiting and connection pooling. + 5. On successful delivery: Updates record status to `Sent`, stamps `DeliveredAt` timestamp, and logs dispatch telemetry (`FR-M25-001`). + 6. On failure: Increments retry counter; if retry limit exceeded, marks as `Failed` with diagnostic error trace. + +### 6.3 Recurring-to-One-Time Job Cascading Pattern + +When a recurring job evaluates a broad batch of records (e.g., scanning thousands of project indicators or verifying compliance expirations), it must not execute long-running operations synchronously in the recurring job thread: +1. The recurring job identifies items requiring heavy processing (e.g. generating an executive PDF digest or calling external webhooks). +2. For each identified item, it enqueues an isolated one-time job via `IJobOrchestrator.SendAsync(specificEvent)`. +3. This releases the recurring job quickly, distributes execution across the 5 available worker threads in `PostFundManagement.Jobs`, and guarantees that a single failed sub-task can be retried independently without stalling the entire recurring cycle. + + diff --git a/README.md b/README.md index e8e88d9..bf112f1 100644 --- a/README.md +++ b/README.md @@ -41,6 +41,16 @@ The application follows a modular, hierarchical architecture where high-level fu ### **5. Notifications & Platform Services** * Multi-channel delivery engine (`NotificationPlatform`) tracking candidate invitations, reminders, and status updates (`NotificationStatus`) across Email, SMS, WhatsApp, In-App, and Webhooks. +### **6. Background Processing & Distributed Jobs Engine** +* **`PostFundManagement.Jobs`**: Dedicated background execution host powered by **Quartz.NET (v4.6+)** with a PostgreSQL clustered JobStore (`pfm-scheduler`). +* **Decoupled Architecture**: `PostFundManagement.Api` serves strictly as a scheduler client (`ZeroSizeThreadPool`), dispatching events into the cluster while leaving execution exclusively to `PostFundManagement.Jobs` (`MaxConcurrency = 5`). +* **Event-Driven Execution**: Domain events implementing `IEvent : INotification` are mediated and serialized into `MediatorJob`. +* **Core Automated Operations**: + - **Automated Project Monitoring & Reminders**: Tracks milestones, evaluates planned vs actual deadlines, flags overdue statuses, and emits reminders to Grantees/Partners. + - **Risk Monitoring & Automated Notifications**: Evaluates risk registers, tracks Key Risk Indicators (KRIs), flags score creep, and escalates overdue mitigation actions. + - **Email Processing**: Asynchronously processes `NotificationPlatform.Email` records via MailKit/MimeKit SMTP. + - **Recurring & Cascading Jobs**: `JobsWorker` registers recurring cron schedules on startup and can trigger isolated one-time jobs when sub-activities cannot be executed directly by another job. + --- ## 🧩 Key System Enums @@ -68,6 +78,7 @@ The application follows a modular, hierarchical architecture where high-level fu * **Object Storage**: S3-compatible storage (Garage S3 / AWS S3) * **Caching & OutputCache Invalidation**: Redis StackExchange + ASP.NET Core `OutputCaching` (2-minute base policy) coupled with TanStack Query v5 and Axios client-side mutation versioning (`_v` parameter) for instantaneous reads and guaranteed zero-staleness on mutations * **Telemetry**: OpenTelemetry Metrics, Traces & Logs with Aspire / OTLP +* **Distributed Background Jobs**: Quartz.NET 4.6+ with clustered PostgreSQL JobStore (`pfm-scheduler`), separating the API (Scheduler Client with `ZeroSizeThreadPool`) from `PostFundManagement.Jobs` (Worker Execution Daemon with `MaxConcurrency = 5`) --- @@ -105,16 +116,18 @@ For in-depth architectural and operational specifications, see: ```text PostFundManagement/ - ├── PostFundManagement.Domain/ # Core Domain Entities, Enums & Interfaces - ├── PostFundManagement.Infrastructure/ # EF Core 10, PostgreSQL, Skoruba STS, Migrations - ├── PostFundManagement.Application/ # CQRS Handlers, DTOs, Business Rules & Validations - ├── PostFundManagement.Api/ # REST Endpoints, Middleware, Auth & Swagger/Scalar - ├── PostFundManagement.Tests/ # Unit & Integration Test Suite - ├── PostFundManagement.ConsoleUI/ # Internal Staff & Admin Web Portal (React 19 + Vite) - ├── PostFundManagement.PartnerUI/ # External Grantee / Recipient Portal (React 19 + Vite) - ├── Architecture.md # System Architecture, RBAC & Maker-Checker Spec - ├── Processes.md # Operational Grant Lifecycle & Process Flows - └── docker-compose.yml # Container orchestration + ├── PostFundManagement.Domain/ # Core Domain Entities, Enums, Value Objects & Abstractions (IEvent) + ├── PostFundManagement.Infrastructure/ # EF Core 10, PostgreSQL (Npgsql), Migrations, Data Protection + ├── PostFundManagement.Application/ # CQRS Handlers, Mediator, DTOs, Validations, S3, Email, Job Orchestrator + ├── PostFundManagement.Api/ # ASP.NET Core Minimal APIs / Carter, Auth (OIDC), Swagger/Scalar, Versioning + ├── PostFundManagement.Jobs/ # Dedicated Quartz.NET Background Worker Host (Recurring & Event-Driven Jobs) + ├── PostFundManagement.Tests/ # Unit & Integration Test Suite (xUnit, Coverlet) + ├── PostFundManagement.ConsoleUI/ # Internal Staff & Admin Web Portal (React 19 + Vite + Tailwind 4) + ├── PostFundManagement.PartnerUI/ # External Grantee / Recipient Portal (React 19 + Vite + WebGL OGL) + ├── Architecture.md # System Architecture, Security, RBAC, Caching & Quartz Spec + ├── Processes.md # Operational Grant Lifecycle, Maker-Checker & Background Workflows + ├── GEMINI.md # Master Technical & Architecture Roadmap (SRS v2.0 Alignment) + └── docker-compose.yml # Container orchestration for API and Frontends ``` ## 🚀 Getting Started