Updated documentation
This commit is contained in:
+131
-5
@@ -22,12 +22,13 @@ The PFM platform is constructed using **Clean Architecture** and **Domain-Driven
|
||||
│ API GATEWAY │
|
||||
│ PostFundManagement.Api (ASP.NET Core Minimal APIs / Carter) │
|
||||
│ Authentication: JWT Bearer + OpenTelemetry Observability │
|
||||
│ Quartz Client: Scheduler-Only (ZeroSizeThreadPool / Enqueues Jobs) │
|
||||
└───────────────────────────────────┬────────────────────────────────────┘
|
||||
│
|
||||
┌───────────────────────────────────▼────────────────────────────────────┐
|
||||
│ APPLICATION LAYER │
|
||||
│ PostFundManagement.Application │
|
||||
│ CQRS (Mediator), Business Validators, Domain Services, Quartz Jobs │
|
||||
│ CQRS (Mediator), Business Validators, Domain Services, Event Handlers │
|
||||
└──────────────────┬─────────────────────────────────┬───────────────────┘
|
||||
│ │
|
||||
┌──────────────────▼──────────────────┐ ┌──────────▼───────────────────┐
|
||||
@@ -35,8 +36,15 @@ The PFM platform is constructed using **Clean Architecture** and **Domain-Driven
|
||||
│ PostFundManagement.Domain │ │ PostFundManagement.Infrastr.│
|
||||
│ Entities, Value Objects, Enums, │ │ PostgreSQL (EF Core 10), │
|
||||
│ Domain Invariants, Audit Models │ │ Skoruba STS, S3 Garage, │
|
||||
└─────────────────────────────────────┘ │ Redis HybridCache │
|
||||
└──────────────────────────────┘
|
||||
└─────────────────────────────────────┘ │ Redis, Quartz (pfm-scheduler)│
|
||||
└──────────────┬───────────────┘
|
||||
│
|
||||
┌────────────────────────────────────────────────────────▼───────────────┐
|
||||
│ BACKGROUND JOBS WORKER HOST │
|
||||
│ PostFundManagement.Jobs (.NET 10 Background Host / Quartz Runner) │
|
||||
│ Recurring Job Registration (JobsWorker) & Mediated Execution Engine │
|
||||
│ Quartz Runner: Full Execution (DefaultThreadPool MaxConcurrency = 5) │
|
||||
└────────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
@@ -113,8 +121,8 @@ Even if an `Executive`, `Manager`, or `Supervisor` possesses the `update` claim
|
||||
flowchart LR
|
||||
A["Maker (User A)<br/>Creates Record"] --> B["Record Persisted<br/>with createdBy = User A"]
|
||||
B --> C{"Checker Evaluation<br/>(Approver Login)"}
|
||||
C -->|Checker == User A| D["Approval BLOCKED<br/>Maker-Checker Violation"]
|
||||
C -->|Checker == User B<br/>with 'update' claim| E["Approval GRANTED<br/>Status -> Approved"]
|
||||
C -->|"Checker is User A"| D["Approval BLOCKED: Maker-Checker Violation"]
|
||||
C -->|"Checker is User B with update claim"| E["Approval GRANTED: Status Approved"]
|
||||
```
|
||||
|
||||
### 4.2 Record Attribution Model
|
||||
@@ -189,4 +197,122 @@ Following the introduction of fine-grained RBAC and Maker-Checker checks, startu
|
||||
* **Immediate Non-Blocking Session Render**: If valid session tokens and user profiles exist in `localStorage`, `AuthProvider` immediately unblocks UI rendering (`isLoading: false`) and runs tenant lockout checks, account synchronization, and claim revalidation asynchronously in the background.
|
||||
* **In-Memory User Sync & Lockout Caching**: `isUserSyncedInBrowser` and `verifyTenantLockout` maintain in-memory references to eliminate synchronous `JSON.parse` operations from the request pipeline.
|
||||
|
||||
---
|
||||
|
||||
## 6. Distributed Background Job Processing Architecture (PostFundManagement.Jobs)
|
||||
|
||||
The PFM platform decouples synchronous API request handling from asynchronous, long-running, and scheduled workloads using a distributed **Quartz.NET (v4.6+)** clustering topology backed by PostgreSQL (`pfm-scheduler`).
|
||||
|
||||
### 6.1 Split-Node Topology: Scheduler Client vs. Execution Host
|
||||
|
||||
To maintain high API throughput, low memory footprint, and complete process isolation, PFM implements a split-node architecture between `PostFundManagement.Api` and `PostFundManagement.Jobs`:
|
||||
|
||||
```
|
||||
┌──────────────────────────────────────────────┐
|
||||
│ PostFundManagement.Api │
|
||||
│ (API Gateway / Scheduler Client) │
|
||||
│ │
|
||||
│ * AddQuartzSchedulerClient() │
|
||||
│ * ZeroSizeThreadPool (0 execution threads) │
|
||||
│ * SchedulerStatus.Created (Standby mode) │
|
||||
│ * Strictly schedules / enqueues events │
|
||||
│ * Never acquires triggers or executes jobs │
|
||||
└──────────────────────┬───────────────────────┘
|
||||
│ Schedules IEvent jobs via IJobOrchestrator
|
||||
▼
|
||||
┌──────────────────────────────────────────────┐
|
||||
│ PostgreSQL JobStore │
|
||||
│ (pfm-scheduler) │
|
||||
│ │
|
||||
│ * Clustered tables (QRTZ_TRIGGERS, etc.) │
|
||||
│ * System.Text.Json serialized IEvent payload│
|
||||
│ * Distributed locking & check-in heartbeat │
|
||||
└──────────────────────▲───────────────────────┘
|
||||
│
|
||||
│ Acquires triggers & executes (Polling loop)
|
||||
┌──────────────────────┴───────────────────────┐
|
||||
│ PostFundManagement.Jobs │
|
||||
│ (Dedicated Worker Service) │
|
||||
│ │
|
||||
│ * AddQuartzScheduler() │
|
||||
│ * DefaultThreadPool (MaxConcurrency = 5) │
|
||||
│ * SchedulerStatus.Running (Started loop) │
|
||||
│ * JobsWorker background service: │
|
||||
│ - Registers recurring cron jobs on start │
|
||||
│ - Enqueues one-time jobs when needed │
|
||||
│ * Executes MediatorJob<TNotification> │
|
||||
│ * Deserializes and publishes via IMediator │
|
||||
└──────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
| Characteristic | `PostFundManagement.Api` | `PostFundManagement.Jobs` |
|
||||
| :--- | :--- | :--- |
|
||||
| **Architectural Role** | API Gateway & Event Producer | Dedicated Execution Daemon & Consumer |
|
||||
| **Registration Extension** | `AddQuartzSchedulerClient(...)` | `AddQuartzScheduler(...)` |
|
||||
| **Thread Pool** | `ZeroSizeThreadPool` (0 threads) | `DefaultThreadPool` (`MaxConcurrency = 5`) |
|
||||
| **Hosted Service** | None (`AddQuartzHostedService` omitted) | `AddQuartzHostedService(WaitForJobsToComplete = true)` |
|
||||
| **Lifecycle State** | `SchedulerStatus.Created` (Standby) | `SchedulerStatus.Running` (Active execution loop) |
|
||||
| **Readiness Check** | Verifies DB connectivity without requiring `Running` | Verifies active execution engine & DB connectivity |
|
||||
| **Trigger Acquisition** | ❌ **Disabled** (Cannot acquire triggers or run jobs) | ✅ **Active** (Polls DB, acquires triggers, runs jobs) |
|
||||
|
||||
### 6.2 Event-Driven Mediated Orchestration (`IEvent` & `MediatorJob`)
|
||||
|
||||
Background tasks are modeled as strongly-typed domain events implementing `IEvent : INotification` (`PostFundManagement.Domain.Abstractions`).
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
autonumber
|
||||
participant API as PostFundManagement.Api
|
||||
participant Orchestrator as IJobOrchestrator
|
||||
participant DB as PostgreSQL pfm-scheduler
|
||||
participant Host as PostFundManagement.Jobs
|
||||
participant QuartzThread as Quartz Execution Engine
|
||||
participant Mediator as IMediator MediatorJob
|
||||
participant Handler as Domain Event Handlers
|
||||
|
||||
Note over API,Orchestrator: 1. Event Scheduling (API Node)
|
||||
API->>Orchestrator: SendAsync(notification) / ScheduleAsync(notification, cron)
|
||||
Orchestrator->>DB: Stores JobDetail and Trigger with serialized Payload JSON
|
||||
Note over DB,Host: 2. Distributed Acquisition (Jobs Node)
|
||||
QuartzThread->>DB: Polls and acquires due trigger with row-level locks
|
||||
QuartzThread->>Host: Dispatches to local thread pool
|
||||
Host->>Mediator: Instantiates MediatorJob[TNotification]
|
||||
Mediator->>Mediator: Deserializes payload from MergedJobDataMap
|
||||
Mediator->>Handler: Publish(notification, cancellationToken)
|
||||
Handler-->>Host: Execution complete and updates business data in pfm DB
|
||||
Host->>DB: Releases trigger and commits execution state
|
||||
```
|
||||
|
||||
1. **Scheduling via `IJobOrchestrator`**:
|
||||
- `SendAsync<TNotification>(notification)`: Schedules a durable, one-time execution in the `onetime-jobs` group with `StartNow()` and `DisallowConcurrentExecution()`.
|
||||
- `ScheduleAsync<TNotification>(notification, cronExpression)`: Configures a durable, cron-driven recurring job in the `scheduled-jobs` group with automatic misfire handling (`CronTriggerMisfireInstruction.IgnoreMisfires`).
|
||||
- `InterruptAsync(eventName, correlationId)`: Allows runtime cancellation of active job executions across the cluster.
|
||||
2. **Transparent Deserialization & Dispatch (`MediatorJob<TNotification>`)**:
|
||||
`MediatorJob<TNotification>` resolves `IMediator` from DI, deserializes the JSON payload, initiates OpenTelemetry distributed tracing (`MediatorTelemetry.Source`), and publishes the event to all registered in-process handlers.
|
||||
|
||||
### 6.3 Recurring vs. One-Time Job Patterns
|
||||
|
||||
* **Recurring Jobs**: Scheduled by `JobsWorker` (`BackgroundService` in `PostFundManagement.Jobs`) during application startup. These jobs govern periodic health sweeps, SLA tracking, notification batching, and risk evaluations on predictable cron intervals.
|
||||
* **One-Time Jobs**: Primarily scheduled by `PostFundManagement.Api` as immediate or deferred side-effects of HTTP requests and lifecycle mutations (e.g. user invites, award registration triggers). Furthermore, a running recurring job in `PostFundManagement.Jobs` can dispatch a one-time job (`jobOrchestrator.SendAsync`) when handling fan-out tasks or invoking decoupled activities that cannot be triggered directly by the originating job.
|
||||
|
||||
### 6.4 Core Background Processing Capabilities
|
||||
|
||||
The background job runner powers four essential operational workloads:
|
||||
|
||||
1. **Automated Project Monitoring & Reminders**:
|
||||
- Sweeps active projects, contractual deliverables, and milestone deadlines.
|
||||
- Computes planned vs. actual progress variance and escalates overdue milestones (`FR-M04-017`, `FR-M07-009`).
|
||||
- Dispatches automated reminders to Grantees/Partners prior to milestone due dates and submission cut-offs.
|
||||
2. **Risk Monitoring & Automated Notifications**:
|
||||
- Evaluates project risk registers, Key Risk Indicators (KRIs), and issue escalation thresholds (`FR-M09-001` - `011`).
|
||||
- Detects risk score creep (Likelihood $\times$ Impact) and triggers automated alerts when risk tolerance thresholds are breached.
|
||||
- Monitors overdue mitigation action plans and escalates unresolved high-severity risks to Programme Managers.
|
||||
3. **Email Notification Processing**:
|
||||
- Continuously consumes and batches pending notification records (`Platform.Email` / `NotificationPlatform.Email`).
|
||||
- Dispatches transactional emails via MailKit / MimeKit SMTP with exponential backoff and retry policies.
|
||||
- Updates delivery status (`Sent`, `Failed`, retry count) and logs dispatch telemetry for audit compliance (`FR-M25-001`).
|
||||
4. **Platform Extensibility**:
|
||||
- New background jobs and recurring routines can be introduced by creating an `IEvent` model in `PostFundManagement.Application` and scheduling it via `JobsWorker` or `IJobOrchestrator`, with zero impact on the API surface.
|
||||
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user