This commit consolidates extensive refactoring across the engine, orchestrators, background tasks, JSON pipelines, caching layers, messaging, and infrastructure. Major improvements include async correctness, cancellation propagation, architectural restructuring, performance optimizations, parser and engine enhancements, UI updates, deployment improvements, and critical bug fixes. Asynchronous, Cancellation, and Concurrency Improvements: * Converted all repository and controller methods to fully asynchronous implementations; removed remaining synchronous database calls. * Introduced centralized cancellation via a `CancellationTokenProvider` in DI and propagated tokens through database, engine, cache, and background processes. * Implemented graceful SIGTERM handling with coordinated shutdown. Most tasks are subject to immediate cancellation, while: - Draining AMQP consumers - Draining Kestrel in-flight requests - Draining async Archive writes - Draining async Case Creation processing * Added middleware to track active requests and emit `Connection: close` during shutdown. * Enhanced exception handling: - Cancellation exceptions logged as INFO - Improved logging for async call failures * Added `/ready` endpoint: - Returns 200 if the engine is ready, 503 otherwise - Integrated readiness signaling from TaskController and ActivationWatcher * Fixed concurrency/timing issues, including LRU cache updates using a semaphore instead of locks. * Ensured proper propagation of cancellation tokens across async call stacks, except where drainage is required. Engine, Orchestrator, and Pipeline Refactoring: * Rebuilt Invoke Orchestrator using a Context–Strategy–Pipeline hybrid pattern. * Extracted JSON parsing and database record parsing into dedicated helper classes. * Standardized instance object state management using a centralized Context object; removed private mutable state from orchestrators and engine. * Reorganized namespaces and moved background tasks to dedicated classes. * Removed god-class, organic-growth code and streamlined dependency flow; dependencies are now passed via Context only. * Added reference date integrity checks: - Prevent processing of future `ReferenceDate` values to prevent purge of records through bad data. - Updating of reference date ignored for records older than the latest entry to prevent invalid purges. * Introduced nested classes for large models and properties to improve readability. Moved very large property sets to nested classes for better context in engine pipelines. * Implemented rich extension functions library for parser and rule coder functionality: - Parser Scans new extensions assembly to add parser security tokens, enabling centralized development of extension functions to enhance rule capabilities. - Documented new tokens under Rule Compilation Tokens and Extensions. * Simplified `Invoke` data extraction logic via helper extension methods. * Centralized Context creation for JSON settings and contract resolvers. * Dropped legacy Response Payload approach, pending improved serialization to support pruning response payload. * Renamed `Report Keys` to `Archive Keys` for consistency with POCOs and database entities. * Reorganized engine files and namespaces to reflect new software architecture. * Implemented a bytes limit for payloads received over the invoke controller, controlled via the `MaxInvokeControllerRequestBytes` environment variable. Cache, Redis, TTL, and Background Task Enhancements: * Replaced Postgres `NOTIFY` in callbacks with Redis-based signaling given Postgres should not be interacted with during real-time flows — only Redis. * Improved callback timeout/removal logic and reduced initial invocation payload sizes. * Added environment variables for SearchKey cache job interval control: `SearchKeyCacheServerIntervalType` and `SearchKeyCacheServerIntervalValue`. * Refactored cache purge with observability tables and supporting migrations to enhance confidence in high-risk processing. * Converted TTL counters to `long` type with corresponding storage updates in the `ArchiveKeys` table. * Ensured only one TTL increment per invocation, irrespective of activation rules. * Updated Redis background scans from `HGETALL` to incremental `HSCAN` to prevent deadlocks and improve performance. - Removed obsolete payload-first cache tracking. * Improved initial LRU cache fill visibility via TaskController integration. * Refactored repository access in the Cache Service and audited DbContext disposal to prevent connection leaks. * Fixed counter metrics in the LRU cache. * Batched SortedSetRemoveAsync and HashDeleteAsync as it was causing long operations in Redis. AMQP, Threading, and Background Workflow Improvements: * Refactored AMQP messaging to resolve deadlocks in async consumers. * Moved RabbitMQ channel creation to consumer tasks and improved heartbeat handling to avoid deadlocks given `IModel` is not thread-safe. * Offloaded expensive case-creation processing to scalable background threads: - Reduces pressure on main transaction threads - Eliminates blocking in Archive storage and reduces memory leak risk (backlogged case creation has a much smaller footprint than backlogged Archive payload) - Threads controlled via the `CaseCreationThreads` environment variable * Introduced `TaskController` to manage all background (non-transactional) tasks. TaskController allows visibility of all background tasks. * Centralized cancellation and graceful shutdown across engine, background tasks, and web host. * On SIGTERM, if a task is not terminated within two seconds — perhaps due to drainage — it is written to console to give feedback on blocking tasks. * Implemented heartbeats in the connection to handle brittle connections and unforgiving .NET RabbitMQ clients, controlled by the `AMQPHeartbeatSeconds` environment variable. JSON, Parser, and Serialization Fixes: * Updated JSON settings, contract resolvers, and DI wiring. * Reduced payload response size on async HTTP invocation to only return the EntityAnalysisModelInstanceEntry GUID for callback, notwithstanding head check and parse still taking place. * Added `DBNull` checks for database led reprocessing deserialization. * Ensured the DictionaryNoBoxing converter is properly added to serialization settings to parse dates using the round-trip format. Performance, Stability, and General Cleanup: * Updated PooledDictionary to fix hash collisions and improve initial sizing. * Return null from PooledDictionary.cs instead of exception. * Ensured DB connections are disposed properly during model sync to prevent pool starvation in the new patterns. * Refactored SMTP client usage and added authentication/connection validations. * Removed unnecessary AutoMapper list mappings and streamlined repository configurations. * Renamed illogical methods and variables for clarity, as came across them. * Applied `ConfigureAwait` consistently across all projects. * Added `Async` suffix to async method names. * Replaced async unsupported `out` variables with tuples UI, Infrastructure, and Deployment Updates: * Updated UI to show case-creation, missing tagging and notification queue balance counters. * Removed obsolete test Blazor files (this project will go Angular). * Added production GC environment variables to `docker-compose.yml`. * Added `Jube.HealthCheck.csproj` for lightweight HTTP health checks in container (avoids installing `curl` as root). * Updated `docker-compose.yml` to bind port `5001` for job process readiness checks, as was already the case with the other process. * Implemented frame-busting on the login page to prevent the authentication page from being loaded in a frame after expiration. Critical Bug Fixes: * Fixed transaction buffer flushes on commit. * Fixed bulk write attempts on empty buffers. Miscellaneous Improvements: * Moved engine and model manager into a unified hybrid context with strategy & pipeline patterns. * Shifted thread pool configuration to startup pipeline. * Reorganized files and namespaces to align with the new architecture. * Improved observability and versioning for `Archive` and `ArchiveKey` tables to support reprocessing. * Dropped unused/complex payload-first cache tracking. * Many additional small bug fixes and refinements discovered during the grand refactor. * The Sandpit had some telemetry to a Chatbot and Google Analytics. Although it was disabled by default — and the Sandpit was intended for Jube use only — it should not have been in the public build. * Updated UI to display case-creation counters and missing tagging/notification fields. * Added a filter to remove instances from the synchronization page that have not sent a heartbeat. * Fixed invoke counter and removed the condition that counters are only written on invoke for more consistent system observability. * Updated the parser to support dates in inline functions, which was previously broken, which meant that dates not available. * Improved thread pool startup parameters to be a factor of two times logical cores for worker threads and four time logical cores for io threads. * Set command timout to zero for archive persistence given sensitivity. In time will create a dead letter archive for total failure, as on failure the records dropped. **Notes:** Functionality remains substantially the same where intended; most changes focus on stability, maintainability, async correctness, and operational robustness.
13 KiB
layout, title, nav_order, parent, grand_parent
| layout | title | nav_order | parent | grand_parent |
|---|---|---|---|---|
| default | HTTP API Concepts | 1 | API | Concepts |
🚀Speed up implementation with hands-on, face-to-face training from the developer.
HTTP API Concepts
There exists an embedded Swagger \ OpenAPI documentation available at the following location:
https://localhost:5001/swagger
Which exposes the following landing page:
Swagger is the central resource for API documentation. The exception to self documentation is the /Invoke/EntityAnalysisModel endpoint which does not map to an object directly, with that request body - which also needs to be JSON - being parsed manually given model definitions (i.e. Request XPath). It follows that except the /Invoke/EntityAnalysisModel, which is documented extensively elsewhere, API endpoints benefit from strong typing.
The use of Swagger is outside the scope of this documentation, and it may be noted that an alternative HTTP request tool has been used throughout this documentation, however it is sufficed to say that in addition to the documentation of the API, there is also the functionality in Swagger to invoke any part of the API directly given JWT authentication token.
For example, the following API seeks to return Case with ID 1:
In the above example, it can be seen that the API parameter is accompanied by a JSON Web Token, Bearer token, without which a 401 http status code would be returned.
Where a JSON Web Token, Bearer Token, is required, Swagger will display required parameter.
The api/Authentication/ByUserNamePassword endpoint is available for the validation of username and password returning a JSON Web Token in the response body:
As is standard, the HTTP header (which must be there on request, but is also echoed in the response) is named authentication with the value being the string "Bearer " concatenated with the JSON Web Token. The HTTP cookie meanwhile is called "authentication" and contains only the JSON Web Token, although it is unlikely that an API led integration would choose to store or use a HTTP cookie, being more likely reserved for user interface page requests.
The JSON web token is encrypted using Environment Variable values:
JWTValidAudience=http://localhost:5001
JWTValidIssuer=http://localhost:5001
JWTKey=ExtraSuperDuperSecretKeyCreatedOnFirstStartup
| Value | Description |
|---|---|
| JWTValidAudience | The server domain that is allowed the token. |
| JWTValidIssuer | The server domain that issued the token. |
| JWTKey | The encryption key for the JWT. This key is created randomly on first startup of a Jube instance executable and stored in the Jube.environment file, but can be changed. |
Notwithstanding the stateless architecture, the token contains a single claim type of Name (http://schemas.xmlsoap.org/ws/2005/05/identity/claims/name) which will contain the username (e.g. Administrator). It follows that in knowing the key values for the JSON Web Token, authentication can take place entirely out of band of the application.
The following HTTP status codes are implemented by the API:
| Status Code | Implementation |
|---|---|
| 401 | Not authenticated. JSON Web Token is either missing or expired. |
| 403 | Not authorised. The authenticated user does not have access to the resource as configured in the permissions functionality. |
| 400 | Bad Request. The request has not passed validation. Error messages will be returned indicating the source of the failure in the JSON response body |
| 204 | Key Not Found. In the case of PUT or DELETE methods, the data was not found for modification. This is often an indication of tenancy problems, where an attempt has been made to update data in a tenant which has not been allocated to the user. |
| 500 | Internal Error. An error fatal to the request. The error messages will only be written to logs server side and not communicated to the end user |
| 404 | Not Found. The resource has not been found. There are very few programmatic uses of Not Found, except in the case of the Invoke APIs, hence the first step should be to check the URL. |
| 408 | Timeout. The resource has not been found in a timeout threshold. In the case of the Invoke APIs callback on asynchronous processing. |
Resources are almost entirely subject to JSON Web Token authentication and permission authorisation except endpoints that are intended for the purpose of gaining authentication or integration. Where an endpoint is outside of JSON Web Token Authentication, there may however be other configuration options available for security:
| Status Code | Implementation | Other Security |
|---|---|---|
| /api/Authentication/ByUserNamePassword | An endpoint that exists to validate a user name and password, returning a JSON Web Token in the response body, the authentication header and placing the same in a cookie called "authenticate". | None |
| /api/Invoke/Sanction | An endpoint for the purposes of integration to perform sanctions checking. | Environment variable EnablePublicInvokeController must be set to True |
| /api/Invoke/Archive/Tag | An endpoint for the purpose of adding tags to events stored in Archive. | Environment variable EnablePublicInvokeController must be set to True |
| /api/Invoke/EntityAnalysisModel/{guid} | An endpoint for the invocation of models. | Environment variable EnablePublicInvokeController must be set to True |
| /api/Invoke/ExhaustiveSearchInstance/{guid} | An endpoint for the invocation of exhaustive machine learning models beyond that available in model invocation. | Environment variable EnablePublicInvokeController must be set to True |
| /api/Invoke/ExampleFraudScoreLocalEndpoint | A mocking endpoint to echo back a score for the purpose of testing. | Environment variable EnablePublicInvokeController must be set to True |
| /api/Invoke/EntityAnalysisModel/Callback/{guid} | An endpoint for the callback of the payload response given asynchronous model invocation. | Environment variable EnablePublicInvokeController must be set to True |
The user interface is entirely Javascript set out on pages that are entirely static (save for some very minimal code behind authorisation). Given that the user interface is Javascript there must therefore be some form of API available to support the functions. Every function that can be enacted in the User Interface is available via the API. API which is prefixed with Invoke are not intended for use on an authenticated basis and are instead for the purpose of integration.
HTTP method verbs are implemented as customary:
| Verb | Implementation |
|---|---|
| GET | HTTP requests to fetch data without a request body with parameterization in the URL stub or querystring. |
| POST | HTTP requests with a JSON request body for creation. |
| PUT | HTTP requests with a JSON request body for update. |
| DELETE | HTTP requests to delete data without a request body with parameterization in the URL stub or querystring. |
HTTP request body is in all instances JSON, with media type header of application/json, as following example, for PUT verb, targeting api/Case:
{
"id": 0,
"diaryDate": "2022-10-18T07:09:16.686Z",
"caseWorkflowStatusId": 0,
"createdDate": "2022-10-18T07:09:16.686Z",
"locked": true,
"lockedUser": "string",
"lockedDate": "2022-10-18T07:09:16.686Z",
"closedStatusId": 0,
"closedDate": "2022-10-18T07:09:16.686Z",
"closedUser": "string",
"caseKey": "string",
"diary": true,
"diaryUser": "string",
"rating": 0,
"json": "string",
"caseKeyValue": "string",
"lastClosedStatus": 0,
"payload": "string"
}
With response example as:
{
"id": 0,
"diaryDate": "2022-10-18T07:10:46.761Z",
"caseWorkflowStatusId": 0,
"createdDate": "2022-10-18T07:10:46.762Z",
"locked": true,
"lockedUser": "string",
"lockedDate": "2022-10-18T07:10:46.762Z",
"closedStatusId": 0,
"closedDate": "2022-10-18T07:10:46.762Z",
"closedUser": "string",
"caseKey": "string",
"diary": true,
"diaryUser": "string",
"rating": 0,
"json": "string",
"caseKeyValue": "string",
"lastClosedStatus": 0,
"payload": "string"
}
In all cases, examples are available in Swagger given the strong typing of the controllers scaffolding the endpoints:
All requests with the exception of Invoke endpoints are subject to validation. In the event validation fails, a Bad Request HTTP status is served, including details of the validation failure, as follows for PUT verb, to api/case:
{
"isValid": false,
"errors": [
{
"propertyName": "Rating",
"errorMessage": "'Rating' must be greater than '0'.",
"attemptedValue": 0,
"customState": null,
"severity": 0,
"errorCode": "GreaterThanValidator",
"formattedMessagePlaceholderValues": {
"ComparisonValue": 0,
"ComparisonProperty": "",
"PropertyName": "Rating",
"PropertyValue": 0
}
}
],
"ruleSetsExecuted": [
"default"
]
}
In the above example validation has failed owing to the rating not being of a correct value.