richard-churchman
5f6bbe45f7
Major bug fixes relating to the Redis implementation in Reprocessing and Search Key Cache were identified and fixed as part of the rekeying development. Included functionality to stop the Exhaustive training. Fixed the Unsupervised learning functionality in Exhaustive Training. Embedded the Mock Data CSV data into a class to make it compatible with Docker without needing to modify the Dockerfile (i.e, build). Included a new table that stores training data to enhance support capabilities for Exhaustive. Reintroduced and validated the Multitenancy functionality. Postgres is now fully deprecated in place of Redis for cache, and all implementations of the Interfaces for Postgres have been removed. No migrations exist to support this deprecation, and Postgres as a cache is now wholly unsupported. Removed the Indexing background thread, as Redis does not require it, given that it is a key-value pair database. Created a RedisCommandFlag Environment Variable to support Fire and Forget write to Redis in Invocation. Implemented functionality to Export, Peek and Download configuration environments using a secure, encrypted scheme. Created a permission to access the functionality via the user interface. Extensive database-based logging is included to provide for a comprehensive audit. Created functionality to Import the encrypted Export file inside a database transaction to ensure consistency. As Docker is now the primary deployment model, log4net instantiation has been moved to an approach that primarily uses programmatic instantiation of the ConsoleLogger and RollingFileAppender given Environment Variables. Implemented functionality to support remote \ mounted Log4Net XML Serialisation files for more advanced log4net configurations. Reintroduced access to counters via the user interface and created a permission. Took the opportunity to implement ReSharper linter suggestions. Disabled the Lock button across the whole application, with that functionality being reserved for future use. It is intended that locking happens programmatically on the basis of dependencies in future commits. Fixed several significant end-use bugs which became apparent during testing of the refactored code relating to case management and visualisations. Fixed certain other minor bugs that came up during testing. Brought documentation up to date for this commit. Brought the Docker Compose file up to date for this commit. Removed Redundant Environment Variables for this commit. Made the README file less verbose and unnecessary.
3.2 KiB
layout, title, nav_order, parent, grand_parent
| layout | title | nav_order | parent | grand_parent |
|---|---|---|---|---|
| default | Reprocessing Eligibility | 1 | Reprocessing | Configuration |
🚀Speed up implementation with hands-on, face-to-face training from the developer.
Reprocessing Eligibility
Reprocessing allows for the contents of the Archive table to be reprocessed one by one retroactively, which is useful if a new rule has been created and some historic management is required.
It should be noted that reprocess is computationally expensive and should not be used in place of a forward only transaction or event methodology. A common use case is to facilitate the reprocessing of an extremely targeted specification, such as a particular high risk client or customer subset. It should also be noted that reprocessing takes it data from the archive and will upset the value to the cache. The upsert step puts some demand on the cache but also the extent to which the cache may grow should be considered if data is being processed outside of a time frame know to otherwise have good coverage in the cache.
There are a number of ways to remove load from the reprocessing task.
The steps for reprocessing are firstly, setting an Activation Rule to be eligible for reprocessing. Navigate to the Activation Rule Volume1DayUSDForIPOver100:
Scroll down the Activation Rule and locate the Reprocessing switch:
Ensure that the Enable Reprocessing is switched to true for this Activation Rule to be eligible for reprocessing.
Reprocessing jobs are responsible for fetching data from the Archive table, filtering and performing model invocation for each event returned:
When reprocessing takes place, model invocation is almost exactly the same as if it were being called real-time, and on a similar one by one basis in order of transaction processing date. However, when it comes to executing Activation Rules as part of model invocation, only rules that have been marked as Reprocessing are evaluated.
To set a rule to be eligible for reprocessing, it is a simple matter of updating the Reprocessing check-box in the Activation Rule. Scroll down to enable reprocessing by setting the switch to true:
Scroll down and update the Activation Rule to create a version:
It is important that the model is synchronised, as reprocessing relied upon the Activation Rule configurations, henceforth, navigate to Models >> Synchronisation and enact.
While the transaction is sourced from the Archive, it is important to note that the transaction is replayed through the invoke engine on the same basis as a real-time transaction. It follows that care must be taken to ensure that time frames of eligibility in the Cache exceed that filtered from the Archive. The availability of cache data must be sufficient to provide coverage for not only the transaction, but crucially the maximum range required of abstraction rules. In the absence of the transaction existing in the cache, it will be upserted, but in such a scenario abstraction rules would assume a standing start, and may not be wholly accurate until the Archive transaction cursor moves past a date range ensuring coverage.