Files
Carly de FrondevilleandGitHub 41993fb63f Reachability optimization: add caching for visibility (#5816)
## What changed?
Add an LRU cache that reachability calls before reaching out to
visibility. Separate cache for open and closed workflow executions so
that there can be separate TTL. TTL configurable via dynamic config.
Include cache hit/miss in `reachabilityExitPoint` so that it can be
logged and tracked in prometheus, making there be 6
`reachabilityExitPoint` options.

## Why?
To reduce load on visibility.

## How did you test it?
Tested with unit tests and existing functional tests.
Added additional functional test that exercises the reachability cache
by doing:
1. Add assignment rule `A` and start workflow with build id `A`
2. Wait for `A` to show up as `running` in visibility db
3. Commit a different build id --> `A` should now only be reachable via
visibility query
4. Query `reachability(A)` --> `reachable` by visibility db, populating
reachability open WF cache with `A: true`
5. Signal workflow with build id `A` so that it completes
6. Wait for `A` to show up as `closed` in visibility db (cache TTL must
be longer than this wait time)
7. Query `reachability(A)` --> `reachable` by reachability cache
8. Wait for `testReachabilityCacheOpenWFsTTL` to pass
9. Query `reachability(A)` --> `closed_only` by visibility db,
populating reachability closed WF cache with `A: true`

## Potential risks
Worst case scenario would be that reachability returns a false-negative.
Due to cache TTL there is the possibility of a false-positive, but
eventually the false positive will resolve due to TTL

## Documentation
Make sure to add the false-positive caveat to docs.

## Is hotfix candidate?
No
2024-05-03 18:35:09 -07:00
..