Dennis KniepandGitHub
011ce043f1
<!-- Describe what has changed in this PR --> **What changed?** Ability to configure a loglevel for AWS SDK, which is used during s3 archival. <!-- Tell your future self why have you made these changes --> **Why?** This is relevant if one want to troubleshoot s3 archival. <!-- How have you verified this change? Tested locally? Added a unit test? Checked in staging env? --> **How did you test it?** Tested locally <!-- Assuming the worst case, what can be broken when deploying this change to production? --> **Potential risks** No risk, because by default it is turned `off`, so no change for existing deployments <!-- Is this PR a hotfix candidate or require that a notification be sent to the broader community? (Yes/No) --> **Is hotfix candidate?** No
3.1 KiB
Amazon S3 blobstore
Configuration
See https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/configuring-sdk.html#specifying-credentials on how to set up authentication against s3
Enabling archival is done by using the configuration below. Region and bucket URI are required
archival:
history:
state: "enabled"
enableRead: true
provider:
s3store:
region: "us-east-1"
logLevel: 0
visibility:
state: "enabled"
enableRead: true
provider:
s3store:
region: "us-east-1"
logLevel: 0
namespaceDefaults:
archival:
history:
state: "enabled"
URI: "s3://<bucket-name>"
visibility:
state: "enabled"
URI: "s3://<bucket-name>"
Visibility query syntax
You can query the visibility store by using the tctl workflow listarchived command
The syntax for the query is based on SQL
Supported column names are
- WorkflowId String
- WorkflowTypeName String
- StartTime Date
- CloseTime Date
- SearchPrecision String - Day, Hour, Minute, Second
WorkflowId or WorkflowTypeName is required. If filtering on date use StartTime or CloseTime in combination with SearchPrecision.
Searching for a record will be done in times in the UTC timezone
SearchPrecision specifies what range you want to search for records. If you use SearchPrecision = 'Day'
it will search all records starting from 2020-01-21T00:00:00Z to 2020-01-21T59:59:59Z
Limitations
- The only operator supported is
=due to how records are stored in s3.
Example
Searches for all records done in day 2020-01-21 with the specified workflow id
./tctl --ns samples-namespace workflow listarchived -q "StartTime = '2020-01-21T00:00:00Z' AND WorkflowId='workflow-id' AND SearchPrecision='Day'"
Storage in S3
Workflow runs are stored in s3 using the following structure
s3://<bucket-name>/<namespace-id>/
history/<workflow-id>/<run-id>
visibility/
workflowTypeName/<workflow-type-name>/
startTimeout/2020-01-21T16:16:11Z/<run-id>
closeTimeout/2020-01-21T16:16:11Z/<run-id>
workflowID/<workflow-id>/
startTimeout/2020-01-21T16:16:11Z/<run-id>
closeTimeout/2020-01-21T16:16:11Z/<run-id>
Enable AWS SDK Logging with config parameter logLevel. For example enable debug logging with logLevel: 4096. Possbile Values:
- LogOff = 0 = 0x0
- LogDebug = 4096 = 0x1000
- LogDebugWithSigning = 4097 = 0x1001
- LogDebugWithHTTPBody = 4098 = 0x1002
- LogDebugWithRequestRetries = 4100 = 0x1004
- LogDebugWithRequestErrors = 4104 = 0x1008
- LogDebugWithEventStreamBody = 4112 = 0x1010
- LogDebugWithDeprecated = 4128 = 0x1020