mirror of
https://github.com/hanzoai/tasks.git
synced 2026-10-04 02:04:03 +00:00
Lanie HeiandGitHub
5f5a0f6a4d
## What changed? -> Part I (History service): @dnr and others have recently implemented a health check to speed up the history service rollout by explicitly marking it as healthy when the history shards have stabilized after startup instead of waiting an arbitrary time period. In this PR, I've reused this check in the history service to ensure that the history service has marked the host as healthy. This avoids a false positive because the host isn't yet receiving error metrics, and so it might not be accurate to say it's serving if history shards haven't ceased shuffling. Added health status starting to differentiate from unhealthy host status to use this upstream and help with error propagation and debugging. (footnote) Further background on the host serving status for documentation and general interest: The status is set [here](https://github.com/temporalio/temporal/blob/main/service/history/service.go#L82) after it has both joined ringpop, and the shards have stabiliized. It reuses a gRPC history health check that was previously implemented, but was unused. -> Part II (Frontend service, aggregator of history.DeepHealthCheck): In the frontend service, I've added the default percentage of 5% based on the guidance from @00Sim that this would be quite a high error percentage, accounting for rollouts. I've added this percentage to the NOT_SERVING metric to avoid a combined outage where some pods are completely down, while some are crashlooping. ## Why? This PR starts to seek improvement for failure detection in the case of crashloops. It's a small improvement as part of the effort to detect crashloopbackoff by not marking pods as healthy until they've been marked as such by the service while starting up. If the pods do not ever "start up", then it's a good indication that something is wrong. ## How did you test it? We can test during rollout by ensuring the failure detection is working as expected. ## Potential risks If the existing check is not marking pods as healthy, we could break the autodetection code. This is not directly workflow (customer)-impacting. ## Documentation N/A ## Is hotfix candidate? No
48 lines
1.4 KiB
Go
48 lines
1.4 KiB
Go
// Code generated by protoc-gen-go-helpers. DO NOT EDIT.
|
|
package enums
|
|
|
|
import (
|
|
"fmt"
|
|
)
|
|
|
|
var (
|
|
ClusterMemberRole_shorthandValue = map[string]int32{
|
|
"Unspecified": 0,
|
|
"Frontend": 1,
|
|
"History": 2,
|
|
"Matching": 3,
|
|
"Worker": 4,
|
|
}
|
|
)
|
|
|
|
// ClusterMemberRoleFromString parses a ClusterMemberRole value from either the protojson
|
|
// canonical SCREAMING_CASE enum or the traditional temporal PascalCase enum to ClusterMemberRole
|
|
func ClusterMemberRoleFromString(s string) (ClusterMemberRole, error) {
|
|
if v, ok := ClusterMemberRole_value[s]; ok {
|
|
return ClusterMemberRole(v), nil
|
|
} else if v, ok := ClusterMemberRole_shorthandValue[s]; ok {
|
|
return ClusterMemberRole(v), nil
|
|
}
|
|
return ClusterMemberRole(0), fmt.Errorf("%s is not a valid ClusterMemberRole", s)
|
|
}
|
|
|
|
var (
|
|
HealthState_shorthandValue = map[string]int32{
|
|
"Unspecified": 0,
|
|
"Serving": 1,
|
|
"NotServing": 2,
|
|
"DeclinedServing": 3,
|
|
}
|
|
)
|
|
|
|
// HealthStateFromString parses a HealthState value from either the protojson
|
|
// canonical SCREAMING_CASE enum or the traditional temporal PascalCase enum to HealthState
|
|
func HealthStateFromString(s string) (HealthState, error) {
|
|
if v, ok := HealthState_value[s]; ok {
|
|
return HealthState(v), nil
|
|
} else if v, ok := HealthState_shorthandValue[s]; ok {
|
|
return HealthState(v), nil
|
|
}
|
|
return HealthState(0), fmt.Errorf("%s is not a valid HealthState", s)
|
|
}
|