Files
Michael CrenshawandGitHub ba889a0c86 feat: source hydrator (#20345)
* feat(hydrator): add sourceHydrator types

Co-authored-by: Alexandre Gaudreault <[email protected]>
Co-authored-by: Omer Azmon <[email protected]>
Co-authored-by: daengdaengLee <[email protected]>
Co-authored-by: Juwon Hwang (Kevin) <[email protected]>
Co-authored-by: thisishwan2 <[email protected]>
Co-authored-by: mirageoasis <[email protected]>
Co-authored-by: Robin Lieb <[email protected]>
Co-authored-by: miiiinju1 <[email protected]>
Signed-off-by: Michael Crenshaw <[email protected]>

fix(codegen): use kube_codegen.sh deepcopy and client gen correctly

Signed-off-by: Michael Crenshaw <[email protected]>

deepcopy gen

Signed-off-by: Michael Crenshaw <[email protected]>

* feat(hydrator): add commit-server component

Co-authored-by: Alexandre Gaudreault <[email protected]>
Co-authored-by: Omer Azmon <[email protected]>
Co-authored-by: daengdaengLee <[email protected]>
Co-authored-by: Juwon Hwang (Kevin) <[email protected]>
Co-authored-by: thisishwan2 <[email protected]>
Co-authored-by: mirageoasis <[email protected]>
Co-authored-by: Robin Lieb <[email protected]>
Co-authored-by: miiiinju1 <[email protected]>
Signed-off-by: Michael Crenshaw <[email protected]>

go mod tidy

Signed-off-by: Michael Crenshaw <[email protected]>

one test file for both implementations

Signed-off-by: Michael Crenshaw <[email protected]>

simplify

Signed-off-by: Michael Crenshaw <[email protected]>

fix test for linux

Signed-off-by: Michael Crenshaw <[email protected]>

fix git client mock

Signed-off-by: Michael Crenshaw <[email protected]>

fix git client mock

Signed-off-by: Michael Crenshaw <[email protected]>

address comments

Signed-off-by: Michael Crenshaw <[email protected]>

unit tests

Signed-off-by: Michael Crenshaw <[email protected]>

lint

Signed-off-by: Michael Crenshaw <[email protected]>

fix image, fix health checks, fix merge issue

Signed-off-by: Michael Crenshaw <[email protected]>

fix lint issues

Signed-off-by: Michael Crenshaw <[email protected]>

remove code that doesn't work for GHE

Signed-off-by: Michael Crenshaw <[email protected]>

changes from comments

Signed-off-by: Michael Crenshaw <[email protected]>

* feat(hydrator): enable controller

Co-authored-by: Alexandre Gaudreault <[email protected]>
Co-authored-by: Omer Azmon <[email protected]>
Co-authored-by: daengdaengLee <[email protected]>
Co-authored-by: Juwon Hwang (Kevin) <[email protected]>
Co-authored-by: thisishwan2 <[email protected]>
Co-authored-by: mirageoasis <[email protected]>
Co-authored-by: Robin Lieb <[email protected]>
Co-authored-by: miiiinju1 <[email protected]>
Signed-off-by: Michael Crenshaw <[email protected]>

feat(hydrator): enable controller

Co-authored-by: Alexandre Gaudreault <[email protected]>
Co-authored-by: Omer Azmon <[email protected]>
Co-authored-by: daengdaengLee <[email protected]>
Co-authored-by: Juwon Hwang (Kevin) <[email protected]>
Co-authored-by: thisishwan2 <[email protected]>
Co-authored-by: mirageoasis <[email protected]>
Co-authored-by: Robin Lieb <[email protected]>
Co-authored-by: miiiinju1 <[email protected]>
Signed-off-by: Michael Crenshaw <[email protected]>

allow opt-in

Signed-off-by: Michael Crenshaw <[email protected]>

separation between app controller and hydrator

Signed-off-by: Michael Crenshaw <[email protected]>

simplify diff

Signed-off-by: Michael Crenshaw <[email protected]>

todos

Signed-off-by: Michael Crenshaw <[email protected]>

simplify

Signed-off-by: Michael Crenshaw <[email protected]>

add dry sha to logs

Signed-off-by: Michael Crenshaw <[email protected]>

add app name to logs

Signed-off-by: Michael Crenshaw <[email protected]>

more logging, no caching

Signed-off-by: Michael Crenshaw <[email protected]>

fix cluster install

Signed-off-by: Michael Crenshaw <[email protected]>

don't interrupt an ongoing hydrate operation

Signed-off-by: Michael Crenshaw <[email protected]>

revert hydrate loop fix

Signed-off-by: Michael Crenshaw <[email protected]>

handle project-scoped repo creds

Signed-off-by: Michael Crenshaw <[email protected]>

codegen

Signed-off-by: Michael Crenshaw <[email protected]>

improve docs

Signed-off-by: Michael Crenshaw <[email protected]>

fixes from comments

Signed-off-by: Michael Crenshaw <[email protected]>

* set hydrator enabled key when using hydrator manifests

Signed-off-by: Michael Crenshaw <[email protected]>

fix manifests

Signed-off-by: Michael Crenshaw <[email protected]>

improve docs

Signed-off-by: Michael Crenshaw <[email protected]>

* feat(hydrator): enable controller

Co-authored-by: Alexandre Gaudreault <[email protected]>
Co-authored-by: Omer Azmon <[email protected]>
Co-authored-by: daengdaengLee <[email protected]>
Co-authored-by: Juwon Hwang (Kevin) <[email protected]>
Co-authored-by: thisishwan2 <[email protected]>
Co-authored-by: mirageoasis <[email protected]>
Co-authored-by: Robin Lieb <[email protected]>
Co-authored-by: miiiinju1 <[email protected]>
Signed-off-by: Michael Crenshaw <[email protected]>

feat(hydrator): enable controller

Co-authored-by: Alexandre Gaudreault <[email protected]>
Co-authored-by: Omer Azmon <[email protected]>
Co-authored-by: daengdaengLee <[email protected]>
Co-authored-by: Juwon Hwang (Kevin) <[email protected]>
Co-authored-by: thisishwan2 <[email protected]>
Co-authored-by: mirageoasis <[email protected]>
Co-authored-by: Robin Lieb <[email protected]>
Co-authored-by: miiiinju1 <[email protected]>
Signed-off-by: Michael Crenshaw <[email protected]>

allow opt-in

Signed-off-by: Michael Crenshaw <[email protected]>

separation between app controller and hydrator

Signed-off-by: Michael Crenshaw <[email protected]>

simplify diff

Signed-off-by: Michael Crenshaw <[email protected]>

todos

Signed-off-by: Michael Crenshaw <[email protected]>

simplify

Signed-off-by: Michael Crenshaw <[email protected]>

add dry sha to logs

Signed-off-by: Michael Crenshaw <[email protected]>

add app name to logs

Signed-off-by: Michael Crenshaw <[email protected]>

more logging, no caching

Signed-off-by: Michael Crenshaw <[email protected]>

fix cluster install

Signed-off-by: Michael Crenshaw <[email protected]>

don't interrupt an ongoing hydrate operation

Signed-off-by: Michael Crenshaw <[email protected]>

revert hydrate loop fix

Signed-off-by: Michael Crenshaw <[email protected]>

handle project-scoped repo creds

Signed-off-by: Michael Crenshaw <[email protected]>

codegen

Signed-off-by: Michael Crenshaw <[email protected]>

improve docs

Signed-off-by: Michael Crenshaw <[email protected]>

fixes from comments

Signed-off-by: Michael Crenshaw <[email protected]>

* feat(hydrator): add sourceHydrator types

Co-authored-by: Alexandre Gaudreault <[email protected]>
Co-authored-by: Omer Azmon <[email protected]>
Co-authored-by: daengdaengLee <[email protected]>
Co-authored-by: Juwon Hwang (Kevin) <[email protected]>
Co-authored-by: thisishwan2 <[email protected]>
Co-authored-by: mirageoasis <[email protected]>
Co-authored-by: Robin Lieb <[email protected]>
Co-authored-by: miiiinju1 <[email protected]>
Signed-off-by: Michael Crenshaw <[email protected]>

fix(codegen): use kube_codegen.sh deepcopy and client gen correctly

Signed-off-by: Michael Crenshaw <[email protected]>

deepcopy gen

Signed-off-by: Michael Crenshaw <[email protected]>

* feat(hydrator): enable controller

Co-authored-by: Alexandre Gaudreault <[email protected]>
Co-authored-by: Omer Azmon <[email protected]>
Co-authored-by: daengdaengLee <[email protected]>
Co-authored-by: Juwon Hwang (Kevin) <[email protected]>
Co-authored-by: thisishwan2 <[email protected]>
Co-authored-by: mirageoasis <[email protected]>
Co-authored-by: Robin Lieb <[email protected]>
Co-authored-by: miiiinju1 <[email protected]>
Signed-off-by: Michael Crenshaw <[email protected]>

feat(hydrator): enable controller

Co-authored-by: Alexandre Gaudreault <[email protected]>
Co-authored-by: Omer Azmon <[email protected]>
Co-authored-by: daengdaengLee <[email protected]>
Co-authored-by: Juwon Hwang (Kevin) <[email protected]>
Co-authored-by: thisishwan2 <[email protected]>
Co-authored-by: mirageoasis <[email protected]>
Co-authored-by: Robin Lieb <[email protected]>
Co-authored-by: miiiinju1 <[email protected]>
Signed-off-by: Michael Crenshaw <[email protected]>

allow opt-in

Signed-off-by: Michael Crenshaw <[email protected]>

separation between app controller and hydrator

Signed-off-by: Michael Crenshaw <[email protected]>

simplify diff

Signed-off-by: Michael Crenshaw <[email protected]>

todos

Signed-off-by: Michael Crenshaw <[email protected]>

simplify

Signed-off-by: Michael Crenshaw <[email protected]>

add dry sha to logs

Signed-off-by: Michael Crenshaw <[email protected]>

add app name to logs

Signed-off-by: Michael Crenshaw <[email protected]>

more logging, no caching

Signed-off-by: Michael Crenshaw <[email protected]>

fix cluster install

Signed-off-by: Michael Crenshaw <[email protected]>

don't interrupt an ongoing hydrate operation

Signed-off-by: Michael Crenshaw <[email protected]>

revert hydrate loop fix

Signed-off-by: Michael Crenshaw <[email protected]>

handle project-scoped repo creds

Signed-off-by: Michael Crenshaw <[email protected]>

codegen

Signed-off-by: Michael Crenshaw <[email protected]>

* feat(hydrator): write credentials handling + UI

Co-authored-by: Alexandre Gaudreault <[email protected]>
Co-authored-by: Omer Azmon <[email protected]>
Co-authored-by: daengdaengLee <[email protected]>
Co-authored-by: Juwon Hwang (Kevin) <[email protected]>
Co-authored-by: thisishwan2 <[email protected]>
Co-authored-by: mirageoasis <[email protected]>
Co-authored-by: Robin Lieb <[email protected]>
Co-authored-by: miiiinju1 <[email protected]>
Signed-off-by: Michael Crenshaw <[email protected]>

feat(hydrator): enable controller

Co-authored-by: Alexandre Gaudreault <[email protected]>
Co-authored-by: Omer Azmon <[email protected]>
Co-authored-by: daengdaengLee <[email protected]>
Co-authored-by: Juwon Hwang (Kevin) <[email protected]>
Co-authored-by: thisishwan2 <[email protected]>
Co-authored-by: mirageoasis <[email protected]>
Co-authored-by: Robin Lieb <[email protected]>
Co-authored-by: miiiinju1 <[email protected]>
Signed-off-by: Michael Crenshaw <[email protected]>

allow opt-in

Signed-off-by: Michael Crenshaw <[email protected]>

separation between app controller and hydrator

Signed-off-by: Michael Crenshaw <[email protected]>

simplify diff

Signed-off-by: Michael Crenshaw <[email protected]>

todos

Signed-off-by: Michael Crenshaw <[email protected]>

simplify

Signed-off-by: Michael Crenshaw <[email protected]>

add dry sha to logs

Signed-off-by: Michael Crenshaw <[email protected]>

add app name to logs

Signed-off-by: Michael Crenshaw <[email protected]>

more logging, no caching

Signed-off-by: Michael Crenshaw <[email protected]>

fix cluster install

Signed-off-by: Michael Crenshaw <[email protected]>

don't interrupt an ongoing hydrate operation

Signed-off-by: Michael Crenshaw <[email protected]>

revert hydrate loop fix

Signed-off-by: Michael Crenshaw <[email protected]>

feat(hydrator): write credentials handling + UI

Signed-off-by: Michael Crenshaw <[email protected]>

WIP: add new APIs for write creds

Signed-off-by: Michael Crenshaw <[email protected]>

write api and template api

Signed-off-by: Michael Crenshaw <[email protected]>

fix time function

Signed-off-by: Michael Crenshaw <[email protected]>

fix lint issues

Signed-off-by: Michael Crenshaw <[email protected]>

don't enrich with read creds

Signed-off-by: Michael Crenshaw <[email protected]>

revert tls change

Signed-off-by: Michael Crenshaw <[email protected]>

don't disable buttons in UI

Signed-off-by: Michael Crenshaw <[email protected]>

ask repo server for specific revision

Signed-off-by: Michael Crenshaw <[email protected]>

fixes

Signed-off-by: Michael Crenshaw <[email protected]>

lint ui

Signed-off-by: Michael Crenshaw <[email protected]>

remove unnecessary change

Signed-off-by: Michael Crenshaw <[email protected]>

fix test and lint

Signed-off-by: Michael Crenshaw <[email protected]>

lint

Signed-off-by: Michael Crenshaw <[email protected]>

enable hydrator for e2e tests

Signed-off-by: Michael Crenshaw <[email protected]>

* free disk space for e2e tests

Signed-off-by: Michael Crenshaw <[email protected]>

don't free disk space

Signed-off-by: Michael Crenshaw <[email protected]>

* free disk space for e2e tests

Signed-off-by: Michael Crenshaw <[email protected]>

* remove comment that breaks auth

Signed-off-by: Michael Crenshaw <[email protected]>

* try removing extra function

Signed-off-by: Michael Crenshaw <[email protected]>

* cleanup from comments

Signed-off-by: Michael Crenshaw <[email protected]>

* fix test

Signed-off-by: Michael Crenshaw <[email protected]>

---------

Signed-off-by: Michael Crenshaw <[email protected]>
2024-12-16 16:59:09 -05:00

158 lines
5.5 KiB
Go

package metrics
import (
"net/http"
"time"
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/collectors"
"github.com/prometheus/client_golang/prometheus/promhttp"
)
// Server is a prometheus server which collects application metrics.
type Server struct {
handler http.Handler
commitPendingRequestsGauge *prometheus.GaugeVec
gitRequestCounter *prometheus.CounterVec
gitRequestHistogram *prometheus.HistogramVec
commitRequestHistogram *prometheus.HistogramVec
userInfoRequestHistogram *prometheus.HistogramVec
commitRequestCounter *prometheus.CounterVec
}
// GitRequestType is the type of git request
type GitRequestType string
const (
// GitRequestTypeLsRemote is a request to list remote refs
GitRequestTypeLsRemote = "ls-remote"
// GitRequestTypeFetch is a request to fetch from remote
GitRequestTypeFetch = "fetch"
// GitRequestTypePush is a request to push to remote
GitRequestTypePush = "push"
)
// CommitResponseType is the type of response for a commit request
type CommitResponseType string
const (
// CommitResponseTypeSuccess is a successful commit request
CommitResponseTypeSuccess CommitResponseType = "success"
// CommitResponseTypeFailure is a failed commit request
CommitResponseTypeFailure CommitResponseType = "failure"
)
// NewMetricsServer returns a new prometheus server which collects application metrics.
func NewMetricsServer() *Server {
registry := prometheus.NewRegistry()
registry.MustRegister(collectors.NewProcessCollector(collectors.ProcessCollectorOpts{}))
registry.MustRegister(collectors.NewGoCollector())
commitPendingRequestsGauge := prometheus.NewGaugeVec(
prometheus.GaugeOpts{
Name: "argocd_commitserver_commit_pending_request_total",
Help: "Number of pending commit requests",
},
[]string{"repo"},
)
registry.MustRegister(commitPendingRequestsGauge)
gitRequestCounter := prometheus.NewCounterVec(
prometheus.CounterOpts{
Name: "argocd_commitserver_git_request_total",
Help: "Number of git requests performed by repo server",
},
[]string{"repo", "request_type"},
)
registry.MustRegister(gitRequestCounter)
gitRequestHistogram := prometheus.NewHistogramVec(
prometheus.HistogramOpts{
Name: "argocd_commitserver_git_request_duration_seconds",
Help: "Git requests duration seconds.",
Buckets: []float64{0.1, 0.25, .5, 1, 2, 4, 10, 20},
},
[]string{"repo", "request_type"},
)
registry.MustRegister(gitRequestHistogram)
commitRequestHistogram := prometheus.NewHistogramVec(
prometheus.HistogramOpts{
Name: "argocd_commitserver_commit_request_duration_seconds",
Help: "Commit request duration seconds.",
Buckets: []float64{0.1, 0.25, .5, 1, 2, 4, 10, 20},
},
[]string{"repo", "response_type"},
)
registry.MustRegister(commitRequestHistogram)
userInfoRequestHistogram := prometheus.NewHistogramVec(
prometheus.HistogramOpts{
Name: "argocd_commitserver_userinfo_request_duration_seconds",
Help: "Userinfo request duration seconds.",
Buckets: []float64{0.1, 0.25, .5, 1, 2, 4, 10, 20},
},
[]string{"repo", "credential_type"},
)
registry.MustRegister(userInfoRequestHistogram)
commitRequestCounter := prometheus.NewCounterVec(
prometheus.CounterOpts{
Name: "argocd_commitserver_commit_request_total",
Help: "Number of commit requests performed handled",
},
[]string{"repo", "response_type"},
)
registry.MustRegister(commitRequestCounter)
return &Server{
handler: promhttp.HandlerFor(registry, promhttp.HandlerOpts{}),
commitPendingRequestsGauge: commitPendingRequestsGauge,
gitRequestCounter: gitRequestCounter,
gitRequestHistogram: gitRequestHistogram,
commitRequestHistogram: commitRequestHistogram,
userInfoRequestHistogram: userInfoRequestHistogram,
commitRequestCounter: commitRequestCounter,
}
}
// GetHandler returns the http.Handler for the prometheus server
func (m *Server) GetHandler() http.Handler {
return m.handler
}
// IncPendingCommitRequest increments the pending commit requests gauge
func (m *Server) IncPendingCommitRequest(repo string) {
m.commitPendingRequestsGauge.WithLabelValues(repo).Inc()
}
// DecPendingCommitRequest decrements the pending commit requests gauge
func (m *Server) DecPendingCommitRequest(repo string) {
m.commitPendingRequestsGauge.WithLabelValues(repo).Dec()
}
// IncGitRequest increments the git requests counter
func (m *Server) IncGitRequest(repo string, requestType GitRequestType) {
m.gitRequestCounter.WithLabelValues(repo, string(requestType)).Inc()
}
// ObserveGitRequestDuration observes the duration of a git request
func (m *Server) ObserveGitRequestDuration(repo string, requestType GitRequestType, duration time.Duration) {
m.gitRequestHistogram.WithLabelValues(repo, string(requestType)).Observe(duration.Seconds())
}
// ObserveCommitRequestDuration observes the duration of a commit request
func (m *Server) ObserveCommitRequestDuration(repo string, rt CommitResponseType, duration time.Duration) {
m.commitRequestHistogram.WithLabelValues(repo, string(rt)).Observe(duration.Seconds())
}
// ObserveUserInfoRequestDuration observes the duration of a userinfo request
func (m *Server) ObserveUserInfoRequestDuration(repo string, credentialType string, duration time.Duration) {
m.userInfoRequestHistogram.WithLabelValues(repo, credentialType).Observe(duration.Seconds())
}
// IncCommitRequest increments the commit request counter
func (m *Server) IncCommitRequest(repo string, rt CommitResponseType) {
m.commitRequestCounter.WithLabelValues(repo, string(rt)).Inc()
}