Co-authored-by: Hanzo Dev <[email protected]>
porto
Server and client building blocks for Go services: a REST server, a combined gRPC / gRPC-Web / REST server, HTTP helpers (errors, identity, correlation, marshalling), and infrastructure packages (retrying HTTP client, gRPC client builder, Redis client and cache, scheduler, TLS reloading, listeners, application bootstrap).
Module: github.com/effective-security/porto. No binaries; import the
packages you need.
Requirements
- Go 1.27 or later (
go.modsetsgo 1.27; the standarduuidpackage is used). - Docker, only to run the Redis-backed tests (
pkg/redisclient,pkg/cache).
go get github.com/effective-security/porto@latest
Packages
| Package | Purpose |
|---|---|
restserver |
httprouter-based HTTP/HTTPS server hosting Service plug-ins with correlation ID, identity, metrics, logging, authz, readiness and CORS middleware. |
restserver/authz |
Path-segment/role access control tree (YAML-configurable) exposed as an HTTP handler and gRPC interceptors. |
restserver/ready |
Readiness-gate middleware answering 503 not_ready until the server reports ready. |
restserver/telemetry |
Request logger and request metrics middleware with skip-path filters. |
gserver |
Combined gRPC + gRPC-Web + REST server with cmux multiplexing, TLS reload, CORS, rate limiting, identity/authz middleware and graceful shutdown. |
gserver/credentials |
gRPC credential bundle: TLS transport credentials plus per-RPC Bearer/DPoP/AWS tokens with automatic refresh. |
gserver/roles |
Identity provider mapping JWT, DPoP, AWS STS and TLS/SPIFFE callers to roles for HTTP and gRPC. |
xhttp/correlation |
Correlation/request ID propagation across HTTP headers, gRPC metadata, contexts and logs. |
xhttp/header |
HTTP header name and content-type constants. |
xhttp/httperror |
Structured API errors with HTTP status, code, gRPC status and request ID, plus gRPC ↔ HTTP mapping. |
xhttp/identity |
Caller identity, client IP and user agent extraction into request contexts for HTTP and gRPC. |
xhttp/limits |
Shared HTTP, protocol detection and TLS handshake timeout defaults, the optional handler deadline, and body-size defaults. |
xhttp/marshal |
JSON response writing (errors, gzip, pretty print) and strict JSON request decoding. |
pkg/retriable |
HTTP client with retry policy, JSON marshalling, header propagation, Bearer/DPoP auth, replay nonces and token storage. See pkg/retriable/README.md. |
pkg/rpcclient |
gRPC client builder: TLS bundle, per-RPC Bearer/DPoP tokens, keepalive, message limits, optional blocking dial. |
pkg/redisclient |
Prefixed go-redis wrapper with JSON values, bounded collections, a distributed lock and a per-window rate limiter. |
pkg/cache |
Provider cache abstraction (TTL, keys, pub/sub) with in-memory, Redis and prefixing-proxy backends. |
pkg/revocation |
JWT revocation list over cache.Provider: Validate and Revoke for jwt.Revocation. |
pkg/location |
Client location (city, region, country) from an IP: Provider interface, ipwho.is lookup and an LRU cache shared per IPv4 /24 or IPv6 /48 network. |
pkg/tasks |
In-process cron-like scheduler: run functions every N units, daily at hh:mm, or weekly, with overlap protection. |
pkg/tlsconfig |
Build tls.Config from PEM files (TLS 1.2+, h2) with OCSP staple support and a file-polling certificate reloader. |
pkg/transport |
TLS listener with eager handshake and CRL checks, keepalive listeners, and file-based TLSInfo server config. |
pkg/appinit |
Service bootstrap: logging flags and rotation, Prometheus/CloudWatch metrics, CPU profiling. |
pkg/appinit/config |
YAML/JSON config struct for the metrics pipeline. |
pkg/discovery |
Minimal in-process service registry resolved by interface. |
pkg/crlcache |
Verifier interface for certificate revocation checks. |
pkg/streamctx |
Override a gRPC ServerStream context in interceptors. |
metricskey |
Metric descriptors emitted by the servers. |
tests/testutils, tests/mockappcontainer |
Test helpers: free ports, JSON comparison, dig container builder. |
Every package has a doc.go with a usage example. The generated API
reference lives in Documentation/api (make docs), and
the navigation index for the code is
Documentation/codemap.md.
Quick start
REST server
type cfg struct{ bind, name, public string }
func (c cfg) GetServerName() string { return c.name }
func (c cfg) GetBindAddr() string { return c.bind }
func (c cfg) GetPublicURL() string { return c.public }
func (c cfg) GetServices() []string { return nil }
type hello struct{}
func (hello) Name() string { return "hello" }
func (hello) IsReady() bool { return true }
func (hello) Close() {}
func (hello) Register(r restserver.Router) {
r.GET("/v1/hello/:name", func(w http.ResponseWriter, req *http.Request, p restserver.Params) {
marshal.WriteJSON(w, req, map[string]string{"hello": p.ByName("name")})
})
}
srv, err := restserver.New("1.0.0", "", cfg{bind: ":8080", name: "api"}, nil) // nil TLS = plain HTTP
if err != nil {
log.Fatal(err)
}
srv.WithCORS(&restserver.CORSOptions{AllowedOrigins: []string{"*"}})
srv.AddService(hello{})
if err := srv.StartHTTP(); err != nil {
log.Fatal(err)
}
defer srv.StopHTTP()
Authorization is configured in YAML and attached with WithAuthz:
authz:
allow:
- /v1/admin:admin
- /v1/items:admin,user
allow_any:
- /v1/status
allow_any_role:
- /v1/me
log_denied: true
az, err := authz.New(&cfg.Authz)
if err != nil {
return err
}
srv.WithAuthz(az).WithIdentityProvider(myMapper) // identity.ProviderFromRequest
Handlers: errors, decoding, identity
func (s *svc) get(w http.ResponseWriter, r *http.Request, p restserver.Params) {
var req ItemRequest
if marshal.DecodeBody(w, r, &req) != nil {
return // 400 invalid_json already written
}
if req.ID == "" {
marshal.WriteJSON(w, r, httperror.InvalidParam("id is required"))
return
}
caller := identity.FromRequest(r)
item, err := s.store.Get(r.Context(), caller.Identity().Tenant(), req.ID)
if err != nil {
marshal.WriteJSON(w, r, httperror.WrapWithCtx(r.Context(), err, "load %s", req.ID)) // 404/408/500
return
}
marshal.WriteJSON(w, r, item) // 200 application/json, gzip when accepted
}
// error body: {"code":"not_found","message":"load 42","request_id":"a1b2c3d4e5f6"}
gRPC + gRPC-Web + REST server
cfg := &gserver.Config{
ListenURLs: []string{"https://0.0.0.0:8443"},
Services: []string{"status"},
ServerTLS: &gserver.TLSInfo{
CertFile: "server.pem",
KeyFile: "server-key.pem",
ClientCAFile: "ca.pem",
},
IdentityMap: &roles.IdentityMap{
JWT: roles.JWTIdentityMap{
Enabled: true,
Issuer: "https://issuer",
Audience: "api",
DefaultAuthenticatedRole: roles.JWTUserRoleName,
},
},
Authz: &authz.Config{AllowAny: []string{"/healthz"}, Allow: []string{"/v1:jwt_user,admin"}},
}
factories := map[string]gserver.ServiceFactory{
"status": func(s gserver.GServer) any {
return func() { s.AddService(status.New(s)) } // implements Service + RouteRegistrator/GRPCRegistrator
},
}
srv, err := gserver.Start("api", cfg, container, factories, gserver.WithMiddleware(myMiddleware))
if err != nil {
log.Fatal(err)
}
defer srv.Close()
select {
case err := <-srv.Err():
log.Fatal(err)
case <-ctx.Done():
}
The same server from YAML:
listen_urls: ["https://0.0.0.0:8443", "unix:///var/run/api.sock"]
services: ["status", "api"]
server_tls:
cert: /etc/tls/server.pem
key: /etc/tls/server-key.pem
client_ca: /etc/tls/ca.pem
cors:
enabled: true
allowed_origins: ["https://app.example.com"]
allow_credentials: true
rate_limit:
enabled: true
requests_per_second: 100
# log_rejections: true
timeouts:
handler: 2m # server-side bound on every request; an earlier client deadline is kept
timeout:
shutdown: 10s # how long Close waits for in-flight requests (formerly timeout.request)
identity_map:
skip_auth: ["/healthz"]
jwt:
enabled: true
issuer: https://issuer
audience: api
default_authenticated_role: jwt_user
roles:
admin: ["[email protected]"]
tls:
enabled: true
default_authenticated_role: tls_user
roles:
admin: ["spiffe://trust/ns/ops"]
authz:
allow_any: ["/healthz"]
allow: ["/v1:jwt_user,admin", "/pb.API/:admin"]
Client IPs use the socket peer by default; X-Forwarded-Proto is ignored
unless the peer is trusted. When a
reverse proxy overwrites incoming forwarding headers, configure
trusted_proxy_cidrs with its network ranges (for example,
["10.2.0.0/16"]). The default rate limiter then uses the resolved client
IP. restserver.HTTPServer supports the same policy; set it before
StartHTTP:
trust, err := identity.ParseTrustedProxies([]string{"10.2.0.0/16"})
if err != nil {
return err
}
srv.WithTrustedProxies(trust)
gserver limits REST, gRPC and gRPC-Web on every listener, with one
bucket per client IP and request path; callers on a Unix socket have no
client IP and are not limited unless headers_ip_lookups is set (without
RemoteAddr, which keys every socket caller as @). A limited
REST request gets HTTP 429 with an X-Correlation-ID; a limited gRPC or
gRPC-Web call gets ResourceExhausted with the request ID. Both are logged
and counted in the metrics like any other response. Unless
headers_ip_lookups is set, the 429 access line's remote is the
resolved client IP the limiter keys on; tollbooth groups an IPv6 client
by its /64 prefix. If
every 429 shows an internal address such as 10.0.x.x, all callers behind
the load balancer share one bucket: add the balancer to
trusted_proxy_cidrs and make sure it sets X-Forwarded-For. Set
log_rejections: true on rate_limit to log each rejection at WARNING
with the limiter key, the socket peer, the User-Agent and the forwarding
headers as received.
Request log lines use the same field names for REST and gRPC: remote is
the client IP resolved under trusted_proxy_cidrs, peer the socket
address (host:port) where a line shows it, and agent the User-Agent
(no-agent when missing).
HTTP client with retries
client, err := retriable.New(retriable.ClientConfig{
Host: "https://api.example.com",
Request: &retriable.RequestPolicy{RetryLimit: 3, Timeout: 5 * time.Second},
}, retriable.WithUserAgent("my-cli"))
if err != nil {
return err
}
if err := client.Config.LoadAuthTokenOrFromEnv("MY_TOKEN"); err == nil {
_ = client.SetAuthorization() // Authorization: Bearer or DPoP
}
var res struct {
ID string `json:"id"`
}
_, status, err := client.Get(ctx, "/v1/items/1", &res)
# retriable.LoadFactory(file)
clients:
prod:
host: https://api.example.com
tls:
trusted_ca: /etc/pki/cabundle.pem
request:
retry_limit: 3
timeout: 2s
storage_folder: ~/.config/demo
gRPC client
cfg := &rpcclient.Config{
Endpoint: "https://api.example.com:443", // TLS requires https:// or unixs://
TLS: tlsCfg,
DialTimeout: 5 * time.Second, // 0 = lazy connect
}
if err := cfg.LoadAuthTokenOrFromEnv("MY_TOKEN"); err != nil {
return err
}
client, err := rpcclient.New(cfg)
if err != nil {
return err
}
defer client.Close()
svc := pb.NewMyServiceClient(client.Conn())
res, err := svc.Call(ctx, req, client.Opts()...)
Redis and cache
rc, err := redisclient.New(&redisclient.Config{Server: "redis://localhost:6379/0", Password: "secret"})
if err != nil {
return err
}
defer rc.Close()
c := rc.WithPrefix("myapp") // keys under /myapp/
_ = c.Set(ctx, "user:1", user, time.Hour) // JSON encoded
var u User
if err := c.Get(ctx, "user:1", &u); redisclient.IsNotFoundError(err) {
// miss
}
token, remaining, err := c.TryLock(ctx, "job", 30*time.Second)
if err != nil {
return err
}
if token == "" {
return fmt.Errorf("job is locked for %s", remaining)
}
defer c.ReleaseLock(ctx, "job", token) // releases only while this token holds the lock
var p cache.Provider
if cfg.Provider == "redis" {
p, err = cache.NewRedisProvider(*cfg.Redis, "/myapp")
} else {
p = cache.NewMemoryProvider("/myapp")
}
_ = p.Set(ctx, "user:1", user, 10*time.Minute) // 0 => default TTL, cache.KeepTTL => no expiry
err = p.Get(ctx, "user:1", &u) // cache.IsNotFoundError(err) on miss
users := cache.NewProxyProvider("users", p) // sub-namespace sharing the connection
Scheduler
s := tasks.NewScheduler(tasks.WithTickerInterval(time.Second))
s.Add(tasks.NewTaskAtIntervals(30, tasks.Seconds).Do("cleanup", cleanup))
s.Add(tasks.NewTaskDaily(2, 30).Do("rotate", rotate, "/var/log"))
t, err := tasks.NewTask("every day 11:15")
if err != nil {
return err
}
s.Add(t.Do("report", report))
if err := s.Start(); err != nil {
return err
}
defer s.Stop()
TLS listeners with certificate reload
info := &transport.TLSInfo{
CertFile: "server.pem",
KeyFile: "server-key.pem",
TrustedCAFile: "ca.pem",
ClientAuthType: tls.VerifyClientCertIfGiven,
}
defer info.Close()
ln, err := net.Listen("tcp", ":8443")
if err != nil {
return err
}
tlsLn, err := transport.NewTLSListener(ln, info) // keypair reloaded every 5 minutes
if err != nil {
_ = ln.Close() // left open when the TLS config fails to load
return err
}
srv := &http.Server{Handler: h, TLSConfig: info.Config(), ReadHeaderTimeout: 10 * time.Second}
return srv.Serve(tlsLn)
// client-side mTLS with rotation
cfg, reloader, err := tlsconfig.NewClientTLSWithReloader("client.pem", "client-key.pem", "ca.pem", 5*time.Minute)
if err != nil {
return err
}
defer reloader.Close()
client := &http.Client{Transport: &http.Transport{TLSClientConfig: cfg}}
Application bootstrap
metrics:
provider: prometheus
prefix: myservice
global_tags: [service, cluster_id]
prometheus:
addr: :9090
expiration: 5m
lc, err := appinit.Logs(&flags.LogConfig, "myservice") // --log-dir enables rotation
if err != nil {
return err
}
if lc != nil {
defer lc.Close()
}
mc, err := appinit.Metrics(&cfg.Metrics, "myservice", cluster, version, commit, nil)
if err != nil {
return err
}
if mc != nil {
defer mc.Close()
}
Development
make toolsinstalls golangci-lint, cov-report, govulncheck and gomarkdoc intobin/.make testruns the tests (Docker required for the Redis packages);make test RACE=trueadds the race detector.make testshortskips the slow tests;make covtest coverageproduces and opens the coverage report.make lintruns gofmt (with the project rewrites),go vet,govulncheckandgolangci-lint.make docsregenerates the API reference underDocumentation/api/.make allruns clean, tools, generate and covtest.
CI runs make lint and make build covtest and requires more than 90% total coverage. Conventions
for contributors and agents are in AGENTS.md; known defects
are tracked in FINDINGS.md and larger work in
ROADMAP.md.
Server network limits
HTTP servers now default to 10s for headers, 30s for whole-request reads, 60s idle and 10 MiB for request bodies. cmux detection and eager TLS handshakes default to 10s. Zero selects a default; negative values disable individual limits. Header deadlines use native net/http semantics: HTTP/2 has no per-stream header deadline. There is no response write deadline. Raise or disable read/body limits for large uploads and long request streams.
gserver.Config accepts timeouts: {header: 10s, read: 30s, idle: 60s, handshake: 10s, handler: 2m} and max_request_body: 10485760 in YAML.
timeout.shutdown (formerly timeout.request, which is still read) is how
long Close waits for in-flight requests; it does not bound a request while
the server is serving. Native gRPC on plaintext listeners retains its
per-message limits; HTTP body limits also apply to TLS gRPC/gRPC-Web
streams. Native gRPC streams have no read deadline, so long-lived client and
bidi streams work on both listener types; on TLS a stream fails with
Unavailable once it has sent max_request_body bytes in total.
The read deadlines end when the request has been read; without
timeouts.handler a request then runs as long as the client's deadline
allows, and without one until the client goes away. timeouts.handler is
the server-side maximum: the context of every REST request and of every
gRPC and gRPC-Web call, streams included, expires after it unless the client
asked for an earlier deadline, which is kept. It is off by default; set it
above the longest operation the server performs. For gRPC it is also the
transport stream deadline, so a stream handler blocked in RecvMsg or
SendMsg behind a stalled client is released: with DeadlineExceeded on a
plaintext listener, and on a TLS listener with DeadlineExceeded for a
receive, while a send blocked on the client's flow-control window fails with
Unavailable one second after the deadline, when the listener resets the
stream and the client sees Internal (net/http cannot deliver trailers
after aborting a write). A handler that returns its expired context error
fails the call with DeadlineExceeded instead of Unknown. Handlers must
honor their context otherwise: one that ignores it outlives the deadline, as
it does a client deadline, and a REST handler blocked writing to a client
that stopped reading is not released (there is no response write deadline).
For REST, configure WithTimeouts(limits.Timeouts{...}) and
WithMaxRequestBody(bytes) before StartHTTP. Both default and custom muxes
are bounded, including by Handler. Native net/http TLS uses the smaller
positive header/read value for its handshake; the Handshake field is for
cmux/eager listeners. transport.TLSInfo.HandshakeTimeout overrides the
eager TLS deadline.
marshal.DecodeBody also enforces the body default when used alone.
marshal.LimitRequestBody(handler, bytes) overrides it. The limiter writes
no response, so it cannot bypass CORS, correlation, logging or metrics. A body
with a known oversized Content-Length fails on its first read before any
byte is read. DecodeBody answers it, and any oversized JSON body, with HTTP
413 request_too_large. Other handlers that read bodies must handle
*http.MaxBytesError. Over gRPC, request_too_large travels as
ResourceExhausted; httperror.NewFromPb restores the 413 from the code
detail that GRPCStatus attaches.
Prometheus accepts the same timeouts and max_request_body fields inside
its metrics config block; its timeouts.handler is the scrape timeout: a
slower scrape is answered 503 (promhttp gathers without consulting the
request context, so the handler timeout of promhttp is used instead of
the context deadline). appinit.Metrics returns bind errors synchronously;
callers must handle them and close its returned closer to stop the endpoint.