20 KiB
hip, title, author, type, category, status, created
| hip | title | author | type | category | status | created |
|---|---|---|---|---|---|---|
| 0028 | Key-Value Store Standard | Hanzo AI Team | Standards Track | Infrastructure | Active | 2025-01-15 |
HIP-0028: Key-Value Store Standard
Abstract
This proposal defines the standard for Hanzo KV, the high-performance key-value store
that serves as the shared caching, session, pub/sub, and streaming backbone for all
services in the Hanzo ecosystem. Hanzo KV is built on KV 8.1, the Linux Foundation
fork of Redis, and is distributed as ghcr.io/hanzoai/kv:latest. It exposes the RESP3
wire protocol on port 6379 and is a drop-in replacement for any KV client.
Repository: github.com/hanzoai/kv
Port: 6379
Docker: ghcr.io/hanzoai/kv:latest and docker.io/hanzoai/kv:latest
License: BSD-3-Clause
Motivation
Every service in the Hanzo ecosystem -- IAM, LLM Gateway, Cloud, Chat, Commerce, Bot, Analytics, Zen -- needs a fast shared store for at least one of the following:
- Session and token caching: OAuth tokens, rate-limit counters, CSRF nonces
- Pub/Sub messaging: real-time event propagation between services
- Streams: append-only logs for audit trails and event sourcing
- Ephemeral state: inference request queues, job locks, circuit-breaker state
- Leaderboard/sorted-set operations: billing rank, usage tracking
Previously, the Hanzo infrastructure relied on the Bitnami KV Helm chart deployed via
helm install redis bitnami/redis. This worked, but introduced three problems:
- Licensing risk: KV Labs changed KV to a dual-license model (RSALv2 + SSPLv1) in March 2024. Both licenses restrict how cloud providers and SaaS platforms can distribute KV. For an infrastructure company like Hanzo that ships managed services, this is a direct legal exposure.
- Operational opacity: The Bitnami chart bundles a metrics sidecar (redis-exporter), init containers, and Sentinel by default. When any of these sidecars fail (e.g., the exporter cannot authenticate to a password-protected instance), the entire pod enters CrashLoopBackOff and the root cause is obscured.
- Image bloat: The Bitnami KV image is ~150MB compressed. A minimal Alpine-based KV image is ~12MB. In a cluster with rolling updates, smaller images mean faster pulls and shorter disruption windows.
Hanzo KV solves all three by replacing the entire Bitnami stack with a single, purpose-built container image based on KV.
Design Philosophy
This section explains every major design decision and why the alternatives were rejected. Infrastructure choices compound -- a wrong call here propagates to every service that touches KV. Each heading below addresses one decision.
Why Single Instance over Cluster Mode
Hanzo KV currently runs as a single-instance StatefulSet with 2Gi of PVC storage and a 2Gi memory limit. This is a deliberate choice, not a shortcut.
Scale math: Our current production dataset (sessions, rate-limit counters, cache entries across all services) occupies approximately 400MB of memory. Even with 10x growth, we stay under 4GB. A single KV instance on modern hardware can saturate a 10Gbps NIC at ~1.2 million ops/sec. Our peak observed throughput is approximately 8,000 ops/sec. We are three orders of magnitude below the single-node ceiling.
Cluster complexity: KV Cluster (and by extension KV Cluster) introduces hash slots, cross-slot restrictions on multi-key operations, MOVED/ASK redirects, and cluster bus gossip traffic. Every KV client library must understand cluster topology. Some operations (MULTI/EXEC across slots, Lua scripts touching multiple keys on different slots) simply do not work. This complexity buys horizontal scaling we do not need.
Failure modes: A single instance has exactly one failure mode -- the pod dies and restarts. With AOF persistence, data loss on restart is bounded to the last fsync interval (1 second by default). A cluster has N failure modes: split-brain during network partition, slot migration failures, gossip protocol desynchronization, and partial availability when a master is down and its replica has not yet been promoted.
Vertical ceiling: Kubernetes nodes support up to 64GB of memory. We can scale the KV StatefulSet to 32GB before even considering cluster mode. When we reach that point (which would imply ~80x current load), we will revisit with a separate HIP.
Why We Removed the Metrics Sidecar
The Bitnami KV chart ships with a redis-exporter sidecar that scrapes INFO output
and exposes Prometheus metrics on port 9121. When we migrated to Hanzo KV with password
authentication, the exporter sidecar could not authenticate because it expected the
password in a different environment variable format than our secret layout provided.
Rather than debug the exporter's authentication logic and add another secret reference, we removed the sidecar entirely. The reasoning:
- KV's built-in
INFOcommand already provides all metrics (memory, connections, keyspace, replication, persistence) in a machine-parseable format. - For our current scale,
kubectl execinto the pod and runningkv-cli INFOis sufficient for debugging. - When we need continuous Prometheus metrics, we will deploy
oliver006/redis_exporteras a separate Deployment (not a sidecar) with its own authentication config, decoupled from the KV pod lifecycle.
Principle: a database pod should contain exactly one process -- the database. Every sidecar is a potential crash-loop vector that takes the database down with it.
Specification
Wire Protocol
Hanzo KV implements RESP3 (REdis Serialization Protocol version 3) as defined by the KV protocol specification. All commands from the KV 7.2 command set are supported. Any client library that speaks RESP2 or RESP3 is compatible.
Connection Parameters
host: localhost
port: 6379
password: <from K8s secret "redis", key "redis-password">
db: 0 # default database
protocol: resp3 # RESP3 preferred, RESP2 accepted
tls: false # intra-cluster, TLS not required
Client Connection String
Services should construct their connection URL as:
redis://:${REDIS_PASSWORD}@redis-master:6379/0
Or for explicit host within the hanzo namespace:
redis://:${REDIS_PASSWORD}@localhost:6379/0
Configuration Reference
The production kv.conf (mounted from ConfigMap):
# Persistence: AOF only, no RDB snapshots
appendonly yes
save ""
# Eviction: LRU when memory limit is reached
maxmemory-policy allkeys-lru
# Safety: disable destructive bulk operations
rename-command FLUSHDB ""
rename-command FLUSHALL ""
Additional settings applied via container command-line arguments:
--requirepass $(REDIS_PASSWORD) # authentication
--dir /data # persistence directory
--bind 0.0.0.0 # accept connections on all interfaces
--maxmemory-policy allkeys-lru # eviction policy (also in kv.conf for safety)
--protected-mode no # allow non-loopback connections (K8s networking)
Health Checks
Readiness probe (is the instance ready to accept commands?):
exec:
command: ["sh", "-c", "kv-cli -a \"$REDIS_PASSWORD\" ping | grep -q PONG"]
initialDelaySeconds: 5
periodSeconds: 10
failureThreshold: 3
Liveness probe (is the instance alive and not deadlocked?):
exec:
command: ["sh", "-c", "kv-cli -a \"$REDIS_PASSWORD\" ping | grep -q PONG"]
initialDelaySeconds: 15
periodSeconds: 30
failureThreshold: 5
The liveness probe has a longer initialDelaySeconds and failureThreshold to avoid
killing a pod that is replaying a large AOF on startup.
Resource Allocation
resources:
requests:
cpu: 100m
memory: 256Mi
limits:
cpu: 1000m
memory: 2Gi
The memory limit (2Gi) acts as a hard ceiling. Combined with allkeys-lru, KV will
evict the least-recently-used keys when approaching this limit rather than crashing with
an OOM error.
Storage
volumeClaimTemplates:
- metadata:
name: redis-data
spec:
accessModes: [ReadWriteOnce]
resources:
requests:
storage: 2Gi
The PVC stores the AOF file. With our current workload, the AOF (after automatic compaction) stays under 100MB. The 2Gi allocation provides 20x headroom.
Implementation
Container Image
The Dockerfile is minimal by design:
ARG KV_VERSION=8.1
FROM valkey/valkey:${KV_VERSION}-alpine AS base
FROM base
LABEL maintainer="[email protected]"
LABEL org.opencontainers.image.source="https://github.com/hanzoai/kv"
LABEL org.opencontainers.image.description="Hanzo KV - High-performance key-value store"
LABEL org.opencontainers.image.vendor="Hanzo AI"
# Install Hanzo KV CLI tools
# Primary names are kv-* ; legacy valkey-* names remain as symlinks
RUN cp /usr/local/bin/valkey-server /usr/local/bin/kv-server \
&& cp /usr/local/bin/valkey-cli /usr/local/bin/kv-cli \
&& ln -sf /usr/local/bin/kv-cli /usr/local/bin/kv \
&& cp /usr/local/bin/valkey-sentinel /usr/local/bin/kv-sentinel 2>/dev/null; \
cp /usr/local/bin/valkey-benchmark /usr/local/bin/kv-benchmark 2>/dev/null; \
cp /usr/local/bin/valkey-check-aof /usr/local/bin/kv-check-aof 2>/dev/null; \
cp /usr/local/bin/valkey-check-rdb /usr/local/bin/kv-check-rdb 2>/dev/null; \
true
EXPOSE 6379
HEALTHCHECK --interval=15s --timeout=3s --start-period=10s --retries=3 \
CMD kv ping | grep -q PONG || exit 1
ENTRYPOINT ["kv-server"]
CMD ["--bind", "0.0.0.0", "--dir", "/data", \
"--maxmemory-policy", "allkeys-lru", "--protected-mode", "no"]
Key points:
- Base image:
valkey/valkey:8.1-alpine(~12MB compressed) - CLI renaming: All KV binaries are copied to
kv-*names. The originalvalkey-*names remain as the originals. This gives operators a clean Hanzo-branded CLI while maintaining compatibility with scripts that referencevalkey-cli. - No custom compilation: We use the upstream KV binary as-is. Custom patches would create a maintenance burden and diverge from upstream security fixes.
CLI Tools
| Command | Description |
|---|---|
kv |
Interactive CLI (symlink to kv-cli) |
kv-server |
Start KV server |
kv-cli |
Command-line client |
kv-sentinel |
High-availability sentinel |
kv-benchmark |
Performance benchmarking tool |
kv-check-aof |
AOF file integrity checker |
kv-check-rdb |
RDB file integrity checker |
CI/CD Pipeline
The deploy workflow (.github/workflows/deploy.yml) has two stages:
Stage 1: Build
- Checkout source from
github.com/hanzoai/kv - Authenticate to Hanzo KMS (Universal Auth) to fetch CI secrets
- Build multi-arch image (
linux/amd64,linux/arm64) via Docker Buildx - Push to GHCR (
ghcr.io/hanzoai/kv) with tags:latest, git SHA, semver - Push to Docker Hub (
docker.io/hanzoai/kv) as fallback (continue-on-error)
Stage 2: Deploy (main branch only)
- Authenticate to Hanzo KMS for DigitalOcean API token
- Configure
kubectlforthe clustercluster viadoctl - Rolling update:
kubectl set image statefulset/redis-master kv=ghcr.io/hanzoai/kv:latest - Wait for rollout:
kubectl rollout status statefulset/redis-master --timeout=120s
Trigger conditions: push to main, tag push (v*), or manual workflow_dispatch.
K8s Manifest Structure
All manifests live in universe/infra/k8s/kv/ and are aggregated via Kustomize:
# kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- statefulset.yaml
- service.yaml
- secret.yaml
- configmap.yaml
Migration from Bitnami KV
The migration from the Bitnami KV Helm chart to Hanzo KV was performed as follows:
- Scale down Bitnami:
helm uninstall redisremoves the Deployment and Service but preserves the PVC (Helm defaultresourcePolicy: keep). - Apply Hanzo KV manifests: The StatefulSet uses the same PVC name (
redis-data), same Service name (redis-master), and same secret name (redis). This means the new pod attaches to the existing PVC with all data intact. - Verify data:
kv-cli -a "$REDIS_PASSWORD" DBSIZEconfirms key count matches pre-migration. - Remove Helm artifacts: Clean up orphaned Helm release secrets.
The migration is zero-downtime because the Service name and selector labels are preserved. Client connections fail for the ~30 seconds between the old pod terminating and the new pod passing its readiness probe, which is within the retry tolerance of all Hanzo services.
Client SDKs
| Language | Package | Install |
|---|---|---|
| Python | hanzo-kv | pip install hanzo-kv |
| Go | hanzo/kv-go | go get github.com/hanzoai/kv-go |
| Node.js | @hanzo/kv | npm install @hanzo/kv |
All three are thin wrappers around standard KV client libraries (redis-py,
go-redis, ioredis) with Hanzo-specific defaults (connection URL construction,
KMS secret resolution, structured logging). Any vanilla KV client works equally well.
Security
Authentication
All connections require a password. The password is stored in a K8s Secret:
apiVersion: v1
kind: Secret
metadata:
name: redis
namespace: hanzo
type: Opaque
stringData:
redis-password: "<generated-value>"
Services receive the password via environment variable injection from this secret.
The secret name (redis) and key (redis-password) match the Bitnami convention to
avoid changing every service deployment manifest.
In production, this secret is synced from Hanzo KMS (kms.hanzo.ai) via the KMS
Operator. The plaintext value in the manifest is a bootstrap default that gets
overwritten on first KMS sync.
Network Isolation
- Service type: ClusterIP (no external exposure)
- No NodePort, no LoadBalancer, no Ingress
- Only pods within the
hanzonamespace (or with appropriate NetworkPolicy) can reach port 6379 - The
--protected-mode noflag is safe because the pod is never exposed outside the cluster. Protected mode is a KV safety net for instances accidentally exposed to the internet without a password; our instance has both network isolation and a password.
Dangerous Command Disablement
As specified in the Configuration section, FLUSHDB and FLUSHALL are renamed to empty
strings (disabled). Additional commands to consider disabling in future:
DEBUG-- can crash the server or dump memoryCONFIG-- can change runtime settings (e.g., disable authentication)SHUTDOWN-- can stop the server
These are not currently disabled because they are useful for debugging in a cluster
environment where only operators have kubectl exec access.
TLS
TLS is available in KV 8.1 but not enabled for intra-cluster communication. The reasoning:
- All traffic stays within the Kubernetes VPC, encrypted at the network layer by DigitalOcean
- TLS adds ~15% latency overhead on every command due to encryption/decryption
- The threat model (attacker with VPC access) implies they already have
kubectlaccess and can read secrets directly
If we add external replication (e.g., cross-cluster) or expose KV outside the VPC, TLS
will be enabled via --tls-port 6380 --tls-cert-file --tls-key-file --tls-ca-cert-file.
Memory Limits
The 2Gi memory limit prevents a runaway client from consuming all node memory and
triggering the Linux OOM killer (which would kill the KV process and potentially other
pods on the same node). With allkeys-lru, KV gracefully evicts cold keys instead
of refusing writes or crashing.
Consumers
Services in the Hanzo ecosystem that connect to KV:
| Service | Use Case | Key Pattern |
|---|---|---|
| IAM (hanzo.id) | Session tokens, OAuth state | iam:session:*, iam:oauth:* |
| LLM Gateway | Rate limiting, response cache | llm:rate:*, llm:cache:* |
| Cloud | Job queues, inference state | cloud:job:*, cloud:inf:* |
| Console | Session cache | console:session:* |
| Chat | Conversation state, pub/sub | chat:conv:*, chat:stream:* |
| Bot | Command state, cooldowns | bot:state:*, bot:cd:* |
| Analytics | Event buffering | analytics:buf:* |
| Zen | Model routing cache | zen:route:* |
| Commerce | Cart state, rate limits | commerce:cart:* |
Key Namespace Convention
All keys SHOULD be prefixed with <service>:<category>:<id>. This enables:
- Per-service monitoring via
kv-cli --statorSCANwith pattern matching - Targeted eviction of one service's keys without affecting others
- Clear ownership when debugging unexpected key growth
Monitoring
Built-in Metrics
KV's INFO command provides comprehensive metrics without any sidecar:
# Memory usage
kv-cli -a "$REDIS_PASSWORD" INFO memory
# Client connections
kv-cli -a "$REDIS_PASSWORD" INFO clients
# Keyspace statistics
kv-cli -a "$REDIS_PASSWORD" INFO keyspace
# Persistence status
kv-cli -a "$REDIS_PASSWORD" INFO persistence
# All metrics
kv-cli -a "$REDIS_PASSWORD" INFO all
Key Metrics to Watch
| Metric | Warning Threshold | Critical Threshold |
|---|---|---|
used_memory |
> 1.5Gi (75% of limit) | > 1.8Gi (90%) |
connected_clients |
> 100 | > 500 |
evicted_keys |
> 0 (indicates memory pressure) | > 1000/min |
rejected_connections |
> 0 | > 10/min |
aof_last_bgrewrite_status |
err |
- |
instantaneous_ops_per_sec |
> 50,000 | > 100,000 |
Future: Prometheus Integration
When continuous monitoring is needed, deploy oliver006/redis_exporter as a standalone
Deployment in the hanzo namespace:
apiVersion: apps/v1
kind: Deployment
metadata:
name: kv-exporter
namespace: hanzo
spec:
replicas: 1
template:
spec:
containers:
- name: exporter
image: oliver006/redis_exporter:latest
env:
- name: REDIS_ADDR
value: redis-master:6379
- name: REDIS_PASSWORD
valueFrom:
secretKeyRef:
name: redis
key: redis-password
ports:
- containerPort: 9121
This runs as a separate pod, not a sidecar. If the exporter crashes, KV is unaffected.
Reference Implementation
Repository: github.com/hanzoai/kv
Key Files:
Dockerfile-- Multi-arch container image based on KV 8.1 Alpine.github/workflows/deploy.yml-- CI/CD: build, push to GHCR/Docker Hub, deploy to K8s.github/workflows/ci.yml-- Upstream KV test suitevalkey.conf-- Full reference configuration (upstream defaults)sentinel.conf-- Sentinel configuration for HA deployments
K8s Manifests (universe/infra/k8s/kv/):
statefulset.yaml-- StatefulSetredis-masterwith PVC and health checksservice.yaml-- ClusterIP Service on port 6379configmap.yaml--kv.conf(AOF, eviction policy, disabled commands)secret.yaml-- KV-compatible password secretkustomization.yaml-- Kustomize aggregation
Status: Implemented and running in production on the cluster (24.199.76.156)
References
- HIP-0: Hanzo AI Architecture Framework
- HIP-4: LLM Gateway
- KV Project -- Linux Foundation fork of KV
- Redis License Change Announcement -- March 2024
- KV 8.1 Release Notes
- RESP3 Protocol Specification
Copyright
Copyright and related rights waived via CC0.