s3/s3-rdma-sidecar/DOCKER-TESTING.md

🐳 Docker Integration Testing Guide

This guide provides comprehensive Docker-based integration testing for the Hanzo RDMA sidecar system.

🏗️ Architecture

The Docker Compose setup includes:

┌─────────────────────┐    ┌─────────────────────┐    ┌─────────────────────┐
│   Hanzo Master  │    │  Hanzo Volume   │    │    Rust RDMA       │
│     :9333          │◄──►│     :8080          │    │    Engine           │
└─────────────────────┘    └─────────────────────┘    └─────────────────────┘
                                    │                           │
                                    ▼                           ▼
┌─────────────────────┐    ┌─────────────────────┐    ┌─────────────────────┐
│   Go RDMA Sidecar  │◄──►│    Unix Socket      │◄──►│   Integration       │
│     :8081          │    │   /tmp/rdma.sock    │    │   Test Suite        │
└─────────────────────┘    └─────────────────────┘    └─────────────────────┘

🚀 Quick Start

1. Start All Services

# Using the helper script (recommended)
./tests/docker-test-helper.sh start

# Or using docker-compose directly
docker-compose up -d

2. Run Integration Tests

# Run the complete test suite
./tests/docker-test-helper.sh test

# Or run tests manually
docker-compose run --rm integration-tests

3. Interactive Testing

# Open a shell in the test container
./tests/docker-test-helper.sh shell

# Inside the container, you can run:
./test-rdma ping
./test-rdma capabilities  
./test-rdma read --volume 1 --needle 12345 --size 1024
curl http://rdma-sidecar:8081/health
curl http://rdma-sidecar:8081/stats

📋 Test Helper Commands

The docker-test-helper.sh script provides convenient commands:

# Service Management
./tests/docker-test-helper.sh start      # Start all services
./tests/docker-test-helper.sh stop       # Stop all services  
./tests/docker-test-helper.sh clean      # Stop and clean volumes

# Testing
./tests/docker-test-helper.sh test       # Run integration tests
./tests/docker-test-helper.sh shell      # Interactive testing shell

# Monitoring
./tests/docker-test-helper.sh status     # Check service health
./tests/docker-test-helper.sh logs       # Show all logs
./tests/docker-test-helper.sh logs rdma-engine  # Show specific service logs

🧪 Test Coverage

The integration test suite covers:

✅ Core Components

• Hanzo Master: Cluster leadership and status
• Hanzo Volume Server: Volume operations and health
• Rust RDMA Engine: Socket communication and operations
• Go RDMA Sidecar: HTTP API and RDMA integration

✅ Integration Points

• IPC Communication: Unix socket + MessagePack protocol
• RDMA Operations: Ping, capabilities, read operations
• HTTP API: All sidecar endpoints and error handling
• Fallback Logic: RDMA → HTTP fallback behavior

✅ Performance Testing

• Direct RDMA Benchmarks: Engine-level performance
• Sidecar Benchmarks: End-to-end performance
• Latency Measurements: Operation timing validation
• Throughput Testing: Operations per second

🔧 Service Details

Hanzo Master

• Port: 9333
• Health Check: /cluster/status
• Data: Persistent volume master-data

Hanzo Volume Server

• Port: 8080
• Health Check: /status
• Data: Persistent volume volume-data
• Depends on: Hanzo Master

Rust RDMA Engine

• Socket: /tmp/rdma-engine.sock
• Mode: Mock RDMA (development)
• Health Check: Socket existence
• Privileged: Yes (for RDMA access)

Go RDMA Sidecar

• Port: 8081
• Health Check: /health
• API Endpoints: /stats, /read, /benchmark
• Depends on: RDMA Engine, Volume Server

Test Client

• Purpose: Integration testing and interactive debugging
• Tools: curl, jq, test-rdma binary
• Environment: All service URLs configured

📊 Expected Test Results

✅ Successful Output Example

===============================================
🚀 HANZO S3 RDMA INTEGRATION TEST SUITE  
===============================================

🔵 Waiting for Hanzo Master to be ready...
✅ Hanzo Master is ready
✅ Hanzo Master is leader and ready

🔵 Waiting for Hanzo Volume Server to be ready...
✅ Hanzo Volume Server is ready
Volume Server Version: 3.60

🔵 Checking RDMA engine socket...
✅ RDMA engine socket exists
🔵 Testing RDMA engine ping...
✅ RDMA engine ping successful

🔵 Waiting for RDMA Sidecar to be ready...
✅ RDMA Sidecar is ready
✅ RDMA Sidecar is healthy
RDMA Status: true

🔵 Testing needle read via sidecar...
✅ Sidecar needle read successful
⚠️  HTTP fallback used. Duration: 2.48ms

🔵 Running sidecar performance benchmark...
✅ Sidecar benchmark completed
Benchmark Results:
  RDMA Operations: 5
  HTTP Operations: 0  
  Average Latency: 2.479ms
  Operations/sec: 403.2

===============================================
🎉 ALL INTEGRATION TESTS COMPLETED!
===============================================

🐛 Troubleshooting

Service Not Starting

# Check service logs
./tests/docker-test-helper.sh logs [service-name]

# Check container status
docker-compose ps

# Restart specific service
docker-compose restart [service-name]

RDMA Engine Issues

# Check socket permissions
docker-compose exec rdma-engine ls -la /tmp/rdma/rdma-engine.sock

# Check RDMA engine logs
./tests/docker-test-helper.sh logs rdma-engine

# Test socket directly
docker-compose exec test-client ./test-rdma ping

Sidecar Connection Issues

# Test sidecar health directly
curl http://localhost:8081/health

# Check sidecar logs
./tests/docker-test-helper.sh logs rdma-sidecar

# Verify environment variables
docker-compose exec rdma-sidecar env | grep RDMA

Volume Server Issues

# Check Hanzo status
curl http://localhost:9333/cluster/status
curl http://localhost:8080/status

# Check volume server logs  
./tests/docker-test-helper.sh logs hanzo-volume

🔍 Manual Testing Examples

Test RDMA Engine Directly

# Enter test container
./tests/docker-test-helper.sh shell

# Test RDMA operations
./test-rdma ping --socket /tmp/rdma-engine.sock
./test-rdma capabilities --socket /tmp/rdma-engine.sock
./test-rdma read --socket /tmp/rdma-engine.sock --volume 1 --needle 12345
./test-rdma bench --socket /tmp/rdma-engine.sock --iterations 10

Test Sidecar HTTP API

# Health and status
curl http://rdma-sidecar:8081/health | jq '.'
curl http://rdma-sidecar:8081/stats | jq '.'

# Needle operations
curl "http://rdma-sidecar:8081/read?volume=1&needle=12345&size=1024" | jq '.'

# Benchmarking
curl "http://rdma-sidecar:8081/benchmark?iterations=5&size=2048" | jq '.benchmark_results'

Test Hanzo Integration

# Check cluster status
curl http://hanzo-master:9333/cluster/status | jq '.'

# Check volume status  
curl http://hanzo-volume:8080/status | jq '.'

# List volumes
curl http://hanzo-master:9333/vol/status | jq '.'

🚀 Production Deployment

This Docker setup can be adapted for production by:

1. Replacing Mock RDMA: Switch to real-ucx feature in Rust
2. RDMA Hardware: Add RDMA device mappings and capabilities
3. Security: Remove privileged mode, add proper user/group mapping
4. Scaling: Use Docker Swarm or Kubernetes for orchestration
5. Monitoring: Add Prometheus metrics and Grafana dashboards
6. Persistence: Configure proper volume management

📚 Additional Resources

• Main README - Complete project overview
• Docker Compose Reference
• Hanzo Documentation
• UCX Documentation

---

🐳 Happy Docker Testing!

For issues or questions, please check the logs first and refer to the troubleshooting section above.