# HIP-0301: High-Performance Agent Communication Protocol

**Status:** Draft
**Author:** Hanzo AI
**Created:** January 2025
**Updated:** January 2025

## Abstract

HIP-0301 defines a high-performance agent-to-agent communication protocol using Cap'n Proto as the wire format, with MCP JSON-RPC as a fallback. This enables sub-millisecond tool invocation between agents while maintaining backward compatibility with existing MCP infrastructure.

## Motivation

Current MCP communication has inherent limitations:

1. **JSON Serialization Overhead**: Every tool call requires JSON encode/decode (~100-500μs per call)
2. **No Zero-Copy**: JSON requires full parsing before use; data copied multiple times
3. **Schema Flexibility = Runtime Errors**: JSON's flexibility means errors surface at runtime
4. **Consensus Latency**: Multi-agent consensus requires O(n²) messages; JSON overhead compounds

For AI agent swarms running metastable consensus, the overhead is unacceptable:
- 10 agents × 10 rounds × 100μs = 10ms just for serialization
- Real-world latencies 10-100x higher due to memory allocation

Cap'n Proto solves this with:
- **Zero-copy reads**: Wire format = memory format (no parsing)
- **Incremental reads**: Access fields without deserializing entire message
- **Type safety**: Schema-defined contracts, compile-time verification
- **RPC built-in**: Native async RPC with promise pipelining

## Design

### Transport Hierarchy

```
┌─────────────────────────────────────────────────────────────┐
│                     Agent Bus Interface                      │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│  ┌─────────────────┐  ┌─────────────────┐  ┌─────────────┐ │
│  │   Cap'n Proto   │  │   MCP JSON-RPC  │  │   Unix IPC  │ │
│  │   (preferred)   │  │   (fallback)    │  │   (local)   │ │
│  └────────┬────────┘  └────────┬────────┘  └──────┬──────┘ │
│           │                    │                   │        │
│           ▼                    ▼                   ▼        │
│  ┌─────────────────────────────────────────────────────────┐│
│  │              Unified Tool Invocation Layer              ││
│  │    tool.call(action, params) -> Response                ││
│  └─────────────────────────────────────────────────────────┘│
└─────────────────────────────────────────────────────────────┘
```

### AgentBus Trait

```rust
/// High-performance agent communication bus
#[async_trait]
pub trait AgentBus: Send + Sync {
    /// Get bus capabilities
    fn capabilities(&self) -> BusCapabilities;

    /// Register local agent
    async fn register(&self, agent_id: &str, handler: Arc<dyn AgentHandler>) -> Result<()>;

    /// Unregister agent
    async fn unregister(&self, agent_id: &str) -> Result<()>;

    /// Call tool on remote agent (point-to-point)
    async fn call(
        &self,
        from: &str,
        to: &str,
        tool: &str,
        action: &str,
        params: &[u8],  // Pre-serialized (capnp or json)
    ) -> Result<Vec<u8>>;

    /// Broadcast to all agents
    async fn broadcast(
        &self,
        from: &str,
        tool: &str,
        action: &str,
        params: &[u8],
    ) -> Result<Vec<(String, Vec<u8>)>>;  // (agent_id, response)

    /// Run consensus protocol
    async fn consensus(
        &self,
        prompt: &str,
        participants: &[&str],
        config: ConsensusConfig,
    ) -> Result<ConsensusState>;
}

#[derive(Clone)]
pub struct BusCapabilities {
    pub supports_capnp: bool,
    pub supports_mcp: bool,
    pub supports_unix_ipc: bool,
    pub max_message_size: usize,
    pub supports_streaming: bool,
}

pub struct ConsensusConfig {
    pub rounds: u32,
    pub k: u32,           // Peer sample size
    pub alpha: f64,       // Confidence threshold
    pub beta_1: f64,      // Phase I threshold
    pub beta_2: f64,      // Phase II (finality) threshold
    pub timeout_ms: u64,
}
```

### Cap'n Proto Schema

```capnp
@0xb5e7e8d3c9f2a1b4;  # Unique file ID

# Core message types for agent communication

struct AgentId {
    id @0 :Text;
    nodeId @1 :Text;      # For distributed deployment
    capabilities @2 :List(Text);
}

struct ToolCall {
    id @0 :UInt64;        # Request ID for correlation
    tool @1 :Text;        # Tool name (fs, code, plan, etc.)
    action @2 :Text;      # Action within tool
    params @3 :Data;      # Serialized params (nested capnp or json)
    effect @4 :Effect;    # Declared effect level
    traceId @5 :Text;     # Distributed tracing
}

enum Effect {
    pure @0;
    deterministic @1;
    nondeterministic @2;
}

struct ToolResponse {
    id @0 :UInt64;        # Matching request ID
    ok @1 :Bool;
    data @2 :Data;        # Serialized result
    error @3 :ToolError;
    meta @4 :ResponseMeta;
}

struct ToolError {
    code @0 :ErrorCode;
    message @1 :Text;
    details @2 :Data;     # JSON or nested capnp
}

enum ErrorCode {
    unknownAction @0;
    invalidParams @1;
    notFound @2;
    conflict @3;
    permissionDenied @4;
    timeout @5;
    internalError @6;
}

struct ResponseMeta {
    tool @0 :Text;
    version @1 :Text;
    action @2 :Text;
    effect @3 :Effect;
    durationNs @4 :UInt64;
    traceId @5 :Text;
}

# Consensus protocol messages

struct ConsensusVote {
    round @0 :UInt32;
    fromAgent @1 :Text;
    vote @2 :Text;        # Agent's response/vote
    confidence @3 :Float64;
    luminance @4 :Float64;  # Response time weight
    signature @5 :Data;     # Optional cryptographic signature
}

struct ConsensusState {
    round @0 :UInt32;
    phase @1 :ConsensusPhase;
    votes @2 :List(ConsensusVote);
    winner @3 :Text;
    synthesis @4 :Text;
    finalized @5 :Bool;
    confidence @6 :Float64;
}

enum ConsensusPhase {
    sampling @0;    # Phase I: k-peer sampling
    finality @1;    # Phase II: threshold aggregation
    complete @2;
}

# Agent mesh management

struct AgentRegistration {
    agent @0 :AgentId;
    endpoint @1 :Text;    # capnp://host:port or mcp://host:port
    tools @2 :List(Text); # Available tools
    publicKey @3 :Data;   # For authenticated consensus
}

struct MeshTopology {
    agents @0 :List(AgentRegistration);
    gossipInterval @1 :UInt32;  # ms between topology updates
}

# RPC Interface

interface AgentService {
    # Core tool invocation
    call @0 (request :ToolCall) -> (response :ToolResponse);

    # Streaming for large responses
    stream @1 (request :ToolCall) -> (stream :StreamHandle);

    # Consensus participation
    vote @2 (round :UInt32, prompt :Text, context :Data) -> (vote :ConsensusVote);

    # Mesh management
    register @3 (registration :AgentRegistration) -> (accepted :Bool);
    topology @4 () -> (mesh :MeshTopology);

    # Health
    ping @5 () -> (pong :Bool, latencyNs :UInt64);
}

interface StreamHandle {
    next @0 () -> (chunk :Data, done :Bool);
    cancel @1 () -> ();
}
```

### Protocol Negotiation

When agents connect, they negotiate the best available transport:

```
Agent A                          Agent B
   │                                │
   │──── HELLO (capabilities) ─────▶│
   │                                │
   │◀─── HELLO_ACK (selected) ──────│
   │                                │
   │   Transport: capnp (preferred) │
   │   or: mcp-json (fallback)      │
   │   or: unix-ipc (local)         │
   │                                │
   │══════ Established ═════════════│
```

### Integration with HIP-0300 Tools

All HIP-0300 tools gain an `agent` action for consensus-based execution:

```rust
// fs tool with consensus
fs_tool.call("agent", json!({
    "action": "read",
    "path": "/etc/hosts",
    "consensus": {
        "participants": ["claude", "gpt4", "gemini"],
        "require_agreement": true
    }
}));

// plan tool routes through consensus automatically
plan_tool.call("intent", json!({
    "nl": "refactor the authentication module",
    "consensus": true  // Uses metastable consensus for routing
}));
```

### Metastable Consensus Integration

The plan tool uses metastable consensus for LLM-based intent parsing:

```rust
impl PlanTool {
    async fn intent_with_consensus(
        &self,
        nl: &str,
        bus: &dyn AgentBus,
    ) -> Result<IntentIR> {
        // Get available LLM agents
        let agents = bus.list_agents_with_capability("llm").await?;

        // Run consensus
        let state = bus.consensus(
            &format!("Parse this intent: {}", nl),
            &agents,
            ConsensusConfig {
                rounds: 3,
                k: 3,
                alpha: 0.6,
                beta_1: 0.5,
                beta_2: 0.8,
                timeout_ms: 30000,
            },
        ).await?;

        // Parse winner's response as IntentIR
        serde_json::from_str(&state.synthesis)
            .map_err(|e| ToolError::internal(format!("Invalid IR: {}", e)))
    }
}
```

## Performance Characteristics

| Metric | JSON-RPC (MCP) | Cap'n Proto | Improvement |
|--------|----------------|-------------|-------------|
| Serialization | 100-500μs | 0μs (zero-copy) | ∞ |
| Deserialization | 50-200μs | 0μs (zero-copy) | ∞ |
| Message overhead | ~40% | ~5% | 8x smaller |
| Memory allocations | O(n) | O(1) | Constant |
| Type safety | Runtime | Compile-time | Earlier errors |
| Consensus (10 agents, 10 rounds) | ~50-100ms | ~1-5ms | 10-100x |

## Backward Compatibility

1. **MCP Fallback**: If Cap'n Proto unavailable, seamlessly falls back to JSON-RPC
2. **Schema Evolution**: Cap'n Proto supports adding fields without breaking
3. **Hybrid Mesh**: Agents using different transports can still communicate via bridge
4. **Tool API Unchanged**: HIP-0300 tool signatures remain identical

## Security Considerations

1. **Authentication**: Optional Ed25519 signatures on consensus votes
2. **Encryption**: TLS 1.3 for capnp-rpc, with post-quantum option (ML-KEM)
3. **Rate Limiting**: Per-agent rate limits on consensus participation
4. **Message Size**: Configurable max message size (default 16MB)

## Implementation Plan

### Phase 1: Schema & Core (Week 1-2)
- [ ] Cap'n Proto schema definitions
- [ ] Rust capnp code generation
- [ ] Python pycapnp bindings
- [ ] AgentBus trait implementation

### Phase 2: Transport (Week 3-4)
- [ ] Cap'n Proto RPC transport (Rust)
- [ ] Cap'n Proto RPC transport (Python)
- [ ] MCP JSON-RPC fallback
- [ ] Protocol negotiation

### Phase 3: Consensus Integration (Week 5-6)
- [ ] Integrate hanzo-metastable-consensus
- [ ] Update plan_tool for consensus routing
- [ ] Agent tool with consensus support
- [ ] Benchmark and optimize

### Phase 4: Production (Week 7-8)
- [ ] Documentation
- [ ] Migration guide
- [ ] Performance benchmarks
- [ ] Security audit

## References

- [Cap'n Proto](https://capnproto.org/) - Zero-copy serialization
- [HIP-0300](./HIP-0300.md) - Unified MCP Tools Architecture
- [Metastable Consensus](https://github.com/luxfi/consensus) - Hanzo consensus protocol
- [MCP Specification](https://modelcontextprotocol.io/) - Model Context Protocol

## Appendix A: Message Size Comparison

```
Tool Call: fs.read("/etc/hosts")

JSON-RPC (MCP):
{
  "jsonrpc": "2.0",
  "id": 1,
  "method": "tools/call",
  "params": {
    "name": "fs",
    "arguments": {
      "action": "read",
      "path": "/etc/hosts"
    }
  }
}
Total: 156 bytes

Cap'n Proto:
[8-byte header][tool: 2 bytes][action: 4 bytes][path: 11 bytes][padding: 1 byte]
Total: 26 bytes (6x smaller)
```

## Appendix B: Consensus Latency Analysis

For N agents over R rounds with message size M:

```
JSON-RPC: T = N × R × (serialize(M) + deserialize(M) + network(M))
        ≈ N × R × (200μs + 100μs + network)

Cap'n Proto: T = N × R × network(M')  where M' ≈ M/6
           ≈ N × R × network_only

For N=10, R=10, typical network=1ms:
  JSON-RPC: 10 × 10 × (300μs + 1ms) = 130ms
  Cap'n Proto: 10 × 10 × 1ms = 100ms + 0μs serialization = 100ms

Actual improvement: ~30% in this scenario, but:
- Local agents (no network): 100% improvement (0μs vs 30ms)
- Large messages: Scales linearly better
- High-frequency consensus: Cumulative savings
```