python-sdk/docs/hip/HIP-0301.md

HIP-0301: High-Performance Agent Communication Protocol

Status: Draft Author: Hanzo AI Created: January 2025 Updated: January 2025

Abstract

HIP-0301 defines a high-performance agent-to-agent communication protocol using Cap'n Proto as the wire format, with MCP JSON-RPC as a fallback. This enables sub-millisecond tool invocation between agents while maintaining backward compatibility with existing MCP infrastructure.

Motivation

Current MCP communication has inherent limitations:

1. JSON Serialization Overhead: Every tool call requires JSON encode/decode (~100-500μs per call)
2. No Zero-Copy: JSON requires full parsing before use; data copied multiple times
3. Schema Flexibility = Runtime Errors: JSON's flexibility means errors surface at runtime
4. Consensus Latency: Multi-agent consensus requires O(n²) messages; JSON overhead compounds

For AI agent swarms running metastable consensus, the overhead is unacceptable:

• 10 agents × 10 rounds × 100μs = 10ms just for serialization
• Real-world latencies 10-100x higher due to memory allocation

Cap'n Proto solves this with:

• Zero-copy reads: Wire format = memory format (no parsing)
• Incremental reads: Access fields without deserializing entire message
• Type safety: Schema-defined contracts, compile-time verification
• RPC built-in: Native async RPC with promise pipelining

Design

Transport Hierarchy

┌─────────────────────────────────────────────────────────────┐
│                     Agent Bus Interface                      │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│  ┌─────────────────┐  ┌─────────────────┐  ┌─────────────┐ │
│  │   Cap'n Proto   │  │   MCP JSON-RPC  │  │   Unix IPC  │ │
│  │   (preferred)   │  │   (fallback)    │  │   (local)   │ │
│  └────────┬────────┘  └────────┬────────┘  └──────┬──────┘ │
│           │                    │                   │        │
│           ▼                    ▼                   ▼        │
│  ┌─────────────────────────────────────────────────────────┐│
│  │              Unified Tool Invocation Layer              ││
│  │    tool.call(action, params) -> Response                ││
│  └─────────────────────────────────────────────────────────┘│
└─────────────────────────────────────────────────────────────┘

AgentBus Trait

/// High-performance agent communication bus
#[async_trait]
pub trait AgentBus: Send + Sync {
    /// Get bus capabilities
    fn capabilities(&self) -> BusCapabilities;

    /// Register local agent
    async fn register(&self, agent_id: &str, handler: Arc<dyn AgentHandler>) -> Result<()>;

    /// Unregister agent
    async fn unregister(&self, agent_id: &str) -> Result<()>;

    /// Call tool on remote agent (point-to-point)
    async fn call(
        &self,
        from: &str,
        to: &str,
        tool: &str,
        action: &str,
        params: &[u8],  // Pre-serialized (capnp or json)
    ) -> Result<Vec<u8>>;

    /// Broadcast to all agents
    async fn broadcast(
        &self,
        from: &str,
        tool: &str,
        action: &str,
        params: &[u8],
    ) -> Result<Vec<(String, Vec<u8>)>>;  // (agent_id, response)

    /// Run consensus protocol
    async fn consensus(
        &self,
        prompt: &str,
        participants: &[&str],
        config: ConsensusConfig,
    ) -> Result<ConsensusState>;
}

#[derive(Clone)]
pub struct BusCapabilities {
    pub supports_capnp: bool,
    pub supports_mcp: bool,
    pub supports_unix_ipc: bool,
    pub max_message_size: usize,
    pub supports_streaming: bool,
}

pub struct ConsensusConfig {
    pub rounds: u32,
    pub k: u32,           // Peer sample size
    pub alpha: f64,       // Confidence threshold
    pub beta_1: f64,      // Phase I threshold
    pub beta_2: f64,      // Phase II (finality) threshold
    pub timeout_ms: u64,
}

Cap'n Proto Schema

@0xb5e7e8d3c9f2a1b4;  # Unique file ID

# Core message types for agent communication

struct AgentId {
    id @0 :Text;
    nodeId @1 :Text;      # For distributed deployment
    capabilities @2 :List(Text);
}

struct ToolCall {
    id @0 :UInt64;        # Request ID for correlation
    tool @1 :Text;        # Tool name (fs, code, plan, etc.)
    action @2 :Text;      # Action within tool
    params @3 :Data;      # Serialized params (nested capnp or json)
    effect @4 :Effect;    # Declared effect level
    traceId @5 :Text;     # Distributed tracing
}

enum Effect {
    pure @0;
    deterministic @1;
    nondeterministic @2;
}

struct ToolResponse {
    id @0 :UInt64;        # Matching request ID
    ok @1 :Bool;
    data @2 :Data;        # Serialized result
    error @3 :ToolError;
    meta @4 :ResponseMeta;
}

struct ToolError {
    code @0 :ErrorCode;
    message @1 :Text;
    details @2 :Data;     # JSON or nested capnp
}

enum ErrorCode {
    unknownAction @0;
    invalidParams @1;
    notFound @2;
    conflict @3;
    permissionDenied @4;
    timeout @5;
    internalError @6;
}

struct ResponseMeta {
    tool @0 :Text;
    version @1 :Text;
    action @2 :Text;
    effect @3 :Effect;
    durationNs @4 :UInt64;
    traceId @5 :Text;
}

# Consensus protocol messages

struct ConsensusVote {
    round @0 :UInt32;
    fromAgent @1 :Text;
    vote @2 :Text;        # Agent's response/vote
    confidence @3 :Float64;
    luminance @4 :Float64;  # Response time weight
    signature @5 :Data;     # Optional cryptographic signature
}

struct ConsensusState {
    round @0 :UInt32;
    phase @1 :ConsensusPhase;
    votes @2 :List(ConsensusVote);
    winner @3 :Text;
    synthesis @4 :Text;
    finalized @5 :Bool;
    confidence @6 :Float64;
}

enum ConsensusPhase {
    sampling @0;    # Phase I: k-peer sampling
    finality @1;    # Phase II: threshold aggregation
    complete @2;
}

# Agent mesh management

struct AgentRegistration {
    agent @0 :AgentId;
    endpoint @1 :Text;    # capnp://host:port or mcp://host:port
    tools @2 :List(Text); # Available tools
    publicKey @3 :Data;   # For authenticated consensus
}

struct MeshTopology {
    agents @0 :List(AgentRegistration);
    gossipInterval @1 :UInt32;  # ms between topology updates
}

# RPC Interface

interface AgentService {
    # Core tool invocation
    call @0 (request :ToolCall) -> (response :ToolResponse);

    # Streaming for large responses
    stream @1 (request :ToolCall) -> (stream :StreamHandle);

    # Consensus participation
    vote @2 (round :UInt32, prompt :Text, context :Data) -> (vote :ConsensusVote);

    # Mesh management
    register @3 (registration :AgentRegistration) -> (accepted :Bool);
    topology @4 () -> (mesh :MeshTopology);

    # Health
    ping @5 () -> (pong :Bool, latencyNs :UInt64);
}

interface StreamHandle {
    next @0 () -> (chunk :Data, done :Bool);
    cancel @1 () -> ();
}

Protocol Negotiation

When agents connect, they negotiate the best available transport:

Agent A                          Agent B
   │                                │
   │──── HELLO (capabilities) ─────▶│
   │                                │
   │◀─── HELLO_ACK (selected) ──────│
   │                                │
   │   Transport: capnp (preferred) │
   │   or: mcp-json (fallback)      │
   │   or: unix-ipc (local)         │
   │                                │
   │══════ Established ═════════════│

Integration with HIP-0300 Tools

All HIP-0300 tools gain an agent action for consensus-based execution:

// fs tool with consensus
fs_tool.call("agent", json!({
    "action": "read",
    "path": "/etc/hosts",
    "consensus": {
        "participants": ["claude", "gpt4", "gemini"],
        "require_agreement": true
    }
}));

// plan tool routes through consensus automatically
plan_tool.call("intent", json!({
    "nl": "refactor the authentication module",
    "consensus": true  // Uses metastable consensus for routing
}));

Metastable Consensus Integration

The plan tool uses metastable consensus for LLM-based intent parsing:

impl PlanTool {
    async fn intent_with_consensus(
        &self,
        nl: &str,
        bus: &dyn AgentBus,
    ) -> Result<IntentIR> {
        // Get available LLM agents
        let agents = bus.list_agents_with_capability("llm").await?;

        // Run consensus
        let state = bus.consensus(
            &format!("Parse this intent: {}", nl),
            &agents,
            ConsensusConfig {
                rounds: 3,
                k: 3,
                alpha: 0.6,
                beta_1: 0.5,
                beta_2: 0.8,
                timeout_ms: 30000,
            },
        ).await?;

        // Parse winner's response as IntentIR
        serde_json::from_str(&state.synthesis)
            .map_err(|e| ToolError::internal(format!("Invalid IR: {}", e)))
    }
}

Performance Characteristics

Metric	JSON-RPC (MCP)	Cap'n Proto	Improvement
Serialization	100-500μs	0μs (zero-copy)	∞
Deserialization	50-200μs	0μs (zero-copy)	∞
Message overhead	~40%	~5%	8x smaller
Memory allocations	O(n)	O(1)	Constant
Type safety	Runtime	Compile-time	Earlier errors
Consensus (10 agents, 10 rounds)	~50-100ms	~1-5ms	10-100x

Backward Compatibility

1. MCP Fallback: If Cap'n Proto unavailable, seamlessly falls back to JSON-RPC
2. Schema Evolution: Cap'n Proto supports adding fields without breaking
3. Hybrid Mesh: Agents using different transports can still communicate via bridge
4. Tool API Unchanged: HIP-0300 tool signatures remain identical

Security Considerations

1. Authentication: Optional Ed25519 signatures on consensus votes
2. Encryption: TLS 1.3 for capnp-rpc, with post-quantum option (ML-KEM)
3. Rate Limiting: Per-agent rate limits on consensus participation
4. Message Size: Configurable max message size (default 16MB)

Implementation Plan

Phase 1: Schema & Core (Week 1-2)

• Cap'n Proto schema definitions
• Rust capnp code generation
• Python pycapnp bindings
• AgentBus trait implementation

Phase 2: Transport (Week 3-4)

• Cap'n Proto RPC transport (Rust)
• Cap'n Proto RPC transport (Python)
• MCP JSON-RPC fallback
• Protocol negotiation

Phase 3: Consensus Integration (Week 5-6)

• Integrate hanzo-metastable-consensus
• Update plan_tool for consensus routing
• Agent tool with consensus support
• Benchmark and optimize

Phase 4: Production (Week 7-8)

• Documentation
• Migration guide
• Performance benchmarks
• Security audit

References

• Cap'n Proto - Zero-copy serialization
• HIP-0300 - Unified MCP Tools Architecture
• Metastable Consensus - Hanzo consensus protocol
• MCP Specification - Model Context Protocol

Appendix A: Message Size Comparison

Tool Call: fs.read("/etc/hosts")

JSON-RPC (MCP):
{
  "jsonrpc": "2.0",
  "id": 1,
  "method": "tools/call",
  "params": {
    "name": "fs",
    "arguments": {
      "action": "read",
      "path": "/etc/hosts"
    }
  }
}
Total: 156 bytes

Cap'n Proto:
[8-byte header][tool: 2 bytes][action: 4 bytes][path: 11 bytes][padding: 1 byte]
Total: 26 bytes (6x smaller)

Appendix B: Consensus Latency Analysis

For N agents over R rounds with message size M:

JSON-RPC: T = N × R × (serialize(M) + deserialize(M) + network(M))
        ≈ N × R × (200μs + 100μs + network)

Cap'n Proto: T = N × R × network(M')  where M' ≈ M/6
           ≈ N × R × network_only

For N=10, R=10, typical network=1ms:
  JSON-RPC: 10 × 10 × (300μs + 1ms) = 130ms
  Cap'n Proto: 10 × 10 × 1ms = 100ms + 0μs serialization = 100ms

Actual improvement: ~30% in this scenario, but:
- Local agents (no network): 100% improvement (0μs vs 30ms)
- Large messages: Scales linearly better
- High-frequency consensus: Cumulative savings