s3/s3-rdma-sidecar/CORRECT-SIDECAR-APPROACH.md

✅ Correct RDMA Sidecar Approach - Simple Parameter-Based

🎯 You're Right - Simplified Architecture

The RDMA sidecar should be simple and just take the volume server address as a parameter. The volume lookup complexity should stay in s3 mount, not in the sidecar.

🏗️ Correct Architecture

1. s3 mount (Client Side) - Does Volume Lookup

// File: s3/mount/filehandle_read.go (integration point)
func (fh *FileHandle) tryRDMARead(ctx context.Context, buff []byte, offset int64) (int64, int64, error) {
    entry := fh.GetEntry()
    
    for _, chunk := range entry.GetEntry().Chunks {
        if offset >= chunk.Offset && offset < chunk.Offset+int64(chunk.Size) {
            // Parse chunk info
            volumeID, needleID, cookie, err := ParseFileId(chunk.FileId)
            if err != nil {
                return 0, 0, err
            }
            
            // 🔍 VOLUME LOOKUP (in s3 mount, not sidecar)
            volumeServerAddr, err := fh.wfs.lookupVolumeServer(ctx, volumeID)
            if err != nil {
                return 0, 0, err
            }
            
            // 🚀 SIMPLE RDMA REQUEST WITH VOLUME SERVER PARAMETER
            data, isRDMA, err := fh.wfs.rdmaClient.ReadNeedleFromServer(
                ctx, volumeServerAddr, volumeID, needleID, cookie, chunkOffset, readSize)
            
            return int64(copy(buff, data)), time.Now().UnixNano(), nil
        }
    }
}

2. RDMA Mount Client - Passes Volume Server Address

// File: s3/mount/rdma_client.go (modify existing)
func (c *RDMAMountClient) ReadNeedleFromServer(ctx context.Context, volumeServerAddr string, volumeID uint32, needleID uint64, cookie uint32, offset, size uint64) ([]byte, bool, error) {
    // Simple HTTP request with volume server as parameter
    reqURL := fmt.Sprintf("http://%s/rdma/read", c.sidecarAddr)
    
    requestBody := map[string]interface{}{
        "volume_server": volumeServerAddr,  // ← KEY: Pass volume server address
        "volume_id":     volumeID,
        "needle_id":     needleID,
        "cookie":        cookie,
        "offset":        offset,
        "size":          size,
    }
    
    // POST request with volume server parameter
    jsonBody, err := json.Marshal(requestBody)
    if err != nil {
        return nil, false, fmt.Errorf("failed to marshal request body: %w", err)
    }
    resp, err := c.httpClient.Post(reqURL, "application/json", bytes.NewBuffer(jsonBody))
    if err != nil {
        return nil, false, fmt.Errorf("http post to sidecar: %w", err)
    }
}

3. RDMA Sidecar - Simple, No Lookup Logic

// File: s3-rdma-sidecar/cmd/demo-server/main.go
func (s *DemoServer) rdmaReadHandler(w http.ResponseWriter, r *http.Request) {
    if r.Method != http.MethodPost {
        http.Error(w, "Method not allowed", http.StatusMethodNotAllowed)
        return
    }
    
    // Parse request body
    var req struct {
        VolumeServer string `json:"volume_server"`  // ← Receive volume server address
        VolumeID     uint32 `json:"volume_id"`
        NeedleID     uint64 `json:"needle_id"`
        Cookie       uint32 `json:"cookie"`
        Offset       uint64 `json:"offset"`
        Size         uint64 `json:"size"`
    }
    
    if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
        http.Error(w, "Invalid request", http.StatusBadRequest)
        return
    }
    
    s.logger.WithFields(logrus.Fields{
        "volume_server": req.VolumeServer,  // ← Use provided volume server
        "volume_id":     req.VolumeID,
        "needle_id":     req.NeedleID,
    }).Info("📖 Processing RDMA read with volume server parameter")
    
    // 🚀 SIMPLE: Use the provided volume server address
    // No complex lookup logic needed!
    resp, err := s.rdmaClient.ReadFromVolumeServer(r.Context(), req.VolumeServer, req.VolumeID, req.NeedleID, req.Cookie, req.Offset, req.Size)
    
    if err != nil {
        http.Error(w, fmt.Sprintf("RDMA read failed: %v", err), http.StatusInternalServerError)
        return
    }
    
    // Return binary data
    w.Header().Set("Content-Type", "application/octet-stream")
    w.Header().Set("X-RDMA-Used", "true")
    w.Write(resp.Data)
}

4. Volume Lookup in s3 mount (Where it belongs)

// File: s3/mount/s3fs.go (add method)
func (wfs *WFS) lookupVolumeServer(ctx context.Context, volumeID uint32) (string, error) {
    // Use existing Hanzo volume lookup logic
    vid := fmt.Sprintf("%d", volumeID)
    
    // Query master server for volume location
    locations, err := operation.LookupVolumeId(wfs.getMasterFn(), wfs.option.GrpcDialOption, vid)
    if err != nil {
        return "", fmt.Errorf("volume lookup failed: %w", err)
    }
    
    if len(locations.Locations) == 0 {
        return "", fmt.Errorf("no locations found for volume %d", volumeID)
    }
    
    // Return first available location (or implement smart selection)
    return locations.Locations[0].Url, nil
}

🎯 Key Differences from Over-Complicated Approach

❌ Over-Complicated (What I Built Before):

• ❌ Sidecar does volume lookup
• ❌ Sidecar has master client integration
• ❌ Sidecar has volume location caching
• ❌ Sidecar forwards requests to remote sidecars
• ❌ Complex distributed logic in sidecar

✅ Correct Simple Approach:

• ✅ s3 mount does volume lookup (where it belongs)
• ✅ s3 mount passes volume server address to sidecar
• ✅ Sidecar is simple and stateless
• ✅ Sidecar just does local RDMA read for given server
• ✅ No complex distributed logic in sidecar

🚀 Request Flow (Corrected)

1. User Application → read() system call
2. FUSE → s3 mount WFS.Read()
3. s3 mount → Volume lookup: "Where is volume 7?"
4. Hanzo Master → "Volume 7 is on server-B:8080"
5. s3 mount → HTTP POST to sidecar: {volume_server: "server-B:8080", volume: 7, needle: 12345}
6. RDMA Sidecar → Connect to server-B:8080, do local RDMA read
7. RDMA Engine → Direct memory access to volume file
8. Response → Binary data back to s3 mount → user

📝 Implementation Changes Needed

1. Simplify Sidecar (Remove Complex Logic)

• Remove DistributedRDMAClient
• Remove volume lookup logic
• Remove master client integration
• Keep simple RDMA engine communication

2. Add Volume Lookup to s3 mount

• Add lookupVolumeServer() method to WFS
• Modify RDMAMountClient to accept volume server parameter
• Integrate with existing Hanzo volume lookup

3. Simple Sidecar API

POST /rdma/read
{
  "volume_server": "server-B:8080",
  "volume_id": 7,
  "needle_id": 12345,
  "cookie": 0,
  "offset": 0,
  "size": 4096
}

✅ Benefits of Simple Approach

• 🎯 Single Responsibility: Sidecar only does RDMA, s3 mount does lookup
• 🔧 Maintainable: Less complex logic in sidecar
• ⚡ Performance: No extra network hops for volume lookup
• 🏗️ Clean Architecture: Separation of concerns
• 🐛 Easier Debugging: Clear responsibility boundaries

You're absolutely right - this is much cleaner! The sidecar should be a simple RDMA accelerator, not a distributed system coordinator.