python-sdk/pkg/hanzo-memory/LLM.md

# Hanzo Memory Service - LLM Context

## Project Overview

Hanzo Memory Service is a high-performance AI memory and knowledge management system built with:
- **InfinityDB**: Embedded vector database for fast similarity search
- **FastEmbed**: Local embedding generation (no API calls)
- **LiteLLM**: Universal LLM interface supporting 100+ providers
- **FastAPI**: Modern async web framework

## Key Architecture Decisions

### 1. InfinityDB over PostgreSQL/pgvector
- Embedded database (no separate server process)
- Optimized for vector similarity search
- Lightweight deployment with file-based storage
- Direct support for multimodal embeddings

### 2. FastEmbed for Local Embeddings
- No dependency on external embedding APIs
- Fast local inference using ONNX runtime
- Default model: BAAI/bge-small-en-v1.5 (384 dimensions)
- Supports custom models via model registry

### 3. LiteLLM for LLM Flexibility
- Single interface for all LLM providers
- Supports OpenAI, Anthropic, Ollama, Azure, etc.
- Easy switching between cloud and local models
- Automatic retry and fallback handling

## Core Components

### Database Schema (InfinityDB Tables)

1. **Projects Database**
   - `projects` table: User projects with metadata

2. **Memories Database**
   - `memories_{user_id}` tables: Per-user memory storage
   - Columns: memory_id, content, embedding, metadata, importance

3. **Knowledge Database**
   - `knowledge_bases` table: Knowledge base definitions
   - `facts_{kb_id}` tables: Facts with parent-child relationships

4. **Chats Database**
   - `chats_{user_id}` tables: Conversation history with embeddings

### Service Layer

1. **EmbeddingService**
   - Manages FastEmbed model lifecycle
   - Batch processing for efficiency
   - Similarity computation (cosine, dot, euclidean)

2. **LLMService**
   - LiteLLM integration for all LLM operations
   - Summarization with knowledge extraction
   - JSON-mode for structured outputs
   - PII stripping capabilities

3. **MemoryService**
   - Memory CRUD operations
   - Semantic search with optional LLM filtering
   - Importance scoring and metadata management

## API Design Patterns

### Authentication
- Bearer token or x-hanzo-api-key header
- Optional apikey field in JSON body (legacy support)
- DISABLE_AUTH environment variable for development

### Request/Response Models
- Lowercase field names for compatibility (userid, messagecontent)
- Pydantic models for validation and documentation
- Consistent error responses with status codes

### Unified Search
- Query embeddings generated locally
- Vector similarity search in InfinityDB
- Optional LLM re-ranking for relevance
- Project and session-based filtering

## Testing Strategy

### Unit Tests
- Service layer testing with mocked dependencies
- Embedding generation and similarity tests
- LLM response parsing and error handling

### Integration Tests
- FastAPI TestClient for endpoint testing
- InfinityDB operations with temporary databases
- End-to-end memory storage and retrieval

### Test Fixtures
- Temporary database paths
- Sample embeddings and content
- Mock LLM responses for deterministic tests

## Performance Optimizations

1. **Local Operations**
   - Embeddings generated in-process
   - No network latency for vector operations
   - Batch processing where possible

2. **Caching Strategy**
   - Optional Redis integration
   - Embedding cache for repeated content
   - LLM response caching for common queries

3. **Async Operations**
   - FastAPI async endpoints
   - Non-blocking database operations
   - Concurrent request handling

## Deployment Considerations

### Docker Deployment
- Multi-stage build for smaller images
- Non-root user for security
- Health checks for orchestration
- Volume mounting for data persistence

### Configuration Management
- Environment variables with HANZO_ prefix
- .env file support for local development
- Sensible defaults for all settings
- Model selection via environment

### Scaling Options
1. **Vertical Scaling**: Larger instances for more memory/CPU
2. **Horizontal Scaling**: Multiple instances with shared storage
3. **Edge Deployment**: Fully offline operation with local models

## Future Enhancements

1. **MCP Server Implementation**
   - Full Model Context Protocol support
   - Tool definitions for memory/knowledge operations
   - Integration with Claude Desktop and other MCP clients

2. **Advanced Features**
   - Multi-modal embeddings (images, audio)
   - Incremental learning and memory consolidation
   - Federated memory sharing between instances
   - Advanced graph operations for knowledge bases

3. **Performance Improvements**
   - GPU acceleration for embeddings
   - Streaming responses for large result sets
   - Distributed vector indices

## Common Patterns

### Adding New Endpoints
1. Define Pydantic models in `models/`
2. Implement service logic in `services/`
3. Add FastAPI endpoint in `server.py`
4. Write tests in `tests/`

### Extending Embedding Support
1. Update FastEmbed model in config
2. Adjust embedding dimensions
3. Regenerate existing embeddings if needed

### Custom LLM Integration
1. Configure LiteLLM model string
2. Set appropriate API base/key
3. Adjust temperature and token limits
4. Test JSON mode compatibility