Files
zeekay 520ce57a83 feat: Complete Gym training platform with BitDelta/DeltaQuant quantization
Major updates:
- Implemented BitDelta 1-bit quantization (10-25× memory reduction)
- Added DeltaQuant flexible quantization (INT2/4/8, Binary, Ternary)
- Created DeltaSoup for Byzantine-robust community aggregation
- Configured accurate Qwen3 models for Zen family (Nano, Eco, Coder, Omni, Next)
- Added GRPO/GSPO training algorithms with memory optimizations
- Simplified CLI from gym-cli to gym
- Updated WebUI with Zoo Labs black monochromatic theme
- Integrated Unsloth optimizations for 2-3× speedup
- Added comprehensive test suite for quantization methods

Zoo Labs Foundation Inc. - Copyright 2025
2025-09-25 13:40:56 -05:00

1.1 KiB

Zen Next Model (32B MoE)

Advanced reasoning and complex problem-solving

Overview

Zen Next represents cutting-edge capabilities for:

  • Advanced reasoning tasks
  • Complex problem-solving
  • Research applications
  • Enterprise solutions
  • High-accuracy requirements

Model Details

  • Base Model: Qwen/Qwen3-Next-32B-MoE-Instruct
  • Parameters: 32B (8B active per token via MoE)
  • Context Length: 4096 tokens
  • Memory Required: ~48GB VRAM

Training

Quick Start

# Train with GRPO algorithm
gym train models/next/configs/gspo_training.yaml

# DeepSpeed integration
gym train models/next/configs/gspo_training.yaml \
  --deepspeed configs/deepspeed/ds_z3_config.json

Configuration

  • Algorithm: GSPO (MoE-optimized for scale)
  • LoRA Rank: 64
  • Learning Rate: 1e-5
  • Batch Size: 1

Performance

  • MMLU: 81.2
  • HumanEval: 78.9
  • MT-Bench: 8.9
  • Speed: 30 tokens/sec

Use Cases

✓ Research tasks ✓ Complex analysis ✓ Strategic planning ✓ Scientific computing ✓ Advanced AI agents


Part of the Zen Model Family by Zoo Labs Foundation