mirror of
https://github.com/hanzo-docs/docs.git
synced 2026-10-29 07:01:44 +00:00
zeekay
cf191aedc2
- Update blog/[slug]/page.tsx to use getAllSlugs() instead of getSortedPosts() for static params - Sweep all blog MDX files to remove upstream model name references (brand policy compliance) - Add new blog posts: 7680-dim-embeddings, agent-nfts, bitdelta-behavioral-compression, case-for-decentralized-science - Delete gspo.mdx and qwen2.5-vl-32b.mdx (consolidated/removed content) - Remove packages/zenlm-models and packages/zenlm-ui entirely - Update lib/blog.ts, docs/server.ts, docs/browser.ts for blog infrastructure - Update pnpm-lock.yaml after package removal
78 lines
2.8 KiB
Plaintext
78 lines
2.8 KiB
Plaintext
---
|
||
title: "Training LLMs on Collective Intelligence"
|
||
description: "How we're approaching training data curation to capture humanity's collective intelligence."
|
||
date: "2021-07-22"
|
||
author: "Zach Kelling"
|
||
tags: ["Research", "Training"]
|
||
---
|
||
|
||
Language models are trained on text. That text represents the accumulated knowledge, reasoning, and creativity of countless individuals. Yet the curation process that selects training data receives surprisingly little attention.
|
||
|
||
## The Data Problem
|
||
|
||
Most large language models are trained on web scrapes filtered by simple heuristics. This approach has several issues:
|
||
|
||
1. **Quality variance** : Web content ranges from expert research to spam
|
||
2. **Hidden biases** : Filtering decisions embed value judgments
|
||
3. **Provenance opacity** : It’s unclear what’s included or excluded
|
||
4. **Legal ambiguity** : Copyright and consent questions remain unresolved
|
||
|
||
## Our Approach: Transparent Curation
|
||
|
||
At Zen, we’re taking a different path. Our data pipeline operates on three principles.
|
||
|
||
### Explicit Criteria
|
||
|
||
Every filtering decision has documented rationale. When we exclude content, we record why. When we weight certain sources higher, we explain the reasoning. This creates an auditable trail.
|
||
|
||
### Community Input
|
||
|
||
Data curation involves value judgments. What counts as “quality”? Which perspectives matter? These aren’t purely technical questions. We’re building mechanisms for community input into curation criteria.
|
||
|
||
### Provenance Tracking
|
||
|
||
Each training example links to its source with metadata about:
|
||
|
||
* Original publication context
|
||
* Author information (when available)
|
||
* Licensing terms
|
||
* Processing steps applied
|
||
|
||
## Technical Implementation
|
||
|
||
We’ve developed a pipeline that processes documents through several stages:
|
||
```
|
||
|
||
Source -> Extraction -> Deduplication -> Quality Scoring -> Attribution -> Storage
|
||
|
||
|
||
```
|
||
|
||
The quality scoring model itself is trained on human judgments, with explicit criteria:
|
||
|
||
* Factual accuracy (where verifiable)
|
||
* Reasoning coherence
|
||
* Writing clarity
|
||
* Information density
|
||
|
||
## Early Results
|
||
|
||
Our initial corpus contains 2.3 trillion tokens with full provenance tracking. Early experiments suggest that careful curation can match larger but noisier datasets:
|
||
|
||
Corpus| Tokens| Benchmark Score
|
||
---|---|---
|
||
Web-raw| 5T| 72.3
|
||
Web-filtered| 3T| 74.1
|
||
Zen-curated| 2.3T| 75.8
|
||
|
||
The numbers are preliminary, but the direction is clear: quality over quantity.
|
||
|
||
## What This Means
|
||
|
||
Training on collective intelligence isn’t just about scraping more data. It’s about respecting the sources, maintaining transparency, and involving the community in decisions that shape AI behavior.
|
||
|
||
We’ll publish our full curation methodology and tooling in the coming months.
|
||
|
||
* * *
|
||
|
||
_Zach Kelling is a co-founder of Zoo Labs Foundation._ |