mirror of
https://github.com/hanzoai/brain.git
synced 2026-10-30 18:46:38 +00:00
zeekay
237a080a80
Studied fortemi/fortemi @ v2026.5.0 and produced PARITY.md — every
algorithm Fortémi ships has a row mapping to a Hanzo destination
(this repo, a sibling hanzo/* repo, or a tracked planned module).
Landed under packages/memory/, all backed by 21 passing tests in
algorithms.test.ts:
- fusion.ts RRF (k=20 BEIR), RSF, adaptive k, adaptive weights
- rerank.ts Maximal Marginal Relevance (Carbonell & Goldstein 1998)
- dedup.ts chunk-aware deduplication (best per chain)
- script.ts Unicode script detection — Latin/CJK/emoji/Cyrillic/
Arabic/Hebrew/Greek/Devanagari
- embed.ts embedding model registry, asymmetric E5 prefixes,
Matryoshka truncation, coarse-dim picker
- temporal.ts UUIDv7 floor/ceiling + named ranges
- captions.ts WebVTT / SRT / RTTM renderers
Exposed from @hanzo/bot-memory v1.4.3.
Remaining gaps tracked in PARITY.md: FTS feature flags, federated search,
runtime config hot-swap, capabilities/task selector, eval harness, SKOS,
FAIR export, SNN/PFNET/Louvain graph maintenance, LLM-typed graph links,
extraction registry — plus sibling-repo wire-ups (search/vector/crypto/
age/replicate/insights/agents).
46 lines
1.4 KiB
TypeScript
46 lines
1.4 KiB
TypeScript
/**
|
|
* Search-result deduplication for chunked documents.
|
|
*
|
|
* When documents are chunked for embedding, multiple chunks from the same
|
|
* source page can land in the result set. Default behavior: keep the
|
|
* highest-scoring chunk per chain.
|
|
*
|
|
* A "chain" is identified by either:
|
|
* - an explicit `chainId` on the hit, or
|
|
* - the slug's parent (everything before the last `#chunk-N` suffix)
|
|
*/
|
|
import type { SearchHit } from "./index.js";
|
|
|
|
export interface DedupOptions {
|
|
/** Custom extractor; default falls back to chain-id then slug-prefix. */
|
|
chainOf?: (hit: SearchHit) => string;
|
|
/** Keep top-N chunks per chain (default 1). */
|
|
perChain?: number;
|
|
}
|
|
|
|
export function dedupHits(hits: SearchHit[], opts: DedupOptions = {}): SearchHit[] {
|
|
const perChain = opts.perChain ?? 1;
|
|
const chainOf = opts.chainOf ?? defaultChainOf;
|
|
const buckets = new Map<string, SearchHit[]>();
|
|
|
|
for (const h of hits) {
|
|
const c = chainOf(h);
|
|
if (!buckets.has(c)) buckets.set(c, []);
|
|
buckets.get(c)!.push(h);
|
|
}
|
|
|
|
const out: SearchHit[] = [];
|
|
for (const list of buckets.values()) {
|
|
list.sort((a, b) => b.score - a.score);
|
|
out.push(...list.slice(0, perChain));
|
|
}
|
|
out.sort((a, b) => b.score - a.score);
|
|
return out;
|
|
}
|
|
|
|
function defaultChainOf(hit: SearchHit & { chainId?: string }): string {
|
|
if (hit.chainId) return hit.chainId;
|
|
// strip "#chunk-N" / "::N" suffixes
|
|
return hit.slug.replace(/#chunk-\d+$/, "").replace(/::\d+$/, "");
|
|
}
|