mirror of
https://github.com/hanzoai/vector.git
synced 2026-10-30 18:18:56 +00:00
ff89694ce6
* Add new tokenizer for multilingual * Make codespell happy * improve handling of when to apply stemming * Fix stemming implementation * Fix Japanese language detection. Add tests for MultilingualV2 * Update lib/segment/src/index/field_index/full_text_index/tokenizers/japanese.rs Co-authored-by: Luis Cossío <[email protected]> * Update lib/segment/src/index/field_index/full_text_index/tokenizers/japanese.rs Co-authored-by: Luis Cossío <[email protected]> * Update lib/segment/src/index/field_index/full_text_index/tokenizers/multilingual.rs Co-authored-by: Luis Cossío <[email protected]> * Make init() private * move model into segment, filter non-alphanumeric tokens * fmt * exclude punctuation from tokenizer result * fm --------- Co-authored-by: Luis Cossío <[email protected]> Co-authored-by: generall <[email protected]>