mirror of
https://github.com/hanzoai/team.git
synced 2026-09-30 01:57:35 +00:00
Ben SchumacherandGitHub
54438407cf
Add LIKE-based CJK (Chinese, Japanese, Korean) search support for PostgreSQL, gated behind a `CJKSearch` feature flag. PostgreSQL's built-in full-text search (`to_tsvector`/`to_tsquery`) does not support CJK languages because it relies on whitespace-based tokenization, which doesn't work for logographic/syllabic scripts that don't use spaces between words. This PR adds a `LIKE`-based fallback for search terms containing CJK characters. **How it works:** When the `CJKSearch` feature flag is enabled and a search term contains CJK characters (Han, Hiragana, Katakana, or Hangul), the query builder generates `LIKE '%term%'` clauses instead of `to_tsvector @@ to_tsquery` expressions. Case-sensitive `LIKE` is used rather than `ILIKE` since CJK scripts have no letter casing, which also allows potential use of standard B-tree indexes.
19 lines
378 B
Go
19 lines
378 B
Go
package model
|
|
|
|
import (
|
|
"unicode"
|
|
)
|
|
|
|
// ContainsCJK returns true if the string contains any CJK (Chinese, Japanese, Korean) characters.
|
|
func ContainsCJK(s string) bool {
|
|
for _, r := range s {
|
|
if unicode.Is(unicode.Han, r) ||
|
|
unicode.Is(unicode.Hiragana, r) ||
|
|
unicode.Is(unicode.Katakana, r) ||
|
|
unicode.Is(unicode.Hangul, r) {
|
|
return true
|
|
}
|
|
}
|
|
return false
|
|
}
|