Files
ai/split/provider_test.go
zeekayandHanzo Dev 3c59399b6b fix: four real bugs the unrunnable tests were hiding
go test ./... in this repo failed in five packages, so nobody ran it, so
nothing in it was read. Underneath were four genuine defects:

1. CREATE DATABASE against SQLite. object/adapter.go switched on driverName
   with a postgres case and a DEFAULT that emits MySQL DDL, so sqlite got
   `CREATE DATABASE IF NOT EXISTS … charset utf8mb4` and every caller took
   `near "DATABASE": syntax error`. SQLite has no CREATE DATABASE at all — the
   database IS the file. A default that assumes one dialect is wrong for every
   dialect it has not met.

2. Markdown chunks came out in RANDOM ORDER. SplitText ranged a map, and Go
   randomises map iteration, so the same document produced the same chunks in a
   different sequence every run. Chunks are reassembled into context
   downstream; document order is the only order that means anything. Now sorted
   by position in the source (not alphabetically — "Section 10" must not sort
   before "Section 9").

3. Tables lost their last row. Every row pattern ends in \\n, so a table running
   to end-of-input dropped its final row into the remainder, where it came back
   out glued to the neighbouring prose — a table silently split across two
   chunks. Sections are sliced per heading, so ending without a newline is the
   COMMON case.

4. GetVideoCoverUrl("") panicked. VodClient is package-level and nil until
   SetVodClient runs, so on any deployment without the Aliyun VOD provider all
   four entry points called through nil, inside request handlers. An
   unconfigured OPTIONAL provider took the process down. Plus resp.Body.Video
   was three unguarded dereferences on a response we do not control.

Also: ExtractMarkdownTree emitted result["root"] = "" for documents opening
with a heading — a key for content that does not exist — and the flush logic
was written twice.

Why they were invisible: 14 files are operator SCRIPTS wearing func Test —
69 panics, ZERO assertions, needing a populated database, live credentials or
fixtures not in this repo. A panicking test aborts the whole binary, so one
missing mp3 took every real test in its package with it. They are now behind
`-tags ai_scripts`; several WRITE to whatever database they find, which
`go test ./...` must never do. TestListGeminiModels had `apiKey := ""`
hardcoded and could only ever panic; it and the txt fixture test now skip,
matching TestGenerateImageDOAI_Live which already did this correctly.

The split tests also said only "did not get the expected result" and "Expected
6 headings, got 7". They now name which section and which heading.

Whole suite green.

Co-authored-by: Hanzo Dev <[email protected]>
2026-07-27 09:10:49 -07:00

125 lines
3.0 KiB
Go

// Copyright 2023-2025 Hanzo AI Inc. All Rights Reserved.
// Portions Copyright 2023 The OpenAgent Authors. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
// These are operator SCRIPTS, not tests: they assert nothing, print results
// with fmt.Printf, panic instead of failing, and need a populated database,
// live provider credentials or local fixture files this repository does not
// contain. Run one deliberately:
//
// go test -tags ai_scripts ./<pkg>/ -run TestName -v
//
// Behind a tag because a panicking test aborts the whole test binary, so one
// missing fixture took every real test in the package down with it. Several
// also WRITE to whatever database they find, which `go test ./...` must not.
//go:build !skipCi && ai_scripts
package split_test
import (
"fmt"
"path/filepath"
"testing"
"github.com/hanzoai/ai/object"
"github.com/hanzoai/ai/split"
"github.com/hanzoai/ai/txt"
"github.com/hanzoai/ai/util"
)
func TestSplit(t *testing.T) {
object.InitConfig()
p, err := split.GetSplitProvider("Default")
if err != nil {
panic(err)
}
storageProvider, err := object.GetProvider("admin/provider-storage-built-in")
if err != nil {
panic(err)
}
path := filepath.Join(storageProvider.ClientId, "test.md")
text := util.ReadStringFromPath(path)
textSections, err := p.SplitText(text)
if err != nil {
panic(err)
}
for i, s := range textSections {
fmt.Printf("[%d] %s\n\n", i, s)
}
}
func TestSplit2(t *testing.T) {
object.InitConfig()
p, err := split.GetSplitProvider("QA")
if err != nil {
panic(err)
}
storageProvider, err := object.GetProvider("admin/provider-storage-built-in")
if err != nil {
panic(err)
}
path := filepath.Join(storageProvider.ClientId, "QAText.docx")
text, err := txt.GetParsedTextFromUrl(path, ".docx", "en")
if err != nil {
panic(err)
}
textSections, err := p.SplitText(text)
if err != nil {
panic(err)
}
for i, s := range textSections {
fmt.Printf("[%d] %s\n\n", i, s)
}
}
func TestSplit3(t *testing.T) {
object.InitConfig()
p, err := split.GetSplitProvider("Default")
if err != nil {
panic(err)
}
storageProvider, err := object.GetProvider("admin/storage-built")
if err != nil {
panic(err)
}
path := filepath.Join(storageProvider.ClientId, "myfile.docx")
text, err := txt.GetParsedTextFromUrl(path, ".docx", "en")
if err != nil {
panic(err)
}
textSections, err := p.SplitText(text)
if err != nil {
panic(err)
}
for i, s := range textSections {
fmt.Printf("[%d] %s\n\n", i, s)
}
}