Files
ai/object/crawl4ai_test.go
zeekayandHanzo Dev 62d6139e0a crawl: /v1/crawl reads through the one crawl, not a host that does not resolve
POST /v1/crawl dialled crawl.hanzo.svc.cluster.local:11235 — a name with no
service behind it. Every request came back success:false with empty markdown, so
the endpoint looked unlucky rather than unwired, and nothing ever escalated.

It now fetches through github.com/hanzoai/cloud/apps/crawl, which is the same
code the answer engine's read stage grounds on. One crawl in the binary for the
product's crawl surface, and it is in-process: no service to deploy, no token to
rotate, no batch to poll.

The replacement is also strictly safer than the dial it replaces. apps/crawl
carries a guarded dialer that resolves and checks EVERY hop, redirects included,
and refuses any address that is not public-unicast. /v1/crawl takes a URL from
the caller — that is textbook SSRF reach — and the crawl4ai leg had no such
guard: a caller could have aimed it at the cluster's own metadata service or an
internal admin port and read the response back out as markdown. It cannot now.

Behaviour that did not change: the request body ({url} or {urls}, merged and
deduped), the ten-URL cap, the write-level auth gate, the balance gate, and the
CrawlResult wire shape. A URL that cannot be read still comes back Success:false
rather than failing the batch — nine pages that read are worth more than one
refusal — and results stay in the order asked, keyed by the url ASKED for so a
batching caller can still match them up. Where a page actually came from is in
Metadata["sourceURL"].

What is NOT ripped, and why: object/scraper.go's `browser` engine still calls
CrawlWithCrawl4AI. It needs the per-page internal/external LINK MAP to run its
BFS, and apps/crawl does not expose one — swapping it would silently reduce a
depth-crawl to a single page. It is already guarded by IsCrawl4AIAvailable() and
degrades to the Go scraper, and whether that host resolves in-cluster is not
something this change can verify from outside. Left honest and flagged rather
than half-migrated.

The crawl4ai-shaped projection behind /v1/crawl is deleted along with its tests;
the wire-decode tests that still cover the scraper's leg stay.

Tests: crawl_test.go proves the refusal (a real local server that WOULD have
served content is still refused, as is 169.254.169.254), that a refusal carries
its reason and no content, that order and requested-url keying hold across the
concurrent batch, and that only an empty request is a whole-request error. The
success path is deliberately not faked: apps/crawl has no env switch and no test
hook for non-public addresses, and a hermetic "it fetched a page" test would have
to weaken the exact property being protected.

Co-authored-by: Hanzo Dev <[email protected]>
2026-08-04 17:07:32 -07:00

109 lines
3.9 KiB
Go

// Copyright 2023-2025 Hanzo AI Inc. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
package object
import (
"encoding/json"
"testing"
)
// TestMarkdownFieldUnmarshalJSON locks the shape-polymorphic decode of Crawl4AI's
// `markdown` field: a bare string (older builds) OR an object
// {fit_markdown, raw_markdown, ...} (deployed 0.8.x/0.9.x). The object form is
// what silently returned empty content before the fix, so both must decode and
// the object form must prefer fit_markdown with raw_markdown as fallback.
func TestMarkdownFieldUnmarshalJSON(t *testing.T) {
cases := []struct {
name string
in string
want string
}{
{"bare string", `"hello world"`, "hello world"},
{"object prefers fit", `{"raw_markdown":"RAW","fit_markdown":"FIT"}`, "FIT"},
{"object falls back to raw", `{"raw_markdown":"RAW","fit_markdown":""}`, "RAW"},
{"object with extra fields", `{"raw_markdown":"R","fit_markdown":"F","markdown_with_citations":"C"}`, "F"},
{"empty string", `""`, ""},
}
for _, c := range cases {
t.Run(c.name, func(t *testing.T) {
var m MarkdownField
if err := json.Unmarshal([]byte(c.in), &m); err != nil {
t.Fatalf("Unmarshal(%s) error: %v", c.in, err)
}
if string(m) != c.want {
t.Fatalf("Unmarshal(%s) = %q, want %q", c.in, string(m), c.want)
}
})
}
}
// TestCrawl4AIResponseDecodeObjectMarkdown is the regression guard: the deployed
// Hanzo Crawl (Crawl4AI 0.8.x) returns a synchronous envelope with a boolean
// `success` (no top-level status, no task_id) whose per-result `markdown` is an
// OBJECT. The whole response must decode, and CrawlWithCrawl4AI's "inline results
// present" branch must be reachable (len(Results) > 0).
func TestCrawl4AIResponseDecodeObjectMarkdown(t *testing.T) {
body := `{
"success": true,
"results": [{
"url": "https://example.com",
"success": true,
"markdown": {"raw_markdown":"# Raw","fit_markdown":"# Example\n\nClean body."},
"metadata": {"title":"Example Domain","description":"An example page."}
}]
}`
var resp Crawl4AIResponse
if err := json.Unmarshal([]byte(body), &resp); err != nil {
t.Fatalf("decode object-markdown envelope failed: %v", err)
}
if len(resp.Results) != 1 {
t.Fatalf("Results len = %d, want 1", len(resp.Results))
}
r := resp.Results[0]
if !r.Success {
t.Fatalf("per-result Success = false, want true")
}
if string(r.Markdown) != "# Example\n\nClean body." {
t.Fatalf("Markdown = %q, want fit_markdown content", string(r.Markdown))
}
}
// TestCrawl4AIResultDecodeHeterogeneousLinks guards the real 0.8.6 link-object
// shape: values are mixed types (string href/text, float score, null head_data).
// A map-of-string value errored the whole decode; the interface{} value typing
// must accept it and the href read must still work.
func TestCrawl4AIResultDecodeHeterogeneousLinks(t *testing.T) {
body := `{
"url": "https://example.com",
"success": true,
"markdown": {"raw_markdown":"# Example Domain","fit_markdown":""},
"links": {"external": [
{"href":"https://iana.org/domains/example","text":"Learn more","title":"",
"base_domain":"iana.org","head_data":null,"intrinsic_score":0.0,"total_score":null}
]},
"metadata": {"title":"Example Domain"}
}`
var r Crawl4AIResult
if err := json.Unmarshal([]byte(body), &r); err != nil {
t.Fatalf("decode heterogeneous links failed: %v", err)
}
// fit_markdown empty → raw_markdown fallback carries the content.
if string(r.Markdown) != "# Example Domain" {
t.Fatalf("Markdown = %q, want raw_markdown fallback", string(r.Markdown))
}
// The structured projection lifts href strings out of the interface{} values.
sr := Crawl4AIResultToScrapeResult(r)
found := false
for _, l := range sr.Links {
if l == "https://iana.org/domains/example" {
found = true
}
}
if !found {
t.Fatalf("expected external href in ScrapeResult.Links, got %v", sr.Links)
}
}