Files
ai/object/crawl_test.go
zeekayandHanzo Dev 62d6139e0a crawl: /v1/crawl reads through the one crawl, not a host that does not resolve
POST /v1/crawl dialled crawl.hanzo.svc.cluster.local:11235 — a name with no
service behind it. Every request came back success:false with empty markdown, so
the endpoint looked unlucky rather than unwired, and nothing ever escalated.

It now fetches through github.com/hanzoai/cloud/apps/crawl, which is the same
code the answer engine's read stage grounds on. One crawl in the binary for the
product's crawl surface, and it is in-process: no service to deploy, no token to
rotate, no batch to poll.

The replacement is also strictly safer than the dial it replaces. apps/crawl
carries a guarded dialer that resolves and checks EVERY hop, redirects included,
and refuses any address that is not public-unicast. /v1/crawl takes a URL from
the caller — that is textbook SSRF reach — and the crawl4ai leg had no such
guard: a caller could have aimed it at the cluster's own metadata service or an
internal admin port and read the response back out as markdown. It cannot now.

Behaviour that did not change: the request body ({url} or {urls}, merged and
deduped), the ten-URL cap, the write-level auth gate, the balance gate, and the
CrawlResult wire shape. A URL that cannot be read still comes back Success:false
rather than failing the batch — nine pages that read are worth more than one
refusal — and results stay in the order asked, keyed by the url ASKED for so a
batching caller can still match them up. Where a page actually came from is in
Metadata["sourceURL"].

What is NOT ripped, and why: object/scraper.go's `browser` engine still calls
CrawlWithCrawl4AI. It needs the per-page internal/external LINK MAP to run its
BFS, and apps/crawl does not expose one — swapping it would silently reduce a
depth-crawl to a single page. It is already guarded by IsCrawl4AIAvailable() and
degrades to the Go scraper, and whether that host resolves in-cluster is not
something this change can verify from outside. Left honest and flagged rather
than half-migrated.

The crawl4ai-shaped projection behind /v1/crawl is deleted along with its tests;
the wire-decode tests that still cover the scraper's leg stay.

Tests: crawl_test.go proves the refusal (a real local server that WOULD have
served content is still refused, as is 169.254.169.254), that a refusal carries
its reason and no content, that order and requested-url keying hold across the
concurrent batch, and that only an empty request is a whole-request error. The
success path is deliberately not faked: apps/crawl has no env switch and no test
hook for non-public addresses, and a hermetic "it fetched a page" test would have
to weaken the exact property being protected.

Co-authored-by: Hanzo Dev <[email protected]>
2026-08-04 17:07:32 -07:00

110 lines
4.1 KiB
Go

// Copyright 2023-2025 Hanzo AI Inc. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package object
// crawl_test.go — proofs for POST /v1/crawl over the native crawl: results come
// back in the order asked and keyed by the url asked for, a page that cannot be
// read degrades to Success:false instead of failing the batch, and a URL aimed at
// the cluster's own internals is REFUSED.
//
// The success path is deliberately NOT faked. apps/crawl refuses every non-public
// address with no env switch and no test hook — which is exactly right for an
// SSRF guard — so a hermetic "it fetched a page" test would have to weaken the
// property it claims to protect. What is asserted here is what can be asserted
// without doing that.
import (
"net/http"
"net/http/httptest"
"testing"
)
// page serves one HTML document on loopback... which the crawl's guarded dialer
// refuses by design. That refusal is itself the security property under test, so
// this helper exists to prove the refusal, not to fake a public page.
func page(t *testing.T, body string) string {
t.Helper()
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
w.Header().Set("Content-Type", "text/html; charset=utf-8")
_, _ = w.Write([]byte(body))
}))
t.Cleanup(srv.Close)
return srv.URL
}
// TestCrawlRefusesNonPublicAddresses is the SSRF proof and the whole reason this
// endpoint reads through apps/crawl rather than its own http.Client. A caller
// hands it a URL; if that URL resolves to loopback, link-local, or any other
// non-public address, the fetch is refused rather than performed on the
// cluster's behalf. The crawl4ai dial this replaced had no such guard.
func TestCrawlRefusesNonPublicAddresses(t *testing.T) {
local := page(t, "<html><head><title>Internal</title></head><body><p>secrets</p></body></html>")
got, err := Crawl([]string{local, "http://169.254.169.254/latest/meta-data/"})
if err != nil {
t.Fatalf("a refused url must not fail the batch: %v", err)
}
if len(got) != 2 {
t.Fatalf("want one result per url, got %d", len(got))
}
for i, r := range got {
if r.Success {
t.Fatalf("result %d: a non-public address must never be fetched: %+v", i, r)
}
if r.Markdown != "" {
t.Fatalf("result %d: a refused fetch must carry no content, got %q", i, r.Markdown)
}
if msg, _ := r.Metadata["error"].(string); msg == "" {
t.Fatalf("result %d: a refusal must say why, got %+v", i, r.Metadata)
}
}
// The result is keyed by the url that was ASKED for, so a batching caller can
// still match every result back to its request.
if got[0].URL != local || got[1].URL != "http://169.254.169.254/latest/meta-data/" {
t.Fatalf("results must carry the requested url: %+v", got)
}
}
// TestCrawlPreservesOrder proves the batch contract: results come back in the
// order asked, whatever order the concurrent fetches actually complete in.
func TestCrawlPreservesOrder(t *testing.T) {
urls := []string{
"http://127.0.0.1:1/a",
"http://127.0.0.1:2/b",
"http://127.0.0.1:3/c",
"not a url at all",
}
got, err := Crawl(urls)
if err != nil {
t.Fatalf("Crawl: %v", err)
}
if len(got) != len(urls) {
t.Fatalf("want %d results, got %d", len(urls), len(got))
}
for i, r := range got {
if r.URL != urls[i] {
t.Fatalf("result %d out of order: %q, want %q", i, r.URL, urls[i])
}
}
}
// TestCrawlEmptyIsAnError proves the one whole-request failure: being handed
// nothing to do. Every other failure is per-URL and does not fail the batch.
func TestCrawlEmptyIsAnError(t *testing.T) {
if _, err := Crawl(nil); err == nil {
t.Fatal("crawling nothing must be an error, not an empty success")
}
}