Files
ai/object/scrape_address_test.go
zeekay 99f2e88361 A tenant's URL is fetched by outbound's rule, and never read as a path
Every address a request names now reaches the network through one client,
github.com/hanzoai/outbound (proxy.Public), which judges the address at
the dial, on the IP the socket connects to, and re-dials and re-judges
every redirect.

What it replaces failed three ways. A message href was checked by
util.Fetchable and then fetched by a different client: the name was
resolved twice, so a rebinding answer passed the check, and a redirect
to 169.254.169.254 was followed unchecked. The check itself was
net.IP's predicates, which admit NAT64 64:ff9b::a9fe:a9fe, 6to4,
Teredo, 198.18/15, 240/4 and the TEST-NETs; proxy/public.go's routable
admitted the same. And /v1/ai/rag/embed's url and /v1/ai/rag/ingest's
files[].url had no check at all: a loopback or metadata URL was
fetched, and a scheme-less string was read off this machine's disk.

A tenant document now goes through txt.GetParsedTextFromAddress (http
and https only, over proxy.Public); txt.GetParsedTextFromUrl is for
addresses this system stored. The Go crawler fetches every page and
robots.txt through proxy.Public, and the browser engine is handed only
addresses outbound.Host finds public.
2026-10-18 22:21:42 -07:00

32 lines
1.1 KiB
Go

// Copyright 2026 Hanzo AI Inc. All Rights Reserved.
package object
import (
"errors"
"testing"
"github.com/hanzoai/outbound"
)
// A crawl's start URL and every link it follows are the tenant's addresses: the Go
// scraper fetches them through outbound's client, and the browser engine is handed
// only addresses outbound.Host finds public.
func TestACrawlOfThisNetworkIsRefused(t *testing.T) {
srv, reached := inside(t)
if _, err := ScrapePage(srv.URL + "/"); !errors.Is(err, outbound.ErrRefused) {
t.Errorf("scrape of %s: err = %v, want outbound.ErrRefused", srv.URL, err)
}
if rules := fetchRobotsTxt(srv.URL); rules == nil {
t.Fatal("robots read nothing and answered nil")
}
if n := reached.Load(); n != 0 {
t.Errorf("the server on this network was reached %d times by a crawl", n)
}
for _, u := range []string{srv.URL + "/", "http://169.254.169.254/latest/meta-data/", "http://[fd00:ec2::254]/", "file:///etc/passwd"} {
if err := addressable(u); !errors.Is(err, outbound.ErrRefused) {
t.Errorf("addressable(%s) = %v, want outbound.ErrRefused", u, err)
}
}
}