Files
zeekayandHanzo Dev 62d6139e0a crawl: /v1/crawl reads through the one crawl, not a host that does not resolve
POST /v1/crawl dialled crawl.hanzo.svc.cluster.local:11235 — a name with no
service behind it. Every request came back success:false with empty markdown, so
the endpoint looked unlucky rather than unwired, and nothing ever escalated.

It now fetches through github.com/hanzoai/cloud/apps/crawl, which is the same
code the answer engine's read stage grounds on. One crawl in the binary for the
product's crawl surface, and it is in-process: no service to deploy, no token to
rotate, no batch to poll.

The replacement is also strictly safer than the dial it replaces. apps/crawl
carries a guarded dialer that resolves and checks EVERY hop, redirects included,
and refuses any address that is not public-unicast. /v1/crawl takes a URL from
the caller — that is textbook SSRF reach — and the crawl4ai leg had no such
guard: a caller could have aimed it at the cluster's own metadata service or an
internal admin port and read the response back out as markdown. It cannot now.

Behaviour that did not change: the request body ({url} or {urls}, merged and
deduped), the ten-URL cap, the write-level auth gate, the balance gate, and the
CrawlResult wire shape. A URL that cannot be read still comes back Success:false
rather than failing the batch — nine pages that read are worth more than one
refusal — and results stay in the order asked, keyed by the url ASKED for so a
batching caller can still match them up. Where a page actually came from is in
Metadata["sourceURL"].

What is NOT ripped, and why: object/scraper.go's `browser` engine still calls
CrawlWithCrawl4AI. It needs the per-page internal/external LINK MAP to run its
BFS, and apps/crawl does not expose one — swapping it would silently reduce a
depth-crawl to a single page. It is already guarded by IsCrawl4AIAvailable() and
degrades to the Go scraper, and whether that host resolves in-cluster is not
something this change can verify from outside. Left honest and flagged rather
than half-migrated.

The crawl4ai-shaped projection behind /v1/crawl is deleted along with its tests;
the wire-decode tests that still cover the scraper's leg stay.

Tests: crawl_test.go proves the refusal (a real local server that WOULD have
served content is still refused, as is 169.254.169.254), that a refusal carries
its reason and no content, that order and requested-url keying hold across the
concurrent batch, and that only an empty request is a whole-request error. The
success path is deliberately not faked: apps/crawl has no env switch and no test
hook for non-public addresses, and a hermetic "it fetched a page" test would have
to weaken the exact property being protected.

Co-authored-by: Hanzo Dev <[email protected]>
2026-08-04 17:07:32 -07:00

103 lines
4.0 KiB
Go

// Copyright 2023-2025 Hanzo AI Inc. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package object
// crawl.go — POST /v1/crawl: fetch a URL, get its content back.
//
// It reads through the ONE crawl, github.com/hanzoai/cloud/apps/crawl, which is
// the same code the answer engine's read stage grounds on. There is no crawl
// SERVICE to call: this used to dial crawl.hanzo.svc.cluster.local:11235, a name
// that does not resolve, so every request returned success:false and the endpoint
// looked merely unlucky rather than unwired.
//
// The native fetch is also strictly safer than the dial it replaces: apps/crawl
// carries a guarded dialer that refuses non-public addresses on EVERY hop,
// including redirects, so a caller-supplied URL cannot be aimed at the cluster's
// own internals. That is the property an SSRF-facing endpoint has to have, and
// the crawl4ai leg never had it.
import (
"context"
"errors"
"sync"
"time"
crawlpkg "github.com/hanzoai/cloud/apps/crawl"
)
// errNoURLs is the only way Crawl fails as a whole: it was handed nothing to do.
var errNoURLs = errors.New("crawl: no urls")
// crawlBudget bounds one /v1/crawl batch. Each URL is an outbound fetch with its
// own transport timeout; this is the ceiling on the whole request, so a batch of
// slow hosts cannot hold a handler open indefinitely.
const crawlBudget = 60 * time.Second
// CrawlResult is the canonical, clean output of POST /v1/crawl for one URL: the
// fetched page as LLM-ready markdown plus lightweight metadata. It is the ONE
// crawl result shape — distinct from ScrapeResult, which is docs-structured for
// search ingest.
type CrawlResult struct {
URL string `json:"url"`
Title string `json:"title,omitempty"`
Description string `json:"description,omitempty"`
Markdown string `json:"markdown"`
Success bool `json:"success"`
Metadata map[string]interface{} `json:"metadata,omitempty"`
}
// Crawl fetches every URL and returns one result each, in the order asked.
//
// A URL that cannot be read is NOT an error: it comes back Success:false with the
// reason in Metadata, because a batch where nine pages read and one was blocked
// has nine pages worth of answer in it. The error return is reserved for a
// request that asked for nothing.
func Crawl(urls []string) ([]CrawlResult, error) {
if len(urls) == 0 {
return nil, errNoURLs
}
ctx, cancel := context.WithTimeout(context.Background(), crawlBudget)
defer cancel()
out := make([]CrawlResult, len(urls))
var wg sync.WaitGroup
for i, u := range urls {
wg.Add(1)
go func() {
defer wg.Done()
out[i] = crawlOne(ctx, u)
}()
}
wg.Wait()
return out, nil
}
// crawlOne reads one URL. The result's URL is the one that was ASKED for, never
// the post-redirect address: a caller that submitted a batch matches results back
// to requests by it, and handing back the redirected address would break that
// match on exactly the pages that redirect. Where the page actually came from is
// in Metadata["sourceURL"].
func crawlOne(ctx context.Context, url string) CrawlResult {
page, err := crawlpkg.Fetch(ctx, url)
if err != nil {
return CrawlResult{URL: url, Metadata: map[string]interface{}{"error": err.Error()}}
}
r := CrawlResult{URL: url, Title: page.Title, Markdown: page.Markdown, Success: true, Metadata: page.Metadata}
if desc, ok := page.Metadata["description"].(string); ok {
r.Description = desc
}
return r
}