fetch: search asks our own API, and says so when it cannot

`fetch:search` scraped html.duckduckgo.com and pulled results out of a regex
over `class="result__title"`. Measured 2026-08-20: that page answers HTTP 202 —
DuckDuckGo's anti-bot challenge — and the markup carries no `result__title` at
all. So the regex matched nothing and the tool returned `{results: [], count: 0}`
for EVERY query: the identical answer it gives when the web genuinely has
nothing. Blind rendered as empty, with no way for a caller to tell them apart,
which is exactly why nobody noticed.

It now asks POST /v1/websearch — the meta-search we ship — reading the base URL
and credential from the same env the tracker tool reads, so the MCP has one way
to reach cloud rather than a second one here.

The failure shape is the point:

  no credential  -> NO_CREDENTIAL, never an empty result list
  cloud refuses  -> its own words verbatim ("sign in to search the web"),
                    because a 401 and a 500 are different problems and
                    flattening both into "search failed" makes an outage
                    look like a bad query
  results        -> carries WHICH ENGINES answered, so a zero-result search
                    with engines listed is a real empty and one with none is
                    a search that did not happen

HONEST GAP: the committed test cannot run. The repo declares typescript ^7.0.2
against ts-jest ^29.4.6, which does not support TS7, so `npx jest` dies in
_resolveTsConfig on "Cannot read properties of undefined (reading 'fileExists')"
— for the pre-existing suites too (edit.test.ts: 0 tests). `tsc --noEmit` is
clean repo-wide; only the transform is dead. The eight assertions were run
against the esbuild-bundled tool instead and all pass; they execute under jest
the moment the toolchain is fixed.

Co-authored-by: Hanzo Dev <[email protected]>

src/tools/unified/fetch.ts

@@ -169,26 +169,75 @@ export const fetchTool: Tool = {
 
         case 'search': {
           if (!args.query) return fail('INVALID_PARAMS', 'query required');
-          // Use DuckDuckGo HTML search (no API key needed)
-          const searchUrl = `https://html.duckduckgo.com/html/?q=${encodeURIComponent(args.query)}`;
-          const searchResp = await fetch(searchUrl, {
-            headers: { 'User-Agent': 'Mozilla/5.0 (compatible; HanzoBot/1.0)' },
+          /**
+           * OUR OWN SEARCH, not a scrape of somebody else's results page.
+           *
+           * This used to GET html.duckduckgo.com and pull results out with a
+           * regex over `class="result__title"`. Measured 2026-08-20, that
+           * returns HTTP 202 — DuckDuckGo's anti-bot challenge — and the markup
+           * carries no `result__title` at all, so the regex matched nothing and
+           * the tool answered `{results: [], count: 0}`: BLIND rendered as
+           * EMPTY, for every query, with no way for a caller to tell the two
+           * apart. A scraper of a third party's HTML is one class rename away
+           * from that on any day, which is the argument for our own endpoint
+           * rather than a better regex.
+           *
+           * Hanzo Cloud serves this at POST /v1/websearch — the same meta-search
+           * the product uses — so the tool now asks the API we ship. Credential
+           * and base URL come from the same env the tracker tool reads, so the
+           * MCP has ONE way to reach cloud rather than a second one here.
+           */
+          const base = process.env.API_URL || 'https://api.hanzo.ai';
+          const key =
+            process.env.HANZO_API_KEY || process.env.API_KEY || process.env.API_TOKEN || process.env.HANZO_TOKEN || '';
+          // Refuse LOUDLY without a credential. Returning an empty result list
+          // here would reproduce the exact defect this replaced: the caller
+          // cannot distinguish "the web has nothing" from "we never asked".
+          if (!key) {
+            return fail(
+              'NO_CREDENTIAL',
+              'web search needs a Hanzo API key: set HANZO_API_KEY (pk- or sk-). Cloud refuses anonymous search.',
+            );
+          }
+          const searchResp = await fetch(`${base}/v1/websearch`, {
+            method: 'POST',
+            headers: {
+              'Content-Type': 'application/json',
+              Accept: 'application/json',
+              Authorization: `Bearer ${key}`,
+            },
+            body: JSON.stringify({ q: args.query }),
             signal: AbortSignal.timeout(args.timeout || 15000),
           });
-          const searchHtml = await searchResp.text();
-          // Extract results with regex (no DOM parser in Node by default)
-          const resultRe = /class="result__title"[^>]*>[\s\S]*?href="([^"]*)"[^>]*>([\s\S]*?)<\/a>[\s\S]*?class="result__snippet"[^>]*>([\s\S]*?)<\//g;
-          const results: any[] = [];
-          let match;
-          const maxResults = args.limit || 10;
-          while ((match = resultRe.exec(searchHtml)) !== null && results.length < maxResults) {
-            results.push({
-              url: match[1].replace(/&amp;/g, '&'),
-              title: match[2].replace(/<[^>]+>/g, '').trim(),
-              snippet: match[3].replace(/<[^>]+>/g, '').trim(),
-            });
+          const searchText = await searchResp.text();
+          if (!searchResp.ok) {
+            // Cloud's own words reach the agent verbatim — "sign in to search the
+            // web" is a different problem from a 500, and flattening both into
+            // "search failed" is what makes an outage look like a bad query.
+            return fail('SEARCH_FAILED', `${searchResp.status}: ${searchText.substring(0, 200)}`);
           }
-          return envelope({ query: args.query, results, count: results.length }, 'search');
+          let payload: any;
+          try {
+            payload = JSON.parse(searchText);
+          } catch {
+            return fail('SEARCH_FAILED', `unreadable response from ${base}/v1/websearch`);
+          }
+          const maxResults = args.limit || 10;
+          const results = (Array.isArray(payload?.results) ? payload.results : [])
+            .slice(0, maxResults)
+            .map((r: any) => ({ url: r?.url ?? '', title: r?.title ?? '', snippet: r?.content ?? r?.snippet ?? '' }));
+          return envelope(
+            {
+              query: payload?.query ?? args.query,
+              results,
+              count: results.length,
+              // Which engines answered. A zero-result search with engines listed
+              // is a real empty; one with none is a search that did not happen.
+              engines: Array.isArray(payload?.engines) ? payload.engines : [],
+              total: payload?.number_of_results ?? null,
+            },
+            'search',
+          );
         }
 
         case 'crawl': {

test/tools/fetch-search.test.ts

@@ -0,0 +1,107 @@
+import { fetchTool } from '../../src/tools/unified/fetch.js';
+
+/**
+ * The regression these pin is BLINDNESS RENDERED AS EMPTINESS.
+ *
+ * `fetch:search` used to scrape html.duckduckgo.com and pull results out with a
+ * regex. Measured 2026-08-20 that page answers HTTP 202 (an anti-bot challenge)
+ * and carries no `result__title`, so the regex matched nothing and the tool
+ * reported `{results: [], count: 0}` — the same answer it gives for a query the
+ * web genuinely has nothing for. A caller could not tell the two apart, which is
+ * the whole reason it went unnoticed.
+ *
+ * So the assertions are about the SHAPE OF A FAILURE, not about search quality:
+ * when we cannot search, the tool must say so.
+ */
+function body(res: any) {
+  return JSON.parse(res.content[0].text);
+}
+
+const KEYS = ['HANZO_API_KEY', 'API_KEY', 'API_TOKEN', 'HANZO_TOKEN', 'API_URL'];
+let saved: Record<string, string | undefined>;
+
+beforeEach(() => {
+  saved = Object.fromEntries(KEYS.map((k) => [k, process.env[k]]));
+  for (const k of KEYS) delete process.env[k];
+});
+afterEach(() => {
+  for (const k of KEYS) {
+    if (saved[k] === undefined) delete process.env[k];
+    else process.env[k] = saved[k];
+  }
+});
+
+describe('fetch:search', () => {
+  it('refuses loudly with no credential, instead of reporting zero results', async () => {
+    const res: any = await fetchTool.handler({ action: 'search', query: 'anything' });
+    expect(res.isError).toBe(true);
+    const b = body(res);
+    expect(b.ok).toBe(false);
+    expect(b.error.code).toBe('NO_CREDENTIAL');
+    // The failure must never be mistakable for a successful empty search.
+    expect(b.data).toBeNull();
+  });
+
+  it('still requires a query', async () => {
+    process.env.HANZO_API_KEY = 'sk-test';
+    const res: any = await fetchTool.handler({ action: 'search' });
+    expect(res.isError).toBe(true);
+    expect(body(res).error.code).toBe('INVALID_PARAMS');
+  });
+
+  it("passes cloud's own refusal through verbatim rather than flattening it", async () => {
+    process.env.HANZO_API_KEY = 'sk-test';
+    process.env.API_URL = 'https://api.example.invalid';
+    const spy = jest.spyOn(globalThis, 'fetch' as any).mockResolvedValue({
+      ok: false,
+      status: 401,
+      text: async () => '{"status":401,"error":"sign in to search the web"}',
+    } as any);
+    try {
+      const res: any = await fetchTool.handler({ action: 'search', query: 'q' });
+      expect(res.isError).toBe(true);
+      const b = body(res);
+      expect(b.error.code).toBe('SEARCH_FAILED');
+      expect(b.error.message).toContain('401');
+      expect(b.error.message).toContain('sign in to search the web');
+    } finally {
+      spy.mockRestore();
+    }
+  });
+
+  it('asks OUR api, and maps its envelope', async () => {
+    process.env.HANZO_API_KEY = 'sk-test';
+    process.env.API_URL = 'https://api.example.invalid';
+    const seen: any = {};
+    const spy = jest.spyOn(globalThis, 'fetch' as any).mockImplementation(async (url: any, init: any) => {
+      seen.url = String(url);
+      seen.auth = init?.headers?.Authorization;
+      seen.body = JSON.parse(init.body);
+      return {
+        ok: true,
+        status: 200,
+        text: async () =>
+          JSON.stringify({
+            query: 'q',
+            number_of_results: 2,
+            engines: ['bing', 'mojeek'],
+            results: [{ url: 'https://a.example', title: 'A', content: 'snippet a' }],
+          }),
+      } as any;
+    });
+    try {
+      const res: any = await fetchTool.handler({ action: 'search', query: 'q' });
+      expect(seen.url).toBe('https://api.example.invalid/v1/websearch');
+      expect(seen.auth).toBe('Bearer sk-test');
+      expect(seen.body).toEqual({ q: 'q' });
+      const b = body(res);
+      expect(b.ok).toBe(true);
+      expect(b.data.results).toEqual([{ url: 'https://a.example', title: 'A', snippet: 'snippet a' }]);
+      // Which engines answered: an empty result WITH engines is a real empty,
+      // one with none is a search that never happened.
+      expect(b.data.engines).toEqual(['bing', 'mojeek']);
+    } finally {
+      spy.mockRestore();
+    }
+  });
+});