bot/docs/tools/firecrawl.md at skill/ml-research

mirror of https://github.com/hanzoai/bot.git synced 2026-10-30 17:24:26 +00:00

Files

T

zeekay acaeda400a Eradicate OpenClaw — resolve merge conflicts and finish the rebrand

Three coordinated things:

1. Resolve the committed upstream/main merge conflicts. 94 files were
   sitting with raw <<<<<<< HEAD ... >>>>>>> upstream/main blocks. Every
   conflict had HanzoBot/Bot on HEAD and OpenClaw on upstream. Resolved
   by keeping HEAD everywhere — 540 conflict blocks.

2. Rewrite remaining OpenClaw text refs.
     OPENCLAW   → HANZO_BOT
     OpenClaw   → HanzoBot          (PascalCase type identifiers)
     openclaw   → bot               (codebase convention — botToken,
                                    botUsername etc.; avoids invalid
                                    hyphenated JS identifiers)
     openclaw[_]/openclaw[A-Z] → bot[_]/bot[A-Z]   (compound)
     ai.openclaw.x → ai.hanzo.bot.x  (JVM package path)

3. Delete or rename openclaw-named file paths (116 of them). Dead
   duplicates with a Bot-named canonical from the partial migration:
   deleted. Otherwise renamed.
     ai/openclaw/**           → deleted (ai/hanzo/bot/** canonical)
     Sources/OpenClaw*/       → deleted (Sources/Bot* canonical)
     Tests/OpenClawIPCTests/  → deleted (Tests/BotIPCTests canonical)
     OpenClawKit/             → deleted (BotKit canonical)
     openclaw-tools.*.ts      → deleted (bot-tools.*.ts canonical)
     openclaw-root.ts etc.    → deleted (bot-root.ts canonical)
     types.openclaw.ts        → deleted (types.bot.ts canonical)
     extensions/*/openclaw.plugin.json → renamed hanzo-bot.plugin.json
     docs/start/openclaw.md   → renamed hanzo-bot.md (+ zh-CN)
     docs/assets/openclaw-*.png, whatsapp-openclaw*.jpg → deleted
     scripts/.../openclaw-*   → deleted (bot/hanzo-bot parallels)
     openclaw.mjs             → deleted (hanzo-bot.mjs is package.bin)

Then patched 65 broken JS/TS import strings where hanzo-bot had ended
up inside an import path ('./types.hanzo-bot.js' → './types.bot.js'
etc., since the on-disk filename uses the brand-neutral bot- prefix).

Pre-existing lint debt cleaned up to get oxlint --type-aware to 0/0:
removed the dead i18n test that referenced a path no longer existing;
defined the missing DIDConfig and WalletConfig types in types.base.ts
(they were imported but never declared); dropped unused imports;
collapsed redundant type assertion and a tautological meta.bot lookup
that the openclaw→bot rename made redundant.

The a2ui.bundle.js generated artifact is kept at its pre-rebrand
contents — it gets regenerated by 'pnpm canvas:a2ui:bundle' from
sources I cannot rebuild in this commit; the source side is clean.

Verified: 0 occurrences of openclaw (case-insensitive) in tree source
content, 0 file or directory paths with openclaw in the name; oxlint
--type-aware src test reports 0/0.

2026-05-11 17:56:29 -07:00

1.7 KiB

Raw Permalink Blame History

summary, read_when, title

summary

read_when

title

Firecrawl fallback for web_fetch (anti-bot + cached extraction)

You want Firecrawl-backed web extraction

You need a Firecrawl API key

You want anti-bot extraction for web_fetch

Firecrawl

HanzoBot can use Firecrawl as a fallback extractor for web_fetch. It is a hosted content extraction service that supports bot circumvention and caching, which helps with JS-heavy sites or pages that block plain HTTP fetches.

Get an API key

Create a Firecrawl account and generate an API key.
Store it in config or set FIRECRAWL_API_KEY in the gateway environment.

Configure Firecrawl

{
  tools: {
    web: {
      fetch: {
        firecrawl: {
          apiKey: "FIRECRAWL_API_KEY_HERE",
          baseUrl: "https://api.firecrawl.dev",
          onlyMainContent: true,
          maxAgeMs: 172800000,
          timeoutSeconds: 60,
        },
      },
    },
  },
}

Notes:

firecrawl.enabled defaults to true when an API key is present.
maxAgeMs controls how old cached results can be (ms). Default is 2 days.

Stealth / bot circumvention

Firecrawl exposes a proxy mode parameter for bot circumvention (basic, stealth, or auto). HanzoBot always uses proxy: "auto" plus storeInCache: true for Firecrawl requests. If proxy is omitted, Firecrawl defaults to auto. auto retries with stealth proxies if a basic attempt fails, which may use more credits than basic-only scraping.

How `web_fetch` uses Firecrawl

web_fetch extraction order:

Readability (local)
Firecrawl (if configured)
Basic HTML cleanup (last fallback)

See Web tools for the full web tool setup.

1.7 KiB Raw Permalink Blame History