Files
agents/src/tools/toolOutputReferences.ts
Danny AvilaandGitHub 136401703c ⚑ feat: Single Tail Prompt-Cache Breakpoint (#245)
* ⚑ feat: Single Tail Prompt-Cache Breakpoint

Replace the rolling "last two user messages" prompt-cache strategy with a
single breakpoint anchored on the conversation tail, mirroring the approach
used by Claude Code. Anthropic/OpenRouter now place exactly one ephemeral
cache_control marker on the last cacheable block of the final non-synthetic
message; Bedrock places a single cachePoint via the new
addBedrockTailCacheControl. Because the marker always rides the true tail,
the whole prefix is written once and read back as history grows append-only,
instead of re-writing large spans every step.

- Add addTailCacheControl / addBedrockTailCacheControl (single tail marker),
  skipping thinking blocks and synthetic skill/meta messages as anchors and
  stripping all stale markers in one pass.
- Wire Graph (Anthropic, OpenRouter, Bedrock), AgentContext system-runnable
  body path, and summarization to the tail strategy by default.
- Keep legacy addCacheControl / addBedrockCacheControl exported for
  compatibility; update affected tests and add cache.tail.test.ts.

* 🩹 fix: Hoist Bedrock cachePoint out of toolResult body for tail breakpoint

The single tail prompt-cache breakpoint frequently anchors on a tool
result, since agent-loop conversations end with a tool turn before the
next model call. addBedrockTailCacheControl writes the cachePoint into
the tool message content, but the Converse converter wrapped the entire
content (cachePoint included) inside toolResult.content.

A cachePoint is a message-level ContentBlock, not a ToolResultContentBlock.
Bedrock does not reject the nested form β€” it silently drops the breakpoint
(verified live: cache_creation/cache_read both stay 0), so the tail
strategy produced ZERO caching for the most common agent-loop shape.

Hoist any cachePoint out of toolResult.content to a message-level sibling
after the toolResult block β€” the only position Bedrock honors. Live
Bedrock Converse now shows the tool-result tail writing the prefix on
turn 1 (cache_creation) and reading it back on turn 2 (cache_read),
matching the Anthropic-direct behavior.

- Hoist cachePoint(s) in convertToolMessageToConverseMessage.
- Add toolResultCachePoint.test.ts (converter hoist + end-to-end).
- Add cache.tail.test.ts case for a trailing string tool-result tail.

* 🩹 fix: Keep tail cache breakpoint on a block that survives conversion

Two edge cases dropped the single tail breakpoint before the model call,
silently regressing to zero message caching (legacy marked human messages,
which avoided both paths):

1. Foreign reasoning tail (Anthropic/OpenRouter): isTailCacheableBlock only
   excluded native `thinking`/`redacted_thinking`, so on a cross-provider
   handoff the marker could anchor on a `reasoning_content`/`reasoning`/
   `think` block β€” which _convertMessagesToAnthropicPayload drops on
   assistant turns. The only breakpoint vanished. Now exclude foreign
   reasoning types from tail anchoring so the marker lands on a surviving
   text/tool block.

2. Thinking-fold ordering: the tail marker was placed before
   ensureThinkingBlockInMessages, which folds a trailing non-thinking AI→Tool
   chain into a `[Previous agent context]` HumanMessage whose builder copies
   text but not cache_control/cachePoint. Move the provider-specific tail
   cache insertion (Anthropic, Bedrock, OpenRouter) to run LAST β€” after
   thinking normalization and orphan sanitization β€” so it anchors on the
   final message list.

Verified by inspecting the final _convertMessagesToAnthropicPayload output:
the breakpoint now survives in both cases (and a guard test asserts the old
mark-before-fold order loses it).

- Exclude reasoning_content/reasoning/think in isTailCacheableBlock.
- Reorder tail cache insertion after ensureThinkingBlock/sanitizeOrphan in Graph.
- Add tailCacheConversion.test.ts and foreign-reasoning cases in cache.tail.test.ts.

* 🩹 fix: Harden tail prompt-cache anchor against dropped/stripped tails

Three more cases where the single tail breakpoint failed to reach the model;
all stem from anchoring on a volatile tail that a later stage drops/rewrites.

1. input_json_delta anchor (Anthropic/OpenRouter): persisted partial tool-input
   deltas are dropped by _convertMessagesToAnthropicPayload (input is restored
   onto the tool_use block). Anchoring the marker there lost it. Excluded
   input_json_delta from tail anchoring (joins the reasoning types), renaming
   the set to NON_ANCHORABLE_BLOCK_TYPES.

2. toolOutputReferences annotation (functional regression): prompt caching
   rewrites a string ToolMessage tail into a text-block array to host its
   marker; annotateMessagesForLLM only applied the live `[ref: …]` annotation
   to STRING tool content, so the common tool-result tail silently lost its
   reference marker once cached. annotateMessagesForLLM now projects the live
   ref (and unresolved warning) onto array tool content too.

3. assistant-prefill strip (Claude 4.6+): stripUnsupportedAssistantPrefill pops
   a trailing assistant prefill right before the API call; if the only tail
   breakpoint rode it, message caching was lost. It now re-anchors the
   breakpoint onto the new tail (only when one was actually removed, so
   caching-off requests stay untouched), reusing addTailCacheControl to honor
   the same exclusions.

Tests: stripPrefillCache.test.ts (re-anchor); array live-ref cases in
annotateMessagesForLLM.test.ts; input_json_delta is covered by the
NON_ANCHORABLE_BLOCK_TYPES exclusion. tsc + lint clean.

* 🩹 fix: Hoist Anthropic tool_result cache_control onto the top-level block

The single tail breakpoint frequently anchors on a tool result. For a string
ToolMessage tail, addTailCacheControl rewrites it to a text-block array carrying
cache_control, and _ensureMessageContents nests that block inside
tool_result.content. The Anthropic API currently honors that nested marker β€”
verified live with an isolated, system-prompt-free large tool result (control
no-marker => cache_creation 0; nested marker => 10232 written then read) β€” so it
is not broken today. But Anthropic documents the top-level messages.content
block as the cacheable position and does not document sub-content caching, so
relying on the nested form is fragile.

Hoist any cache_control off the inner tool-result content onto the generated
tool_result block itself (mirrors the Bedrock cachePoint hoist). Live-verified
end to end: control no-marker => cache_creation 0; hoisted marker => 12354
written on turn 1, read on turn 2.

- Add hoistToolResultCacheControl; apply it in _ensureMessageContents.
- tailCacheConversion.test.ts now asserts the marker lands on the tool_result
  block, not nested.

* 🩹 fix: Keep orphan sanitization enabled for prompt-cached sends

Moving the tail cache marker to run after sanitizeOrphanToolBlocks (so the
marker survives the thinking fold) had a side effect: the marker no longer
reassigns finalMessages before the `needsOrphanSanitize` gate is evaluated. For
a prompt-cached Anthropic/Bedrock send whose pruner returned the context
unchanged (finalMessages === messagesToUse), the gate went false and orphaned
AI/tool pairs from persisted history could reach the provider and fail
structural validation β€” whereas the pre-move code always reassigned first.

Compute the prompt-cache strategy up front and add `willAddTailCache` to the
sanitize gate, so cached sends are cleaned before the marker is applied
(restoring the pre-move guarantee). Collapses the cache-insertion branch to the
same up-front booleans.

* 🩹 fix: Orphan-sanitize system-runnable prompt-cached sends too

The previous gate used "this node will add the marker" (which excludes the
system-runnable path via !systemRunnable). But when a system runnable owns the
system prompt, AgentContext still adds the body cache marker β€” so those are
cached sends that must be orphan-sanitized as well. With prompt caching +
system runnable + a pruner that returned the context unchanged, orphaned
AI/tool pairs from persisted history could still reach the provider.

Track two separate facts: `providerPromptCacheEnabled` (caching is on for the
provider at all β€” drives orphan cleanup, system-runnable included) vs. the
node-adds-the-marker condition (Anthropic/OpenRouter minus systemRunnable, or
Bedrock β€” drives the insertion). The sanitize gate now uses the former.

* 🩹 fix: Break import cycle from the prefill re-anchor

The P3-1 re-anchor imported addTailCacheControl from @/messages/cache into the
Anthropic converter, closing a cycle:
  messages/format.ts -> llm/anthropic/utils/message_inputs.ts
    -> messages/cache.ts -> messages/format.ts
which the bundler's circular-dependency check (npm run build:dev) flags.

Replace the cross-module reuse with a small local re-anchor that operates on the
already-converted Anthropic payload. This is also more correct: at that stage
the converter has already dropped foreign-reasoning / input_json_delta blocks,
so only native thinking blocks need excluding, and the post-strip tail is always
a user message. Live-reverified: turn1 cache_creation=6264, turn2 read=6264.

* πŸ“Š test: Live reproducible prompt-cache benchmark (tail vs legacy)

Add a committed, live benchmark that empirically justifies the single tail
breakpoint over the legacy "last two user messages" strategy, plus a doc with
representative results.

bench-prompt-cache.ts replays three realistic harness shapes (agent tool loop,
multi-turn chat, realistic agent) under BOTH strategies over the same
conversations in separate cache namespaces, against a real provider, and reports
per-call cache token breakdowns. `fresh` (uncached, full-price input) is derived
provider-agnostically from total_tokens-output_tokens minus the cache buckets,
since Anthropic folds cache tokens into input_tokens while Bedrock reports them
separately.

Result (live, claude-sonnet-4-5): the tail strategy is cheaper in every scenario
on both Anthropic and Bedrock. Legacy reprocesses tens of thousands of
full-price tokens in any tool-bearing conversation (its lone user-message marker
leaves the growing transcript uncached); tail reduces that to ~0 and reads the
prefix back. Effective cost βˆ’30..βˆ’38% (Anthropic), βˆ’9..βˆ’15% (Bedrock); even
legacy's best case (frequent user messages) ties-or-wins.

- src/scripts/bench-prompt-cache.ts (excluded from build/CI; real paid calls)
- npm run bench:cache [-- --provider bedrock|anthropic --rounds N --model id]
- docs/prompt-cache-benchmark.md

* πŸ“Š test: Add post-compaction scenario to the prompt-cache benchmark

Covers the two transcript-mutating harness behaviors raised in review:

- Tool truncation: a non-issue for caching β€” applied once at tool-exec with a
  model-fixed (turn-invariant) cap by the already-tested, deterministic
  truncateToolResultContent, so a truncated result is a stable prefix block.
  Documented; no separate scenario needed (existing tool-loop already exercises
  tool results in the cached prefix).
- Compaction (summarization): add a post-compaction scenario β€” a few pre-
  compaction tool rounds, a head→summary swap (one-time cache miss for any
  strategy), then continued tool rounds. Confirms the tail strategy
  re-establishes append-only caching on the new summary-headed prefix.

Live result (claude-sonnet-4-5): tail wins 4/4 scenarios on BOTH Anthropic and
Bedrock. Post-compaction is among the largest wins (Anthropic effective βˆ’41%,
read +76%) because after compaction the summary is the only user message, so
legacy re-sends all continued tool work uncached (fresh 63k β†’ 42).

docs/prompt-cache-benchmark.md updated with the 4-scenario tables and a
truncation/compaction section.
2026-06-17 13:45:15 -04:00

840 lines
29 KiB
TypeScript

/**
* Tool output reference registry.
*
* When enabled via `RunConfig.toolOutputReferences.enabled`, ToolNode
* stores each successful tool output under a stable key
* (`tool<idx>turn<turn>`) where `idx` is the tool's position within a
* ToolNode batch and `turn` is the batch index within the run
* (incremented once per ToolNode invocation).
*
* Subsequent tool calls can pipe a previous output into their args by
* embedding `{{tool<idx>turn<turn>}}` inside any string argument;
* {@link ToolOutputReferenceRegistry.resolve} walks the args and
* substitutes the placeholders immediately before invocation.
*
* The registry stores the *raw, untruncated* tool output so a later
* `{{…}}` substitution pipes the full payload into the next tool β€”
* even when the LLM only saw a head+tail-truncated preview in
* `ToolMessage.content`. Outputs are stored without any annotation
* (the `_ref` key or the `[ref: ...]` prefix seen by the LLM is
* strictly a UX signal attached to `ToolMessage.content`). Keeping the
* registry pristine means downstream bash/jq piping receives the
* complete, verbatim output with no injected fields.
*/
import { ToolMessage } from '@langchain/core/messages';
import type { BaseMessage } from '@langchain/core/messages';
import {
calculateMaxTotalToolOutputSize,
HARD_MAX_TOOL_RESULT_CHARS,
HARD_MAX_TOTAL_TOOL_OUTPUT_SIZE,
} from '@/utils/truncation';
/**
* Non-global matcher for a single `{{tool<i>turn<n>}}` placeholder.
* Exported for consumers that want to detect references (e.g., syntax
* highlighting, docs). The stateful `g` variant lives inside the
* registry so nobody trips on `lastIndex`.
*/
export const TOOL_OUTPUT_REF_PATTERN = /\{\{(tool\d+turn\d+)\}\}/;
/** Object key used when a parsed-object output has `_ref` injected. */
export const TOOL_OUTPUT_REF_KEY = '_ref';
/**
* Object key used to carry unresolved reference warnings on a parsed-
* object output. Using a dedicated field instead of a trailing text
* line keeps the annotated `ToolMessage.content` parseable as JSON for
* downstream consumers that rely on the object shape.
*/
export const TOOL_OUTPUT_UNRESOLVED_KEY = '_unresolved_refs';
/** Single-line prefix prepended to non-object tool outputs so the LLM sees the reference key. */
export function buildReferencePrefix(key: string): string {
return `[ref: ${key}]`;
}
/** Stable registry key for a tool output. */
export function buildReferenceKey(toolIndex: number, turn: number): string {
return `tool${toolIndex}turn${turn}`;
}
export type ToolOutputReferenceRegistryOptions = {
/** Maximum characters stored per registered output. */
maxOutputSize?: number;
/** Maximum total characters retained across all registered outputs. */
maxTotalSize?: number;
/**
* Upper bound on the number of concurrently-tracked runs. When
* exceeded, the oldest run bucket is evicted (FIFO). Defaults to 32.
*/
maxActiveRuns?: number;
};
/**
* Result of resolving placeholders in tool args.
*/
export type ResolveResult<T> = {
/** Arguments with placeholders replaced. Same shape as the input. */
resolved: T;
/** Reference keys that were referenced but had no stored value. */
unresolved: string[];
};
/**
* Read-only view over a frozen registry snapshot. Returned by
* {@link ToolOutputReferenceRegistry.snapshot} for callers that need
* to resolve placeholders against the registry state at a specific
* point in time, ignoring any subsequent registrations.
*/
export interface ToolOutputResolveView {
resolve<T>(args: T): ResolveResult<T>;
}
/**
* Pre-resolved arg map keyed by `toolCallId`. Used by the mixed
* direct+event dispatch path to feed event calls' resolved args
* (captured pre-batch) into the dispatcher without re-resolving
* against the now-stale live registry.
*/
export type PreResolvedArgsMap = Map<
string,
{ resolved: Record<string, unknown>; unresolved: string[] }
>;
/**
* Per-call sink for resolved args, keyed by `toolCallId`. Threaded
* as a per-batch local map so concurrent `ToolNode.run()` calls do
* not race on shared sink state.
*/
export type ResolvedArgsByCallId = Map<string, Record<string, unknown>>;
const EMPTY_ENTRIES: ReadonlyMap<string, string> = new Map<string, string>();
/**
* Per-run state bucket held inside the registry. Each distinct
* `run_id` gets its own bucket so overlapping concurrent runs on a
* shared registry cannot leak outputs, turn counters, or warn-memos
* into one another.
*/
class RunStateBucket {
entries: Map<string, string> = new Map();
totalSize: number = 0;
turnCounter: number = 0;
warnedNonStringTools: Set<string> = new Set();
}
/**
* Anonymous (`run_id` absent) bucket key. Anonymous batches are
* treated as fresh runs on every invocation β€” see `nextTurn`.
*/
const ANON_RUN_KEY = '\0anon';
/**
* Default upper bound on the number of concurrently-tracked runs per
* registry. When exceeded, the oldest run's bucket (by insertion
* order) is evicted. Keeps memory bounded when a ToolNode is reused
* across many runs without explicit `releaseRun` calls.
*/
const DEFAULT_MAX_ACTIVE_RUNS = 32;
/**
* Ordered map of reference-key β†’ stored output, partitioned by run so
* concurrent / interleaved runs sharing one registry cannot leak
* outputs between each other.
*
* Each public method takes a `runId` which selects the run's bucket.
* Hosts typically get one registry per run via `Graph`, in which
* case only a single bucket is ever populated; the partitioning
* exists so the registry also behaves correctly when a single
* instance is reused directly.
*/
export class ToolOutputReferenceRegistry {
private runStates: Map<string, RunStateBucket> = new Map();
private readonly maxOutputSize: number;
private readonly maxTotalSize: number;
private readonly maxActiveRuns: number;
/**
* Local stateful matcher used only by `replaceInString`. Kept
* off-module so callers of the exported `TOOL_OUTPUT_REF_PATTERN`
* never see a stale `lastIndex`.
*/
private static readonly PLACEHOLDER_MATCHER = /\{\{(tool\d+turn\d+)\}\}/g;
constructor(options: ToolOutputReferenceRegistryOptions = {}) {
/**
* Per-output default is the same ~400 KB budget as the standard
* tool-result truncation (`HARD_MAX_TOOL_RESULT_CHARS`). This
* keeps a single `{{…}}` substitution at a size that is safe to
* pass through typical shell `ARG_MAX` limits and matches what
* the LLM would otherwise have seen. Hosts that want larger per-
* output payloads (API consumers, long JSON streams) can raise
* the cap explicitly up to the 5 MB total budget.
*/
const perOutput =
options.maxOutputSize != null && options.maxOutputSize > 0
? options.maxOutputSize
: HARD_MAX_TOOL_RESULT_CHARS;
/**
* Clamp a caller-supplied `maxTotalSize` to
* `HARD_MAX_TOTAL_TOOL_OUTPUT_SIZE` (5 MB) so the documented
* absolute cap is enforced regardless of host config β€”
* `calculateMaxTotalToolOutputSize` already applies the same
* upper bound on its computed default, but the user-provided
* branch was bypassing it.
*/
const totalRaw =
options.maxTotalSize != null && options.maxTotalSize > 0
? Math.min(options.maxTotalSize, HARD_MAX_TOTAL_TOOL_OUTPUT_SIZE)
: calculateMaxTotalToolOutputSize(perOutput);
this.maxTotalSize = totalRaw;
/**
* The per-output cap can never exceed the per-run aggregate cap:
* if a single entry were allowed to be larger than `maxTotalSize`,
* the eviction loop would either blow the cap (to keep the entry)
* or self-evict a just-stored value. Clamping here turns
* `maxTotalSize` into a hard upper bound on *any* state the
* registry retains per run.
*/
this.maxOutputSize = Math.min(perOutput, totalRaw);
this.maxActiveRuns =
options.maxActiveRuns != null && options.maxActiveRuns > 0
? options.maxActiveRuns
: DEFAULT_MAX_ACTIVE_RUNS;
}
private keyFor(runId: string | undefined): string {
return runId ?? ANON_RUN_KEY;
}
private getOrCreate(runId: string | undefined): RunStateBucket {
const key = this.keyFor(runId);
let state = this.runStates.get(key);
if (state == null) {
state = new RunStateBucket();
this.runStates.set(key, state);
if (this.runStates.size > this.maxActiveRuns) {
const oldest = this.runStates.keys().next().value;
if (oldest != null && oldest !== key) {
this.runStates.delete(oldest);
}
}
}
return state;
}
/** Registers (or replaces) the output stored under `key` for `runId`. */
set(runId: string | undefined, key: string, value: string): void {
const bucket = this.getOrCreate(runId);
const clipped =
value.length > this.maxOutputSize
? value.slice(0, this.maxOutputSize)
: value;
const existing = bucket.entries.get(key);
if (existing != null) {
bucket.totalSize -= existing.length;
bucket.entries.delete(key);
}
bucket.entries.set(key, clipped);
bucket.totalSize += clipped.length;
this.evictWithinBucket(bucket);
}
/** Returns the stored value for `key` in `runId`'s bucket, or `undefined`. */
get(runId: string | undefined, key: string): string | undefined {
return this.runStates.get(this.keyFor(runId))?.entries.get(key);
}
/**
* Returns `true` when `key` is currently stored in `runId`'s bucket.
* Used by {@link annotateMessagesForLLM} to gate transient annotation
* on whether the registry still owns the referenced output (a stale
* `_refKey` from a prior run silently no-ops here).
*/
has(runId: string | undefined, key: string): boolean {
return this.runStates.get(this.keyFor(runId))?.entries.has(key) ?? false;
}
/** Total number of registered outputs across every run bucket. */
get size(): number {
let n = 0;
for (const bucket of this.runStates.values()) {
n += bucket.entries.size;
}
return n;
}
/** Maximum characters retained per output (post-clip). */
get perOutputLimit(): number {
return this.maxOutputSize;
}
/** Maximum total characters retained *per run*. */
get totalLimit(): number {
return this.maxTotalSize;
}
/** Drops every run's state. */
clear(): void {
this.runStates.clear();
}
/**
* Explicitly release `runId`'s state. Safe to call when a run has
* finished. Hosts sharing one registry across runs should call this
* to reclaim memory deterministically; otherwise LRU eviction kicks
* in when `maxActiveRuns` runs accumulate.
*/
releaseRun(runId: string | undefined): void {
this.runStates.delete(this.keyFor(runId));
}
/**
* Claims the next batch turn synchronously from `runId`'s bucket.
*
* Must be called once at the start of each ToolNode batch before
* any `await`, so concurrent invocations within the same run see
* distinct turn values (reads are effectively atomic by JS's
* single-threaded execution of the sync prefix).
*
* If `runId` is missing the anonymous bucket is dropped and a
* fresh one created so each anonymous call behaves as its own run.
*/
nextTurn(runId: string | undefined): number {
if (runId == null) {
this.runStates.delete(ANON_RUN_KEY);
}
const bucket = this.getOrCreate(runId);
return bucket.turnCounter++;
}
/**
* Records that `toolName` has been warned about in `runId` (returns
* `true` on the first call per run, `false` after). Used by
* ToolNode to emit one log line per offending tool per run when a
* `ToolMessage.content` isn't a string.
*/
claimWarnOnce(runId: string | undefined, toolName: string): boolean {
const bucket = this.getOrCreate(runId);
if (bucket.warnedNonStringTools.has(toolName)) {
return false;
}
bucket.warnedNonStringTools.add(toolName);
return true;
}
/**
* Walks `args` and replaces every `{{tool<i>turn<n>}}` placeholder in
* string values with the stored output *from `runId`'s bucket*. Non-
* string values and object keys are left untouched. Unresolved
* references are left in-place and reported so the caller can
* surface them to the LLM. When no placeholder appears anywhere in
* the serialized args, the original input is returned without
* walking the tree.
*/
resolve<T>(runId: string | undefined, args: T): ResolveResult<T> {
if (!hasAnyPlaceholder(args)) {
return { resolved: args, unresolved: [] };
}
const bucket = this.runStates.get(this.keyFor(runId));
return this.resolveAgainst(bucket?.entries ?? EMPTY_ENTRIES, args);
}
/**
* Captures a frozen snapshot of `runId`'s current entries and
* returns a view that resolves placeholders against *only* that
* snapshot. The snapshot is decoupled from the live registry, so
* subsequent `set()` calls (for example, same-turn direct outputs
* registering while an event branch is still in flight) are
* invisible to the snapshot's `resolve`. Used by the mixed
* direct+event dispatch path to preserve same-turn isolation when
* a `PreToolUse` hook rewrites event args after directs have
* completed.
*/
snapshot(runId: string | undefined): ToolOutputResolveView {
const bucket = this.runStates.get(this.keyFor(runId));
const entries: ReadonlyMap<string, string> = bucket
? new Map(bucket.entries)
: EMPTY_ENTRIES;
return {
resolve: <T>(args: T): ResolveResult<T> =>
this.resolveAgainst(entries, args),
};
}
private resolveAgainst<T>(
entries: ReadonlyMap<string, string>,
args: T
): ResolveResult<T> {
if (!hasAnyPlaceholder(args)) {
return { resolved: args, unresolved: [] };
}
const unresolved = new Set<string>();
const resolved = this.transform(entries, args, unresolved) as T;
return { resolved, unresolved: Array.from(unresolved) };
}
private transform(
entries: ReadonlyMap<string, string>,
value: unknown,
unresolved: Set<string>
): unknown {
if (typeof value === 'string') {
return this.replaceInString(entries, value, unresolved);
}
if (Array.isArray(value)) {
return value.map((item) => this.transform(entries, item, unresolved));
}
if (value !== null && typeof value === 'object') {
const source = value as Record<string, unknown>;
const next: Record<string, unknown> = {};
for (const [key, item] of Object.entries(source)) {
next[key] = this.transform(entries, item, unresolved);
}
return next;
}
return value;
}
private replaceInString(
entries: ReadonlyMap<string, string>,
input: string,
unresolved: Set<string>
): string {
if (input.indexOf('{{tool') === -1) {
return input;
}
return input.replace(
ToolOutputReferenceRegistry.PLACEHOLDER_MATCHER,
(match, key: string) => {
const stored = entries.get(key);
if (stored == null) {
unresolved.add(key);
return match;
}
return stored;
}
);
}
private evictWithinBucket(bucket: RunStateBucket): void {
if (bucket.totalSize <= this.maxTotalSize) {
return;
}
for (const key of bucket.entries.keys()) {
if (bucket.totalSize <= this.maxTotalSize) {
return;
}
const entry = bucket.entries.get(key);
if (entry == null) {
continue;
}
bucket.totalSize -= entry.length;
bucket.entries.delete(key);
}
}
}
/**
* Cheap pre-check: returns true if any string value in `args` contains
* the `{{tool` substring. Lets `resolve()` skip the deep tree walk (and
* its object allocations) for the common case of plain args.
*/
function hasAnyPlaceholder(value: unknown): boolean {
if (typeof value === 'string') {
return value.indexOf('{{tool') !== -1;
}
if (Array.isArray(value)) {
for (const item of value) {
if (hasAnyPlaceholder(item)) {
return true;
}
}
return false;
}
if (value !== null && typeof value === 'object') {
for (const item of Object.values(value as Record<string, unknown>)) {
if (hasAnyPlaceholder(item)) {
return true;
}
}
return false;
}
return false;
}
/**
* Annotates `content` with a reference key and/or unresolved-ref
* warnings so the LLM sees both alongside the tool output.
*
* Behavior:
* - If `content` parses as a plain (non-array, non-null) JSON object
* and the object does not already have a conflicting `_ref` key,
* the reference key and (when present) `_unresolved_refs` array
* are injected as object fields, preserving JSON validity for
* downstream consumers that parse the output.
* - Otherwise (string output, JSON array/primitive, parse failure,
* or `_ref` collision), a `[ref: <key>]\n` prefix line is
* prepended and unresolved refs are appended as a trailing
* `[unresolved refs: …]` line.
*
* The annotated string is what the LLM sees as `ToolMessage.content`.
* The *original* (un-annotated) value is what gets stored in the
* registry, so downstream piping remains pristine.
*
* @param content Raw (post-truncation) tool output.
* @param key Reference key for this output, or undefined when
* there is nothing to register (errors etc.).
* @param unresolved Reference keys that failed to resolve during
* argument substitution. Surfaced so the LLM can
* self-correct its next tool call.
*/
export function annotateToolOutputWithReference(
content: string,
key: string | undefined,
unresolved: string[] = []
): string {
const hasRefKey = key != null;
const hasUnresolved = unresolved.length > 0;
if (!hasRefKey && !hasUnresolved) {
return content;
}
const trimmed = content.trimStart();
if (trimmed.startsWith('{')) {
const annotated = tryInjectRefIntoJsonObject(content, key, unresolved);
if (annotated != null) {
return annotated;
}
}
const prefix = hasRefKey ? `${buildReferencePrefix(key!)}\n` : '';
const trailer = hasUnresolved
? `\n[unresolved refs: ${unresolved.join(', ')}]`
: '';
return `${prefix}${content}${trailer}`;
}
function tryInjectRefIntoJsonObject(
content: string,
key: string | undefined,
unresolved: string[]
): string | null {
let parsed: unknown;
try {
parsed = JSON.parse(content);
} catch {
return null;
}
if (parsed === null || typeof parsed !== 'object' || Array.isArray(parsed)) {
return null;
}
const obj = parsed as Record<string, unknown>;
const injectingRef = key != null;
const injectingUnresolved = unresolved.length > 0;
/**
* Reject the JSON-injection path (fall back to prefix form) when
* either of our keys collides with real payload data:
* - `_ref` collision: existing value is non-null and differs from
* the key we're about to inject.
* - `_unresolved_refs` collision: existing value is non-null and
* is not a deep-equal match for the array we'd inject.
* This keeps us from silently overwriting legitimate tool output.
*/
if (
injectingRef &&
TOOL_OUTPUT_REF_KEY in obj &&
obj[TOOL_OUTPUT_REF_KEY] !== key &&
obj[TOOL_OUTPUT_REF_KEY] != null
) {
return null;
}
if (
injectingUnresolved &&
TOOL_OUTPUT_UNRESOLVED_KEY in obj &&
obj[TOOL_OUTPUT_UNRESOLVED_KEY] != null &&
!arraysShallowEqual(obj[TOOL_OUTPUT_UNRESOLVED_KEY], unresolved)
) {
return null;
}
/**
* Only strip the framework-owned key we're actually injecting β€”
* leave everything else (including a pre-existing `_ref` on the
* unresolved-only path, or a pre-existing `_unresolved_refs` on a
* plain-annotation path) untouched so we annotate rather than
* mutate downstream payload data. Our injected keys land first in
* the serialized JSON so the LLM sees them before the body.
*/
const omitKeys = new Set<string>();
if (injectingRef) omitKeys.add(TOOL_OUTPUT_REF_KEY);
if (injectingUnresolved) omitKeys.add(TOOL_OUTPUT_UNRESOLVED_KEY);
const rest: Record<string, unknown> = {};
for (const [k, v] of Object.entries(obj)) {
if (!omitKeys.has(k)) {
rest[k] = v;
}
}
const injected: Record<string, unknown> = {};
if (injectingRef) {
injected[TOOL_OUTPUT_REF_KEY] = key;
}
if (injectingUnresolved) {
injected[TOOL_OUTPUT_UNRESOLVED_KEY] = unresolved;
}
Object.assign(injected, rest);
const pretty = /^\{\s*\n/.test(content);
return pretty ? JSON.stringify(injected, null, 2) : JSON.stringify(injected);
}
function arraysShallowEqual(a: unknown, b: readonly string[]): boolean {
if (!Array.isArray(a) || a.length !== b.length) {
return false;
}
for (let i = 0; i < a.length; i++) {
if (a[i] !== b[i]) {
return false;
}
}
return true;
}
/**
* Lazy projection that, given a registry and a runId, returns a new
* `messages` array where each `ToolMessage` carrying ref metadata is
* projected into a transient copy with annotated content (when the ref
* is live in the registry) and with the framework-owned `additional_
* kwargs` keys (`_refKey`, `_refScope`, `_unresolvedRefs`) stripped
* regardless of whether annotation applied. The original input array
* and its messages are never mutated.
*
* Annotation is gated on registry presence: a stale `_refKey` from a
* prior run (e.g. one that survived in persisted history) silently
* no-ops on the *content* side. The strip-metadata side still runs so
* stale framework keys never leak onto the wire under any custom or
* future provider serializer that might transmit `additional_kwargs`.
* `_unresolvedRefs` is always meaningful and is not gated.
*
* **Feature-disabled fast path:** when the host hasn't enabled the
* tool-output-reference feature, the registry is `undefined` and this
* function returns the input array reference-equal *without iterating
* a single message*. The loop is exclusive to the feature-enabled
* code path.
*/
export function annotateMessagesForLLM(
messages: BaseMessage[],
registry: ToolOutputReferenceRegistry | undefined,
runId: string | undefined
): BaseMessage[] {
if (registry == null) return messages;
/**
* Lazy-allocate the output array so the common case (no ToolMessage
* carries framework metadata) returns the input reference-equal with
* zero allocations beyond the per-message predicate checks.
*/
let out: BaseMessage[] | undefined;
for (let i = 0; i < messages.length; i++) {
const m = messages[i];
if (m._getType() !== 'tool') continue;
/**
* `additional_kwargs` is untyped at the LangChain layer
* (`Record<string, unknown>`), so persisted or client-supplied
* ToolMessages can carry arbitrary shapes β€” including primitives
* (a malformed serializer might write a string, or `null`).
* Guard with a runtime object check before the `in` probes
* because the `in` operator throws `TypeError` on primitives.
* A single malformed message must never crash the provider call
* path; skip its annotation/strip and continue.
*/
const rawMeta = m.additional_kwargs as unknown;
if (rawMeta == null || typeof rawMeta !== 'object') continue;
const meta = rawMeta as Record<string, unknown>;
const hasRefKey = '_refKey' in meta;
const hasRefScope = '_refScope' in meta;
const hasUnresolvedField = '_unresolvedRefs' in meta;
if (!hasRefKey && !hasRefScope && !hasUnresolvedField) continue;
const refKey = readRefKey(meta);
const unresolved = readUnresolvedRefs(meta);
/**
* Prefer the message-stamped `_refScope` for the registry lookup.
* For named runs it equals the current `runId`; for anonymous
* invocations it carries the per-batch synthetic scope minted by
* ToolNode (`\0anon-<n>`), which `runId` from config cannot
* recover. Falling back to `runId` keeps backward compatibility
* with messages stamped before this field existed.
*/
const lookupScope = readRefScope(meta) ?? runId;
const liveRef =
refKey != null && registry.has(lookupScope, refKey) ? refKey : undefined;
const annotates = liveRef != null || unresolved.length > 0;
const tm = m as ToolMessage;
let nextContent: ToolMessage['content'] = tm.content;
if (annotates && typeof tm.content === 'string') {
nextContent = annotateToolOutputWithReference(
tm.content,
liveRef,
unresolved
);
} else if (annotates && Array.isArray(tm.content)) {
/**
* Array tool content. The string annotator can't run β€” this notably
* includes a tail tool result that prompt caching rewrote from a string
* into a text-block array to host its `cache_control` / `cachePoint`
* marker (the `_refKey` survives on `additional_kwargs`). Project the
* same markers the string path would, as leading text blocks: the live
* `[ref: …]` prefix and/or the unresolved-refs warning. Without this the
* common tool-result tail loses its reference marker once cached.
*
* `as unknown as ToolMessage['content']` is unavoidable: LangChain's
* content union does not accept a freshly built mixed array literal even
* though the structural shape is valid at runtime. The double-cast is
* structurally safe β€” every original block is preserved and only
* `{ type: 'text', text }` blocks (which all providers accept) are
* prepended.
*/
const prefixBlocks: Array<{ type: 'text'; text: string }> = [];
if (liveRef != null) {
prefixBlocks.push({
type: 'text',
text: buildReferencePrefix(liveRef),
});
}
if (unresolved.length > 0) {
prefixBlocks.push({
type: 'text',
text: `[unresolved refs: ${unresolved.join(', ')}]`,
});
}
if (prefixBlocks.length > 0) {
nextContent = [
...prefixBlocks,
...tm.content,
] as unknown as ToolMessage['content'];
}
}
/**
* Project unconditionally: even when no annotation applies (stale
* `_refKey` or non-annotatable content), `cloneToolMessageWithContent`
* runs `stripFrameworkRefMetadata` on `additional_kwargs` so the
* framework-owned keys never reach the wire.
*/
out ??= messages.slice();
out[i] = cloneToolMessageWithContent(tm, nextContent);
}
return out ?? messages;
}
/**
* Reads `_refKey` defensively from untyped `additional_kwargs`. Returns
* undefined for non-string values so a malformed field cannot poison
* the registry lookup or downstream string operations.
*/
function readRefKey(
meta: Record<string, unknown> | undefined
): string | undefined {
const v = meta?._refKey;
return typeof v === 'string' ? v : undefined;
}
/**
* Reads `_refScope` defensively from untyped `additional_kwargs`.
* Mirrors {@link readRefKey} β€” non-string scopes are dropped (the
* caller falls back to the run-derived scope) rather than passed into
* the registry as a malformed key.
*/
function readRefScope(
meta: Record<string, unknown> | undefined
): string | undefined {
const v = meta?._refScope;
return typeof v === 'string' ? v : undefined;
}
/**
* Reads `_unresolvedRefs` defensively from untyped `additional_kwargs`.
* Returns an empty array for any non-array value, and filters out
* non-string entries from a real array. Without this guard, a hydrated
* ToolMessage carrying e.g. `_unresolvedRefs: 'tool0turn0'` would crash
* `attemptInvoke` on the eventual `.length` / `.join(...)` call.
*/
function readUnresolvedRefs(
meta: Record<string, unknown> | undefined
): string[] {
const v = meta?._unresolvedRefs;
if (!Array.isArray(v)) return [];
const out: string[] = [];
for (const item of v) {
if (typeof item === 'string') out.push(item);
}
return out;
}
/**
* Builds a fresh `ToolMessage` that mirrors `tm`'s identity fields with
* the supplied `content`. Every `ToolMessage` field but `content` is
* carried over so the projection is structurally identical to the
* original from a LangChain serializer's perspective.
*
* `additional_kwargs` is rebuilt with the framework-owned ref keys
* stripped. Defensive: LangChain's standard provider serializers do not
* transmit `additional_kwargs` to provider HTTP APIs, but a custom
* adapter or future LangChain change could. Stripping keeps the
* implementation correct under any serializer behavior at the cost of a
* shallow object spread per annotated message.
*/
function cloneToolMessageWithContent(
tm: ToolMessage,
content: ToolMessage['content']
): ToolMessage {
return new ToolMessage({
id: tm.id,
name: tm.name,
status: tm.status,
artifact: tm.artifact,
tool_call_id: tm.tool_call_id,
response_metadata: tm.response_metadata,
additional_kwargs: stripFrameworkRefMetadata(tm.additional_kwargs),
content,
});
}
/**
* Returns a copy of `kwargs` with `_refKey`, `_refScope`, and
* `_unresolvedRefs` removed. Returns the input reference-equal when
* none of those keys are present so the no-strip path stays cheap;
* returns `undefined` when stripping leaves the object empty so the
* caller can drop the field entirely.
*/
function stripFrameworkRefMetadata(
kwargs: Record<string, unknown> | undefined
): Record<string, unknown> | undefined {
if (kwargs == null) return undefined;
if (
!('_refKey' in kwargs) &&
!('_refScope' in kwargs) &&
!('_unresolvedRefs' in kwargs)
) {
return kwargs;
}
const { _refKey, _refScope, _unresolvedRefs, ...rest } = kwargs as Record<
string,
unknown
> & {
_refKey?: unknown;
_refScope?: unknown;
_unresolvedRefs?: unknown;
};
void _refKey;
void _refScope;
void _unresolvedRefs;
return Object.keys(rest).length === 0 ? undefined : rest;
}