hanzoai/insights
2
0
Fork 0
mirror of https://github.com/hanzoai/insights.git synced 2026-10-30 16:15:31 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
main
insights/products/experiments/mcp/tools.yaml
T
zeekay bc4ddf332b chore: rebase the tree onto upstream master fa1a9a60 
Mechanically re-derived: bin/debrand runs over a fresh import of
posthog/posthog@fa1a9a60d4 (2026-08-06).
The pipeline (.hanzo/workflows), the Dockerfiles and bin/docker*
entrypoints, compose files, LLM.md and the squashed initial migration
are restored from the shipped tree, same as the previous import.

The prune's import-closure guard kept 21 products the new core now
imports; dropping them is a refactor now, recorded rather than forced.
2026-08-06 01:25:36 -07:00

1326 lines
60 KiB
YAML
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# MCP tool definition — tool entries are scaffolded from the OpenAPI schema.
# The tool list and operation IDs are kept in sync automatically:
# pnpm --filter=@hanzo/mcp run scaffold-yaml -- --sync-all
#
# To enable a tool, set enabled: true and add required scopes + annotations.
# All other fields (title, description, enrich_url, etc.) are yours to configure.
category: Experiments
feature: experiments
url_prefix: /experiments
ui_apps:
experiment:
type: detail
view_prop: experiment
experiment-list:
type: list
detail_tool: experiment-get
detail_args: '{ id: item.id }'
experiment-results:
type: detail
view_prop: data
tools:
experiment-activity:
operation: experiments_activity_retrieve
enabled: true
scopes:
- activity_log:read
annotations:
readOnly: true
destructive: false
idempotent: true
title: Get experiment change history
description: >
Get the audit trail for a specific experiment by ID. Returns a paginated list of changes to the experiment
and its holdouts and shared metrics: who made each change, what was changed (field-level before/after
values), and when, newest first. Lifecycle actions (launch, end, ship variant, etc.) appear as updates to
the underlying fields (start_date, end_date, conclusion, ...). Use limit and page query params for
pagination. Note: changes to the experiment's linked feature flag (rollout, targeting) are logged on the
flag itself — use feature-flags-activity-retrieve with the flag's ID for those.
param_overrides:
id:
cast: string-int
experiment-archive:
operation: experiments_archive_create
enabled: true
scopes:
- experiment:write
annotations:
readOnly: false
destructive: false
idempotent: false
enrich_url: '{id}'
title: Archive experiment
description: >
Requires an experiment ID. If you don't have the ID, load the finding-experiments skill to resolve the
user's reference first. Load the managing-experiment-lifecycle skill for preconditions and side effects.
Archive a stopped experiment to hide it from the default list view. The experiment must be stopped (end_date
set) — cannot archive draft or running experiments.
Returns 400 if already archived ("Experiment is already archived.") or not yet stopped ("Experiment must be
ended before it can be archived.").
No request body needed. Can be restored later via the unarchive tool.
param_overrides:
id:
cast: string-int
ui_app: experiment
experiment-calculate-running-time:
operation: experiments_calculate_running_time_create
enabled: true
scopes:
- experiment:read
annotations:
readOnly: true
destructive: false
idempotent: true
title: Calculate experiment running time
description: >
Estimate the recommended sample size and how many days an experiment needs to run to detect an effect. This
is a stateless statistical calculation — it does not read or modify any experiment, so it works for planning
before an experiment exists.
Required: metric_type ('funnel' for conversion rates, 'mean_count' for events per user, 'mean_sum_or_avg'
for summed property values per user, 'ratio' or 'retention' for ratio-style metrics) and
minimum_detectable_effect (the smallest relative change to detect, as a percentage — e.g. 5 means a 5%
lift). number_of_variants defaults to 2 (control + one test).
Provide the baseline one of two ways: pass baseline_value directly (a conversion rate as a fraction 0-1 for
funnels, or an average per user for mean metrics), or pass baseline_stats (raw control-group statistics) and
let the server derive it. Ratio and retention metrics REQUIRE baseline_stats (with denominator_sum /
denominator_sum_squares / numerator_denominator_sum_product for the delta-method variance) or an explicit
variance — baseline_value alone is not enough.
Optionally pass exposure_rate_per_day (expected exposures per day across all variants) to also get
recommended_running_time_days. The response returns baseline_value, variance (null for funnels),
recommended_sample_size (total across variants), and recommended_running_time_days.
experiment-copy-to-project:
operation: experiments_copy_to_project_create
enabled: true
scopes:
- experiment:write
annotations:
readOnly: false
destructive: false
idempotent: false
title: Copy experiment to another project
description: >
Requires an experiment ID. If you don't have the ID, load the finding-experiments skill to resolve the
user's reference first. Load the managing-experiment-lifecycle skill for preconditions and side effects.
REQUIRES EXPLICIT USER CONFIRMATION BEFORE CALLING. This writes a new experiment into a DIFFERENT project
than the one the user is currently looking at. Resolve the target project from the user's wording to a
concrete team id, then confirm the source experiment and the target project by name before invoking.
Copies an experiment into another project in the SAME organization as a new draft. The target project must
belong to the same organization — this CANNOT copy across organizations or regions. Use experiment-duplicate
instead when the copy should land in the same project.
What IS copied: name (defaults to "Original Name (Copy)", de-duplicated with a numeric suffix if that name
already exists in the target), description, type, parameters (variant split, rollout), filters, primary and
secondary metrics (each with freshly regenerated uuids and preserved ordering), stats config, scheduling
config, exposure criteria, and the only_count_matured_users setting.
What is NOT copied: saved-metric references (saved metrics are project-scoped, so they are dropped on a
cross-project copy), holdout, exposure cohort, start/end dates, results, and conclusion. The copy always
starts as a fresh draft.
Feature flag: pass feature_flag_key to control the flag key created in the target project. If omitted, the
source experiment's flag key is reused — and if a flag with that key already exists in the target project,
the copy SHARES that existing flag (its variants are reused) rather than creating a new one. A shared flag
means lifecycle operations on either experiment (shipping a variant, pausing) affect both. To avoid this,
pass a feature_flag_key that does not already exist in the target project. If an existing target flag is
reused, it must be multivariate with 2 to 20 variants, otherwise the call returns 400 ("Feature flag must
have at least 2 variants (a baseline and at least one test variant)" or "Feature flag must have at most 20
variants"). No specific variant key is required — the analysis baseline defaults to the variant keyed
"control" when present, else the first variant. Exception: copying a web experiment requires the reused
target flag to have a variant keyed "control", otherwise the call returns 400 ("Web experiments require a
variant with key 'control'").
Returns 400 if the source experiment uses legacy metrics ("Copying is not supported for experiments using
legacy metrics."). Returns 404 if the target project is not found in the organization ("Target team not
found."). Returns 403 if you lack write access to the target project ("You do not have write access to the
target project.").
The returned experiment (including its id) belongs to the TARGET project, not the source project.
param_overrides:
id:
cast: string-int
target_team_id:
cast: string-int
response:
include:
- id
- name
- description
- type
- feature_flag_key
- status
- archived
- start_date
- end_date
- created_at
- parameters
- metrics
- metrics_secondary
- conclusion
- conclusion_comment
experiment-create:
operation: experiments_create
enabled: true
scopes:
- experiment:write
annotations:
readOnly: false
destructive: false
idempotent: false
enrich_url: '{id}'
title: Create experiment
description: >
RULES (follow before calling):
1. Load the creating-experiments skill for the full
creation workflow.
2. If the user mentions a rollout percentage (e.g. "25%"),
load the configuring-experiment-rollout skill and
ask the user to clarify BEFORE calling this tool.
A percentage is always ambiguous — it can mean a
variant split change OR an overall rollout change.
3. Do NOT pass metrics on creation. Create the draft
first, then add metrics via experiment-update. Use
the configuring-experiment-analytics skill for
metric guidance.
Creates a new experiment in draft status (ignore the "type" param, since no-code/toolbar experiments must be
created in the Insights UI). Requires name and feature_flag_key. Feature flag is auto-created — do not create
one separately. Defaults: 50/50 control/test, 100% rollout, Bayesian stats. Launch the experiment with the
launch tool when ready.
include_params:
- name
- description
- feature_flag_key
- feature_flag
- parameters
- stats_config
- holdout_id
- allow_unknown_events
- exposure_criteria
param_overrides:
feature_flag:
description: >
Variant split, rollout scope, payloads, and experience continuity for the auto-created feature flag,
in the flag's own filters shape. This is the canonical input for flag config. If the user mentions a
specific percentage, load the configuring-experiment-rollout skill and clarify before setting these
values. Set filters.multivariate.variants (each with key and rollout_percentage; percentages must
sum to 100) to customize the variant split. Set filters.groups to a single group [{"properties": [],
"rollout_percentage": N}] (0-100) to control the overall fraction of users entering the experiment.
Default: 50/50 control/test, 100% rollout. Omit this parameter entirely when feature_flag_key refers
to a pre-existing flag: the experiment links to that flag as-is and explicit config is rejected.
No specific variant key is required. The analysis baseline defaults to the variant keyed `control`
(lowercase) when present, else the first variant; override with stats_config.baseline_variant_key.
Convention: when the user describes variants as "A/B", "old/new", "original/redesign", or any other
natural-language pair without naming explicit keys, key the baseline `control` and keep their
wording in the variant `name`. When the user asks for specific keys, use them as-is and put the
baseline first.
ui_app: experiment
response:
include:
- id
- name
- description
- type
- feature_flag_key
- status
- archived
- start_date
- end_date
- created_at
- parameters
- metrics
- metrics_secondary
- conclusion
- conclusion_comment
experiment-create-from-prompt:
operation: experiments_create_from_prompt_create
enabled: true
scopes:
- experiment:write
- llm_prompt:read
annotations:
readOnly: false
destructive: false
idempotent: false
enrich_url: '{id}'
title: Create experiment from LLM prompt
description: >
Create a draft experiment that compares versions of an existing LLM prompt (managed with the llma-prompt
tools). Pass the prompt's exact name (call llma-prompt-list to discover names), two or more of its version
numbers (the first entry becomes the control variant), and one or more metric templates — call
experiment-prompt-templates for the allowed template keys. Builds one variant per version, each carrying a
feature flag payload with the prompt name and version so the SDK can resolve which version to serve, and
attaches one primary metric per template, scoped to the prompt. The feature flag is auto-created — do not
create one separately. Launch the experiment with the launch tool when ready.
ui_app: experiment
response:
include:
- id
- name
- description
- type
- feature_flag_key
- status
- archived
- start_date
- end_date
- created_at
- parameters
- metrics
- metrics_secondary
- conclusion
- conclusion_comment
experiment-delete:
operation: experiments_destroy
enabled: true
scopes:
- experiment:write
annotations:
readOnly: false
destructive: true
idempotent: true
title: Delete experiment
description: >
Delete an experiment by ID. If you don't have the ID, load the finding-experiments skill to resolve the
user's reference first. Always confirm by name before deleting.
param_overrides:
id:
cast: string-int
soft_delete: true
response:
include:
- id
- name
- deleted
experiment-duplicate:
operation: experiments_duplicate_create
enabled: true
scopes:
- experiment:write
annotations:
readOnly: false
destructive: false
idempotent: false
title: Duplicate experiment
description: >
Load the managing-experiment-lifecycle skill for preconditions and side effects.
Create a copy of an experiment as a new draft. The duplicate includes the original's metrics, parameters,
and configuration but starts fresh with no dates or results. Rejects experiments that use legacy metrics
(ExperimentTrendsQuery/ExperimentFunnelsQuery).
IMPORTANT: always provide a unique feature_flag_key that differs from the original experiment's flag key. If
omitted or if the same key is provided, the duplicate will reuse the original experiment's feature flag —
meaning changes to one experiment's flag (e.g. shipping a variant, pausing) will affect both experiments.
Always confirm the new flag key with the user before duplicating. Optionally provide a custom name (defaults
to "Original Name (Copy)").
include_params:
- name
- feature_flag_key
param_overrides:
id:
cast: string-int
experiment-end:
operation: experiments_end_create
enabled: true
scopes:
- experiment:write
annotations:
readOnly: false
destructive: false
idempotent: false
enrich_url: '{id}'
title: End experiment
description: >
Requires an experiment ID. If you don't have the ID, load the finding-experiments skill to resolve the
user's reference first. Load the managing-experiment-lifecycle skill for preconditions and side effects.
End a running experiment without shipping a variant. Sets end_date to now and transitions status to stopped.
The feature flag is NOT modified — users continue seeing their assigned variants and exposure events
continue to fire, but only data up to end_date is included in results.
Optionally provide conclusion ("won", "lost", "inconclusive", "stopped_early", "invalid") and
conclusion_comment.
Returns 400 if the experiment is in draft ("Experiment has not been launched yet.") or already stopped
("Experiment has already ended.").
Other options: use experiment-ship-variant to end AND roll out a winner. Use pause to temporarily deactivate
the flag without ending.
param_overrides:
id:
cast: string-int
ui_app: experiment
experiment-freeze-exposure:
operation: experiments_freeze_exposure_create
enabled: true
scopes:
- experiment:write
annotations:
readOnly: false
destructive: false
idempotent: false
enrich_url: '{id}'
title: Freeze experiment exposure
description: >
Requires an experiment ID. If you don't have the ID, load the finding-experiments skill to resolve the
user's reference first. Load the managing-experiment-lifecycle skill for preconditions, limitations, and
side effects.
Freeze exposure on a running experiment: stop enrolling NEW users while already-enrolled users keep their
variant and metrics keep flowing (end_date stays null). Snapshots the already-exposed users into a static
cohort and narrows every release condition on the feature flag to that cohort. Use for long-horizon metrics
(revenue, LTV, retention, renewals) where enrollment should stop but measurement should continue — unlike
end (stops measurement at end_date) or pause (deactivates the flag for everyone). Status becomes
"exposure_frozen". Reversible via experiment-unfreeze-exposure.
SYNCHRONOUS AND POTENTIALLY SLOW: the exposure scan and cohort snapshot run inline in this call, and
duration scales with the number of exposed persons — experiments with tens of thousands of exposed users can
take tens of seconds. Wait for the response rather than assuming failure or retrying.
Not applicable (returns 400) for: experiments that aren't running ("Experiment has not been launched yet.",
"Experiment has already ended.", "Cannot freeze a paused experiment. Resume it first.", "Experiment exposure
is already frozen."), group-aggregated experiments (a person cohort cannot freeze group-based matching),
experiments in a holdout (holdout assignment is evaluated before release conditions), flags with early
access conditions (also evaluated before release conditions), flags that are deleted, missing, or have no
release conditions, exposed sets too large to snapshot synchronously (the scan is bounded by a person cap
and a timeout), and experiments where more than a small share of exposures are anonymous ("personless")
users — they can never match a person cohort, so experiments on logged-out surfaces are not freezable. When
a freeze is rejected, explain the limitation to the user instead of retrying.
Good to know: SDKs using local evaluation cannot resolve static cohorts, so a frozen flag evaluates via the
/decide endpoint; exposures ingested in the final moments before freezing may miss the snapshot;
ship-variant and reset strip the freeze, while end leaves the flag narrowed to the snapshot cohort.
No request body needed.
param_overrides:
id:
cast: string-int
ui_app: experiment
experiment-get:
operation: experiments_retrieve
enabled: true
scopes:
- experiment:read
annotations:
readOnly: true
destructive: false
idempotent: true
enrich_url: '{id}'
title: Get experiment details
description: >
Get a single experiment by ID. If you don't have the ID, load the finding-experiments skill to resolve the
user's reference first.
Returns the full experiment object including: status (draft/running/paused/exposure_frozen/stopped —
"paused" is a virtual value returned when the experiment is launched but its feature flag has been
deactivated, "exposure_frozen" when its flag's release groups are narrowed to the already-exposed cohort so
enrollment is closed while metrics keep flowing), start_date and end_date, linked feature_flag (with
variants, filters, active state), metrics and metrics_secondary arrays, conclusion and conclusion_comment if
ended, parameters (including feature_flag_variants and recommended_running_time), stats_config, type
(web/product), and resolved_exposure_event (the event exposures are counted on when no custom exposure event
is configured — resolved server-side to $feature_flag_called or $experiment_exposure; use it in SQL instead
of hardcoding either name).
param_overrides:
id:
cast: string-int
ui_app: experiment
experiment-holdouts-create:
operation: experiment_holdouts_create
enabled: true
scopes:
- experiment_holdout:write
annotations:
readOnly: false
destructive: false
idempotent: false
title: Create experiment holdout
description: >
Create a holdout group — a reserved slice of users excluded from experiment exposure, used as a held-back
baseline. Requires name and a non-empty filters array. Link experiments to it afterwards by passing the
returned id as holdout_id on experiment-create or experiment-update. Note that you can't change which
holdout an experiment points to once that experiment is running; the holdout itself stays editable (see
experiment-holdouts-partial-update, whose edits cascade to all linked experiments).
include_params:
- name
- description
- filters
param_overrides:
filters:
description: >
Non-empty list of release-condition groups defining the held-out population. Each element needs a
`rollout_percentage` (0–100, may be fractional) — this is the EXCLUSION percentage, i.e. the share
of users held back from every experiment that references this holdout. `properties` optionally
narrows the group by person/group properties (same shape as feature-flag release conditions); pass
an empty array for an unconditional holdout. Do NOT set `variant` — the server sets it to
`holdout-{id}` automatically. Only the first element's `rollout_percentage` is embedded into each
linked experiment's feature flag. Example: [{ "rollout_percentage": 10, "properties": [] }].
experiment-holdouts-destroy:
operation: experiment_holdouts_destroy
enabled: true
scopes:
- experiment_holdout:write
annotations:
readOnly: false
destructive: true
idempotent: true
title: Delete experiment holdout
description: >
Delete a holdout group by ID. Use experiment-holdouts-list first if you don't have the ID, and always
confirm by name before deleting.
This clears the holdout reference from every linked experiment and feature flag (sets the flag's holdout to
null and nulls Experiment.holdout) — the affected experiments stop holding back any users. Returns 204 on
success.
param_overrides:
id:
cast: string-int
experiment-holdouts-list:
operation: experiment_holdouts_list
enabled: true
scopes:
- experiment_holdout:read
annotations:
readOnly: true
destructive: false
idempotent: true
list: true
title: List experiment holdouts
description: >
List the experiment holdout groups in the current project. A holdout reserves a stable slice of users who
are excluded from experiment exposure, so you can measure the long-term effect of all experiments against a
held-back baseline. A holdout is NOT a feature flag — it is a separate object that experiments reference via
holdout_id.
Each holdout has a name, optional description, and a filters array (the held-out population definition).
Holdouts are project-scoped. Use the returned id with experiment-create / experiment-update's holdout_id to
link an experiment to a holdout.
response:
include:
- id
- name
- description
- filters
- created_at
- updated_at
experiment-holdouts-partial-update:
operation: experiment_holdouts_partial_update
enabled: true
scopes:
- experiment_holdout:write
annotations:
readOnly: false
destructive: false
idempotent: false
title: Update experiment holdout
description: >
Update a holdout group's name, description, or filters by ID. Use experiment-holdouts-list first if you
don't have the ID.
CAUTION: editing the filters (e.g. the rollout/exclusion percentage) cascades atomically to EVERY experiment
linked to this holdout — it rewrites the exclusion_percentage on each linked feature flag, silently
re-assigning the held-out population for all of those experiments at once. Confirm with the user before
changing filters on a holdout that experiments depend on.
param_overrides:
id:
cast: string-int
filters:
description: >
Non-empty list of release-condition groups defining the held-out population. Each element needs a
`rollout_percentage` (0–100, may be fractional) — the EXCLUSION percentage. Do NOT set `variant`
(the server manages it as `holdout-{id}`). Changing this cascades to every linked experiment's
feature flag. Example: [{ "rollout_percentage": 10, "properties": [] }].
experiment-holdouts-retrieve:
operation: experiment_holdouts_retrieve
enabled: true
scopes:
- experiment_holdout:read
annotations:
readOnly: true
destructive: false
idempotent: true
title: Get experiment holdout details
description: >
Get a single experiment holdout group by ID, including its name, description, and filters array (the
held-out population definition). Use experiment-holdouts-list first if you don't have the ID.
param_overrides:
id:
cast: string-int
experiment-holdouts-update:
operation: experiment_holdouts_update
enabled: false
experiment-launch:
operation: experiments_launch_create
enabled: true
scopes:
- experiment:write
annotations:
readOnly: false
destructive: false
idempotent: false
enrich_url: '{id}'
title: Launch experiment
description: >
Requires an experiment ID. If you don't have the ID, load the finding-experiments skill to resolve the
user's reference first. Load the managing-experiment-lifecycle skill for preconditions and side effects.
Launch a draft experiment. Validates the feature flag is multivariate with 2 to 20 variants. Activates the
feature flag (sets active=true), sets start_date to the current server time, recomputes metric fingerprints,
and transitions status from draft to running.
Returns 400 if the experiment has already been launched ("Experiment has already been launched.") or if the
flag configuration is invalid.
No request body needed.
param_overrides:
id:
cast: string-int
ui_app: experiment
response:
include:
- id
- name
- description
- type
- feature_flag_key
- status
- archived
- start_date
- end_date
- created_at
- parameters
- metrics
- metrics_secondary
- conclusion
- conclusion_comment
experiment-list:
operation: experiments_list
enabled: true
scopes:
- experiment:read
annotations:
readOnly: true
destructive: false
idempotent: true
enrich_url: '{id}'
list: true
title: Get all experiments
description: >
List experiments in the current project. This is the primary tool for resolving experiment references — load
the finding-experiments skill for guidance on searching by name, status, recency, or description.
Supports filtering by status ("draft", "running", "paused", "exposure_frozen", "stopped", "complete" which
maps to stopped, or "all"), archived state (defaults to non-archived), feature_flag_id, created_by_id, and
free-text search on name. Supports ordering by an allowlisted set of fields (model fields plus computed
"duration" and "status"). Returns paginated results with each experiment's status, dates, feature flag key,
and metrics summary. Use the returned ID for get/update/lifecycle tools.
param_overrides:
status:
description: >
Filter by experiment status. Values: "draft" (not yet launched), "running" (launched, flag active),
"paused" (launched, flag deactivated — mutually exclusive with running), "exposure_frozen"
(launched, enrollment frozen to the already-exposed cohort while metrics keep flowing), "stopped" or
"complete" (both mean ended), "all" (no filter). Defaults to all non-archived experiments.
limit:
cast: string-int
offset:
cast: string-int
feature_flag_id:
cast: string-int
ui_app: experiment-list
response:
include:
- id
- name
- description
- feature_flag_key
- feature_flag
- start_date
- end_date
- archived
- type
- status
- created_at
- updated_at
experiment-metrics-recalculation-create:
operation: experiments_metrics_recalculation_create
enabled: true
scopes:
- experiment:write
annotations:
readOnly: false
destructive: false
idempotent: true
enrich_url: '{params.id}'
title: Recalculate experiment metrics
description: >
Requires an experiment ID. If you don't have the ID, load the finding-experiments skill to resolve the
user's reference first.
Start a fresh calculation of every metric on this experiment against the latest data. Use this when the user
asks to refresh, recalculate, or re-run results, or when results returned by
experiment-metrics-recalculation-latest look stale. Reading results does NOT trigger a new run, this tool is
the only way to start one.
This queues background work that can take minutes on large experiments. Safe to call more than once: if a
run is already pending or in progress, it returns that run (with is_existing true, HTTP 200) instead of
starting a second one.
The response has no results array, the run has only just been queued. Take the returned id and poll
experiment-metrics-recalculation-retrieve with it, or call experiment-metrics-recalculation-latest, until
status is completed or failed. Watch rows_read and completed_metrics for progress. Do not poll in a tight
loop; wait a few seconds between checks.
exclude_params:
- trigger
param_overrides:
id:
cast: string-int
experiment-metrics-recalculation-latest-retrieve:
operation: experiments_metrics_recalculation_latest_retrieve
enabled: true
scopes:
- experiment:read
annotations:
readOnly: true
destructive: false
idempotent: true
enrich_url: '{params.id}'
title: Get experiment results
description: >
The default way to read an experiment's current results. Requires an experiment ID. If you don't have the
ID, load the finding-experiments skill to resolve the user's reference first. To interpret what the numbers
mean, load the diagnosing-experiment-results skill.
Returns the most recent completed metrics run for the experiment: a results array with one entry per metric,
each carrying per-variant statistics (exposures, counts, means, credible intervals, chance to beat control,
significance), plus run-level status, created_at, completed_at, and query_to (the data freshness cutoff
these numbers were computed against, use it to tell the user how fresh the results are).
This is a pure read and never starts a calculation, so the numbers can be stale. If the user wants fresh
numbers, or query_to is old, call experiment-metrics-recalculation-create and then poll.
Two fields matter for reading the response correctly. active_run is present when a run is currently
executing, the returned results are from the previous run, and you can poll active_run.id with
experiment-metrics-recalculation-retrieve for live progress. result_source is 'timeseries_fallback' when the
experiment has never completed a run and these numbers are a cold-start placeholder built from daily
timeseries data, rather than 'recalculation' for a real run; say so if you report those numbers.
Returns 404 when the experiment has no results at all yet, which usually means it has not run long enough or
has never been calculated.
param_overrides:
id:
cast: string-int
experiment-metrics-recalculation-retrieve:
operation: experiments_metrics_recalculation_retrieve
enabled: true
scopes:
- experiment:read
annotations:
readOnly: true
destructive: false
idempotent: true
enrich_url: '{params.id}'
title: Get experiment recalculation run
description: >
Get one specific metrics recalculation run by its UUID, for the experiment it belongs to. Use this to poll a
run you started with experiment-metrics-recalculation-create, or the active_run.id returned by
experiment-metrics-recalculation-latest.
To read an experiment's current results without a specific run id, use
experiment-metrics-recalculation-latest instead.
Returns the run's status (pending, in_progress, completed, failed), progress counters (completed_metrics and
failed_metrics out of total_metrics, plus rows_read against estimated_rows_total), the results array once
metrics finish, and metric_errors for metrics that failed. metric_retries holds transient state for metrics
between failed attempts; ignore entries for metrics that already have a result, as those are stale.
Returns 404 if the run id does not exist or belongs to a different experiment.
param_overrides:
id:
cast: string-int
experiment-pause:
operation: experiments_pause_create
enabled: true
scopes:
- experiment:write
annotations:
readOnly: false
destructive: false
idempotent: false
enrich_url: '{id}'
title: Pause experiment
description: >
Requires an experiment ID. If you don't have the ID, load the finding-experiments skill to resolve the
user's reference first. Load the managing-experiment-lifecycle skill for preconditions and side effects.
Pause a running experiment by deactivating its feature flag (sets flag active=false). The flag is no longer
returned by the /decide endpoint, so users fall back to the application default (typically control). No new
exposure events are recorded while paused. The experiment stays in running status — it is not ended.
Returns 400 if: experiment is in draft ("Experiment has not been launched yet."), already stopped
("Experiment has already ended."), no linked flag ("Experiment does not have a feature flag linked."), or
already paused ("Experiment is already paused.").
No request body needed. Use resume to reactivate.
param_overrides:
id:
cast: string-int
ui_app: experiment
experiment-prompt-templates:
operation: experiments_prompt_templates_retrieve
enabled: true
scopes:
- experiment:read
annotations:
readOnly: true
destructive: false
idempotent: true
title: List LLM prompt metric templates
description: >
List the metric templates accepted by experiment-create-from-prompt. Each entry has a key (the value to pass
in that tool's templates parameter), a label, and a description of what the metric measures.
experiment-reset:
operation: experiments_reset_create
enabled: true
scopes:
- experiment:write
annotations:
readOnly: false
destructive: false
idempotent: false
enrich_url: '{id}'
title: Reset experiment
description: >
Requires an experiment ID. If you don't have the ID, load the finding-experiments skill to resolve the
user's reference first. Load the managing-experiment-lifecycle skill for preconditions and side effects.
Reset an experiment back to draft state. Clears start_date, end_date, conclusion, conclusion_comment, and
archived flag (all set to null/false). The feature flag is left unchanged — users continue to see their
currently assigned variants.
Previously collected events still exist in the database but won't be included in results unless start_date
is manually adjusted after re-launch.
Returns 400 if the experiment is already in draft ("Experiment is already in draft state.").
No request body needed.
param_overrides:
id:
cast: string-int
ui_app: experiment
experiment-resume:
operation: experiments_resume_create
enabled: true
scopes:
- experiment:write
annotations:
readOnly: false
destructive: false
idempotent: false
enrich_url: '{id}'
title: Resume experiment
description: >
Requires an experiment ID. If you don't have the ID, load the finding-experiments skill to resolve the
user's reference first. Load the managing-experiment-lifecycle skill for preconditions and side effects.
Resume a paused experiment by reactivating its feature flag (sets flag active=true). Users are re-bucketed
deterministically into the same variants they had before the pause, and exposure tracking resumes.
Returns 400 if: experiment is in draft ("Experiment has not been launched yet."), already stopped
("Experiment has already ended."), no linked flag ("Experiment does not have a feature flag linked."), or
not currently paused ("Experiment is not paused.").
No request body needed.
param_overrides:
id:
cast: string-int
ui_app: experiment
experiment-saved-metrics-create:
operation: experiment_saved_metrics_create
enabled: true
scopes:
- experiment_saved_metric:write
annotations:
readOnly: false
destructive: false
idempotent: false
title: Create shared experiment metric
description: >
Create a reusable shared metric. Requires name (unique per project, case-insensitive) and query
(kind='ExperimentMetric', metric_type one of 'mean', 'funnel', 'ratio', 'retention'). Optional: description,
tags.
Load the configuring-experiment-analytics skill before constructing the query, and use read-data-schema to
confirm event names. Legacy kinds (ExperimentTrendsQuery, ExperimentFunnelsQuery) are rejected.
To attach to an experiment, call experiment-update with saved_metrics_ids — each item has 'id' and
'metadata.type' ('primary' or 'secondary').
experiment-saved-metrics-destroy:
operation: experiment_saved_metrics_destroy
enabled: true
scopes:
- experiment_saved_metric:write
annotations:
readOnly: false
destructive: true
idempotent: true
title: Delete shared experiment metric
description: >
Delete a shared metric by ID. REQUIRES EXPLICIT USER CONFIRMATION — confirm by name first, and warn that any
experiments using it will lose access.
param_overrides:
id:
cast: string-int
experiment-saved-metrics-list:
operation: experiment_saved_metrics_list
enabled: true
scopes:
- experiment_saved_metric:read
annotations:
readOnly: true
destructive: false
idempotent: true
list: true
title: List shared experiment metrics
description: >
List shared metrics in the current project. Returns paginated results with id, name, description, query,
created_at, updated_at, tags.
To REUSE a metric by what it measures (rather than by its name): pass ?event=<the event you're measuring> to
find metrics that reference it — directly or via an action — then compare each candidate's 'query'
(metric_type + math) against the metric you'd otherwise build. Use 'search' (name/description/tags) only
when the user named a specific metric to attach.
param_overrides:
limit:
cast: string-int
offset:
cast: string-int
event:
description: >
Filter to shared metrics whose query references this event name — matched directly (an EventsNode)
or via the step events of any action the metric references. For finding a reusable metric by what it
measures; then confirm the match against each row's 'query'.
response:
include:
- id
- name
- description
- query
- created_at
- updated_at
- tags
experiment-saved-metrics-partial-update:
operation: experiment_saved_metrics_partial_update
enabled: true
scopes:
- experiment_saved_metric:write
annotations:
readOnly: false
destructive: false
idempotent: true
title: Update shared experiment metric
description: >
Partially update a shared metric. Only name, description, and query are mutable. Edits propagate to every
experiment the metric is attached to — REQUIRES EXPLICIT USER CONFIRMATION before changing query or name on
a metric attached to running experiments.
Legacy-format metrics (kind=ExperimentTrendsQuery / ExperimentFunnelsQuery) cannot be updated.
param_overrides:
id:
cast: string-int
experiment-saved-metrics-retrieve:
operation: experiment_saved_metrics_retrieve
enabled: true
scopes:
- experiment_saved_metric:read
annotations:
readOnly: true
destructive: false
idempotent: true
title: Get shared experiment metric
description: >
Get a single shared metric by ID. Returns the full query JSON (metric_type, source, math, filters, etc.). If
you don't have the ID, call experiment-saved-metrics-list first.
param_overrides:
id:
cast: string-int
experiment-saved-metrics-update:
operation: experiment_saved_metrics_update
enabled: false
experiment-ship-variant:
operation: experiments_ship_variant_create
enabled: true
scopes:
- experiment:write
annotations:
readOnly: false
destructive: true
idempotent: false
enrich_url: '{id}'
title: Ship experiment variant
description: >
Requires an experiment ID. If you don't have the ID, load the finding-experiments skill to resolve the
user's reference first. Load the managing-experiment-lifecycle skill for preconditions and side effects.
REQUIRES EXPLICIT USER CONFIRMATION BEFORE CALLING. This permanently rewrites the linked feature flag and
cannot be undone via the API. Two release modes — confirm which one the user wants before invoking:
- Default (release_to_everyone=false): updates only the variant distribution so the selected variant gets
100%. Existing release conditions and per-user variant overrides on the flag are preserved untouched — the
variant is served only to users who already match them.
- release_to_everyone=true: in addition to flipping the variant distribution, prepends a catch-all release
condition that rolls the variant out to 100% of users. This OVERRIDES any existing release conditions and
per-user variant overrides on the flag. Only use this when the user explicitly wants to release beyond the
experiment's existing population.
Even when the user has asked you to ship a specific variant, ask them to confirm with the variant key,
target experiment, and release mode before invoking this tool.
Ship a variant to 100% of users by rewriting the feature flag. Requires variant_key — the key of the variant
to ship (e.g. "test" or "control"). The flag is rewritten so the selected variant gets 100% rollout via a
new catch-all release group prepended to existing groups (preserved for rollback).
Can be called on both running and stopped experiments. If the experiment is still running, it is also ended
(end_date set, status becomes stopped). If already stopped, only the flag is rewritten — supports the "end
first, ship later" workflow.
Optionally provide conclusion ("won", "lost", "inconclusive", "stopped_early", "invalid") and
conclusion_comment.
Returns 400 if: experiment is in draft ("Experiment has not been launched yet."), variant_key not found on
the flag, or no linked feature flag. Returns 409 if an approval policy requires review before the flag
change takes effect — includes a change_request_id in the response.
param_overrides:
id:
cast: string-int
ui_app: experiment
experiment-stats:
operation: experiments_stats_retrieve
enabled: true
scopes:
- experiment:read
annotations:
readOnly: true
destructive: false
idempotent: true
title: Get experiment velocity stats
description: >
Get aggregate experiment velocity statistics for the current project. Returns: launched_last_30d
(experiments launched in the last 30 days), launched_previous_30d (the 30 days before that), percent_change
between the two periods, active_experiments (currently running count), and completed_last_30d (experiments
ended in the last 30 days). No parameters needed.
experiment-timeseries-results:
operation: experiments_timeseries_results_retrieve
enabled: true
scopes:
- experiment:read
annotations:
readOnly: true
destructive: false
idempotent: true
title: Get experiment metric results over time
description: >
Only for the day-by-day history of a SINGLE metric — how its results moved over the course of the
experiment. For an experiment's current results across all metrics, which is what "how is this experiment
doing" almost always means, use experiment-metrics-recalculation-latest instead.
Requires metric_uuid and fingerprint as query parameters (both available from the experiment's metrics array
— each metric has a uuid and the fingerprint is computed from its configuration), so you generally need to
fetch the experiment first.
Returns a timeseries object keyed by date (YYYY-MM-DD) with per-variant statistical results for each day,
overall status ("pending", "completed", "partial", "failed"), computed_at timestamp, and
recalculation_status if a recalculation is in progress.
param_overrides:
id:
cast: string-int
ui_app: experiment-results
experiment-unarchive:
operation: experiments_unarchive_create
enabled: true
scopes:
- experiment:write
annotations:
readOnly: false
destructive: false
idempotent: false
enrich_url: '{id}'
title: Unarchive experiment
description: >
Requires an experiment ID. If you don't have the ID, load the finding-experiments skill to resolve the
user's reference first. Load the managing-experiment-lifecycle skill for preconditions and side effects.
Unarchive an archived experiment to restore it to the default list view.
Returns 400 if the experiment is not currently archived ("Experiment is not archived.").
No request body needed.
param_overrides:
id:
cast: string-int
ui_app: experiment
experiment-unfreeze-exposure:
operation: experiments_unfreeze_exposure_create
enabled: true
scopes:
- experiment:write
annotations:
readOnly: false
destructive: false
idempotent: false
enrich_url: '{id}'
title: Unfreeze experiment exposure
description: >
Requires an experiment ID. If you don't have the ID, load the finding-experiments skill to resolve the
user's reference first. Load the managing-experiment-lifecycle skill for preconditions and side effects.
Reopen enrollment on an exposure-frozen experiment. Removes the snapshot-cohort condition and freeze markers
from every release group, restoring the flag's original targeting: new users can enroll again and
already-enrolled users keep their variant (deterministic bucketing). The snapshot cohort is deleted. Status
returns to "running".
WARNING — can introduce bias: reopening enrollment re-exposes the flag to a potentially new population.
Users who enrolled before the freeze and users who enroll after the unfreeze joined at different times (and
possibly under different conditions), so mixing the two cohorts in one analysis can bias the results. Warn
the user about this before unfreezing, especially if the freeze lasted a while or something about the
audience or product changed in between. If they only wanted to sanity-check the frozen results, they may not
need to unfreeze at all.
Returns 400 if the experiment is a draft ("Experiment has not been launched yet."), already ended
("Experiment has already ended."), or its exposure is not frozen ("Experiment exposure is not frozen.").
No request body needed.
param_overrides:
id:
cast: string-int
ui_app: experiment
experiment-update:
operation: experiments_partial_update
enabled: true
scopes:
- experiment:write
annotations:
readOnly: false
destructive: false
idempotent: true
enrich_url: '{id}'
title: Update experiment
description: >
RULES (follow before calling):
1. If changing rollout or variants on a RUNNING
experiment, you MUST warn the user about user
experience AND statistical implications BEFORE
making the change and get explicit confirmation.
Load the configuring-experiment-rollout skill for
the full warning. Do NOT silently apply the change
— even if the user asked directly.
2. If the user mentions a percentage, clarify whether
they mean variant split or overall rollout BEFORE
calling.
3. If adding or changing metrics, load the
configuring-experiment-analytics skill first. Call
read-data-schema to discover available events BEFORE
suggesting or using any event name. Never suggest
event names you haven't confirmed exist.
4. For running experiments, set
update_feature_flag_params=true when changing
feature_flag config — without it, the request is
rejected because the change would not sync to the
feature flag.
5. Setting allow_unknown_events=true REQUIRES EXPLICIT
USER CONFIRMATION. When the API returns
"Event(s) '...' not found", do NOT silently retry
with allow_unknown_events=true. Default to
course-correcting: surface the missing event names
and ask whether the user meant a different (existing)
event. Only set allow_unknown_events=true when the
user explicitly confirms the event is intentional
(e.g. about to be instrumented). Course-correcting
to a known event is the expected default; flipping
the flag is the exception.
Update an experiment by ID. If you don't have the ID, load the finding-experiments skill to resolve the
user's reference first. Metrics can be changed at any time including while running. Cannot add/remove
variants on non-draft experiments (changing percentages between existing variants is allowed). Cannot change
holdout on non-draft experiments.
Do NOT use this to change lifecycle state — use the dedicated launch, end, pause, resume, archive, or
experiment-ship-variant tools instead.
include_params:
- name
- description
- feature_flag
- parameters
- running_time_calculation
- excluded_variants
- metrics
- metrics_secondary
- saved_metrics_ids
- stats_config
- exposure_criteria
- holdout_id
- conclusion
- conclusion_comment
- archived
- allow_unknown_events
- update_feature_flag_params
param_overrides:
id:
cast: string-int
feature_flag:
description: >
Variant split, rollout scope, payloads, and experience continuity for the linked feature flag, in
the flag's own filters shape. This is the canonical input for flag config. Set
filters.multivariate.variants (each with key and rollout_percentage; percentages must sum to 100;
the analysis baseline defaults to the variant keyed 'control' when present, else the first variant —
except web experiments, which must keep a variant keyed 'control') to change the variant split. Set
filters.groups to a single group [{"properties": [], "rollout_percentage": N}] (0-100) to change the
overall rollout. Config this object omits is preserved from the flag's current state. On a running
experiment this requires update_feature_flag_params=true (see rule 1: warn the user first).
running_time_calculation:
description: >
Persist a running-time / sample-size plan onto the experiment (the planning target shown in the
experiment's running-time panel). Object with optional keys: minimum_detectable_effect (percentage,
e.g. 20 for a 20% lift), recommended_sample_size (total across all variants),
recommended_running_time (days), and exposure_estimate_config.
saved_metrics_ids:
input_schema: SavedMetricsAttachSchema
ui_app: experiment
response:
include:
- id
- name
- description
- type
- feature_flag_key
- status
- archived
- start_date
- end_date
- created_at
- parameters
- running_time_calculation
- metrics
- metrics_secondary
- saved_metrics
- conclusion
- conclusion_comment
experiments-create-exposure-cohort:
operation: experiments_create_exposure_cohort_for_experiment_create
enabled: false
experiments-flag-cleanup-target-retrieve:
operation: experiments_flag_cleanup_target_retrieve
enabled: false
experiments-flag-cleanup-task-retrieve:
operation: experiments_flag_cleanup_task_retrieve
enabled: false
experiments-recalculate-timeseries:
operation: experiments_recalculate_timeseries_create
enabled: false
experiments-session-buckets-create:
operation: experiments_session_buckets_create
enabled: false
experiments-session-context-retrieve:
operation: experiments_session_context_retrieve
enabled: false
experiments-session-contexts-create:
operation: experiments_session_contexts_create
enabled: false
experiments-update:
operation: experiments_update
enabled: false
Reference in New Issue View Git Blame Copy Permalink