Files
zeekay 189ce08e51 Regenerate against the document the deployment answers for
The projection these were generated from was a month old and predated the
compute migration, so every one of them described /v1/visor — which the API
serves at zero paths now. That is why a hand-written console client and a
generated one had drifted the same way at the same time.

No /v1/visor path survives here. The word still appears as a source-plane
value and inside gvisor and supervisor, which are not this rename.
2026-09-05 18:07:13 -07:00

255 lines
19 KiB
C++

/**
* Hanzo Cloud API
* The Hanzo Cloud API as a customer calls it: every operation under /v1/ except the operator's admin product, relay routes, legacy spellings and capabilities still reached by flag. Tagged by product: the first path segment after /v1/.
*
* The version of the OpenAPI document: v1
*
* NOTE: This class is auto generated by OpenAPI-Generator 7.14.0.
* https://openapi-generator.tech
* Do not edit the class manually.
*/
/*
* EvalApi.h
*
*
*/
#ifndef HANZO_API_EvalApi_H_
#define HANZO_API_EvalApi_H_
#include "hanzo/ApiClient.h"
#include "hanzo/model/Board.h"
#include "hanzo/model/DatasetList.h"
#include "hanzo/model/DatasetReq.h"
#include "hanzo/model/DatasetView.h"
#include "hanzo/model/EvaluatorList.h"
#include "hanzo/model/EvaluatorReq.h"
#include "hanzo/model/EvaluatorView.h"
#include "hanzo/model/ItemList.h"
#include "hanzo/model/ItemReq.h"
#include "hanzo/model/ItemView.h"
#include "hanzo/model/RunRequest.h"
#include "hanzo/model/RunSummary.h"
#include "hanzo/model/Runs.h"
#include "hanzo/model/ScoreConfigList.h"
#include "hanzo/model/ScoreConfigReq.h"
#include "hanzo/model/ScoreConfigView.h"
#include "hanzo/model/ScoreList.h"
#include "hanzo/model/ScoreReq.h"
#include "hanzo/model/ScoreView.h"
#include "hanzo/model/TraceList.h"
#include <cpprest/details/basic_types.h>
#include <boost/optional.hpp>
namespace hanzo {
namespace api {
using namespace hanzo::model;
class EvalApi
{
public:
explicit EvalApi( std::shared_ptr<const ApiClient> apiClient );
virtual ~EvalApi();
/// <summary>
/// Removes the named dataset of the caller&#39;s org AND all of its examples, in one transaction.
/// </summary>
/// <remarks>
/// Removes the named dataset of the caller&#39;s org AND all of its examples, in one transaction. This is not a detach: the examples are gone with the set, so a dataset cannot be resurrected by re-creating the name. A name this org does not have is 404 — never a silent success — and a name belonging to another tenant is the same 404, because the delete is predicated on the validated org. Requires a validated principal; 403 without one. Runs and scores already recorded against the dataset are telemetry events and are NOT deleted with it.
/// </remarks>
/// <param name="name">Name is the dataset the URL names.</param>
pplx::task<void> deleteEvalDatasetsByName(
utility::string_t name
) const;
/// <summary>
/// Is the datasets your org has, each with its name, description, metadata and timestamps.
/// </summary>
/// <remarks>
/// Is the datasets your org has, each with its name, description, metadata and timestamps. It is the only way to enumerate what an org holds. Requires a validated principal; 403 without one. Every row is filtered on the validated org, so there is no parameter that reaches another tenant&#39;s datasets. The item count is NOT populated here — read one dataset to get it.
/// </remarks>
/// <param name="limit">Limit caps the rows returned. It defaults to 100 and is capped at 500; a non-positive or unparseable value falls back to the default rather than failing, because a typo about paging is not a reason to refuse a read. (optional, default to 0L)</param>
pplx::task<std::shared_ptr<DatasetList>> getEvalDatasets(
boost::optional<int64_t> limit
) const;
/// <summary>
/// Returns one dataset of the caller&#39;s org by name, together with its live item count — the one read that answers how big the set actually is.
/// </summary>
/// <remarks>
/// Returns one dataset of the caller&#39;s org by name, together with its live item count — the one read that answers how big the set actually is. A name this org does not have is 404, which is also what another tenant&#39;s dataset looks like from here. Requires a validated principal; 403 without one.
/// </remarks>
/// <param name="name">Name is the dataset the URL names.</param>
pplx::task<std::shared_ptr<DatasetView>> getEvalDatasetsByName(
utility::string_t name
) const;
/// <summary>
/// Is the examples in one of your datasets — the set is named in the path, because this collection only exists inside one.
/// </summary>
/// <remarks>
/// Is the examples in one of your datasets — the set is named in the path, because this collection only exists inside one. Archived examples are included, so the caller sees the whole set rather than only what a run would use. Requires a validated principal; 403 without one, and the read is filtered on the validated org, so naming another tenant&#39;s dataset returns nothing rather than its contents.
/// </remarks>
/// <param name="name">Dataset is the set to read, from the path — this collection only exists inside one.</param>
/// <param name="limit"> (optional, default to 0L)</param>
pplx::task<std::shared_ptr<ItemList>> getEvalDatasetsByNameItems(
utility::string_t name,
boost::optional<int64_t> limit
) const;
/// <summary>
/// Is the judges your org has defined, each with its judge model, criteria and the score name it writes under.
/// </summary>
/// <remarks>
/// Is the judges your org has defined, each with its judge model, criteria and the score name it writes under. Requires a validated principal; 403 without one, and the listing is filtered on the validated org.
/// </remarks>
/// <param name="limit">Limit caps the rows returned. It defaults to 100 and is capped at 500; a non-positive or unparseable value falls back to the default rather than failing, because a typo about paging is not a reason to refuse a read. (optional, default to 0L)</param>
pplx::task<std::shared_ptr<EvaluatorList>> getEvalEvaluators(
boost::optional<int64_t> limit
) const;
/// <summary>
/// Is your org&#39;s AI overview board over a window: totals (generations, prompt and completion tokens, cost in cents, errors, success rate, distinct models and users), a gap-filled time series, a per-model breakdown with the long tail folded into \&quot;other\&quot;, and latency percentiles read from the GenAI spans.
/// </summary>
/// <remarks>
/// Is your org&#39;s AI overview board over a window: totals (generations, prompt and completion tokens, cost in cents, errors, success rate, distinct models and users), a gap-filled time series, a per-model breakdown with the long tail folded into \&quot;other\&quot;, and latency percentiles read from the GenAI spans. The window the answer was actually computed over is echoed back, so a client never has to infer it. A platform admin sees the board across ALL orgs; everyone else sees their own. The board is HONEST-EMPTY where it cannot be computed: with no datastore wired, or under a named project scope the usage ledger does not yet carry, it answers a valid board with zero totals and a flat series rather than a fabricated number or a 500. Requires a validated principal; 403 without one.
/// </remarks>
/// <param name="range">Range is 24h (the default), 7d or 30d. Anything else normalises to 24h rather than failing, so the board always has a valid window. (optional, default to utility::conversions::to_string_t(&quot;&quot;))</param>
/// <param name="interval">Interval overrides the bucket the series is grouped into: \&quot;hour\&quot; or \&quot;day\&quot;. Any other value leaves the range&#39;s own default in place. (optional, default to utility::conversions::to_string_t(&quot;&quot;))</param>
pplx::task<std::shared_ptr<Board>> getEvalMetrics(
boost::optional<utility::string_t> range,
boost::optional<utility::string_t> interval
) const;
/// <summary>
/// Is the score shapes your org has declared — each name&#39;s data type, its numeric bounds and its allowed categories.
/// </summary>
/// <remarks>
/// Is the score shapes your org has declared — each name&#39;s data type, its numeric bounds and its allowed categories. Requires a validated principal; 403 without one, and the listing is filtered on the validated org.
/// </remarks>
/// <param name="limit">Limit caps the rows returned. It defaults to 100 and is capped at 500; a non-positive or unparseable value falls back to the default rather than failing, because a typo about paging is not a reason to refuse a read. (optional, default to 0L)</param>
pplx::task<std::shared_ptr<ScoreConfigList>> getEvalRubrics(
boost::optional<int64_t> limit
) const;
/// <summary>
/// Is your past runs and how they scored — the dataset and model, the judge model, how many examples were attempted and how many scored, the average score, and when it happened.
/// </summary>
/// <remarks>
/// Is your past runs and how they scored — the dataset and model, the judge model, how many examples were attempted and how many scored, the average score, and when it happened. Requires a validated principal; 403 without one, and rows are filtered on the validated org. These records come from the metastore rather than the datastore, so they are readable on a deployment with no telemetry wired — but a run&#39;s traces and scores are not.
/// </remarks>
/// <param name="datasetName">Dataset narrows to the runs against one dataset. (optional, default to utility::conversions::to_string_t(&quot;&quot;))</param>
/// <param name="limit"> (optional, default to 0L)</param>
pplx::task<std::shared_ptr<Runs>> getEvalRuns(
boost::optional<utility::string_t> datasetName,
boost::optional<int64_t> limit
) const;
/// <summary>
/// Is the score events your org has recorded, narrowed by any of name, runName and traceId.
/// </summary>
/// <remarks>
/// Is the score events your org has recorded, narrowed by any of name, runName and traceId. The org is bound as an authoritative predicate on the query, never taken from a header, so a filter can narrow the caller&#39;s own scores but can never widen past them. Requires a validated principal; 403 without one. Scores live in the datastore, so a deployment with none wired answers 503 rather than an empty page that would read as \&quot;no scores\&quot;.
/// </remarks>
/// <param name="name">Name narrows to one score name. (optional, default to utility::conversions::to_string_t(&quot;&quot;))</param>
/// <param name="runName">RunName narrows to the scores of one run. (optional, default to utility::conversions::to_string_t(&quot;&quot;))</param>
/// <param name="traceId">TraceID narrows to the scores on one model call. (optional, default to utility::conversions::to_string_t(&quot;&quot;))</param>
/// <param name="limit"> (optional, default to 0L)</param>
pplx::task<std::shared_ptr<ScoreList>> getEvalScores(
boost::optional<utility::string_t> name,
boost::optional<utility::string_t> runName,
boost::optional<utility::string_t> traceId,
boost::optional<int64_t> limit
) const;
/// <summary>
/// Is the traces behind your evaluations — one per model call an evaluation made, carrying its input, output, model and timing — narrowed by any of sessionId, runName and datasetName.
/// </summary>
/// <remarks>
/// Is the traces behind your evaluations — one per model call an evaluation made, carrying its input, output, model and timing — narrowed by any of sessionId, runName and datasetName. Scoped by org AND by project: the project is the caller&#39;s server-minted scope, not a parameter, so it cannot be widened by asking. Requires a validated principal; 403 without one. Traces live in the datastore, so a deployment with none wired answers 503 rather than an empty page.
/// </remarks>
/// <param name="sessionId">SessionID narrows to one session, which for an evaluation is one run. (optional, default to utility::conversions::to_string_t(&quot;&quot;))</param>
/// <param name="runName">RunName narrows to the calls one run made. (optional, default to utility::conversions::to_string_t(&quot;&quot;))</param>
/// <param name="datasetName">Dataset narrows to the calls made against one dataset. (optional, default to utility::conversions::to_string_t(&quot;&quot;))</param>
/// <param name="limit"> (optional, default to 0L)</param>
pplx::task<std::shared_ptr<TraceList>> getEvalTraces(
boost::optional<utility::string_t> sessionId,
boost::optional<utility::string_t> runName,
boost::optional<utility::string_t> datasetName,
boost::optional<int64_t> limit
) const;
/// <summary>
/// Writes a dataset — the named set of graded examples a run scores a model against — under the caller&#39;s org and answers 201 with it.
/// </summary>
/// <remarks>
/// Writes a dataset — the named set of graded examples a run scores a model against — under the caller&#39;s org and answers 201 with it. The NAME is the key, not an id: posting a name the org already has updates that dataset&#39;s description and metadata and keeps its original creation time, so this is create-or-edit and never a duplicate. Its items are untouched. Requires a validated principal; 403 without one. The org comes from the validated owner claim, never from a client X-Org-Id, so a dataset can only ever be written under the caller&#39;s own tenant. A description over 64 KiB is 400.
/// </remarks>
/// <param name="datasetReq"></param>
pplx::task<std::shared_ptr<DatasetView>> postEvalDatasets(
std::shared_ptr<DatasetReq> datasetReq
) const;
/// <summary>
/// Writes one graded example — its input, its expected output, free-form metadata and a status — into the dataset named in the path, and answers 201 with it.
/// </summary>
/// <remarks>
/// Writes one graded example — its input, its expected output, free-form metadata and a status — into the dataset named in the path, and answers 201 with it. That dataset MUST already exist for this org: an unknown one is 404, never a silent create. Requires a validated principal; 403 without one.
/// </remarks>
/// <param name="name"></param>
/// <param name="itemReq"></param>
pplx::task<std::shared_ptr<ItemView>> postEvalDatasetsByNameItems(
utility::string_t name,
std::shared_ptr<ItemReq> itemReq
) const;
/// <summary>
/// Saves a reusable judge for the caller&#39;s org — the judge model and the written criteria it grades against — and answers 201 with it.
/// </summary>
/// <remarks>
/// Saves a reusable judge for the caller&#39;s org — the judge model and the written criteria it grades against — and answers 201 with it. Like a dataset, the NAME is the key: re-posting a name edits that judge rather than adding a second one. Requires a validated principal; 403 without one.
/// </remarks>
/// <param name="evaluatorReq"></param>
pplx::task<std::shared_ptr<EvaluatorView>> postEvalEvaluators(
std::shared_ptr<EvaluatorReq> evaluatorReq
) const;
/// <summary>
/// Defines the shape of one score name for the caller&#39;s org and answers 201 with it.
/// </summary>
/// <remarks>
/// Defines the shape of one score name for the caller&#39;s org and answers 201 with it. This is the integrity contract, not documentation: once a rubric exists for a name, every score recorded under that name is checked against it and the rubric&#39;s data type is AUTHORITATIVE — a caller cannot claim a different one. Out-of-range values, unlisted labels and non-finite numbers are refused at write time. A CATEGORICAL rubric with no categories is 400, as is a non-finite bound or a minValue above maxValue. Requires a validated principal; 403 without one.
/// </remarks>
/// <param name="scoreConfigReq"></param>
pplx::task<std::shared_ptr<ScoreConfigView>> postEvalRubrics(
std::shared_ptr<ScoreConfigReq> scoreConfigReq
) const;
/// <summary>
/// Runs a real evaluation and answers the summary when it is finished — this is synchronous work, not a job id.
/// </summary>
/// <remarks>
/// Runs a real evaluation and answers the summary when it is finished — this is synchronous work, not a job id. For each ACTIVE example in the dataset it calls the model under test, records a trace, calls the LLM-as-judge, and records the judge&#39;s score with its reasoning. The answer carries the per-item results (item id, trace id, score, output or error) alongside items, scored and avgScore. The dataset must belong to the caller&#39;s org (404 otherwise) and must have at least one ACTIVE example (422 otherwise). It runs as YOU: the caller&#39;s own Authorization bearer drives the model gateway, so a request without one is 401 rather than a run made anonymously or under a service identity. Only a non-reversible hash of that credential is recorded on the traces. Bounded and honest about it: an org may have at most 4 runs in flight and the fifth is 429 rather than queued, and the whole run is capped at 10 minutes — examples past the deadline come back with an error instead of a score, and scored counts only real successes. A run where NOTHING scored answers 502, not a 200 that looks like an evaluation. A run must be able to persist what it produces, so a deployment with no datastore wired is 503 up front. Requires a validated principal; 403 without one.
/// </remarks>
/// <param name="runRequest"></param>
/// <param name="authorization"> (optional, default to utility::conversions::to_string_t(&quot;&quot;))</param>
pplx::task<std::shared_ptr<RunSummary>> postEvalRuns(
std::shared_ptr<RunRequest> runRequest,
boost::optional<utility::string_t> authorization
) const;
/// <summary>
/// Files one score event for the caller&#39;s org and answers 201 with it.
/// </summary>
/// <remarks>
/// Files one score event for the caller&#39;s org and answers 201 with it. This is how human review and out-of-band graders land beside the automatic ones: name the score, give it a value (or a stringValue for a categorical label), and attach it to a trace, a run, a dataset example, or any combination. Scores are validated fail-closed. A value must be FINITE — NaN and Inf are 400 — and if the org has declared a rubric for this name, that rubric decides the type and the value must satisfy it: inside the numeric bounds, or one of the allowed categories. A caller cannot override the declared type by sending a different dataType. A score is TELEMETRY, not metadata, so it needs the datastore: a deployment with none wired answers 503 rather than accepting a score it cannot persist. Requires a validated principal; 403 without one, and the org is stamped from the validated claim rather than read off the body.
/// </remarks>
/// <param name="scoreReq"></param>
pplx::task<std::shared_ptr<ScoreView>> postEvalScores(
std::shared_ptr<ScoreReq> scoreReq
) const;
protected:
std::shared_ptr<const ApiClient> m_ApiClient;
};
}
}
#endif /* HANZO_API_EvalApi_H_ */