mirror of
https://github.com/hanzo-cpp/sdk.git
synced 2026-11-20 19:55:32 +00:00
The projection was 113 commits behind the document it claims to describe, and `generate.py --all --check` reported every one of the thirteen clients DRIFTED — the same +19 models in each, because they all fell behind the same emission. Regenerated in place from hanzo-inc/cloud openapi.yaml at 939b09d4 (sha256 087024f6…), which is the only description of this API that cannot name a route the binary does not serve. `--check` is clean across all thirteen after. .spec-lock moved with the code. The generator READS that file and never writes it — hanzoai/ci`s lane does — so a hand run leaves every lock naming the previous document. A lock that disagrees with the tree beside it is worse than no lock: the next `--check` without `--spec` fetches what production serves, compares it to a digest that was already wrong, and reports drift that is really staleness.
165 lines
12 KiB
C++
165 lines
12 KiB
C++
/**
|
|
* Hanzo Cloud API
|
|
* The Hanzo Cloud API as a customer calls it: every operation under /v1/ except the operator's admin product, relay routes, legacy spellings and capabilities still reached by flag. Tagged by product: the first path segment after /v1/.
|
|
*
|
|
* The version of the OpenAPI document: v1
|
|
*
|
|
* NOTE: This class is auto generated by OpenAPI-Generator 7.14.0.
|
|
* https://openapi-generator.tech
|
|
* Do not edit the class manually.
|
|
*/
|
|
|
|
/*
|
|
* BenchmarkApi.h
|
|
*
|
|
*
|
|
*/
|
|
|
|
#ifndef HANZO_API_BenchmarkApi_H_
|
|
#define HANZO_API_BenchmarkApi_H_
|
|
|
|
|
|
|
|
#include "hanzo/ApiClient.h"
|
|
|
|
#include "hanzo/model/Admission.h"
|
|
#include "hanzo/model/BenchmarkCatalog.h"
|
|
#include "hanzo/model/ClaimsOut.h"
|
|
#include "hanzo/model/HistoryOut.h"
|
|
#include "hanzo/model/Leaderboard.h"
|
|
#include "hanzo/model/Pairing.h"
|
|
#include "hanzo/model/Preset.h"
|
|
#include "hanzo/model/PresetAccepted.h"
|
|
#include "hanzo/model/PresetList.h"
|
|
#include "hanzo/model/PutClaimsIn.h"
|
|
#include "hanzo/model/PutClaimsOut.h"
|
|
#include "hanzo/model/Suite.h"
|
|
#include <cpprest/details/basic_types.h>
|
|
#include <boost/optional.hpp>
|
|
|
|
namespace hanzo {
|
|
namespace api {
|
|
|
|
using namespace hanzo::model;
|
|
|
|
|
|
|
|
class BenchmarkApi
|
|
{
|
|
public:
|
|
|
|
explicit BenchmarkApi( std::shared_ptr<const ApiClient> apiClient );
|
|
|
|
virtual ~BenchmarkApi();
|
|
|
|
/// <summary>
|
|
/// Is the canonical public benchmarks this arena runs — the id, title, axis, item count and upstream source of each, with native marking the ones the standardized harness runs today; the rest are registered and adapter-pending.
|
|
/// </summary>
|
|
/// <remarks>
|
|
/// Is the canonical public benchmarks this arena runs — the id, title, axis, item count and upstream source of each, with native marking the ones the standardized harness runs today; the rest are registered and adapter-pending. These ids are the vocabulary the rest of the surface takes: a run names them, and the leaderboard and compare read them from ?benchmark=. The catalog is deployment-wide and identical for every caller — there is no tenant in it.
|
|
/// </remarks>
|
|
pplx::task<std::shared_ptr<BenchmarkCatalog>> getBenchmarkCatalog(
|
|
) const;
|
|
/// <summary>
|
|
/// Lists the effective published claims: what the leaderboard will use for each (benchmark, model) after the seed, the import and any stored correction are layered.
|
|
/// </summary>
|
|
/// <remarks>
|
|
/// Lists the effective published claims: what the leaderboard will use for each (benchmark, model) after the seed, the import and any stored correction are layered. It answers the operator's question — what does this arena currently believe someone else reported, and did we ship that or fix it. Effective values only. The history of a key lives in the append-only file and is not what this op is for; a list that returned every superseded row would make the common question the hard one.
|
|
/// </remarks>
|
|
/// <param name="benchmark">Benchmark filters to one benchmark id. Empty returns every benchmark. (optional, default to utility::conversions::to_string_t(""))</param>
|
|
/// <param name="model">Model filters to one model. Empty returns every model. (optional, default to utility::conversions::to_string_t(""))</param>
|
|
/// <param name="provider">Provider filters to one lab or leaderboard — the way to read what a single source claims across every model it covers. (optional, default to utility::conversions::to_string_t(""))</param>
|
|
/// <param name="source">Source filters to one citation, which is the finest grain there is: a source is what makes two claims about one model independent rather than a restatement of each other. (optional, default to utility::conversions::to_string_t(""))</param>
|
|
/// <param name="protocol">Protocol filters by HOW a claim was scored, so provider cards can be read apart from third parties running their own harness. (optional, default to utility::conversions::to_string_t(""))</param>
|
|
pplx::task<std::shared_ptr<ClaimsOut>> getBenchmarkClaims(
|
|
boost::optional<utility::string_t> benchmark,
|
|
boost::optional<utility::string_t> model,
|
|
boost::optional<utility::string_t> provider,
|
|
boost::optional<utility::string_t> source,
|
|
boost::optional<utility::string_t> protocol
|
|
) const;
|
|
/// <summary>
|
|
/// Is the ONLY valid arm-vs-arm test: it pairs the two models on the items BOTH completed, and answers rescue and damage counts with an exact-McNemar p.
|
|
/// </summary>
|
|
/// <remarks>
|
|
/// Is the ONLY valid arm-vs-arm test: it pairs the two models on the items BOTH completed, and answers rescue and damage counts with an exact-McNemar p. Pairing is what prevents the subset artifact — comparing one model's easy subset against another's full run — so n_common, not either arm's own coverage, is the number to read this by. Both a and b are required. The benchmark defaults to gpqa_diamond.
|
|
/// </remarks>
|
|
/// <param name="a">A is the first model id. It is required.</param>
|
|
/// <param name="b">B is the second model id. It is required.</param>
|
|
/// <param name="benchmark">Benchmark is the catalog id to compare on, defaulting to gpqa_diamond. (optional, default to utility::conversions::to_string_t(""))</param>
|
|
pplx::task<std::shared_ptr<Pairing>> getBenchmarkCompare(
|
|
utility::string_t a,
|
|
utility::string_t b,
|
|
boost::optional<utility::string_t> benchmark
|
|
) const;
|
|
/// <summary>
|
|
/// Returns each model's measured score per run over time, oldest first, with the change between runs.
|
|
/// </summary>
|
|
/// <remarks>
|
|
/// Returns each model's measured score per run over time, oldest first, with the change between runs. This is the counterweight to a leaderboard: the board shows the latest run because that is what \"how good is it\" means, and a single latest number cannot distinguish a model that has always been strong from one that just improved, or from one that regressed after a provider changed something. Both matter for routing, and only one of them is visible on a board. Runs with no id — attempts recorded before runs existed — group under the empty run, which is honestly what they are: one undated measurement.
|
|
/// </remarks>
|
|
/// <param name="benchmark">Benchmark is the catalog id to read, defaulting to gpqa_diamond. (optional, default to utility::conversions::to_string_t(""))</param>
|
|
/// <param name="model">Model filters to one model. Empty returns every model measured. (optional, default to utility::conversions::to_string_t(""))</param>
|
|
pplx::task<std::shared_ptr<HistoryOut>> getBenchmarkHistory(
|
|
boost::optional<utility::string_t> benchmark,
|
|
boost::optional<utility::string_t> model
|
|
) const;
|
|
/// <summary>
|
|
/// Answers one row per model for the benchmark named — what our own harness measured, beside what the vendor claims, and the gap between them.
|
|
/// </summary>
|
|
/// <remarks>
|
|
/// Answers one row per model for the benchmark named — what our own harness measured, beside what the vendor claims, and the gap between them. The gap is the point of the arena; provider-reported claims have run materially hot against one standardized harness. The two planes are NEVER blended, and that is the rule to read the rows by: a model we have measured but no vendor has claimed for shows published null, a model with only a claim shows measured null, and gap exists only where both do. n is coverage and is not decoration: two measured numbers taken over different item counts are not comparable, so read the row's n before reading its accuracy.
|
|
/// </remarks>
|
|
/// <param name="benchmark">Benchmark is the catalog id to read, defaulting to gpqa_diamond. (optional, default to utility::conversions::to_string_t(""))</param>
|
|
pplx::task<std::shared_ptr<Leaderboard>> getBenchmarkLeaderboard(
|
|
boost::optional<utility::string_t> benchmark
|
|
) const;
|
|
/// <summary>
|
|
/// Are the router blends available to compose from — a named set of model arms, the rank they escalate through and the panel width that bounds fan-out — each served by the model layer as enso-<name>.
|
|
/// </summary>
|
|
/// <remarks>
|
|
/// Are the router blends available to compose from — a named set of model arms, the rank they escalate through and the panel width that bounds fan-out — each served by the model layer as enso-<name>. Today it answers exactly one row, the reference blend: a worked example written in models we name, published as an example of the FORM. It is deliberately not the composition of a Hanzo-served tier — the tier name exists to abstract that — so fork it and swap arms by what the leaderboard measures on your own tasks rather than reading it as a disclosure.
|
|
/// </remarks>
|
|
pplx::task<std::shared_ptr<PresetList>> getBenchmarkPresets(
|
|
) const;
|
|
/// <summary>
|
|
/// Records published claims: one to correct a number, many to import a leaderboard.
|
|
/// </summary>
|
|
/// <remarks>
|
|
/// Records published claims: one to correct a number, many to import a leaderboard. Every row must carry a Source, because a claim without its citation is a number nobody can check — and an unattributed number in the published plane is indistinguishable from a measurement, which is the one confusion this whole surface is built to prevent. Writes are append-only, so this never destroys the value it replaces. A vendor restating a score leaves both rows on disk, which is how the restating itself becomes visible.
|
|
/// </remarks>
|
|
/// <param name="putClaimsIn"></param>
|
|
pplx::task<std::shared_ptr<PutClaimsOut>> postBenchmarkClaims(
|
|
std::shared_ptr<PutClaimsIn> putClaimsIn
|
|
) const;
|
|
/// <summary>
|
|
/// Validates a router blend — its name, its arms, the rank they escalate through and the panel fan-out width — and answers 202 with the preset and the enso-<name> it would be served as.
|
|
/// </summary>
|
|
/// <remarks>
|
|
/// Validates a router blend — its name, its arms, the rank they escalate through and the panel fan-out width — and answers 202 with the preset and the enso-<name> it would be served as. It VALIDATES AND ECHOES: the definition is not persisted yet, so a preset accepted here is not one the model layer will resolve. Treat the response as a check on the blend, not a promise to serve it. Defaults fill the shape rather than refusing it: an omitted rank becomes the arms in declared order and a panel below 1 becomes 1. The one real invariant is that rank may only name arms the blend declares — the same rule the model catalog enforces — and a rank naming anything else is a 422 listing exactly which entries were undeclared. A blend with no name or no arms is a 400.
|
|
/// </remarks>
|
|
/// <param name="preset"></param>
|
|
pplx::task<std::shared_ptr<PresetAccepted>> postBenchmarkPresets(
|
|
std::shared_ptr<Preset> preset
|
|
) const;
|
|
/// <summary>
|
|
/// Admits and queues a benchmark run against a model or your own endpoint, and answers 202 with the receipt.
|
|
/// </summary>
|
|
/// <remarks>
|
|
/// Admits and queues a benchmark run against a model or your own endpoint, and answers 202 with the receipt. It is an ADMISSION, not a result: the work is done by the harness afterwards and the numbers appear on the leaderboard as it completes them. Cost is bounded by the store rather than by a quota: attempts are append-only and keyed by (benchmark, item, model), so an (item, model) pair already attempted is skipped instead of re-spent, and re-queuing the same run is close to free. Validation is up front and total — a request with neither model nor endpoint is a 400, one with no benchmarks is a 400, and any benchmark id outside the catalog is a 422 naming exactly which ids were unknown, so a typo never silently queues a partial run.
|
|
/// </remarks>
|
|
/// <param name="suite"></param>
|
|
pplx::task<std::shared_ptr<Admission>> postBenchmarkRuns(
|
|
std::shared_ptr<Suite> suite
|
|
) const;
|
|
|
|
protected:
|
|
std::shared_ptr<const ApiClient> m_ApiClient;
|
|
};
|
|
|
|
}
|
|
}
|
|
|
|
#endif /* HANZO_API_BenchmarkApi_H_ */
|
|
|