mirror of
https://github.com/hanzoai/ml.git
synced 2026-11-20 23:08:29 +00:00
A crate whose whole interface is two verbs. `Config` holds hyperparameters and has no `predict`. `Fit` maps `Config -> Data -> Model`. `Model` is immutable, has private fields and no public constructor, so an unfitted model is not a state to be checked for — it is a value that cannot be spoken, and `check_is_fitted` has no job to do. `Predict::Answer` is `f64` for a regressor and `Class` for a classifier: the two do not unify, so a regressor and a classifier are different types rather than one type with a mode flag. There is no `get_params`, no `BaseEstimator`, no clone protocol and no `Pipeline` — those exist in Python because a config and a fitted model are the same object there. Every fitted model is named by `Address`, a content address over its shape, its hyperparameters and its learned parameters, hashing floats as IEEE-754 bits and length-prefixing every sequence so the encoding is injective. Same construction and same reasoning as the risk plane's published model values, so a fit can be stored, cited by a decision, compared against a challenger and rolled back to without a registry. Three estimators: least squares by Householder QR on centred data (QR not the normal equations, which square the condition number; centring not a ones column, so the intercept never enters the factorisation), binary logistic regression by Newton steps on a strictly convex penalised objective, and gradient boosted CART regression trees. A TREE READS FEATURES AT IEEE-754 BINARY32, and that is the algorithm rather than a detail. scikit-learn casts X to float32 before its splitter sees it, XGBoost and LightGBM do the same, and ai.onnx.ml's TreeEnsembleRegressor stores nodes_values as floats. A tree never does arithmetic on a feature value — it only orders values and places thresholds between them — so precision decides only which side of a boundary a sample falls on. Placing the threshold at single(a)/2 + single(b)/2 instead of a/2 + b/2 moves it by 2e-8, which is enough to send a sample down the other branch. Measured over the 40 depth-1 trees of the stumps fixture, whose roots all see the full sample set: the single-precision midpoint reproduces scikit-learn's threshold bit-for-bit 40 times out of 40, the double-precision midpoint 0 times out of 40. Correctness is asserted against scikit-learn itself. Fixtures generated by 1.9.0 are committed and read as data, so `cargo test` needs no Python and the oracle's version is a recorded fact. 195 boosted trees / 1,977 split nodes agree node for node — split feature, threshold and both children all exact — with leaf values to 1.3e-15 and predictions to 8.9e-16 against a 1e-10 bound; least squares agrees to 1.1e-15 against 1e-10 and logistic to 1.2e-8 against 1e-7, with hard labels exact. Each tolerance is derived from the mathematics and every assertion prints the deviation it observed, so a regression that stays inside a bound is still visible. The generator refuses to record an oracle that is not a function of its data: at scikit-learn's default min_samples_leaf a tree's structure varies with random_state (20 structures over 40 seeds, prediction spread 3.9), so every case pins the tie away and proves determinism over 12 seeds before writing a number. This crate breaks split ties by lowest feature index instead, because a content address over a model an RNG helped choose would name one draw rather than one fit. The exhaustive split search is not lowered to hanzo-kernel, and the obstacle is specific: a launch geometry must be stated before the launch, and across all 69 launch sites in that crate Grid::Static is the only variant used, every geometry computed on the host from a count it already knows. A CART node's sample count is not such a number. The histogram formulation dissolves this — bin count is a hyperparameter, so the per-level geometry is host-knowable — but it quantises thresholds and so cannot hold the node-for-node bar. It is named, not built. scikit-learn is a test oracle only. No code or docstring is taken from it, and NOTICE records that, along with the obligation to add BSD-3 attribution if that ever changes. The crate also carries, unchanged and unclaimed, the anomaly-detection and preprocessing modules written concurrently on another track (isolation, neighbour, transform, scale, impute, encode, metric, moment, split, twister, plus benches and the perf oracle). They share lib.rs and tests/sklearn.rs with the estimator seam, so the crate cannot be committed in halves. Their `Transform` and `Outlier` traits extend this seam without changing `Fit` or `Predict`. One of their fixture tests, `the_imbalanced_metrics_match_scikit_learn`, is red on a fixture key that measure.json does not yet carry — which is why this is a branch and not main.
245 lines
8.9 KiB
Python
245 lines
8.9 KiB
Python
#!/usr/bin/env python3
|
|
"""The scikit-learn half of the benchmark in `benches/learn.rs`.
|
|
|
|
WHY THIS IS COMPARABLE, which is the only interesting thing about a benchmark:
|
|
|
|
SAME DATA, provably. Both sides draw from numpy's legacy Mersenne stream at the
|
|
same seed. `hanzo_learn::twister::Twister::next_real` IS
|
|
`RandomState.random_sample` bit for bit — `twister`'s own unit test pins it
|
|
against numpy's output — so the two benchmarks see identical VALUES and not
|
|
merely the same distribution. That matters for the estimators whose cost
|
|
depends on the values rather than on the shape: a tree's depth and a neighbour
|
|
frontier's insert rate both do.
|
|
|
|
SAME WORK. Each case below is the scikit-learn call that answers the same
|
|
question as the Rust one, at the same shape, with the parameters matched. Where
|
|
scikit-learn does MORE work by default the difference is named in the notes
|
|
column of `README.md` rather than quietly left in the number.
|
|
|
|
SAME MEASUREMENT. Median of `REPEATS` timed runs after one warm-up, wall clock,
|
|
`perf_counter`. criterion reports a median too, so the two columns are the same
|
|
statistic. Allocation of inputs is outside the timed region on both sides.
|
|
|
|
ONE CORE, DECLARED. scikit-learn's IsolationForest and LocalOutlierFactor take
|
|
`n_jobs`; this runs them at the default (1) and hanzo-learn's forest fit spans
|
|
cores through rayon. That is a REAL difference and it is reported as one — the
|
|
Rust fit column says how many threads it had, and the `n_jobs=-1` row is
|
|
measured too so the single-core comparison is available.
|
|
|
|
Run:
|
|
<venv>/bin/python hanzo-learn/oracle/bench.py # the default shapes
|
|
<venv>/bin/python hanzo-learn/oracle/bench.py --quick # skip 1m rows
|
|
"""
|
|
|
|
import argparse
|
|
import json
|
|
import platform
|
|
import sys
|
|
import time
|
|
|
|
import numpy as np
|
|
import sklearn
|
|
from sklearn.ensemble import IsolationForest
|
|
from sklearn.impute import SimpleImputer
|
|
from sklearn.metrics import (
|
|
average_precision_score,
|
|
log_loss,
|
|
precision_recall_curve,
|
|
roc_auc_score,
|
|
roc_curve,
|
|
)
|
|
from sklearn.model_selection import KFold, StratifiedKFold, train_test_split
|
|
from sklearn.neighbors import LocalOutlierFactor
|
|
from sklearn.preprocessing import (
|
|
LabelEncoder,
|
|
MinMaxScaler,
|
|
OneHotEncoder,
|
|
StandardScaler,
|
|
)
|
|
|
|
REPEATS = 5
|
|
|
|
SHAPES = [("10k_x_20", 10_000, 20), ("100k_x_20", 100_000, 20), ("1m_x_20", 1_000_000, 20)]
|
|
NEAR_SHAPES = [("2k_x_20", 2_000, 20), ("20k_x_20", 20_000, 20)]
|
|
|
|
|
|
def design(n, p, seed):
|
|
"""Bit-identical to `benches/learn.rs::design`."""
|
|
return np.random.RandomState(seed).random_sample(n * p).reshape(n, p) * 2.0 - 1.0
|
|
|
|
|
|
def planted(n, p, seed):
|
|
"""Bit-identical to `benches/learn.rs::planted`."""
|
|
x = design(n, p, seed)
|
|
x[: max(n // 100, 1), :] += 9.0
|
|
return x
|
|
|
|
|
|
def time_it(fn):
|
|
"""Median of REPEATS runs, in seconds, after one untimed warm-up."""
|
|
fn()
|
|
took = []
|
|
for _ in range(REPEATS):
|
|
start = time.perf_counter()
|
|
fn()
|
|
took.append(time.perf_counter() - start)
|
|
return float(np.median(took))
|
|
|
|
|
|
def main():
|
|
ap = argparse.ArgumentParser()
|
|
ap.add_argument("--quick", action="store_true", help="skip the 1m-row shapes")
|
|
args = ap.parse_args()
|
|
shapes = SHAPES[:-1] if args.quick else SHAPES
|
|
|
|
out = {
|
|
"oracle": {
|
|
"sklearn": sklearn.__version__,
|
|
"numpy": np.__version__,
|
|
"python": sys.version.split()[0],
|
|
"machine": platform.machine(),
|
|
"processor": platform.processor(),
|
|
},
|
|
"repeats": REPEATS,
|
|
"cases": {},
|
|
}
|
|
|
|
def record(group, case, seconds):
|
|
out["cases"].setdefault(group, {})[case] = seconds
|
|
print(f" {group:11s} {case:32s} {seconds * 1e3:12.3f} ms")
|
|
|
|
print(f"scikit-learn {sklearn.__version__} / numpy {np.__version__}, median of {REPEATS}")
|
|
|
|
for name, n, p in shapes:
|
|
x = design(n, p, 1)
|
|
record("scale", f"standard_fit/{name}", time_it(lambda: StandardScaler().fit(x)))
|
|
fitted = StandardScaler().fit(x)
|
|
record("scale", f"standard_apply/{name}", time_it(lambda: fitted.transform(x)))
|
|
record("scale", f"range_fit/{name}", time_it(lambda: MinMaxScaler().fit(x)))
|
|
ranged = MinMaxScaler().fit(x)
|
|
record("scale", f"range_apply/{name}", time_it(lambda: ranged.transform(x)))
|
|
|
|
for name, n, p in shapes:
|
|
x = design(n, p, 2)
|
|
# Stride 19 against 20 columns, coprime, so the gaps walk across every column. A
|
|
# stride of p would empty ONE column entirely; scikit-learn then drops it with a
|
|
# warning and the two sides are no longer doing the same work.
|
|
gapped = x.copy().ravel()
|
|
gapped[np.arange(gapped.size) % 19 == 7] = np.nan
|
|
gapped = gapped.reshape(n, p)
|
|
record(
|
|
"impute",
|
|
f"mean_fit/{name}",
|
|
time_it(lambda: SimpleImputer(strategy="mean").fit(gapped)),
|
|
)
|
|
record(
|
|
"impute",
|
|
f"median_fit/{name}",
|
|
time_it(lambda: SimpleImputer(strategy="median").fit(gapped)),
|
|
)
|
|
fill = SimpleImputer(strategy="mean").fit(gapped)
|
|
record("impute", f"apply/{name}", time_it(lambda: fill.transform(gapped)))
|
|
|
|
for name, n, _ in shapes:
|
|
raw = np.random.RandomState(3).randint(0, 10, size=(n, 8))
|
|
flat = raw.ravel()
|
|
record("encode", f"label_fit/{name}", time_it(lambda: LabelEncoder().fit(flat)))
|
|
le = LabelEncoder().fit(flat)
|
|
record("encode", f"label_codes/{name}", time_it(lambda: le.transform(flat)))
|
|
oh = OneHotEncoder(sparse_output=False, categories=[list(range(10))] * 8).fit(raw)
|
|
record("encode", f"onehot_apply/{name}", time_it(lambda: oh.transform(raw)))
|
|
|
|
for name, n, _ in shapes:
|
|
index = np.arange(n)
|
|
y = (np.random.RandomState(4).random_sample(n) < 0.08).astype(int)
|
|
record(
|
|
"split",
|
|
f"train_test/{name}",
|
|
time_it(lambda: train_test_split(index, test_size=0.25, random_state=0)),
|
|
)
|
|
record(
|
|
"split",
|
|
f"kfold_5/{name}",
|
|
time_it(lambda: list(KFold(n_splits=5, shuffle=True, random_state=0).split(index))),
|
|
)
|
|
zeros = np.zeros((n, 1))
|
|
record(
|
|
"split",
|
|
f"stratified_5/{name}",
|
|
time_it(
|
|
lambda: list(
|
|
StratifiedKFold(n_splits=5, shuffle=True, random_state=0).split(zeros, y)
|
|
)
|
|
),
|
|
)
|
|
|
|
for name, n, _ in shapes:
|
|
r = np.random.RandomState(5)
|
|
truth = (r.random_sample(n) < 0.07).astype(int)
|
|
score = np.where(truth == 1, 0.35, 0.0) + r.random_sample(n)
|
|
# `Curve::of` builds the whole confusion ladder once; roc_curve is the
|
|
# nearest single scikit-learn call that does the same accumulation.
|
|
record(
|
|
"metric",
|
|
f"curve_of/{name}",
|
|
time_it(lambda: roc_curve(truth, score, drop_intermediate=False)),
|
|
)
|
|
record("metric", f"roc_auc/{name}", time_it(lambda: roc_auc_score(truth, score)))
|
|
record(
|
|
"metric",
|
|
f"average_precision/{name}",
|
|
time_it(lambda: average_precision_score(truth, score)),
|
|
)
|
|
record(
|
|
"metric",
|
|
f"precision_recall/{name}",
|
|
time_it(lambda: precision_recall_curve(truth, score)),
|
|
)
|
|
# log_loss needs a probability; the score is squashed so the call is legal.
|
|
prob = 1.0 / (1.0 + np.exp(-score))
|
|
record("metric", f"log_loss/{name}", time_it(lambda: log_loss(truth, prob, labels=[0, 1])))
|
|
|
|
for name, n, p in shapes:
|
|
x = planted(n, p, 6)
|
|
record(
|
|
"isolation",
|
|
f"fit_100_trees/{name}",
|
|
time_it(lambda: IsolationForest(n_estimators=100, max_samples=256).fit(x)),
|
|
)
|
|
record(
|
|
"isolation",
|
|
f"fit_100_trees_njobs-1/{name}",
|
|
time_it(lambda: IsolationForest(n_estimators=100, max_samples=256, n_jobs=-1).fit(x)),
|
|
)
|
|
forest = IsolationForest(n_estimators=100, max_samples=256).fit(x)
|
|
record("isolation", f"outlier/{name}", time_it(lambda: forest.score_samples(x)))
|
|
|
|
for name, n, p in NEAR_SHAPES:
|
|
x = planted(n, p, 7)
|
|
record(
|
|
"neighbour",
|
|
f"fit_k20/{name}",
|
|
time_it(lambda: LocalOutlierFactor(n_neighbors=20, novelty=True).fit(x)),
|
|
)
|
|
record(
|
|
"neighbour",
|
|
f"fit_k20_njobs-1/{name}",
|
|
time_it(lambda: LocalOutlierFactor(n_neighbors=20, novelty=True, n_jobs=-1).fit(x)),
|
|
)
|
|
fitted = LocalOutlierFactor(n_neighbors=20, novelty=True).fit(x)
|
|
query = design(1_000, p, 8)
|
|
record(
|
|
"neighbour",
|
|
f"outlier_1k_queries/{name}",
|
|
time_it(lambda: fitted.score_samples(query)),
|
|
)
|
|
|
|
path = __file__.rsplit("/", 1)[0] + "/sklearn.json"
|
|
with open(path, "w") as fh:
|
|
json.dump(out, fh, indent=1, sort_keys=True)
|
|
print(f"\nwrote {path}")
|
|
|
|
|
|
if __name__ == "__main__":
|
|
main()
|