mirror of
https://github.com/hanzoai/ml.git
synced 2026-11-20 23:08:29 +00:00
A crate whose whole interface is two verbs. `Config` holds hyperparameters and has no `predict`. `Fit` maps `Config -> Data -> Model`. `Model` is immutable, has private fields and no public constructor, so an unfitted model is not a state to be checked for — it is a value that cannot be spoken, and `check_is_fitted` has no job to do. `Predict::Answer` is `f64` for a regressor and `Class` for a classifier: the two do not unify, so a regressor and a classifier are different types rather than one type with a mode flag. There is no `get_params`, no `BaseEstimator`, no clone protocol and no `Pipeline` — those exist in Python because a config and a fitted model are the same object there. Every fitted model is named by `Address`, a content address over its shape, its hyperparameters and its learned parameters, hashing floats as IEEE-754 bits and length-prefixing every sequence so the encoding is injective. Same construction and same reasoning as the risk plane's published model values, so a fit can be stored, cited by a decision, compared against a challenger and rolled back to without a registry. Three estimators: least squares by Householder QR on centred data (QR not the normal equations, which square the condition number; centring not a ones column, so the intercept never enters the factorisation), binary logistic regression by Newton steps on a strictly convex penalised objective, and gradient boosted CART regression trees. A TREE READS FEATURES AT IEEE-754 BINARY32, and that is the algorithm rather than a detail. scikit-learn casts X to float32 before its splitter sees it, XGBoost and LightGBM do the same, and ai.onnx.ml's TreeEnsembleRegressor stores nodes_values as floats. A tree never does arithmetic on a feature value — it only orders values and places thresholds between them — so precision decides only which side of a boundary a sample falls on. Placing the threshold at single(a)/2 + single(b)/2 instead of a/2 + b/2 moves it by 2e-8, which is enough to send a sample down the other branch. Measured over the 40 depth-1 trees of the stumps fixture, whose roots all see the full sample set: the single-precision midpoint reproduces scikit-learn's threshold bit-for-bit 40 times out of 40, the double-precision midpoint 0 times out of 40. Correctness is asserted against scikit-learn itself. Fixtures generated by 1.9.0 are committed and read as data, so `cargo test` needs no Python and the oracle's version is a recorded fact. 195 boosted trees / 1,977 split nodes agree node for node — split feature, threshold and both children all exact — with leaf values to 1.3e-15 and predictions to 8.9e-16 against a 1e-10 bound; least squares agrees to 1.1e-15 against 1e-10 and logistic to 1.2e-8 against 1e-7, with hard labels exact. Each tolerance is derived from the mathematics and every assertion prints the deviation it observed, so a regression that stays inside a bound is still visible. The generator refuses to record an oracle that is not a function of its data: at scikit-learn's default min_samples_leaf a tree's structure varies with random_state (20 structures over 40 seeds, prediction spread 3.9), so every case pins the tie away and proves determinism over 12 seeds before writing a number. This crate breaks split ties by lowest feature index instead, because a content address over a model an RNG helped choose would name one draw rather than one fit. The exhaustive split search is not lowered to hanzo-kernel, and the obstacle is specific: a launch geometry must be stated before the launch, and across all 69 launch sites in that crate Grid::Static is the only variant used, every geometry computed on the host from a count it already knows. A CART node's sample count is not such a number. The histogram formulation dissolves this — bin count is a hyperparameter, so the per-level geometry is host-knowable — but it quantises thresholds and so cannot hold the node-for-node bar. It is named, not built. scikit-learn is a test oracle only. No code or docstring is taken from it, and NOTICE records that, along with the obligation to add BSD-3 attribution if that ever changes. The crate also carries, unchanged and unclaimed, the anomaly-detection and preprocessing modules written concurrently on another track (isolation, neighbour, transform, scale, impute, encode, metric, moment, split, twister, plus benches and the perf oracle). They share lib.rs and tests/sklearn.rs with the estimator seam, so the crate cannot be committed in halves. Their `Transform` and `Outlier` traits extend this seam without changing `Fit` or `Predict`. One of their fixture tests, `the_imbalanced_metrics_match_scikit_learn`, is red on a fixture key that measure.json does not yet carry — which is why this is a branch and not main.
323 lines
10 KiB
Rust
323 lines
10 KiB
Rust
//! Putting features on comparable footing.
|
|
//!
|
|
//! Both scalers fit from a [`crate::moment`] statistic, so both stream: rows unbounded,
|
|
//! state `O(p)`, and a fit spans threads, chunks or devices by combining statistics rather
|
|
//! than by moving data. Neither ever needs the design resident.
|
|
//!
|
|
//! # Scale, concretely
|
|
//!
|
|
//! `fit` takes `&Matrix`, so the data it is handed is resident and owned by the caller —
|
|
//! `8·n·p` bytes, which on this 128 GB box is about `1.6·10⁸` rows of 100 `f64` features
|
|
//! with room to work in. Past that, do not hand it one matrix: fold [`crate::moment`]
|
|
//! statistics over chunks and call [`Standard::of`] on the result. The fitted value is
|
|
//! `O(p)` either way — three `Vec<f64>` of length `p`, whatever `n` was.
|
|
//!
|
|
//! Single device. There is no GPU path, and the measured reason is in `oracle/README.md`:
|
|
//! a scaler is one pass of one multiply-add per value, so it runs at memory bandwidth and
|
|
//! a transfer to a device would cost more than the arithmetic it saves.
|
|
|
|
use crate::moment::{usable, Extent, Moments};
|
|
use crate::{Error, Matrix, Result, Transform};
|
|
|
|
/// Centre each feature on its mean and divide by its standard deviation.
|
|
///
|
|
/// The fitted value is three vectors of length `p` and nothing else, whatever `n` was.
|
|
/// [`Standard::fit`] and [`Standard::of`] are the only ways to obtain one, so an unfitted
|
|
/// scaler is not a value that can be spoken.
|
|
///
|
|
/// # What is deliberately absent
|
|
///
|
|
/// scikit-learn's `with_mean=False` exists so that centring cannot densify a sparse
|
|
/// matrix. [`Matrix`] is dense, so the flag would be a mode with nothing to protect. A
|
|
/// `Standard` always centres and always scales.
|
|
#[derive(Debug, Clone, PartialEq)]
|
|
pub struct Standard {
|
|
centre: Vec<f64>,
|
|
scale: Vec<f64>,
|
|
variance: Vec<f64>,
|
|
}
|
|
|
|
impl Standard {
|
|
/// Fit on one design matrix.
|
|
pub fn fit(x: &Matrix) -> Self {
|
|
Self::of(&Moments::par(x))
|
|
}
|
|
|
|
/// Fit from an already-accumulated statistic: the streaming, parallel and
|
|
/// multi-device entry point.
|
|
///
|
|
/// Total — every `Moments` implies a scaler. A feature with no spread gets a divisor
|
|
/// of one rather than an error, because a constant column is information a caller
|
|
/// usually wants passed through as zero rather than refused.
|
|
pub fn of(m: &Moments) -> Self {
|
|
let variance = m.variance();
|
|
Self {
|
|
centre: m.mean().to_vec(),
|
|
scale: variance.iter().map(|v| usable(v.sqrt())).collect(),
|
|
variance,
|
|
}
|
|
}
|
|
|
|
/// The per-feature mean that gets subtracted.
|
|
pub fn centre(&self) -> &[f64] {
|
|
&self.centre
|
|
}
|
|
|
|
/// The per-feature divisor: the standard deviation, or one for a feature with no
|
|
/// spread.
|
|
pub fn scale(&self) -> &[f64] {
|
|
&self.scale
|
|
}
|
|
|
|
/// The per-feature variance as fitted, so a feature with no spread reads zero here
|
|
/// even though its divisor reads one.
|
|
pub fn variance(&self) -> &[f64] {
|
|
&self.variance
|
|
}
|
|
|
|
/// Undo the transform.
|
|
pub fn invert(&self, x: &Matrix) -> Result<Matrix> {
|
|
width(self.centre.len(), x)?;
|
|
let mut out = Vec::with_capacity(x.n() * x.p());
|
|
for i in 0..x.n() {
|
|
for (j, v) in x.row(i).iter().enumerate() {
|
|
out.push(v * self.scale[j] + self.centre[j]);
|
|
}
|
|
}
|
|
Matrix::new(x.n(), x.p(), out)
|
|
}
|
|
}
|
|
|
|
impl Transform for Standard {
|
|
fn features(&self) -> usize {
|
|
self.centre.len()
|
|
}
|
|
|
|
fn width(&self) -> usize {
|
|
self.centre.len()
|
|
}
|
|
|
|
fn apply(&self, x: &Matrix) -> Result<Matrix> {
|
|
width(self.centre.len(), x)?;
|
|
let mut out = Vec::with_capacity(x.n() * x.p());
|
|
for i in 0..x.n() {
|
|
for (j, v) in x.row(i).iter().enumerate() {
|
|
out.push((v - self.centre[j]) / self.scale[j]);
|
|
}
|
|
}
|
|
Matrix::new(x.n(), x.p(), out)
|
|
}
|
|
}
|
|
|
|
/// The interval a [`Range`] maps into.
|
|
///
|
|
/// Constructed only from a low bound strictly below a finite high one, so an inverted or
|
|
/// degenerate target is not a value that exists and no scaler has to check for one.
|
|
#[derive(Debug, Clone, Copy, PartialEq)]
|
|
pub struct Span {
|
|
low: f64,
|
|
high: f64,
|
|
}
|
|
|
|
impl Span {
|
|
/// Zero to one, which is what almost every caller wants.
|
|
pub const UNIT: Span = Span {
|
|
low: 0.0,
|
|
high: 1.0,
|
|
};
|
|
|
|
/// An interval from `low` up to `high`.
|
|
pub fn new(low: f64, high: f64) -> Result<Self> {
|
|
if !low.is_finite() || !high.is_finite() || low >= high {
|
|
return Err(Error::Config(format!(
|
|
"a span runs from a finite low bound up to a strictly greater finite high \
|
|
bound, not {low} to {high}"
|
|
)));
|
|
}
|
|
Ok(Self { low, high })
|
|
}
|
|
|
|
/// The bottom of the interval.
|
|
pub fn low(&self) -> f64 {
|
|
self.low
|
|
}
|
|
|
|
/// The top of the interval.
|
|
pub fn high(&self) -> f64 {
|
|
self.high
|
|
}
|
|
}
|
|
|
|
/// Map each feature's observed range onto a fixed interval.
|
|
///
|
|
/// Values outside the range seen while fitting land outside the target interval: the
|
|
/// transform is affine and is not clipped, which is scikit-learn's behaviour. Clipping
|
|
/// would hide the distribution shift a caller usually wants to see.
|
|
#[derive(Debug, Clone, PartialEq)]
|
|
pub struct Range {
|
|
low: Vec<f64>,
|
|
high: Vec<f64>,
|
|
scale: Vec<f64>,
|
|
shift: Vec<f64>,
|
|
span: Span,
|
|
}
|
|
|
|
impl Range {
|
|
/// Fit on one design matrix, mapping into `span`.
|
|
pub fn fit(x: &Matrix, span: Span) -> Self {
|
|
Self::of(&Extent::par(x), span)
|
|
}
|
|
|
|
/// Fit from an already-accumulated statistic: the streaming and multi-device path.
|
|
pub fn of(e: &Extent, span: Span) -> Self {
|
|
let reach = span.high - span.low;
|
|
let scale: Vec<f64> = (0..e.features())
|
|
.map(|j| reach / usable(e.high()[j] - e.low()[j]))
|
|
.collect();
|
|
let shift = (0..e.features())
|
|
.map(|j| span.low - e.low()[j] * scale[j])
|
|
.collect();
|
|
Self {
|
|
low: e.low().to_vec(),
|
|
high: e.high().to_vec(),
|
|
scale,
|
|
shift,
|
|
span,
|
|
}
|
|
}
|
|
|
|
/// The smallest value seen per feature.
|
|
pub fn low(&self) -> &[f64] {
|
|
&self.low
|
|
}
|
|
|
|
/// The largest value seen per feature.
|
|
pub fn high(&self) -> &[f64] {
|
|
&self.high
|
|
}
|
|
|
|
/// The interval fitted into.
|
|
pub fn span(&self) -> Span {
|
|
self.span
|
|
}
|
|
|
|
/// Undo the transform.
|
|
pub fn invert(&self, x: &Matrix) -> Result<Matrix> {
|
|
width(self.low.len(), x)?;
|
|
let mut out = Vec::with_capacity(x.n() * x.p());
|
|
for i in 0..x.n() {
|
|
for (j, v) in x.row(i).iter().enumerate() {
|
|
out.push((v - self.shift[j]) / self.scale[j]);
|
|
}
|
|
}
|
|
Matrix::new(x.n(), x.p(), out)
|
|
}
|
|
}
|
|
|
|
impl Transform for Range {
|
|
fn features(&self) -> usize {
|
|
self.low.len()
|
|
}
|
|
|
|
fn width(&self) -> usize {
|
|
self.low.len()
|
|
}
|
|
|
|
fn apply(&self, x: &Matrix) -> Result<Matrix> {
|
|
width(self.low.len(), x)?;
|
|
let mut out = Vec::with_capacity(x.n() * x.p());
|
|
for i in 0..x.n() {
|
|
for (j, v) in x.row(i).iter().enumerate() {
|
|
out.push(v * self.scale[j] + self.shift[j]);
|
|
}
|
|
}
|
|
Matrix::new(x.n(), x.p(), out)
|
|
}
|
|
}
|
|
|
|
/// The one width check every fitted transform runs, stated once.
|
|
pub(crate) fn width(fitted: usize, x: &Matrix) -> Result<()> {
|
|
if x.p() != fitted {
|
|
return Err(Error::Shape(format!(
|
|
"fitted on {fitted} features, given {}",
|
|
x.p()
|
|
)));
|
|
}
|
|
Ok(())
|
|
}
|
|
|
|
#[cfg(test)]
|
|
mod tests {
|
|
use super::*;
|
|
|
|
#[test]
|
|
fn a_constant_feature_scales_to_zero_and_not_to_infinity() {
|
|
let x = Matrix::new(3, 2, vec![5.0, 1.0, 5.0, 2.0, 5.0, 3.0]).unwrap();
|
|
let s = Standard::fit(&x);
|
|
assert_eq!(s.scale()[0], 1.0);
|
|
assert_eq!(s.variance()[0], 0.0);
|
|
// Matrix::new refuses non-finite values, so a finite result is what makes this
|
|
// call succeed at all.
|
|
let out = s.apply(&x).unwrap();
|
|
assert_eq!(out.at(0, 0), 0.0);
|
|
assert!(Range::fit(&x, Span::UNIT).apply(&x).is_ok());
|
|
}
|
|
|
|
#[test]
|
|
fn a_width_mismatch_is_named_and_not_guessed() {
|
|
let x = Matrix::new(2, 2, vec![1.0, 2.0, 3.0, 4.0]).unwrap();
|
|
let narrow = Matrix::new(4, 1, vec![1.0, 2.0, 3.0, 4.0]).unwrap();
|
|
let e = Standard::fit(&x).apply(&narrow).unwrap_err();
|
|
assert!(
|
|
format!("{e}").contains("fitted on 2 features, given 1"),
|
|
"{e}"
|
|
);
|
|
}
|
|
|
|
#[test]
|
|
fn inverting_returns_what_went_in() {
|
|
let data: Vec<f64> = (0..60).map(|i| (i as f64 * 0.37).sin() * 40.0).collect();
|
|
let x = Matrix::new(20, 3, data).unwrap();
|
|
let s = Standard::fit(&x);
|
|
let back = s.invert(&s.apply(&x).unwrap()).unwrap();
|
|
for i in 0..x.n() {
|
|
for j in 0..x.p() {
|
|
assert!((back.at(i, j) - x.at(i, j)).abs() < 1e-9);
|
|
}
|
|
}
|
|
let r = Range::fit(&x, Span::new(-3.0, 7.0).unwrap());
|
|
let back = r.invert(&r.apply(&x).unwrap()).unwrap();
|
|
for i in 0..x.n() {
|
|
for j in 0..x.p() {
|
|
assert!((back.at(i, j) - x.at(i, j)).abs() < 1e-9);
|
|
}
|
|
}
|
|
}
|
|
|
|
#[test]
|
|
fn an_impossible_span_is_not_a_value() {
|
|
assert!(Span::new(1.0, 1.0).is_err());
|
|
assert!(Span::new(2.0, 1.0).is_err());
|
|
assert!(Span::new(f64::NAN, 1.0).is_err());
|
|
assert!(Span::new(0.0, f64::INFINITY).is_err());
|
|
assert!(Span::new(-3.0, 7.0).is_ok());
|
|
}
|
|
|
|
#[test]
|
|
fn a_chunked_fit_equals_a_whole_fit() {
|
|
let data: Vec<f64> = (0..1200)
|
|
.map(|i| (i as f64 * 0.11).cos() * 9.0 + 3.0)
|
|
.collect();
|
|
let x = Matrix::new(300, 4, data).unwrap();
|
|
let whole = Standard::fit(&x);
|
|
let cuts = [(0usize, 43usize), (43, 100), (100, 271), (271, 300)];
|
|
let streamed = Standard::of(&cuts.iter().fold(Moments::zero(4), |a, &(s, e)| {
|
|
a.merge(&Moments::rows(&x, s, e))
|
|
}));
|
|
for j in 0..4 {
|
|
assert!((whole.centre()[j] - streamed.centre()[j]).abs() < 1e-9);
|
|
assert!((whole.scale()[j] - streamed.scale()[j]).abs() < 1e-9);
|
|
}
|
|
}
|
|
}
|