You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Least squares and ridge — Least squares minimizes squared prediction errors; ridge adds a squared-coefficient penalty to stabilize fits with correlated predictors. Includes implementation and solver guidance.
Lasso — 1996. Constrains the sum of absolute regression coefficients, shrinking some exactly to zero. Combines estimation and variable selection.
Elastic net — Combines L1 and L2 penalties, allowing sparse coefficients while retaining ridge-style shrinkage. A useful alternative to lasso for correlated predictors.
Generalized linear models — 1972. Connects an exponential-family response mean to a linear predictor through a link function. Unifies models fitted by iteratively weighted linear regression.
Logistic regression — Models binary class probability through a logistic link applied to a linear score. The documented implementation also supports multiclass classification and regularization.
Generalized linear mixed models — 1993. Adds random effects to GLMs for correlated observations and overdispersion. Breslow–Clayton develops approximate inference using penalized quasi-likelihood.
Tests as linear models — 2019. Worked connections between regression, t-tests, correlation and ANOVA. Rank-based analogues are also discussed; some equivalences are approximate.
Additive models and adaptive splines
Generalized additive models — 1986. Replaces a GLM's linear predictor terms with smooth functions of individual inputs. Local scoring extends additive regression to different response distributions.
MARS — 1991. Builds regression models from products of spline basis functions, selecting their number, variables and knot locations from data. Separates additive contributions from interactions.
Intelligible classification and regression — 2012. Studies additive models whose individual feature contributions can be plotted, comparing fitting methods and the accuracy cost of restricting interactions.
GA2M — 2013. Extends additive models with selected pairwise interactions. FAST ranks candidate feature pairs, keeping explanations as one- and two-dimensional functions.
Intelligible healthcare models — 2015. Pneumonia and readmission case studies use additive models with pairwise interactions to expose unexpected learned associations and revise models.
GAMs with autoregressive terms — 2012. Combines nonlinear covariate terms with autoregression for serially correlated outcomes. Demonstrated on temperature and daily mortality.
Neural Additive Models — 2020. Jointly trains a separate neural network for each input feature, then combines their outputs linearly. Learns flexible univariate shapes while preserving additive structure.
How trustworthy are GAMs? — 2020. Shows that similarly accurate GAMs can learn conflicting feature shapes. Compares inductive biases, sparsity and behavior on rare subpopulations; accuracy alone does not validate an explanation.
Trees and tabular prediction
CART — Recursively partitions feature space into regions with simple predictions. Depth and pruning control complexity; individual trees can be unstable under changes to training data.
Random forests — 2001. Combines randomized trees, including random feature selection at splits. Analyzes how individual tree strength and dependence affect ensemble error.
Bayesian additive regression trees — 2008. Represents a response as a sum of trees with a regularization prior that keeps individual trees weak. Bayesian backfitting samples the model and its predictive uncertainty.
XGBoost — 2016. Gradient-boosted trees with sparsity-aware split finding and a weighted quantile sketch for approximate tree learning. The paper also addresses cache access and distributed computation.
CatBoost — 2017. Uses ordered boosting and categorical-feature processing to address prediction shift caused by target leakage during training.
TabPFN: original classifier — 2022. Pretrains a transformer on synthetic classification tasks. At inference, labeled training rows provide context for predictions without updating the model weights; the original scope is small tabular datasets.
TabPFN: classification and regression — 2025. Extends prior-data-fitted tabular prediction to regression, categorical inputs and missing values. A pretrained transformer performs in-context prediction from labeled examples.
TabM — 2024. Builds parameter-sharing ensembles of multilayer perceptrons for tabular prediction. Joint training produces multiple predictions without storing a separate full network for every ensemble member.
TabICL — 2025. Combines representations of table rows with a transformer for in-context classification. Synthetic-task pretraining supports prediction on new tables without task-specific weight updates.
Observation protocols
Repeated measurements - 2026 preprint. More readings with independent noise at fixed inputs can harm new-input predictions. Priors coupling input density and responses can count repeated locations as fresh input evidence; TabPFN-v3 also depends on row count.
Shallow TFMs - 2026 preprint. Uses affine block replacement, following ReplaceMe, with per-feature maps for dual-axis models. Across 171 classification datasets and eight models, halving depth loses at most 0.002 median AUROC per model.
Compression limits - On 41 regression datasets, retaining one block costs TabPFN-3 0.132 median R2. Regression windows use labelled calibration R2; classification uses output KL. Maps are fitted to the target table's activations.
Compression cost - MIMIC-III: one-block TabPFN-3 on RTX 5090 gives 1.2x speedup for 512-query re-encoding, 2.4x for a cached single query. On i9-14900K (fp32, batch 512), re-encoding gains are 1.4x for TabPFN-3 and 12.4x for TabPFNv2.
Local and conditional-independence classifiers
Nearest-neighbor classification — 1967. Predicts from nearby labeled examples. Cover and Hart bound asymptotic one-neighbor error by twice Bayes error under their assumptions; this is neither a finite-sample guarantee nor a guarantee for an arbitrary learned distance.
Naive Bayes — 1997 analysis. Multiplies class-conditional feature probabilities under an independence model. Domingos and Pazzani show that correct class decisions can survive some dependence violations even when estimated probabilities are inaccurate.
Support-vector methods
Support-vector networks — 1995. Constructs a linear decision surface in a nonlinear feature space and extends the approach to nonseparable training data. A foundation for kernel-based classification.
SVM implementations — Linear and kernel classification, regression and outlier detection. Decision scores are not probabilities; probability estimation adds a separate fitting step.
Probabilistic prediction, survival and treatment effects
Quantile regression forests — 2006. Extends random forests to estimate the conditional response distribution and its quantiles, rather than only a conditional mean.
Extreme NO2 forecasting — 2020. Compares probabilistic models for high-pollution episodes, including threshold-exceedance probabilities. A case study in evaluating forecast distributions rather than only mean predictions.
Kaplan–Meier — 1958. Estimates a survival curve from incomplete lifetime observations using a product-limit estimator. Independent censoring is a substantive assumption, explicitly discussed in the paper.
Cox regression — 1972. Models covariate effects as multipliers of an unspecified baseline hazard. Handles censored event times without choosing a parametric baseline distribution.
DeepSurv — 2016. Uses a neural network for the risk function in a Cox proportional-hazards model, allowing nonlinear covariate interactions in survival prediction.
Treatment-effect meta-learners — 2017. Builds estimators of conditional average treatment effects from existing regression learners. Introduces the X-learner and studies settings with unequal treatment-group sizes.
Simpson's paradox — 2019. Constructs examples where conditioning on additional variables reverses an association. Illustrates the dependence of aggregated comparisons on the variables included in an analysis.
NGBoost — 2019. Boosts parameters of a conditional probability distribution using natural gradients, extending boosting from point estimates to probabilistic regression.
ARIMA and seasonal ARIMA — 2021. Combines differencing, autoregressive terms and moving-average errors. Seasonal extensions add corresponding terms at seasonal lags. Reference: FPP3, Chapter 9.
Holt–Winters and ETS — 2021. Smooths level, trend and seasonality recursively. ETS supplies statistical state-space formulations and forecast distributions. Reference: FPP3, Chapter 8.
STL decomposition — 2021. Uses Loess smoothing to separate trend, seasonal and remainder components. Seasonal variation may change over time; robust fitting limits outlier influence on the smooth components.
Dynamic regression — 2021. Combines regression predictors with ARIMA errors. Forecasts require future predictor values or scenarios; ordinary intervals condition on these values rather than propagating their uncertainty.
Vector autoregression — 2021. Models several series jointly, regressing each on lags of all series. Stationarity and differencing determine the appropriate specification.
GARCH — 1986. Generalizes ARCH by allowing past conditional variances into the current variance equation. Studies stationarity, estimation and testing for time-varying volatility.
Kalman filtering — 1960. Formulates linear filtering through state transitions and recursive error-covariance updates. Treats stationary and nonstationary prediction problems in a common framework.
Hidden Markov models — 1989. Rabiner reviews Markov models with hidden states and probabilistic observations, with applications to speech recognition. A reference for the model's statistical structure and inference problems.
Infinite HMM — 2001. Uses Dirichlet processes to extend HMMs to countably many possible hidden states, integrating out transition parameters rather than fixing a finite state count in advance.
Dynamic time warping — 1978. Aligns sequences by optimizing a nonlinear time-warping path with dynamic programming. Sakoe–Chiba studies path constraints for spoken-word matching; this is an alignment method, not a forecasting model.
Neural forecasting
DeepAR — 2017. Trains an autoregressive recurrent network across related time series to produce probabilistic forecasts, sharing statistical information across series.
N-BEATS — 2019. Uses fully connected blocks with forward and backward residual connections for forecasting. Provides generic and explicitly interpretable configurations.
Temporal Fusion Transformer — 2019. Combines recurrent processing and attention for multi-horizon forecasts. Separately accommodates static covariates, known future inputs and historically observed variables.
Chronos — 2024. Scales and quantizes time-series values into tokens, then trains transformer language models with cross-entropy. Sampling token sequences produces probabilistic forecasts.
PatchTST — 2022. Treats time-series patches as tokens and processes each channel separately with shared embedding and transformer weights; designed for forecasting and self-supervised pretraining.
Dynamical and mechanistic models
Simple discrete dynamics — 1976. Shows how simple first-order nonlinear difference equations can produce stable equilibria, cycles or irregular fluctuations. A foundation for studying deterministic complexity.
Predator–prey inference — 2018. Fits Lotka–Volterra differential equations with a statistical observation model to lynx–hare pelt counts. Demonstrates Bayesian parameter inference and posterior predictive checks.
Lorenz dynamics — 1963. Studies a deterministic nonlinear convection model whose nearby initial states can diverge. Connects nonperiodic dynamics to limits of long-range prediction.
Bass diffusion — 1969. Models initial product adoption using innovative and imitative behavior: purchase timing depends on previous buyers. The original study tests the model on consumer durables.
Bass model retrospective — 2004. Reviews the origins, applications and extensions of the product-diffusion model, identifying subsequent research directions.
SIR epidemic dynamics — 1927. Models infection and removal in a susceptible population under simplifying assumptions about contact and immunity. Epidemics can end while susceptible individuals remain.
DICE — 1992. Describes an integrated climate–economy model for dynamic optimization of climate policy. Documents the structure and assumptions behind its cost–benefit analysis.
Evaluation and software
Conformal prediction overview — 2021. Reviews calibration-score methods for prediction sets with finite-sample marginal coverage under exchangeability. This guarantee does not imply coverage at each individual input.
Forecast evaluation — 2021. Compare against naïve and seasonal-naïve baselines. Evaluate held-out point forecasts and distributions separately; scaled errors support comparisons across series with different units.
Rolling-origin validation — 2021. Repeatedly trains on past observations and evaluates later ones. Use forecast horizons matching deployment; fitting residuals do not measure future forecast accuracy.
rpart — R implementation of recursive partitioning for classification, regression and survival trees, based on CART.
party — R toolbox for conditional-inference trees, forests and model-based recursive partitioning. Supports response types including censored and multivariate outcomes.
mgcv — R implementation of generalized additive and additive mixed models, with smoothing-parameter estimation and several response-distribution families.
