AI Tech Inspire

Differentiable Parametric Curves in PyTorch: B‑Splines, Legendre, Vectorized Training

What if a small, focused library could turn curves into first-class citizens in your models—and keep everything trainable end to end? At AI Tech Inspire, we spotted a compact release that does exactly that for PyTorch: differentiable parametric curves you can drop into your usual training loops and backprop through like any other layer.

Key facts at a glance

  • A new PyTorch library implements differentiable parametric curves with gradients flowing to both curve inputs and parameters.
  • Current primitives: B‑Spline curves (implemented efficiently by exploiting sparsity) and Legendre Polynomials.
  • Everything is vectorized: over mini-batches and across multiple curves in parallel.
  • Applications called out by the author: continuous embeddings for embedding-based models (e.g., factorization machines, transformers), KANs, and shape-restricted models (e.g., monotonic win-probability vs. bid).
  • Not limited to B‑Splines; any well-approximating basis for learned activations can be used.
  • Source and docs: github.com/alexshtf/torchcurves.
  • Motivation: consolidates ad‑hoc curve implementations from past projects into a general-purpose library.

Why differentiable curves matter

Most deep learning stacks focus on tensors, layers, and activations. Curves—explicit, smooth, low-dimensional functions—are often implicit or approximated via MLPs. This library makes curves explicit and trainable. Because the curves are differentiable, gradients can flow to both the inputs of the curve and its parameters. That unlocks several design patterns:

  • Learned continuous embeddings: Replace discrete embedding tables with continuous functions that map a scalar or low-dimensional input to a dense representation. Useful for IDs with natural ordering (e.g., time, price, position) where continuity and smoothness help.
  • Feature calibrators: Model per-feature transformations as smooth curves (calibration layers) before a downstream model. It’s a modern spin on Generalized Additive Models with neural training.
  • Shape constraints: Need monotonicity? With B‑Splines, ordering of coefficients can enforce increasing/decreasing behavior. That’s handy for problems like “probability of winning an auction vs. bid” where the function should be increasing in the bid.

Key takeaway: Treating curves as differentiable layers can impose structure, smoothness, and interpretability—without giving up end‑to‑end training.

What’s inside: B‑Splines and Legendre Polynomials

The library ships with two families of bases:

  • B‑Splines: Flexible, piecewise polynomial curves with local support. Locality yields a sparse basis matrix, and the implementation exploits that sparsity for efficiency. In practice, this means fast evaluations and gradients, especially when you increase the number of knots.
  • Legendre Polynomials: An orthogonal polynomial basis on a bounded interval. Orthogonality can help with numerical stability and disentangling the effect of different coefficients.

Both are vectorized over the batch dimension and over sets of curves. That’s particularly useful if you want one curve per feature (e.g., a calibrator for each column) or one curve per item in a batch of tasks. With PyTorch tensors and CUDA acceleration, the entire setup slots into existing pipelines without bespoke kernels.

Because the curves are built on top of autograd-friendly ops, you keep the usual tooling—optimizer.step(), .backward(), mixed precision, and the rest. If you need gradients w.r.t. inputs (e.g., for sensitivity analysis), you can toggle requires_grad_(True) on inputs and propagate signals through the curve.

Use cases worth trying this week

  • Continuous embeddings for tabular or ID-like features: Instead of a huge lookup table, parameterize a smooth curve per feature. This can shrink memory and impose continuity. It also plays well with models like factorization machines and Transformers when you want a compact, continuous positional or feature encoding.
  • KANs (Kolmogorov–Arnold Networks): If you’re experimenting with KAN-style architectures that learn per-dimension functions combined later, a curve library provides a modular way to implement the univariate function blocks and tune their smoothness.
  • Monotonic models for economics and auctions: Predicting win probability as a function of bid should be monotonic increasing. By predicting B‑Spline coefficients with a neural network and constraining their increments to be nonnegative, you can enforce the right shape while still learning from data.
  • Time-series smoothing and calibration: Add a curve layer to denoise or calibrate signals before feeding into an RNN, TCN, or Transformer. Smoothness can act as a structural prior.
  • Interpretable per-feature transformations: When stakeholders ask, “What does your model do with feature X?”, a plotted B‑Spline curve is easier to interpret than raw MLP weights.

How it fits your PyTorch workflow

From a developer’s perspective, the attraction is that curves behave like modules or functions you can compose. Define a curve, feed it inputs, get outputs, and optimize everything jointly with the rest of your model. A mental model looks like this:

  • Inputs: x (possibly with x.requires_grad = True if you want input sensitivities).
  • Curve: a parameterized mapping, e.g., a B‑Spline with knot vector and learnable coefficients c.
  • Output: y = curve(x), with gradients flowing to c and optionally to x.

Because evaluation is vectorized, batching is natural: feed [batch, features] through a bank of curves shaped [features, basis_dim] and let the library handle broadcasting. If you run on GPU, the underlying ops align with PyTorch’s device semantics.

Getting started typically looks like: pip install torchcurves, import a curve class, and place it alongside your layers. You can keep your usual training loop, dataloaders, and metrics, which is a big win for adoption.

Practical notes and gotchas

  • Knot placement matters (B‑Splines): Uniform knots are simple but not always optimal. If your data has sharp changes, consider denser knots in those regions. Guard against overfitting by regularizing coefficient magnitudes or curve roughness.
  • Boundary behavior: Decide how you want the curve to behave outside the knot interval—clamping, extrapolation, or padding with boundary knots. This affects stability at inference time.
  • Degree vs. smoothness trade-offs: Higher-degree splines are smoother but may require more careful regularization. Legendre degree controls approximation power; watch for oscillations at high degrees (Runge-like effects).
  • Shape constraints: For monotonicity, constrain coefficient differences (e.g., via softplus). It’s usually easier to enforce constraints at the parameter level than at outputs post hoc.
  • Performance: The implementation exploits sparsity for B‑Splines and uses vectorized ops. Still, monitor memory when you scale curves × basis size. Mixed precision can help on GPU.
  • Gradients w.r.t. inputs: Useful but can be costly. Enable only if you need them for regularizers, adversarial robustness, or sensitivity analyses.

Why it might change your modeling habits

Neural networks are fantastic function approximators, but they don’t always encode domain structure. Curves shine when you need smoothness, locality, or shape constraints with controllable complexity. Instead of contorting an MLP to behave monotonically or be smooth in one dimension, you can put that prior directly into a curve and let the rest of the network learn around it.

In practical terms, that can mean fewer parameters than a wide MLP, more interpretable components, and potentially better sample efficiency when the inductive bias matches the task. The fact that it’s all trainable inside PyTorch removes much of the friction: no manual finite differences, no custom backward passes.

Questions to explore

  • For your dataset, does a per-feature B‑Spline calibrator improve stability or fairness metrics versus a baseline MLP?
  • Can you replace learned positional embeddings with a curve-based encoding and reduce parameter counts without hurting accuracy?
  • How do monotonic constraints affect calibration in auction or pricing models—and can they reduce ROI variance in deployment?
  • In KAN-style architectures, what curve basis (B‑Spline vs. Legendre) trains faster or generalizes better?

If you’ve been wanting a clean, autograd-friendly way to bring explicit curves into your models, this release is worth a look. The implementation details—sparse B‑Splines, vectorization across batches and curves—make it feel native in modern pipelines. The repo is here: torchcurves on GitHub. If you test it in production or research, AI Tech Inspire would love to hear how it changes your modeling stack.

Recommended Resources

As an Amazon Associate, I earn from qualifying purchases.