Principle:Avhz RustQuant Automatic Differentiation

Knowledge Sources	Baydin et al. - Automatic Differentiation in Machine Learning: a Survey Griewank & Walther - Evaluating Derivatives
Domains	Optimization, Numerical_Methods, Machine_Learning
Last Updated	2026-02-07 20:00 GMT

Overview

A technique for computing exact derivatives of numerical functions by decomposing computations into elementary operations and applying the chain rule systematically via a computation graph.

Description

Automatic Differentiation (AD) computes derivatives of computer programs exactly (up to floating-point precision), without the approximation errors of finite differences or the algebraic complexity of symbolic differentiation.

Reverse-mode AD (backpropagation) is particularly efficient for functions f: R^n -> R, computing all n partial derivatives in a single backward pass. This makes it ideal for gradient-based optimization where the objective function maps many parameters to a scalar loss.

The computation is recorded on a Wengert List (computation graph), which tracks the sequence of operations and their partial derivatives. Each node in the graph stores:

• Parent indices (operands)
• Partial derivatives with respect to parents (adjoints)

RustQuant implements reverse-mode AD with the Graph and Variable types.

Usage

Use automatic differentiation when you need exact gradients for optimization (e.g., model calibration via gradient descent) or when computing analytic Greeks of Black-Scholes formulas through algorithmic means. AD eliminates the need to derive and implement gradient formulas by hand.

Theoretical Basis

Given a composite function:

$y=f(g(h(x_1,x_2)))$

Chain rule application:

$\frac{\partial y}{\partial x_i}=\frac{\partial y}{\partial f}\cdot \frac{\partial f}{\partial g}\cdot \frac{\partial g}{\partial h}\cdot \frac{\partial h}{\partial x_i}$

Reverse mode accumulation:

1. Forward pass: compute function value, recording operations on graph
2. Backward pass: propagate adjoints from output to inputs

Pseudo-code:

// Abstract algorithm (NOT real implementation)
graph = new_graph()
x = graph.var(value)       // create variable node
y = f(x)                   // build computation graph
gradients = y.accumulate() // reverse-mode sweep
dy_dx = gradients.wrt(&x)  // extract partial derivative

Complexity: The backward pass costs at most 4-5x the forward pass, independent of the number of input variables.

Related Pages

Implemented By

• Implementation:Avhz_RustQuant_Graph_and_Variable