AUTOMATIC DIFFERENTIATION

Automatic differentiation - A process for evaluating derivatives of a function that depends only on an algorithmic specification of the function, such as a computer program. (The term "automatic" stems from having the derivatives produced when executing the program to evaluate the function.)

There are two modes:
1. forward - direct parse and evaluation, introducing intermediate variables in the usual way, as rules of differentiation are applied.
2. reverse - evaluates entries in the code list first, and lastly differentiates the intermediate and independent variables in reverse order.

While the forward mode is more straightforward and easier to implement, the reverse mode is computationally superior for scalar functions, as its complexity is independent of the number of intermediate variables. More generally, if F:R^n-->R^m, the forward mode has complexity of O(n), and the backward mode has complexity of O(m).