% Optimisation.
%%%%%%%%%%%%%%%

\chapter{Optimisation}
\label{ch: optimisation}
\index{optimisation|textbf}



% Implementation.
%%%%%%%%%%%%%%%%%

\section{Implementation}



% The interface.
%~~~~~~~~~~~~~~~

\subsection{The interface}

Optimisation or minimisation is available in relax via the \uf{minimise.grid\ufus{}search} and \uf{minimise.execute} user functions.
The mathematical model optimised depends on the current data pipe type -- it is implemented differently for each specific analysis.
For analyses such as the steady state NOE (Chapter~\ref{ch: NOE}) or reduced spectral density mapping (Chapter~\ref{ch: J(w) mapping}), the solution can be found by direct calculation rather than optimisation.
In these cases, the \uf{minimise.calculate} user function should be used instead.



% Minfx.
%~~~~~~~

\subsection{The minfx package}
\index{optimisation!minfx|textbf}

To minimise target functions within relax, the minfx optimisation library is used (\href{https://gna.org/projects/minfx/}{https://gna.org/projects/minfx/}).
This Python package is bundled with the official relax distribution archives.
If you are using a version of relax checked out directly from the source code repository, you will need to manually install minfx as a standard Python package.

The minfx library originated as one of relax's packages, but has been spun off as its own project for the benefit of other scientific, analytical, or numerical projects.
Minfx is complete, very stable, well tested.
Numerous optimisation algorithms are supported and can be clustered into three major categories -- the line search methods, the trust-region methods, and the conjugate gradient methods.

The supported line search methods include:
\begin{itemize}
  \item Steepest descent,
  \item Back-and-forth coordinate descent,
  \item Quasi-Newton BFGS,
  \item Newton,
  \item Newton-CG.
\end{itemize}

The supported trust-region methods include:
\begin{itemize}
  \item Cauchy point,
  \item Dogleg,
  \item CG-Steihaug,
  \item Exact trust region.
\end{itemize}

The supported conjugate gradient methods include:
\begin{itemize}
  \item Fletcher-Reeves,
  \item Polak-Ribi\`ere,
  \item Polak-Ribi\`ere +,
  \item Hestenes-Stiefel.
\end{itemize}

In addition, the following miscellaneous algorithms are implemented:
\begin{itemize}
  \item Grid search,
  \item Nelder-Mead simplex,
  \item Levenberg-Marquardt.
\end{itemize}

The step selection subalgorithms include:
\begin{itemize}
  \item Backtracking line search,
  \item Nocedal and Wright interpolation based line search,
  \item Nocedal and Wright line search for the Wolfe conditions,
  \item More and Thuente line search,
  \item No line search.
\end{itemize}

The Hessian modification subalgorithms include: 
\begin{itemize}
  \item Unmodified Hessian,
  \item Eigenvalue modification,
  \item Cholesky with added multiple of the identity,
  \item The Gill, Murray, and Wright modified Cholesky algorithm,
  \item The Schnabel and Eskow 1999 algorithm.
\end{itemize}

All methods can be constrained by:
\begin{itemize}
  \item The Method of Multipliers (also known as the Augmented Lagrangian),
  \item The logarithmic barrier function.
\end{itemize}

These lists may be out of date, so please see the minfx website for additional information.



% The optimisation space.
%%%%%%%%%%%%%%%%%%%%%%%%%

\section{The optimisation space}
\index{optimisation!space|textbf}

The optimisation of the parameters of an arbitrary model is dependent on a function $f$ which takes the current parameter values $\theta \in \mathbb{R}^n$ and returns a single real value $f(\theta) \in \mathbb{R}$ corresponding to position $\theta$ in the $n$-dimensional space.
For it is that single value which is minimised as
\begin{equation}
 \hat\theta = \arg \min_\theta f(\theta),
\end{equation}

\noindent where $\hat\theta$ is the parameter vector which is equal to the argument which minimises the function $f(\theta)$.
In most analyses in relax, $f(\theta)$ is the chi-squared\index{chi-squared|textbf} equation
\begin{equation} \label{eq: chi2}
 \chi^2(\theta) = \sum_{i=1}^n \frac{(y_i - y_i(\theta))^2}{\sigma_i^2},
\end{equation}

\noindent where $i$ is the summation index over all data, $y_i$ is the experimental data, $y_i(\theta)$ is the back calculated data, and $\sigma_i$ is the experimental error.



% Topology of the space.
%%%%%%%%%%%%%%%%%%%%%%%%

\section{Topology of the space}
\index{optimisation!topology|textbf}

The problem of finding the minimum is complicated by the fact that optimisation algorithms are blind to the curvature of the complete space.
Instead they rely on topological information about the current and, sometimes, the previous parameter positions in the space.
The techniques use this information to walk iteratively downhill to the minimum.



% The function value.
%~~~~~~~~~~~~~~~~~~~~

\subsection{The function value}

At the simplest level all minimisation techniques require at least a function which will supply a single value for different parameter values $\theta$.
Conceptually this is the height of the space at the current position.
For certain algorithms, such a simplex minimisation\index{optimisation!algorithm!Nelder-Mead simplex}, this single value suffices.



% The gradient.
%~~~~~~~~~~~~~~

\subsection{The gradient}
\label{sect: gradient}
\index{gradient|textbf}

Most techniques also utilise the gradient at the current position.
Although symbolically complex in the case of model-free analysis, for example, the gradient can simply be calculated as the vector of first partial derivatives of the chi-squared\index{chi-squared} equation with respect to each parameter.
It is defined as
\begin{equation}
 \nabla = \begin{pmatrix}
  \frac{\partial}{\partial \theta_1} \\
  \frac{\partial}{\partial \theta_2} \\
  \vdots \\
  \frac{\partial}{\partial \theta_n} \\
 \end{pmatrix}
\end{equation}

\noindent where $n$ is the total number of parameters in the model.

The gradient is supplied as a second function to the algorithm which is then utilised in diverse ways by different optimisation techniques.
The function value together with the gradient can be combined to construct a linear or planar description of the space at the current parameter position by first-order Taylor series approximation
\begin{equation} \label{eq: linear model}
 f(\theta_k + x) \approx f_k  +  x^T \nabla f_k,
\end{equation}

\noindent where $f_k$ is the function value at the current parameter position $\theta_k$, $\nabla f_k$ is the gradient at the same position, and $x$ is an arbitrary vector.
By accumulating information from previous parameter positions a more comprehensive geometric description of the curvature of the space can be exploited by the algorithm for more efficient optimisation.

An example of a powerful algorithm which requires both the value and gradient at current parameter values is the BFGS quasi-Newton minimisation\index{optimisation!algorithm!BFGS}.
The gradient is also essential for the use of the Method of Multipliers constraints algorithm (also known as the Augmented Lagrangian algorithm).



% The Hessian.
%~~~~~~~~~~~~~

\subsection{The Hessian}
\label{sect: Hessian}
\index{Hessian|textbf}

The best and most comprehensive description of the space is given by the quadratic approximation of the topology which is generated from the combination of the function value, the gradient, and the Hessian.
From the second-order Taylor series expansion the quadratic model of the space is
\begin{equation} \label{eq: quadratic model}
 f(\theta_k + x) \approx f_k  +  x^T \nabla f_k  +  \tfrac{1}{2} x^T \nabla^2 f_k x,
\end{equation}

\noindent where $\nabla^2 f_k$ is the Hessian, which is the symmetric matrix of second partial derivatives of the function, at the position $\theta_k$.
The Hessian is the matrix of second partial derivatives and is defined as
\begin{equation}
 \nabla^2 = \begin{pmatrix}
  \frac{\partial^2}{{\partial \theta_1}^2}                       & \frac{\partial^2}{\partial \theta_1 \cdot \partial \theta_2}  & \ldots    & \frac{\partial^2}{\partial \theta_1 \cdot \partial \theta_n} \\
  \frac{\partial^2}{\partial \theta_2 \cdot \partial \theta_1} & \frac{\partial^2}{{\partial \theta_2}^2}                        & \ldots    & \frac{\partial^2}{\partial \theta_2 \cdot \partial \theta_n} \\
  \vdots                                                       & \vdots                                                        & \ddots    & \vdots \\
  \frac{\partial^2}{\partial \theta_n \cdot \partial \theta_1} & \frac{\partial^2}{\partial \theta_n \cdot \partial \theta_2}  & \ldots    & \frac{\partial^2}{{\partial \theta_n}^2} \\
 \end{pmatrix}.
\end{equation}

\noindent The order in which the partial derivatives are calculated is inconsequential, hence the Hessian is symmetric.

As the Hessian is computationally expensive a number of optimisation algorithms try to approximate it, the BFGS algorithm being a notable example.
The most powerful minimisation algorithm for model-free analysis -- Newton optimisation\index{optimisation!algorithm!Newton} -- requires the value, gradient, and Hessian at the current parameter values.



% Optimisation algorithms.
%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Optimisation algorithms}
\index{optimisation!algorithm|textbf}

Prior to minimisation, all optimisation algorithms require a starting position within the optimisation space.
This initial parameter vector is found by employing a coarse grid search -- chi-squared\index{chi-squared} values at regular positions spanning the space are calculated and the grid point with the lowest value becomes the starting position.
The grid search itself is an optimisation technique.
As it is computationally expensive the number of grid points needs to be kept to a minimum.
Hence the initial parameter values are a rough and imprecise approximation of the local minimum.

Once the starting position has been determined by the grid search the optimisation algorithm can be executed.
The number of algorithms developed within the mathematical field of optimisation is considerable.
They can nevertheless be grouped into one of a small number of major categories based on the fundamental principles of the technique.
These include the line search methods, the trust region methods, and the conjugate gradient methods.
For more details on the algorithms described below see \citet{NocedalWright99}.



% Line search methods.
%~~~~~~~~~~~~~~~~~~~~~

\subsection{Line search methods}

The defining characteristic of a line search algorithm is to choose a search direction $p_k$ and then to find the minimum along that vector starting from $\theta_k$ \citep{NocedalWright99}.
The distance travelled along $p_k$ is the step length $\alpha_k$ and the parameter values for the next iteration are
\begin{equation}
 \theta_{k+1} = \theta_k + \alpha_k p_k.
\end{equation}

\noindent  The line search algorithm determines the search direction $p_k$ whereas the value of $\alpha_k$ is found using an auxiliary step-length selection algorithm.


% The steepest descent algorithm.
\subsubsection{The steepest descent algorithm}
\index{optimisation!algorithm!steepest descent|textbf}

One of the simplest line search methods is the steepest descent algorithm.
The search direction is simply the negative gradient, $p_k = -\nabla f_k$, and hence the direction of maximal descent is always followed.
This method is inefficient -- the linear rate of convergence requires many iterations of the algorithm to reach the minimum and it is susceptible to being trapped on saddle points within the space.


% The coordinate descent algorithm.
\subsubsection{The coordinate descent algorithm}
\index{optimisation!algorithm!coordinate descent|textbf}

The coordinate descent algorithms are a simplistic group of line search methods whereby the search directions alternate between vectors parallel to the parameter axes.
For the back-and-forth coordinate descent the search directions cycle in one direction and then back again.
For example for a three parameter model the search directions cycle $\theta_1, \theta_2, \theta_3, \theta_2, \theta_1, \theta_2, \hdots$, which means that each parameter of the model is optimised one by one.
The method becomes less efficient when approaching the minimum as the step length $\alpha_k$ continually decreases (ibid.).


% The BFGS algorithm.
\subsubsection{The BFGS algorithm}
\index{optimisation!algorithm!BFGS|textbf}

The quasi-Newton methods begin with an initial guess of the Hessian and update it at each iteration using the function value and gradient.
Therefore the benefits of using the quadratic model of \eqref{eq: quadratic model} are obtained without calculating the computationally expensive Hessian.
The Hessian approximation $B_k$ is updated using various formulae, the most common being the BFGS formula \citep{Broyden70,Fletcher70,Goldfarb70,Shanno70}.
The search direction is given by the equation $p_k = -B_k^{-1} \nabla f_k$.
The quasi-Newton algorithms can attain a superlinear rate of convergence, being superior to the steepest descent or coordinate descent methods.


% The Newton algorithm.
\subsubsection{The Newton algorithm}
\index{optimisation!algorithm!Newton|textbf}

The most powerful line search method when close to the minimum is the Newton search direction
\begin{equation} \label{eq: Newton dir}
 p_k = - \nabla^2 f_k^{-1} \nabla f_k.
\end{equation}

\noindent This direction is obtained from the derivative of \eqref{eq: quadratic model} which is assumed to be zero at the minimum of the quadratic model.
The vector $p_k$ points from the current position to the exact minimum of the quadratic model of the space.
The rate of convergence is quadratic, being superior to both linear and superlinear convergence.
The technique is computationally expensive due to the calculation of the Hessian.
It is also susceptible to failure when optimisation commences from distant positions in the space as the Hessian may not be positive definite and hence not convex, a condition required for the search direction both to point downhill and to be reasonably oriented.
In these cases the quadratic model is a poor description of the space.
This algorithm is also known as the Newton-Raphson method.


% The Newton conjugate gradient algorithm.
\subsubsection{The Newton conjugate gradient algorithm}
\index{optimisation!algorithm!Newton-CG|textbf}

A practical Newton algorithm which is robust for distant starting points is the Newton conjugate gradient method (Newton-CG).
This line search method, which is also called the truncated Newton algorithm, finds an approximate solution to Equation~\eqref{eq: Newton dir} by using a conjugate gradient (CG) sub-algorithm.
Retaining the performance of the pure Newton algorithm, the CG sub-algorithm guarantees that the search direction is always downhill as the method terminates when negative curvature is encountered.


% The auxiliary step-length selection algorithm.
\subsubsection{The auxiliary step-length selection algorithm}
\index{optimisation!step-length selection algorithm|textbf}

Once the search direction has been determined by the above algorithms the minimum along that direction needs to be determined.
Not to be confused with the methodology for determining the search direction $p_k$, the line search itself is performed by an auxiliary step-length selection algorithm to find the value $\alpha_k$.
A number of step-length selection methods can be used to find a minimum along the line $\theta_k + \alpha_k p_k$.
One is the backtracking line search of \citet{NocedalWright99}.
This method is inexact -- it takes a starting step length $\alpha_k$ and decreases the value until a sufficient decrease in the function is found.
Another is the line search method of \citet{MoreThuente94}.
Designed to be robust, the MT algorithm finds the exact minimum along the search direction and guarantees sufficient decrease.



% Trust region methods.
%~~~~~~~~~~~~~~~~~~~~~~

\subsection{Trust region methods}

In the trust region class of algorithms the curvature of the space is modelled quadratically by \eqref{eq: quadratic model}.
This model is assumed to be reliable only within a region of trust defined by the inequality $\lVert p \rVert \leqslant \Delta_k$ where $p$ is the step taken by the algorithm and $\Delta_k$ is the radius of the $n$-dimensional sphere of trust \citep{NocedalWright99}.
The solution sought for each iteration of the algorithm is
\begin{equation} \label{eq: trust region}
 \min_{p \in \mathbb{R}^n} m_k(p) = f_k  +  p^{T} \nabla f_k  +  \tfrac{1}{2} p^{T} B_k p,  \qquad \textrm{s.t. } \lVert p \rVert \leqslant \Delta_k,
\end{equation}

\noindent where $m_k(p)$ is the quadratic model, $B_k$ is a positive definite matrix which can be the true Hessian as in the Newton model or an approximation such as the BFGS\index{optimisation!algorithm!BFGS} matrix, and $\lVert p \rVert$ is the Euclidean norm of $p$.
The trust region radius $\Delta_k$ is modified dynamically during optimisation -- if the quadratic model is found to be a poor representation of the space the radius is decreased whereas if the quadratic model is found to be reasonable the radius is increased to allow larger, more efficient steps to be taken.


% The Cauchy point algorithm.
\subsubsection{The Cauchy point algorithm}
\index{optimisation!algorithm!Cauchy point|textbf}

The Cauchy point algorithm is similar in concept to the steepest descent\index{optimisation!algorithm!steepest descent} line search algorithm.
The Cauchy point is the point lying on the gradient which minimises the quadratic model subject to the step being within the trust region.
By iteratively finding the Cauchy point the local minimum can be found.
The convergence of the technique is inefficient, being similar to that of the steepest descent algorithm.


% The dogleg algorithm.
\subsubsection{The dogleg algorithm}
\index{optimisation!algorithm!dogleg|textbf}

In changing the trust region radius the exact solutions to \eqref{eq: trust region} map out a curved trajectory which starts parallel to the gradient for small radii.
The end of the trajectory, which occurs for radii greater than the step length, is the bottom of the quadratic model.
The dogleg algorithm attempts to follow a similar path by first finding the minimum along the gradient and then finding the minimum along a trajectory from the current point to the bottom of the quadratic model.
The minimum along the second path is either the trust region boundary or the quadratic solution.
The matrix $B_k$ of \eqref{eq: trust region} can be the BFGS matrix, the unmodified Hessian, or a Hessian modified to be positive definite.


% Steihaug's algorithm.
\subsubsection{Steihaug's algorithm}
\index{optimisation!algorithm!CG-Steihaug|textbf}

Another trust region algorithm is Steihaug's modified conjugate gradient approach \citep{Steihaug83}.
For each step $k$ an iterative technique is used which is almost identical to the standard conjugate gradient procedure except for two additional termination conditions.
The first is if the next step is outside the trust region, the second is if a direction of zero or negative curvature is encountered.


% The exact trust region.
\subsubsection{The exact trust region}
\index{optimisation!algorithm!exact trust region|textbf}

An almost exact solution to \eqref{eq: trust region} can be found using an algorithm described in \citet{NocedalWright99}.
This exact trust region algorithm aims to precisely find the minimum of the quadratic model $m_k$ of the space within the trust region $\Delta_k$.
Any matrix $B_k$ can be used to construct the quadratic model.
However, the technique is computationally expensive.



% Conjugate gradient methods.
%~~~~~~~~~~~~~~~~~~~~~~~~~~~~

\subsection{Conjugate gradient methods}

The conjugate gradient algorithm (CG) was originally designed as a mathematical technique for solving a large system of linear equations \citet{HestenesStiefel52}, but was later adapted to solving nonlinear optimisation problems \citep{FletcherReeves64}.
The technique loops over a set of directions $p_0$, $p_1$, $\hdots$, $p_n$ which are all conjugate to the Hessian \citep{NocedalWright99}, a property defined as
\begin{equation}
 p_i^T \nabla^2 f_k p_j = 0,  \qquad \textrm{for all } i \ne j.
\end{equation}

\noindent By performing line searches over all directions $p_j$ the solution to the quadratic model \eqref{eq: quadratic model} of the position $\theta_k$ will be found in $n$ or less iterations of the CG algorithm where $n$ is the total number of parameters in the model.
The technique performs well on large problems with many parameters as no matrices are calculated or stored.
The algorithms perform better than the steepest descent method and preconditioning of the system is used to improve optimisation.
Preconditioned techniques include the Fletcher-Reeves\index{optimisation!algorithm!Fletcher-Reeves|textbf} algorithm which was the original conjugate gradient optimisation technique \citep{FletcherReeves64}, the Polak-Ribi\`ere\index{optimisation!algorithm!Polak-Ribi\`ere|textbf} method \citep{PolakRibiere69}, a modified Polak-Ribi\`ere method called the Polak-Ribi\`ere +\index{optimisation!algorithm!Polak-Ribi\`ere~+|textbf} method \citep{NocedalWright99}, and the Hestenes-Stiefel\index{optimisation!algorithm!Hestenes-Stiefel|textbf} algorithm which originates from a formula in \citet{HestenesStiefel52}.
As a line search is performed to find the minimum along each conjugate direction both the backtracking and Mor\'e and Thuente auxiliary step-length selection algorithms will be tested with the CG algorithms.



% Hessian modifications.
%~~~~~~~~~~~~~~~~~~~~~~~

\subsection{Hessian modifications}

The Newton search direction, used in both the line search and trust region methods, is dependent on the Hessian being positive definite for the quadratic model to be convex so that the search direction points sufficiently downhill.
This is not always the case as saddle points and other non-quadratic features of the space can be problematic.
Two classes of algorithms can be used to handle this situation.
The first involves using the conjugate gradient method as a sub-algorithm for solving the Newton problem for the step $k$.
The Newton-CG\index{optimisation!algorithm!Newton-CG} line search algorithm described above is one such example.
The second class involves modifying the Hessian prior to, or at the same time as, finding the Newton step to guarantee that the replacement matrix $B_k$ is positive definite.
The convexity of $B_k$ is ensured by its eigenvalues all being positive.

The first modification uses the Cholesky factorisation of the matrix $B_k$, initialised to the true Hessian, to test for convexity (Algorithm 6.3 of \citet{NocedalWright99}).
If factorisation fails the matrix is not positive definite and a constant $\tau_k$ times the identity matrix $I$ is then added to $B_k$.
The constant originates from the Robbins norm of the Hessian $\lVert \nabla^2 f_k \rVert_F$ and is steadily increased until the factorisation is successful.
The resultant Cholesky lower triangular matrix $L$ can then be used to find the approximate Newton direction.
If $\tau_k$ is too large the convergence of this technique can approach that of the steepest descent\index{optimisation!algorithm!steepest descent} algorithm.

The second method is the Gill, Murray, and Wright (GMW) algorithm \citep{GMW81} which modifies the Hessian during the execution of the Cholesky factorisation $\nabla^2 f_k = LIL^T$, where $L$ is a lower triangular matrix and $D$ is a diagonal matrix.
Only a single factorisation is required.
As rows and columns are interchanged during the algorithm the technique may be slow for large problems such as the optimisation of the model-free parameters of all spins together with the diffusion tensor parameters.
The rate of convergence of the technique is quadratic.



% Other methods.
%~~~~~~~~~~~~~~~

\subsection{Other methods}

% Nelder-Mead simplex.
\subsubsection{Nelder-Mead simplex}
\label{sect: Nelder-Mead simplex}
\index{optimisation!algorithm!Nelder-Mead simplex|textbf}

Some optimisation algorithms cannot be classified within line search, trust region, or conjugate gradient categories.
For example the well known Nelder-Mead simplex optimisation algorithm.
The technique is often used as the only the function value is employed and hence the derivation of the gradient and Hessian can be avoided.
The simplex is created as an $n$-dimensional geometric object with $n+1$ vertices.
The first vertex is the starting position.
Each of the other $n$ vertices are created by shifting the starting position by a small amount parallel to one of unit vectors defining the coordinate system of the space.
Four simple rules are used to move the simplex through the space: reflection, extension, contraction, and a shrinkage of the entire simplex.
The result of these movements is that the simplex moves in an ameoboid-like fashion downhill, shrinking to pass through tight gaps and expanding to quickly move through non-convoluted space, eventually finding the minimum.

Key to these four movements is the pivot point, the centre of the face created by the $n$ vertices with the lowest function values.
The first movement is a reflection -- the vertex with the greatest function value is reflected through the pivot point on the opposite face of the simplex.
If the function value at this new position is less than all others the simplex is allowed to extend -- the point is moved along the line to twice the distance between the current position and the pivot point.
Otherwise if the function value is greater than the second highest value but less than the highest value, the reflected simplex is contracted.
The reflected point is moved to be closer to the simplex, its position being half way between the reflected position and the pivot point.
Otherwise if the function value at the reflected point is greater than all other vertices, then the original simplex is contracted -- the highest vertex is moved to a position half way between the current position and the pivot point.
Finally if none of these four movements yield an improvement, then the simplex is shrunk halfway towards the vertex with the lowest function value.


% Levenberg-Marquardt algorithm.
\subsubsection{Levenberg-Marquardt algorithm}
\index{optimisation!algorithm!Levenberg-Marquardt|textbf}

Another algorithm is the commonly used Levenberg-Marquardt algorithm \citep{Levenberg44,Marquardt63}.
This is the algorithm used by the model-free analysis programs Modelfree4\index{software!Modelfree}, Dasha\index{software!Dasha}, and Tensor2\index{software!Tensor}.
This technique is designed for least-squares problems to which the chi-squared\index{chi-squared} equation \eqref{eq: chi2} belongs.
The key to the algorithm is the replacement of the Hessian with the Levenberg-Marquardt matrix $J^T J + \lambda I$, where $J$ is the Jacobian of the system calculated as the matrix of partial derivatives of the residuals, $\lambda > 0$ is a factor related to the trust-region radius, and $I$ is the identity matrix.
The algorithm is conceptually allied to the trust region methods and its performance varies finely between that of the steepest descent and the pure Newton step.
When far from the minimum $\lambda$ is large and the algorithm takes steps close to the gradient; when in vicinity of the minimum $\lambda$ heads towards zero and the steps taken approximate the Newton direction.
Hence the algorithm avoids the problems of the Newton\index{optimisation!algorithm!Newton} algorithm when non-convex curvature is encountered and approximates the Newton step in convex regions of the space.

The technique does have one weak point though which is often mentioned only in the small print.
That is that the algorithm catastrophically fails when the Levenberg-Marquardt matrix is singular.
This occurs whenever a parameter is undefined.
For example in a model-free analysis if the order parameter is one, then the corresponding internal correlation time can take any value.
Performing a grid search with such undefined points greatly amplifies the problem and is the reason why many published model-free papers contain results with order parameters fixed at one \citep{dAuvergneGooley08a}.



% Constraint algorithms.
%%%%%%%%%%%%%%%%%%%%%%%%

\section{Constraint algorithms}
\label{sect: constraint algorithms}

To guarantee that the minimum will still be reached the implementation of constraints limiting the parameter values together with optimisation algorithms is not a triviality.
For this to occur the space and its boundaries must remain smooth thereby allowing the algorithm to move along the boundary to either find the minimum along the limit or to slide along the limit and then move back into the centre of the constrained space once the curvature allows it.


% Method of Multipliers algorithm.
%~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

\subsection{Method of Multipliers algorithm}
\index{optimisation!constraint algorithm!Method of Multipliers|textbf}

One of the most powerful approaches is the Method of Multipliers \citep{NocedalWright99}, also known as the Augmented Lagrangian.
Instead of a single optimisation the algorithm is iterative with each iteration consisting of an independent unconstrained minimisation on a sequentially modified space.
When inside the limits the function value is unchanged but when outside a penalty, which is proportional to the distance outside the limit, is added to the function value.
This penalty, which is based on the Lagrange multipliers, is smooth and hence the gradient and Hessian are continuous at and beyond the constraints.
For each iteration of the Method of Multipliers the penalty is increased until it becomes impossible for the parameter vector to be in violation of the limits.
This approach allows the parameter vector $\theta$ outside the limits yet the successive iterations ensure that the final results will not be in violation of the constraint.

For inequality constraints, each iteration of the Method of Multipliers attempts to solve the quadratic sub-problem
\begin{equation} \label{eq: Augmented Lagrangian}
    \min_\theta \mathfrak{L}_A(\theta, \lambda^k; \mu_k) \stackrel{\mathrm{def}}{=} f(\theta) + \sum_{i \in \mathfrak{I}} \Psi(c_i(\theta), \lambda_i^k; \mu_k),
\end{equation}

\noindent where the function $\Psi$ is defined as
\begin{equation}
    \Psi(c_i(\theta), \lambda^k; \mu_k) = \begin{cases}
        -\lambda^k c_i(\theta) + \frac{1}{2\mu_k} c_i^2(\theta) & \textrm{if } c_i(\theta) - \mu_k \lambda^k \leqslant 0, \\
        -\frac{\mu_k}{2} (\lambda^k)^2 & \textrm{otherwise}.
    \end{cases}
\end{equation}

\noindent  In \eqref{eq: Augmented Lagrangian}, $\theta$ is the parameter vector; $\mathfrak{L}_A$ is the Augmented Lagrangian function; $k$ is the current iteration of the Method of Multipliers; $\lambda^k$ are the Lagrange multipliers which are positive factors such that, at the minimum $\hat\theta$, $\nabla f(\hat\theta) = \lambda_i \nabla c_i(\hat\theta)$; $\mu_k > 0$ is the penalty parameter which decreases to zero as $k \to \infty$; $\mathfrak{I}$ is the set of inequality constraints; and $c_i(\theta)$ is an individual constraint value.
The Lagrange multipliers are updated using the formula
\begin{equation}
    \lambda_i^{k+1} = \max(\lambda_i^k - c_i(\theta)/\mu_k, 0), \qquad \textrm{for all } i \in \mathfrak{I}.
\end{equation}

The gradient of the Augmented Lagrangian is
\begin{equation}
    \nabla \mathfrak{L}_A(\theta, \lambda^k; \mu_k) = 
        \nabla f(\theta)
        - \sum_{i \in \mathfrak{I} | c_i(\theta) \leqslant \mu_k \lambda_i^k}
            \left( \lambda_i^k - \frac{c_i(\theta)}{\mu_k} \right) \nabla c_i(\theta),
\end{equation}

\noindent and the Hessian is
\begin{equation}
    \nabla^2 \mathfrak{L}_A(\theta, \lambda^k; \mu_k) = 
        \nabla^2 f(\theta)
        + \sum_{i \in \mathfrak{I} | c_i(\theta) \leqslant \mu_k \lambda_i^k}
            \left[
                \frac{1}{\mu_k} \nabla c_i^2(\theta)
                - \left( \lambda_i^k - \frac{c_i(\theta)}{\mu_k} \right) \nabla^2 c_i(\theta)
            \right].
\end{equation}

The Augmented Lagrangian algorithm can accept any set of three arbitrary constraint functions $c(\theta)$, $\nabla c(\theta)$, and $\nabla^2 c(\theta)$.
When given the current parameter values $c(\theta)$ returns a vector of constraint values whereby each position corresponds to one of the model parameters.
The constraint is defined as $c_i \geqslant 0$.
The function $\nabla c(\theta)$ returns the matrix of constraint gradients and $\nabla^2 c(\theta)$ is the constraint Hessian function which should return the 3D matrix of constraint Hessians.

A more specific set of constraints accepted by the Method of Multipliers are bound constraints.
These are defined by the function
\begin{equation}
    l \leqslant \theta \leqslant u,
\end{equation}

\noindent where $l$ and $u$ are the vectors of lower and upper bounds respectively and $\theta$ is the parameter vector.
For example for model-free model $m4$ to place lower and upper bounds on the order parameter and lower bounds on the correlation time and chemical exchange parameters, the vectors are
\begin{equation}
    \begin{pmatrix}
        0 \\
        0 \\
        0 \\
    \end{pmatrix}
    \leqslant
    \begin{pmatrix}
        S^2 \\
        \tau_e \\
        R_{ex} \\
    \end{pmatrix}
    \leqslant
    \begin{pmatrix}
        1 \\
        \infty \\
        \infty \\
    \end{pmatrix}.
\end{equation}

The default setting in the program relax\index{software!relax} is to use linear constraints which are defined as
\begin{equation} \label{eq: linear constraint}
    A \cdot \theta \geqslant b,
\end{equation}

\noindent where $A$ is an $m \times n$ matrix where the rows are the transposed vectors $a_i$ of length $n$; the elements of $a_i$ are the coefficients of the model parameters; $\theta$ is the vector of model parameters of dimension $n$; $b$ is the vector of scalars of dimension $m$; $m$ is the number of constraints; and $n$ is the number of model parameters.

In rearranging \eqref{eq: linear constraint} the linear constraint function $c(\theta)$ returns the vector $A \cdot \theta - b$.
Because of the linearity of the constraints the gradient and Hessian are greatly simplified.
The gradient $\nabla c(\theta)$ is simply the matrix $A$ and the Hessian $\nabla^2 c(\theta)$ is zero.



% Logarithmic barrier algorithm.
%~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

\subsection{Logarithmic barrier constraint algorithm}
\label{sect: Log-barrier constraint algorithm}
\index{optimisation!constraint algorithm!Logarithmic barrier|textbf}

Another constraint method is that of the logarithmic barrier algorithm.
As in the Method of Multipliers this method is iterative.
The function being minimised is replaced with
\begin{equation}
  \Phi(\theta) = \begin{cases}
    \epsilon \sum_{i=1}^m -\log(b_i - A_i^T\theta) & \textrm{if } A \cdot \theta < b, \\
    +\infty & \textrm{otherwise}.
    \end{cases}
\end{equation}

The value of $\epsilon$ is increased with each iteration, increase the logarithmic penalty.
An advantage of this method over the Method of Multipliers is that gradients are not required.



% Diagonal scaling.
%%%%%%%%%%%%%%%%%%%

\section{Diagonal scaling}
\label{sect: diagonal scaling}

Model scaling can have a significant effect on the optimisation algorithm -- a poorly scaled model can cause certain techniques to fail.
When two parameters of the model lie on very different numeric scales the model is said to be poorly scaled.
For example in model-free analysis the order of magnitude of the order parameters is one whereas for the internal correlation times the order of magnitude is between $1e^{-12}$ to $1e^{-8}$.
Most effected are the trust region algorithms -- the multidimensional sphere of trust will either be completely ineffective against the correlation time parameters or severely restrict optimisation in the order parameter dimensions.
Again in model-free analyses the significant scaling disparity can even cause failure of optimisation due to amplified effects of machine precision.
Therefore the model parameters need to be scaled.

This can be done by supplying the optimisation algorithm with the scaled rather than unscaled parameters.
When the chi-squared\index{chi-squared} function, gradient\index{chi-squared gradient}, and Hessian\index{chi-squared Hessian} are called the vector is then premultiplied with a diagonal matrix in which the diagonal elements are the scaling factors.
