#
# = Nonlinear Least-Squares Fitting
# This chapter describes functions for multidimensional nonlinear least-squares
# fitting. The library provides low level components for a variety of iterative
# solvers and convergence tests. These can be combined by the user to achieve
# the desired solution, with full access to the intermediate steps of the
# iteration. Each class of methods uses the same framework, so that you can
# switch between solvers at runtime without needing to recompile your program.
# Each instance of a solver keeps track of its own state, allowing the solvers
# to be used in multi-threaded programs.
#
# Contents:
# 1. {Overview}[link:rdoc/nonlinearfit_rdoc.html#label-Overview]
# 1. {Initializing the Solver}[link:rdoc/nonlinearfit_rdoc.html#label-Initializing+the+Solver]
#    1. {GSL::MultiFit::FdfSolver class}[link:rdoc/nonlinearfit_rdoc.html#label-FdfSolver+class]
# 1. {Providing the function to be minimized}[link:rdoc/nonlinearfit_rdoc.html#label-Providing+the+function+to+be+minimized]
#    1. {GSL::MultiFit::Function_fdf class}[link:rdoc/nonlinearfit_rdoc.html#label-Function_fdf+class]
# 1. {Iteration}[link:rdoc/nonlinearfit_rdoc.html#label-Iteration]
# 1. {Search Stopping Parameters}[link:rdoc/nonlinearfit_rdoc.html#label-Search+Stopping+Parameters]
# 1. {Computing the covariance matrix of best fit parameters}[link:rdoc/nonlinearfit_rdoc.html#label-Computing+the+covariance+matrix+of+best+fit+parameters]
# 1. {Higher level interfaces}[link:rdoc/nonlinearfit_rdoc.html#label-Higher+level+interfaces]
# 1. {Examples}[link:rdoc/nonlinearfit_rdoc.html#label-Examples]
#    1. {Fitting to user-defined functions}[link:rdoc/nonlinearfit_rdoc.html#label-Fitting+to+user-defined+functions]
#    1. {Fitting to built-in functions}[link:rdoc/nonlinearfit_rdoc.html#label-Fitting+to+built-in+functions]
#
# == Overview
# The problem of multidimensional nonlinear least-squares fitting requires the
# minimization of the squared residuals of n functions, f_i, in p parameters,
# x_i, All algorithms proceed from an initial guess using the linearization,
# where x is the initial point, p is the proposed step and J is the Jacobian
# matrix J_{ij} = d f_i / d x_j. Additional strategies are used to enlarge the
# region of convergence. These include requiring a decrease in the norm ||F||
# on each step or using a trust region to avoid steps which fall outside the
# linear regime.
#
# To perform a weighted least-squares fit of a nonlinear model Y(x,t) to data
# (t_i, y_i) with independent gaussian errors \sigma_i, use function components
# of the following form, Note that the model parameters are denoted by x in this
# chapter since the non-linear least-squares algorithms are described
# geometrically (i.e. finding the minimum of a surface). The independent
# variable of any data to be fitted is denoted by t.
#
# With the definition above the Jacobian is
# J_{ij} =(1 / \sigma_i) d Y_i / d x_j, where Y_i = Y(x,t_i).
#
# == Initializing the Solver
#
# === FdfSolver class
# ---
# * GSL::MultiFit::FdfSolver.alloc(T, n, p)
#
#   This creates an instance of the <tt>GSL::MultiFit::FdfSolver</tt> class of
#   type <tt>T</tt> for <tt>n</tt> observations and <tt>p</tt> parameters. The type <tt>T</tt>
#   is given by a <tt>Fixnum</tt> constant or a <tt>String</tt>,
#   * <tt>GSL::MultiFit::LMSDER</tt> or <tt>"lmsder"</tt>
#   * <tt>GSL::MultiFit::LMDER</tt> or <tt>"lmder"</tt>
#   For example, the following code creates an instance of a Levenberg-Marquardt
#   solver for 100 data points and 3 parameters,
#
#       solver = MultiFit::FdfSolver.alloc(MultiFit::LMDER, 100, 3)
#
# ---
# * GSL::MultiFit::FdfSolver#set(f, x)
#
#   This method initializes, or reinitializes, an existing solver <tt>self</tt>
#   to use the function <tt>f</tt> and the initial guess <tt>x</tt>. The function <tt>f</tt>
#   is an instance of the <tt>GSL::MultiFit::Function_fdf</tt> class (see below). The
#   initial guess of the parameters <tt>x</tt> is given by a {GSL::Vector}[link:rdoc/vector_rdoc.html] object.
#
# ---
# * GSL::MultiFit::FdfSolver#name
#
#   This returns the name of the solver <tt>self</tt> as a String.
#
#
# ---
# * GSL::MultiFit::FdfSolver#x
# * GSL::MultiFit::FdfSolver#dx
# * GSL::MultiFit::FdfSolver#f
# * GSL::MultiFit::FdfSolver#J
# * GSL::MultiFit::FdfSolver#jacobian
# * GSL::MultiFit::FdfSolver#jac
#
#   Access to the members (see <tt>gsl_multifit_nlin.h</tt>)
#
# == Providing the function to be minimized
# === Function_fdf class
# ---
# * GSL::MultiFit::Function_fdf.alloc()
# * GSL::MultiFit::Function_fdf.alloc(f, df, p)
# * GSL::MultiFit::Function_fdf.alloc(f, df, fdf, p)
#
#   Constructor for the <tt>Function_fdf</tt> class, to a
#   function with <tt>p</tt> parameters, The first two or three arguments are Ruby Proc objects
#   to evaluate the function to minimize and its derivative (Jacobian).
#
# ---
# * GSL::MultiFit::Function_fdf#set_procs(f, df, p)
# * GSL::MultiFit::Function_fdf#set_procs(f, df, fdf, p)
#
#   This initialize of reinitialize the function <tt>self</tt> with <tt>p</tt> parameters
#   by two or three Proc objects <tt>f, df</tt> and <tt>fdf</tt>.
#
# ---
# * GSL::MultiFit::Function_fdf#set_data(t, y)
# * GSL::MultiFit::Function_fdf#set_data(t, y, sigma)
#
#   This sets the data <tt>t, y, sigma</tt> of length <tt>n</tt>, to the function <tt>self</tt>.
#
# == Iteration
# ---
# * GSL::MultiFit::FdfSolver#iterate
#
#   THis performs a single iteration of the solver <tt>self</tt>. If the iteration
#   encounters an unexpected problem then an error code will be returned.
#   The solver maintains a current estimate of the best-fit parameters at all
#   times. This information can be accessed with the method <tt>position</tt>.
#
# ---
# * GSL::MultiFit::FdfSolver#position
#
#   This returns the current position (i.e. best-fit parameters) of the solver
#   <tt>self</tt>, as a <tt>GSL::Vector</tt> object.
#
#
# == Search Stopping Parameters
# A minimization procedure should stop when one of the following conditions is true:
# * A minimum has been found to within the user-specified precision.
# * A user-specified maximum number of iterations has been reached.
# * An error has occurred.
# The handling of these conditions is under user control. The method below allows
# the user to test the current estimate of the best-fit parameters.
#
# ---
# * GSL::MultiFit::FdfSolver#test_delta(epsabs, epsrel)
#
#   This method tests for the convergence of the sequence by comparing the last
#   step with the absolute error <tt>epsabs</tt> and relative error (<tt>epsrel</tt>
#   to the current position. The test returns <tt>GSL::SUCCESS</tt> if the following
#   condition is achieved,
#     |dx_i| < epsabs + epsrel |x_i|
#   for each component of <tt>x</tt> and returns <tt>GSL::CONTINUE</tt> otherwise.
#
# ---
# * GSL::MultiFit::FdfSolver#test_gradient(g, epsabs)
# * GSL::MultiFit::FdfSolver#test_gradient(epsabs)
#
#   This function tests the residual gradient <tt>g</tt> against the absolute error
#   bound <tt>epsabs</tt>. If <tt>g</tt> is not given, it is calculated internally.
#   Mathematically, the gradient should be exactly zero at the minimum.
#   The test returns <tt>GSL::SUCCESS</tt> if the following condition is achieved,
#     \sum_i |g_i| < epsabs
#   and returns <tt>GSL::CONTINUE</tt> otherwise. This criterion is suitable for
#   situations where the precise location of the minimum, x, is unimportant provided
#   a value can be found where the gradient is small enough.
#
# ---
# * GSL::MultiFit::FdfSolver#gradient
#
#   This method returns the gradient g of \Phi(x) = (1/2) ||F(x)||^2 from the
#   Jacobian matrix and the function values, using the formula g = J^T f.
#
# ---
# * GSL::MultiFit.test_delta(dx, x, epsabs, epsrel)
# * GSL::MultiFit.test_gradient(g, epsabs)
# * GSL::MultiFit.gradient(jac, f, g)
# * GSL::MultiFit.covar(jac, epsrel)
# * GSL::MultiFit.covar(jac, epsrel, covar)
#
#   Singleton methods of the <tt>GSL::MultiFit</tt> module.
#
#
# == Computing the covariance matrix of best fit parameters
# ---
# * GSL::MultiFit.covar(J, epsrel)
# * GSL::MultiFit.covar(J, epsrel, covar)
#
#   This method uses the Jacobian matrix <tt>J</tt> to compute the covariance
#   matrix of the best-fit parameters. If an existing matrix <tt>covar</tt> is given,
#   it is overwritten, and if not, this method returns a new matrix.
#   The parameter <tt>epsrel</tt> is used to remove linear-dependent columns when
#   <tt>J</tt> is rank deficient.
#
#   The covariance matrix is given by,
#      covar = (J^T J)^{-1}
#   and is computed by QR decomposition of <tt>J</tt> with column-pivoting.
#   Any columns of R which satisfy
#     |R_{kk}| <= epsrel |R_{11}|
#   are considered linearly-dependent and are excluded from the covariance matrix
#   (the corresponding rows and columns of the covariance matrix are set to zero).
#
# == Higher level interfaces
# ---
# * GSL::MultiFit::FdfSolver.fit(x, y, type[, guess])
# * GSL::MultiFit::FdfSolver.fit(x, w, y, type[, guess])
#
#   This method uses <tt>FdfSolver</tt> with the LMSDER algorithm to fit the data
#   <tt>[x, y]</tt> to a function of type <tt>type</tt>. The returned value is
#   an array of 4 elements, <tt>[coef, err, chisq, dof]</tt>,
#   where <tt>coef</tt> is an array of the fitting coefficients, <tt>err</tt> contains
#   errors in estimating <tt>coef</tt>, <tt>chisq</tt> is the
#   chi-squared, and <tt>dof</tt> is the degree-of-freedom in the fitting
#   which equals to (data length - number of fitting coefficients). The optional
#   argument <tt>guess</tt> is an array of initial guess of the coefficients.
#   The fitting type <tt>type</tt> is given by a <tt>String</tt> as follows.
#   * <tt>"gaussian"</tt>: Gaussian fit,
#     <tt>y = y0 + A exp(-(x-x0)^2/2/var)</tt>, <tt>coef = [y0, A, x0, var]</tt>
#   * <tt>"gaussian_2peaks"</tt>: 2-peak Gaussian fit,
#     <tt>y = y0 + A1 exp(-(x-x1)^2/2/var1) + A2 exp(-(x-x2)^2/2/var2)</tt>, <tt>coef = [y0, A1, x1, var1, A2, x2, var2]</tt>
#   * <tt>"exp"</tt>: Exponential fit,
#     <tt>y = y0 + A exp(-b x)</tt>, <tt>coef = [y0, A, b]</tt>
#   * <tt>"dblexp"</tt>: Double exponential fit,
#     <tt>y = y0 + A1 exp(-b1 x) + A2 exp(-b2 x)</tt>, <tt>coef = [y0, A1, b1, A2, b2]</tt>
#   * <tt>"sin"</tt>: Sinusoidal fit,
#     <tt>y = y0 + A sin(f x + phi)</tt>, <tt>coef = [y0, A, f, phi]</tt>
#   * <tt>"lor"</tt>: Lorentzian peak fit,
#     <tt>y = y0 + A/((x-x0)^2 + B)</tt>, <tt>coef = [y0, A, x0, B]</tt>
#   * <tt>"hill"</tt>: Hill's equation fit,
#     <tt>y = y0 + (m - y0)/(1 + (xhalf/x)^r)</tt>, <tt>coef = [y0, n, xhalf, r]</tt>
#   * <tt>"sigmoid"</tt>: Sigmoid (Fermi-Dirac) function fit,
#     <tt>y = y0 + m/(1 + exp((x0-x)/r))</tt>, <tt>coef = [y0, m, x0, r]</tt>
#   * <tt>"power"</tt>: Power-law fit,
#     <tt>y = y0 + A x^r</tt>, <tt>coef = [y0, A, r]</tt>
#   * <tt>"lognormal"</tt>: Lognormal peak fit,
#     <tt>y = y0 + A exp[ -(log(x/x0)/width)^2 ]</tt>, <tt>coef = [y0, A, x0, width]</tt>
#
#   See {Linear fitting}[link:rdoc/fit_rdoc.html#label-Higer+level+interface] for linear and polynomical fittings.
#
# == Examples
# === Fitting to user-defined functions
#
# The following example program fits a weighted exponential model with background
# to experimental data, Y = A exp(-lambda t) + b. The first part of the program sets
# up the functions <tt>procf</tt> and <tt>procdf</tt> to calculate the model and its Jacobian.
# The appropriate fitting function is given by,
#   f_i = ((A exp(-lambda t_i) + b) - y_i)/sigma_i
# where we have chosen t_i = i. The Jacobian matrix <tt>jac</tt> is the derivative of
# these functions with respect to the three parameters (A, lambda, b). It is given by,
#   J_{ij} = d f_i / d x_j
# where x_0 = A, x_1 = lambda and x_2 = b.
#
#
#   require("gsl")
#   include GSL::MultiFit
#
#   # x: Vector, list of the parameters to determine
#   # t, y, sigma: Vectors, observational data
#   # f: Vector, function to minimize
#   procf = Proc.new { |x, t, y, sigma, f|
#     a = x[0]
#     lambda = x[1]
#     b = x[2]
#     n = t.size
#     for i in 0...n do
#       yi = a*Math::exp(-lambda*t[i]) + b
#       f[i] = (yi - y[i])/sigma[i]
#     end
#   }
#
#   # jac: Matrix, Jacobian
#   procdf = Proc.new { |x, t, y, sigma, jac|
#     a = x[0]
#     lambda = x[1]
#     n = t.size
#     for i in 0...n do
#       ti = t[i]
#       si = sigma[i]
#       ei = Math::exp(-lambda*ti)
#       jac.set(i, 0, ei/si)
#       jac.set(i, 1, -ti*a*ei/si)
#       jac.set(i, 2, 1.0/si)
#     end
#   }
#
#   f = GSL::MultiFit::Function_fdf.alloc(procf, procdf, 2)
#
#   # Create data
#   r = GSL::Rng.alloc()
#   t = GSL::Vector.alloc(n)
#   y = GSL::Vector.alloc(n)
#   sigma = Vector.alloc(n)
#   for i in 0...n do
#     t[i] = i
#     y[i] = 1.0 + 5*Math::exp(-0.1*t[i]) + r.gaussian(0.1)
#     sigma[i] = 0.1
#   end
#
#   f.set_data(t, y, sigma)
#   x = GSL::Vector.alloc(1.0, 0.0, 0.0)    # initial guess
#
#   solver = GSL::FdfSolver.alloc(FdfSolver::LMSDER, n, np)
#
#   solver.set(f, x)
#
#   iter = 0
#   solver.print_state(iter)
#   begin
#     iter += 1
#     status = solver.iterate
#     solver.print_state(iter)
#     status = solver.test_delta(1e-4, 1e-4)
#   end while status == GSL::CONTINUE and iter < 500
#
#   covar = solver.covar(0.0)
#   position = solver.position
#   chi2 = pow_2(solver.f.dnrm2)
#   dof = n - np
#   printf("A      = %.5f +/- %.5f\n", position[0], Math::sqrt(chi2/dof*covar[0][0]))
#   printf("lambda = %.5f +/- %.5f\n", position[1], Math::sqrt(chi2/dof*covar[1][1]))
#   printf("b      = %.5f +/- %.5f\n", position[2], Math::sqrt(chi2/dof*covar[2][2]))
#
#
# === Fitting to built-in functions
#   #!/usr/bin/env ruby
#   require("gsl")
#   include MultiFit
#
#   N = 100
#
#   y0 = 1.0
#   A = 2.0
#   x0 = 3.0
#   w = 0.5
#
#   r = Rng.alloc
#   x = Vector.linspace(0.01, 10, N)
#   sig = 1
#   # Lognormal function with noise
#   y =  y0 + A*Sf::exp(-pow_2(Sf::log(x/x0)/w)) + 0.1*Ran::gaussian(r, sig, N)
#
#   guess = [0, 3, 2, 1]
#   coef, err, chi2, dof = MultiFit::FdfSolver.fit(x, y, "lognormal", guess)
#   y0 = coef[0]
#   amp = coef[1]
#   x0 = coef[2]
#   w = coef[3]
#
#   graph(x, y, y0+amp*Sf::exp(-pow_2(Sf::log(x/x0)/w)))
#
#
# {prev}[link:rdoc/fit_rdoc.html]
# {next}[link:rdoc/bspline_rdoc.html]
#
# {Reference index}[link:rdoc/ref_rdoc.html]
# {top}[link:index.html]
#
#
#
