\documentclass[11pt]{article}

\usepackage{graphicx}
% \usepackage{pdfpages}
\usepackage{fancyhdr}
\usepackage{listings}
\usepackage{xcolor}

% GILDAS specific definitions
\include{gildas-def}

\makeindex{}

\makeatletter
\textheight 625pt
\textwidth 460pt
\oddsidemargin 0pt
\evensidemargin 0pt
\marginparwidth 50pt
\topmargin 0pt
\brokenpenalty=10000

% Makes fancy headings and footers.
\pagestyle{fancy}

\fancyhf{} % Clears all fields.

\lhead{\scshape Bardeau, de Souza, Pety, 2020}
\rhead{\scshape \nouppercase{\rightmark}}   %
\fancyfoot[C]{\bfseries \thepage{}}

\def\uvfit{UV\textbackslash FIT}

\renewcommand{\headrulewidth}{0pt}    %
\renewcommand{\footrulewidth}{0pt}    %

\renewcommand{\sectionmark}[1]{%
  \markright{\thesection. \MakeLowercase{#1}}}
\renewcommand{\subsectionmark}[1]{}

\lstset{language=[90]Fortran,
  basicstyle=\ttfamily,
  keywordstyle=\color{blue},
  commentstyle=\color{gray},
  frame=single,
  tabsize=2,
  morecomment=[l]{!\ }% Comment only with 
}

\fancypagestyle{firststyle}
{
   \fancyhf{}
   \fancyhead[C]{Original version at \tt http://iram-institute.org/medias/uploads/mapping-for-developers.pdf}
   \fancyfoot[C]{\bfseries \thepage{}}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\title{CUBE developers documentation}
\author{S.~Bardeau$^{1}$, V.~de~Souza~Magalhaes$^{1}$, J.~Pety$^{1,2}$\\
 \vspace{0.3cm}
  1. IRAM\\
  2. Observatoire de Paris
 }
\makeindex{}

\begin{document}

\maketitle

%\begin{rawhtml}
%  Note: this is the on-line version of the ``GREG 2012 UV table format''
%  document for Gildas users and programmers.
%  A <A HREF="../../pdf/greg-uvt-v2.pdf"> PDF version</A>
%  is also available.
%
%  Related information is available in:
%  <UL>
%  <LI> <A HREF="../greg-html/greg.html"> GREG:</A> Grenoble Graphics
%  <LI> <A HREF="../sic-html/sic.html"> SIC:</A> Simple Interpretor of Commands
%  </UL>
%\end{rawhtml}

\begin{abstract}
  Some abstract.
\end{abstract}

\newpage
\tableofcontents{}

\newpage

\section{User workflow}

User workflow is not yet finalized in \cube. This depends on the way
we deal with cubes in memory.

\section{IO}

\subsection{Nomenclature}

In the IO library, cubes are considered either as a collection of
\emph{channels} (a channel being a 2D subset along the V dimension),
or as a collection of \emph{pixels} (a pixel being a 1D subset along
the XY dimensions). Other actors might use different names for a
similar object: channels can be seen as 2D position-position images,
and pixels as 1D spectra (collection of scalar channels for a given XY
position).

\subsection{Data ordering and contiguous access}

Position-position-spectroscopy cubes can be stored in memory or disk
in several different ways. Only two major orders are offered as they
cover all the use-cases (see Fig~\ref{fig:cube-io-ordering}):
\begin{enumerate}
\item LMV: the spatial dimensions come first, channels or range of
  channels are contiguous subset of the cube;
\item VLM: the spectroscopic dimension comes first, pixels or ranges
  of pixels are contiguous subset of the cube.
\end{enumerate}
Each order is the transposition of the other one.

\begin{figure}[htb]
  \centering
  \includegraphics[width=0.45\textwidth]{cube-lmv}
  \includegraphics[width=0.45\textwidth]{cube-vlm}
  \caption{Left: LMV-ordered cube. Right: VLM-ordered cube. Both cubes
    are represented as a juxtaposition of LM (XY) images (one image
    per channel). In both figures, orientation is chosen so that the
    first (resp. last) element in memory or disk is displayed at the
    bottom-left-front (resp. top-right-back) corner. Contiguity goes
    from left to right, then bottom to top, then front to back. Red
    lines illustrate the contiguity path. The grey planes are
    contiguous 2D subsets.}
  \label{fig:cube-io-ordering}
\end{figure}

In order to support both memory and disk modes (see
subsection~\ref{txt:access-modes}), the IO library only offers access
to the data through subset objects, namely channels or pixels. For
best efficiency, accesses should be done\footnote{The IO library
  offers the possibility to read a non-contiguous object, e.g. a
  channel from a VLM cube, but this is possible only in memory mode,
  and this is less efficient than the contiguous access} with the
object which is contiguous for the given cube (i.e. channels for LMV
cubes, pixels for VLM cubes).

\subsection{Memory and disk modes}
\label{txt:access-modes}

\begin{figure}[htb]
  \centering
  \includegraphics[width=0.80\textwidth]{cube-io-blocks}
  \caption{How to IO library defines the current block to be
    processed. Left: in the memory mode, the whole cube is loaded in
    memory and is processed as a single block. Right: in the disk
    mode, a contiguous subset of the cube is loaded in memory; there
    will be several blocks to be processed to cover the full
    cube. Note that all the blocks may not have strictly the same size
    due to rounding issues in the general case.}
  \label{fig:cube-io-blocks}
\end{figure}

The IO library is thought to offer IO accesses to cubes of arbitrary
large sizes\footnote{More precisely: the whole 3D cube might not fit
  in memory, as long as at least a 2D subset along the third dimension
  does.}. Internally, the IO library can:
\begin{itemize}
\item store the whole cube in memory at once (\emph{memory
    mode}). This is suited for cubes smaller than a user-defined
  limit,
\item store contiguous subsets of the cube (\emph{disk
    mode}). \emph{Contiguous} being, as described in
  Fig~\ref{fig:cube-io-ordering}), a contiguous range of channels (LMV
  cubes) or a contiguous range of Y planes (VLM cubes). The size of
  the resulting block is ruled by the command {\tt
    SET$\backslash$BUFFERING /BLOCK}.
\end{itemize}
The figure~\ref{fig:cube-io-blocks} illustrates how the same cube can be
divided in blocks depending on the mode.

\clearpage

\subsection{Serial and parallel executions}
\label{txt:execution-modes}

\begin{figure}[htb]
  \centering
  \includegraphics[width=0.80\textwidth]{cube-io-tasks}
  \caption{How to IO library defines the tasks to be executed. Left:
    in case of serial execution, the current block is treated entirely
    by a single task (a single processor thread). Right: in case of
    parallel execution, the current block is divided in contiguous
    subsets, each of them given for processing to a different parallel
    task.}
  \label{fig:cube-io-tasks}
\end{figure}

CUBE supports parallelization for both memory and disk modes. This is
achieved thanks to the blocks (as explained in
section~\ref{txt:access-modes}), as the IO library ensures a block is
entirely loaded in memory for processing. Once this block is
available, parallelization is performed with \emph{OpenMP tasks} (see
section~\ref{txt:parallelization}). Once the current block has been
processed, it is flushed and a new one (if any) is loaded in memory
and processed again by several tasks.

Note that the serial case is treated as a particular case of the
parallelization: a single \emph{pseudo-task} is defined to process the
current block by a single thread (Fig.~\ref{fig:cube-io-tasks}, left).

\subsection{Access to data by entries}
\label{txt:entries}

\begin{figure}[htb]
  \centering
  \includegraphics[width=0.80\textwidth]{cube-io-entries}
  \caption{The 3 data granularity offered to the external API.}
  \label{fig:cube-io-entries}
\end{figure}

The IO external API does not expose any difference between the disk
and memory modes, and between the serial and parallel execution. The
programmer just has to write the code which processes the data
associated to a task. This data can accessed with 3 granularity
levels:
\begin{itemize}
\item \emph{Image} access: the data is accessed (get-modify-put
  sequence) through a 2D contiguous object (channel/image); this
  access is suited for image-related processing like reprojection of
  LMV cubes.
\item \emph{Spectrum} access: the data is accessed through a 1D
  contiguous object (pixel/spectrum); this access is suited for
  spectrum-related processing like frequency resampling of VLM cubes.
\item \emph{Subcube} access: the data is accessed through a 3D
  contiguous object; this access is suited for processing where data
  order is not relevant (\eg\ data rescaling when converting its
  unit). The larger granularity minimizes the overheads of the 2
  previous accesses.
\end{itemize}
Given the granularity chosen by the programmer, a task can be
considered as collection of entries to be processed.

\clearpage

\section{Parallelization}
\label{txt:parallelization}

\subsection{Setting up the task iterator}

The key part of the parallelization is to properly split the data for
parallel processing. In other words, which piece of data each task has
to process. This problem is ruled by the parameter {\tt
  SET$\backslash$BUFFERING /TASK}, controling the size of data (per
cube) to be processed by a single task, as shown in
Fig.~\ref{fig:cube-io-tasks}.

Note that the total task size (sum of all the input and output cube
slices process by the task) should be large enough to avoid domination
of task overheads (\ie\ processing each single plane by a single task
is not the best, one task should process several planes), but small
enough to ensure many tasks can be defined to benefit parallel
execution (\ie\ there should be at least as many tasks as available
threads).

\begin{figure}[htb]
  \centering
  \includegraphics[width=0.9\textwidth]{cube-iterator-axes}
  \caption{The various cases encountered by the iterator. The lines
    represent the 3rd (iterated) axis of cubes processed by the
    command. For a given input axis (blue), it shows how the other
    (input or output) axes can be aligned or not.}
  \label{fig:cube-iterator-axes}
\end{figure}

The Fig.~\ref{fig:cube-iterator-axes} shows the various cases
encountered by the iterator when it tries to iterate the 3rd axis. The
3rd axis is either the spectral channel axis for a LVM cube, or the Y
spatial axis for a VLM cube. Note that the 2 leading dimensions
(\ie\ one plane) can be identical from one cube to another or not, the
only effect is that a slice with the same 3rd dimension width may
result in different quantity of data.
\begin{enumerate}
  \item In the simplest case, the command performs a 1-to-1
    transformation: one cube as input, one cube as output, same number
    of planes, all aligned one by one.
  \item A bit more complicated case is the region case (e.g. {\tt
    /RANGE 0 100 km/s}). The number of planes differ, but the output
    3rd axis is a subset of the input one.
  \item Another case is the {\tt COMPRESS [/RANGE]} case: not the same
    number of planes, not the same increment, but still aligned with a
    N-to-1 relationship.
  \item And finally the generic case like {\tt RESAMPLE}: the axes
    overlap, with no particular alignment.
\end{enumerate}
In order to deal with all these cases, before starting to work on the
data, the iterator looks at all the cube 3rd axes (input and output
cubes). From these it selects arbitrarily a \emph{reference} cube
(usually the first output cube). This reference cube defines the
\emph{reference axis} for this command. The entry numbering
(e.g. image number 1) refers to this axis (and only this axis).\\

Then the iterator chooses the best number of \emph{reference planes}
per task according to {\tt SET$\backslash$BUFFERING /TASK}. The plane
size of each cube is known (it might differ from one cube to another):
the maximum number of planes to be processed per task and per cube is
thus easily deduced. All these values can be translated each as a
number \emph{reference planes} to be processed. The smallest value is
chosen to avoid overflowing the limit\footnote{Obviously with at least
one reference plane has to be processed by one task}.\\

At the end of all this process, we have retrieved or computed:
\begin{enumerate}
  \item the iterated axis of each cube.
  \item a reference axis,
  \item the number of reference planes to be processed by each task,
\end{enumerate}

\subsection{Setting up the tasks}

Each task is designed to process the desired number of reference
planes. When a task processes the entries, each entry-get and each
entry-put is associated with an entry number. This entry number is in
the reference axis domain. Knowing the reference axis and the actual
axis, the iterator engine is able to do the proper conversion to
select the actual entry to be read or written. Going back to the 4
cases described in Fig.~\ref{fig:cube-iterator-axes}, this translates
as:
\begin{enumerate}
  \item Identical axes: reference entry number and actual entry number
    are identical,
  \item Subset axis: the actual entry number if shifted by an integer
    constant (the number of leading planes dropped by {\tt /RANGE}),
  \item Compressed axis: since there is no plane-to-plane alignment,
    only subcube access is possible. The internal engine knows anyway
    which range of data is to be read or written. It is the
    responsibility of the command to \emph{compress} the input subcube
    into the output one,
  \item Generic case: for sanity reasons, the entry-get or entry-put
    process (whatever the entry kind) will detect and reject this
    case. The current design of the iterator is able to support this
    case, but it is proposed to offer a different API with less sanity
    check, should this use case be requested.
\end{enumerate}

\begin{figure}[p]
  \centering
  \includegraphics[width=0.55\textwidth]{cube-parallelization-1}
  \caption{How the \emph{adm} library in CUBE fills the \emph{OpenMP
      task pool}. In short, a master thread divides the current block
    in smaller parts, and feeds them to OpenMP for \emph{later}
    processing by the parallel threads. In this example, the block is
    divided in 5 tasks.}
  \label{fig:cube-parallelization-1}
\end{figure}

Note that, while the \emph{adm} library has the charge of dividing the
input and output blocks into tasks, reading the data, flushing it,
etc, it does not have the responsibility to \emph{execute} the tasks:
it is the OpenMP responsibility (see next section).

\clearpage

\subsection{Iterating the tasks}

The successful execution in parallel of all the tasks in the pool is
achieved by filling it correctly. This is done by this piece of code
which is executed by each command:

\begin{verbatim}
    call cubeadm_datainit_all(iter,prog%region,error)
    if (error) return
    !$OMP PARALLEL DEFAULT(none) SHARED(prog,error) FIRSTPRIVATE(iter)
    !$OMP SINGLE
    do while (cubeadm_dataiterate_all(iter,error))
       if (error) exit
       !$OMP TASK SHARED(prog,error) FIRSTPRIVATE(iter)
       if (.not.error) &
         call prog%loop(iter,error)
       !$OMP END TASK
    enddo
    !$OMP END SINGLE
    !$OMP END PARALLEL
\end{verbatim}

The subroutine {\tt cubeadm\_datainit\_all} prepares all the input and
output cubes before processing. It checks the requested accesses
(\eg\ {\tt imaset}) and their consistency among all cubces, prepares
the data transposition if needed (but do not execute it yet). Knowing
the cube details (size in all dimensions, iterated axes), it prepares
the task iterations in the {\tt iter} structure. It also initializes
the timers.\\

Once the previous elements are prepared (in particular the {\tt iter}
structure), a master thread ({\tt !\$OMP SINGLE}) is in charge on
filling the \emph{OpenMP Task Pool} with all the tasks needed to
process the current block. There can be (much) more tasks than
computing threads, this is part of the OpenMP tasks design.\\

In details, the {\tt do while} loop increments an internal counter of
the tasks. During the first iteration, the subroutine {\tt
  cubeadm\_dataiterate\_all} sets up the first task, described in a
sub-component of the {\tt iter} object. It also ensures that the
internal buffers (memory mode or disk mode) provide the data necessary
to process this task. If the data are not available (as for example for
the very first task), the data is read from disk at this moment (if
needed, transposition happens here). If the data are already available
(as for example for the second iteration in memory mode), the
subroutine {\tt cubeadm\_dataiterate\_all} sets up the second task in
the {\tt iter} object but this is almost instantaneous. When one
execution of {\tt cubeadm\_dataiterate\_all} is done, the code
internal to the {\tt do while} loop is executed.\\

This part of the code creates an OpenMP task ({\tt !\$OMP TASK}). Note
that this task shares the {\tt prog} structure (which provides access
to the whole data buffers) but has its own copy ({\tt FIRSTPRIVATE})
of the {\tt iter} structure (in particular the components describing
the piece of data to be processed by the task in the current
iteration). This task is put by OpenMP in the pool AND the {\tt do
  while} execution is resumed without waiting for the task to be
executed.\\

The next {\tt cubeadm\_dataiterate\_all} iteration is thus executed,
overwriting the {\tt iter} structure for the next task
execution. Another task is created with its own copy of the {\tt iter}
structure, and so on, filling the task pool with several tasks waiting
for their execution.\\

This {\tt do while} loop creating new tasks is repeated without
interruption until:
\begin{itemize}
\item all the remaining tasks are created (for example in memory mode:
  all the tasks are created in a row),
\item or the data needed by the next task are not available. In this
  case, according to what was explained for the very first task, the
  data buffers should be filled with the relevant data. But we know
  that the previous tasks need the buffer previous contents: these
  buffers should not be released as long as a task needs them. This
  behaviour is achieved thanks to an {\tt !\$OMP TASKWAIT} statement
  in the code in charge of the data buffering: it waits for all the
  previous tasks to be terminated before modifying the buffers. This
  is the core of the execution of the disk mode.
\end{itemize}

\begin{figure}[htb]
  \centering
  \includegraphics[width=0.7\textwidth]{cube-parallelization-2}
  \caption{How OpenMP distributes the tasks for execution by the
    threads. In this example, 3 threads are requested to execute 5
    tasks.}
  \label{fig:cube-parallelization-2}
\end{figure}

In parallel of the tasks creation described above, the OpenMP engine
has the responsibility to actually execute them (\ie\ the call to each
{\tt prog\%loop()}). As soon as there is one or more task waiting for
execution in the pool, the OpenMP engine starts to address them for
execution by parallel threads. The parallelization precisely comes at
this level of the code. \cube{} has no control on the tasks
execution. In particular, there is no predictable order for executing
them; which thread will execute which pending task is unknown. Threads
are working as long as there remains tasks to be executed in the
pool.\\

When the task pool is emptied, either all the data has been processed
and the {\tt do while} loop is over\footnote{The {\tt do while} loop
last iteration was actually terminated before its task is actually
executed. There is a remaining piece of code, not detailed here, which
flushes the data buffers at the end of the command. Again, this flush
can be done only if the {\tt !\$OMP TASKWAIT} allows for it.}, or the
{\tt !\$OMP TASKWAIT} is released and the next piece of data is
buffered and new tasks are created.\\

Note that in serial execution, the {\tt !\$OMP} statements are
ignored. There are no tasks created. The {\tt prog\%loop()} subroutine
is executed as it is encountered, at each iteration process. The
buffering scheme is the same, except there is no risk to unbuffer data
which is still needed since the loops are executed sequentially, in
order (no need for a \emph{wait} mechanism).

\subsection{Execution of one task}

\begin{figure}[htb]
  \centering
  \includegraphics[width=0.70\textwidth]{cube-parallelization-3}
  \caption{How each thread processes the data in the task. This relies
    on the entry kind, with internal loop in the task.}
  \label{fig:cube-parallelization-3}
\end{figure}

Within a given task executed by one thread, the processing runs
serially. For example, if the task provides N images, the task
internal loop will process the N images in order, one after the
other. Same rule applies for spectra within a task. Regarding
subcubes, they are meant to be treated as a whole entity within the
task.

\section{Conventions}

\subsection{Frequency vs Velocity}

In radio astronomy it is very common to use velocity in km/s as the
unit of spectral axes. However the sampling of the signal by a radio
telescope is done on frequency, and hence the native unit of the
spectral axes. All velocity axes are hence derived from an original
frequency axis based on a linear transformation that depends on a rest
frequency.

Due to this we have taken the decision to privilege frequency over
velocity when dealing with spectral axes in \cube{}. An example is the
derivation of the velocity axis in the header types. All the velocity
related quantities are computed from the frequency related ones and
not taken directly from the header of the file. Another example is the
spectral stitching of cubes, which is always done in frequency.

\subsection{Axis descriptions}

In GILDAS we use 4 values to describe a linear axis, be it a spatial
or a spectral axis. These values are:
\begin{itemize}
\item[$n$] The number of elements in the axis (long)
\item[$ref$] The reference pixel position (double)
\item[$val$] The value of the reference pixel in the axis unit (double)
\item[$inc$] The increment between two elements of the axis in the axis unit (double)
\end{itemize}

From these values the elements of an axis can be reconstructed
according to the following equation:
\begin{equation}
  ax[i] = (i-ref)*inc+val, \forall i \in \mathbf{N} < n
\end{equation}
Where $ax[i]$ is the $i$-eth element of the axis.

In the case of spectral axes, we use the convention that the reference
channel will be the same for both the velocity and frequency
axis. $val$ will be the Systemic velocity in the velocity axis and the
rest frequency at the systemic velocity in the case of the frequency
axis.

In the case of spatial axes, the convention is that these are always
offset axes, i.e. $val$ is always 0.0 and reflects the position of the
projection center.

In both cases $ref$ is a double as the reference pixel/channel does
not need to precisely coincide with an actual pixel/channel.

\section{Data Commands Structure}

Commands that work on the data in cubes follow a well defined
structure. This structure is defined by three main objects, command,
user and program.

\subsection{Command Object}

The command object is the first object encountered when executing a
command, this object is created at the moment a command is registered
and exists throughout the execution of cube. This object is called
into action when SIC finds out which command to execute. Currently the
subroutine register and command are not part of the command object due
to constraints in the interface with SIC. Currently this object
contains two important methods: parse, which fills the user object
from the user input; main, which creates the program object from the
user object and calls the program object for the execution of the data
manipulation. The command object type is always named {\tt
  commname\_comm\_t}.

\subsection{User Object}

The user object is created from the parsing. The members of this
object reflect {\emph ipsis literis} what the user has typed but in a
structured way. Usually the User object only has one method, this
method is called toprog. The toprog method makes the translation
between user inputs and program inputs and hence generates the program
object. The user object type is always named {\tt commname\_user\_t}.

\subsection{Program Object}
After the creation of the program object from the user object the
program object controls the flow of actions. The first method to be
invoked is usually the header method, this method constructs the
header of the eventual output cube(s). The following method to be
called is usually the data method. The data method may sometimes be
preceded by some pre-processing methods, or be succeeded by
post-processing methods. The data method is where the actual data
manipulation is split into different tasks which are sent to different
threads to be processed in parallel. Each task will then call
sub-methods of the data method to execute the actual data manipulation.
The program object type is always named {\tt commname\_prog\_t}.


\end{document}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
