chap_DIA.tex

\documentclass[../main/NEMO_manual]{subfiles}

\begin{document}

\chapter{Output and Diagnostics (IOM, DIA, TRD, FLO)}
\label{chap:DIA}

%    {\em 4.0} & {\em Mirek Andrejczuk, Massimiliano Drudi} & {\em }  \\
%    {\em }      & {\em Dorotea Iovino, Nicolas Martin} & {\em }  \\
%    {\em 3.6} & {\em Gurvan Madec, Sebastien Masson } & {\em }  \\
%    {\em 3.4} & {\em Gurvan Madec, Rachid Benshila, Andrew Coward } & {\em }  \\
%    {\em }      & {\em Christian Ethe, Sebastien Masson } & {\em }  \\

\chaptertoc

\paragraph{Changes record} ~\\

{\footnotesize
  \begin{tabularx}{\textwidth}{l||X|X}
    Release & Author(s) & Modifications \\
    \hline
    {\em   4.0} & {\em ...} & {\em ...} \\
    {\em   3.6} & {\em ...} & {\em ...} \\
    {\em   3.4} & {\em ...} & {\em ...} \\
    {\em <=3.4} & {\em ...} & {\em ...}
  \end{tabularx}
}

\clearpage

%% =================================================================================================
\section{Model output}
\label{sec:DIA_io_old}

The model outputs are of three types: the output log/progress listings; 
the diagnostic output file(s); and the restart file(s)


The output listing and file(s) are predefined but should be checked and eventually adapted
to the user's needs. The output listing is stored in the \textit{ocean.output} file. The
information is printed from within the code on the logical unit \texttt{numout}. To
locate these prints, use the UNIX command "\textit{grep -i numout}" in the source code
directory. The \textit{ocean.output} file is the first place to check if something appears
to have gone wrong with the model since any detectable errors will be reported here.
Additional progress information can be requested using the options explained in
\autoref{subsec:MISC_statusinfo}.

Diagnostic output files are written in NetCDF format and the structure of legacy output files is predefined.
The legacy output files contain by default the time averaged fields, but by activating the macro \key{diainstant} 
it is possible to switch the full output to the instantaneous time value. 
When compiled with \key{xios}, \NEMO\ can employ the full capability of an I/O server (XIOS)
which provides flexibility in the choice of the fields to be written as well as how the writing tasks are distributed
over the processors in a massively parallel computing environment. A complete description
of the use of this I/O server is presented in the next section.

The restart file is used by the code when the user wants to start the model with initial
conditions defined by a previous simulation.  Restart files are NetCDF files containing
all the information that is necessary in order for there to be no changes in the model
results (even at the computer precision) between a run performed with several stops and
restarts and the same run performed in one continuous integration step.  It should be
noted that this requires that the restart file contains two consecutive time steps for all
the prognostic variables. The default behaviour of \NEMO\ is to generate a restart file
for each MPP region. These files will be read in by the same regions on restarting.
However, if a change in MPP decomposition is required, then the invidual restart files
must first be combined into a whole domain restart file. This can be done using the
\forcode{REBUILD_NEMO} tool.  Alternatively, users may experiment with the new options in
4.2 to write restarts via XIOS (see \autoref{subsec:XIOS_restarts}) with which it is possible to
write a whole domain restart file from a running model.

%% =================================================================================================
\section{Standard model output (\texttt{iom\_put})}
\label{sec:DIA_iom}

Since version 3.2, \rou{iom\_put} is the \NEMO\ output interface of choice.
It has been designed to be simple to use, flexible and efficient.
The two main purposes of \rou{iom\_put} are:

\begin{enumerate}
\item The complete and flexible control of the output files through external XML files adapted by
  the user from standard templates.
\item To achieve high performance and scalable output through the optional distribution of
  all diagnostic output related tasks to dedicated processes.
\end{enumerate}

\noindent The first functionality allows the user to specify, without code changes or recompilation,
aspects of the diagnostic output stream, such as:

\begin{itemize}
\item The choice of output frequencies that can be different for each file (including real months and years).
\item The choice of file contents; includes complete flexibility over which data are written in which files
  (the same data can be written in different files).
\item The possibility to split output files at a chosen frequency.
\item The possibility to extract a vertical or an horizontal subdomain.
\item The choice of the temporal operation to perform, \eg: average, accumulate, instantaneous, min, max and once.
\item Control over metadata via a large XML "database" of possible output fields.
\item Control over the compression and/or precision of output fields (subject to certain conditions)
\end{itemize}

\noindent In addition, \rou{iom\_put}  allows the user to add in the code the output of any new
variable (scalar, 1D, 2D or 3D) in a very easy way.  All details of \rou{iom\_put}
functionalities are listed in the following subsections.  An example of the main XML file
that control the outputs can be found in \path{cfgs/ORCA2_ICE_PISCES/EXPREF/iodef.xml}.\\

\noindent The second functionality targets output performance when running in parallel.  XIOS
provides the possibility to specify N dedicated I/O processes (in addition to the \NEMO\
processes) to collect and write the outputs.  With an appropriate choice of N by the user,
the bottleneck associated with the writing of the output files can be greatly reduced.

Since version 3.6, the \rou{iom\_put} interface depends on an external code called \XIOS, 
which is developed independently and has its own repository and support pages. Further details
are available in the \href{https://sites.nemo-ocean.io/user-guide/}{NEMO User guide}. 
Note that by default the model is interfaced with the trunk version of 
\href{https://forge.ipsl.jussieu.fr/ioserver/browser/XIOS2/trunk}{XIOS2}, but it is possible to use also 
the more recent version \href{https://forge.ipsl.jussieu.fr/ioserver/browser/XIOS3/trunk}{XIOS3} 
by adding the macro \key{xios3} in the configuration cpp file.\\

The IO server can also take advantage of the parallel I/O functionality of NetCDF4 to
create a single output file and therefore to bypass any rebuilding phase. This facility is ideal for
small to moderate size configurations but can be problematic with large models due to the large memory
requirements and the inability to use NetCDF4's compression capabilities in this "one\_file" mode.
XIOS now has the option of using two levels of I/O servers so it may be possible, in some circumstances,
to use a single I/O server at the second level to enable compression. In many cases. though, it is
often more robust to use "multiple\_file" mode (where each XIOS server writes its own file) and to
recombine these files as a post-processing step. The \forcode{REBUILD_NEMO} tool in the \forcode{tools}
directory is provided for this purpose.

Note that writing in parallel into the same NetCDF files requires that your NetCDF4 library is linked to
an HDF5 library that has been correctly compiled (\ie\ with the configure option $--$enable-parallel).
Note that the files created by \rou{iom\_put} through XIOS are incompatible with NetCDF3.
All post-processsing and visualization tools must therefore be compatible with NetCDF4 and not only NetCDF3.

Even when not using the "one\_file" functionality of NetCDF4, using N dedicated I/O
servers, where N is typically much less than the number of \NEMO\ processors, will reduce
the number of output files created.  This can greatly reduce the post-processing burden
otherwise associated with using large numbers of \NEMO\ processors.  Note that for smaller
configurations, the rebuilding phase can be avoided, even without a parallel-enabled
NetCDF4 library, simply by employing only one dedicated I/O server.

%% =================================================================================================
\subsection{XIOS: Reading and writing restart file}
\label{subsec:XIOS_restarts}


New from version 4.2, XIOS may be used to read from a single file restart produced by \NEMO. 
This does not add a new functionality (since NEMO has long had the capability for all
processes to read their patch from a single, combined restart file) but it may be advantageous 
on systems which struggle with too many simultaneous accesses to one file.  The
variables written to file \forcode{numror} (OCE), \forcode{numrir} (SI3), \forcode{numrtr}
(TOP), \forcode{numrsr} (SED) can be handled by XIOS.  To activate restart reading using
XIOS, set \np[=.true. ]{ln_xios_read}{ln\_xios\_read} in \textit{namelist\_cfg}. This
setting will be ignored when multiple restart files are present, and default \NEMO
functionality will be used for reading. There is no need to change iodef.xml file to use
XIOS to read restart, all definitions are done within the \NEMO\ code. For high resolution
configurations, however, there may be a need to add the following line in iodef.xml (xios
context):

\begin{xmllines}
<variable id="recv_field_timeout"        type="double">1800</variable>
\end{xmllines}

\noindent This variable sets timeout for reading.

\noindent If XIOS is to be used to read restart from files generated with an earlier \NEMO\ version (3.6 for instance),
dimension \forcode{z} defined in restart file must be renamed to \forcode{nav_lev}.\\

XIOS can also be used to write \NEMO\ restarts. A namelist parameter
\np{nn_wxios}{nn\_wxios} is used to determine the type of restart \NEMO\ will write. If it
is set to 0, default \NEMO\ functionality will be used - each processor writes its own
restart file; if it is set to 1 XIOS will write restart into a single file; for
\np[=2]{nn_wxios}{nn\_wxios} the restart will be written by XIOS into multiple files, one
for each XIOS server.  Note, however, that \textbf{\NEMO\ will not read restart generated
by XIOS when \np[=2]{nn_wxios}{nn\_wxios}}. The restart will have to be rebuilt before
continuing the run. This option aims to reduce number of restart files generated by \NEMO\
only, and may be useful when there is a need to change number of processors used to run
simulation.

The use of XIOS to read and write restart files is in preparation of NEMO for exascale
computing platforms. There may not be any performance gains on current clusters but it
should reduce file system bottlenecks in any future attempts to run NEMO on hundreds of
thousands of cores.

%% =================================================================================================
\subsection{XIOS: XML Inputs-Outputs Server}

%% =================================================================================================
\subsubsection{Attached or detached mode?}

\rou{iom\_put} is based on \href{http://forge.ipsl.jussieu.fr/ioserver/wiki}{XIOS},
the io\_server developed by Yann Meurdesoif from IPSL.
The behaviour of the I/O subsystem is controlled by settings in the external XML files listed above.
Key settings in the iodef.xml file are the tags associated with each defined file.

\xmlline|<variable id="using_server" type="bool"></variable>|

The \texttt{using\_server} setting determines whether or not the server will be used in
\textit{attached mode}
(as a library) [\texttt{.false.}] or in \textit{detached mode}
(as an external executable on N additional, dedicated cpus) [\texttt{.true.}].
The \textit{attached mode} is simpler to use but much less efficient for
massively parallel applications.