chap_DIA.tex

\documentclass[../main/NEMO_manual]{subfiles}

\begin{document}

\chapter{Output and Diagnostics (IOM, DIA, TRD)}
\label{chap:DIA}

%    {\em 4.0} & {\em Mirek Andrejczuk, Massimiliano Drudi} & {\em }  \\
%    {\em }      & {\em Dorotea Iovino, Nicolas Martin} & {\em }  \\
%    {\em 3.6} & {\em Gurvan Madec, Sebastien Masson } & {\em }  \\
%    {\em 3.4} & {\em Gurvan Madec, Rachid Benshila, Andrew Coward } & {\em }  \\
%    {\em }      & {\em Christian Ethe, Sebastien Masson } & {\em }  \\

\chaptertoc

\paragraph{Changes record} ~\\

{\footnotesize
  \begin{tabularx}{\textwidth}{l||X|X}
    Release & Author(s) & Modifications \\
    \hline
    {\em   5.0} &
        {\em D. Calvert, A. Coward and S. M{\" u}ller} &
            {\em General revisions; addition of \autoref{sec:DIA_diamlr}; expansion of \autoref{sec:DIA_dia25h}} \\
    {\em   4.0} & {\em ...} & {\em ...} \\
    {\em   3.6} & {\em ...} & {\em ...} \\
    {\em   3.4} & {\em ...} & {\em ...} \\
    {\em <=3.4} & {\em ...} & {\em ...}
  \end{tabularx}
}

\clearpage

%% =================================================================================================
\section{Model outputs}
\label{sec:DIA_io_old}

The model outputs are of three types: the output log/progress listings; 
the diagnostic output file(s); and the restart file(s).

The output log and progress listings are output in the \textit{ocean.output} file(s), which contains
information printed from within the code on the logical unit \texttt{numout}. To
locate these prints, use the UNIX command "\textit{grep -i numout}" in the source code
directory. Model errors that are caught by \NEMO\ (via the \rou{ctl\_stop} subroutine) will issue a
return code of \texttt{123} and information on the errors will be written to the \textit{ocean.output} file.
Additional progress information can be requested using the options explained in
\autoref{subsec:MISC_statusinfo}.

Diagnostic output files are written in NetCDF4 format and are generated by one of two available methods.
With the legacy method (used when \key{xios} is not specified), output files have a predefined structure
and contain time averaged diagnostics. If \key{diainstant} is specified, instantaneous diagnostics are instead output.
With the standard method (used when \key{xios} is specified), \NEMO\ can employ the full capability of the XIOS I/O server,
which provides flexibility in the choice of the fields to be written as well as how the writing tasks are distributed
over the processors in a massively parallel computing environment. A complete description
of the use of this I/O server is presented in the next section.

The restart file is used by the code when the user wants to start the model with initial
conditions defined by a previous simulation.  Restart files are NetCDF files containing
all the information that is necessary in order for there to be no changes in the model
results (even at the computer precision) between a run performed with several stops and
restarts and the same run performed in one continuous integration step.  It should be
noted that this requires that the restart file contains two consecutive time steps for all
the prognostic variables.

Two methods are available to read and write restart files.
The default method is for \NEMO\ to perform these tasks: \NEMO\ will generate a restart file for each MPP subdomain,
which will then be read by the same subdomain on restarting. Therefore if a change in MPP decomposition is required
between runs, then the individual restart files
must first be combined into a single restart file for the full domain. This can be done using the
\href{https://sites.nemo-ocean.io/user-guide/tools.html#rebuild-nemo}{REBUILD\_NEMO} tool.
The alternative method is to use XIOS to read and/or write restart files (see \autoref{sec:XIOS_restarts}).
This functionality was introduced in \NEMO\ v4.2 and includes the ability to write restart data directly to a single file
for the full domain.

%% =================================================================================================
\section[Standard model diagnostic output with XIOS (\texttt{iom\_put}, \texttt{key\_xios})]{Standard model diagnostic output with XIOS (\protect\rou{iom\_put}, \protect\key{xios})}
\label{sec:DIA_iom}

The standard \NEMO\ diagnostic output method (activated when \key{xios} is specified) uses an
external I/O library and server named \XIOS\, with the \NEMO\ subroutine \rou{iom\_put} serving as the main interface
to this library.

XIOS is developed by Yann Meurdesoif and his team at IPSL, and has its own
\href{http://forge.ipsl.jussieu.fr/ioserver/wiki}{repository and support pages}.
\NEMO\ v5 can be used with either \href{https://forge.ipsl.jussieu.fr/ioserver/browser/XIOS2/trunk}{version 2}
or \href{https://forge.ipsl.jussieu.fr/ioserver/browser/XIOS3/trunk}{version 3} of XIOS.
\NEMO\ expects XIOS v2 by default and requires at least SVN revision 2131 of this version of the library.
The use of XIOS v3 requires that the \NEMO\ key \key{xios3} be specified.
Further details are available in the \href{https://sites.nemo-ocean.io/user-guide/}{NEMO user guide}.
XIOS will create output files in NetCDF4 format, which is incompatible with the older NetCDF3
libraries. Post-processing and visualization tools must therefore be linked to NetCDF4 libraries to be able to
handle the NetCDF files created by XIOS.//

XIOS has been designed to be simple to use, flexible and efficient.
Its two main purposes are:

\begin{enumerate}
\item The complete and flexible control of the output files through external XML files adapted by
  the user from standard templates.
\item To achieve high performance and scalable output through the optional distribution of
  all diagnostic output related tasks to dedicated processes.
\end{enumerate}

\noindent The first functionality allows the user to specify, without code changes or recompilation,
aspects of the diagnostic output stream, such as:

\begin{itemize}
\item The choice of output frequencies that can be different for each file (including real months and years).
\item The choice of file contents; includes complete flexibility over which data are written in which files
  (the same data can be written in different files).
\item The possibility to split output files at a chosen frequency.
\item The possibility to extract a vertical or an horizontal subdomain.
\item The choice of the temporal operation to perform, \eg: average, accumulate, instantaneous, min, max and once.
\item Control over metadata via a large XML "database" of possible output fields.
\item Control over the compression and/or precision of output fields (subject to certain conditions)
\end{itemize}

\noindent In addition, the \rou{iom\_put} interface allows the user to add in the \NEMO\ code the output of any new
variable (scalar, 1D, 2D or 3D) in a very easy way. The functionalities of XIOS and the \rou{iom\_put} interface
are listed in the following subsections. \\

\noindent The second functionality targets output performance when running in parallel. XIOS
provides the possibility to specify $N$ dedicated I/O servers (in addition to the \NEMO\
processes) to collect and write the outputs.  With an appropriate choice of $N$ by the user,
the bottleneck associated with the writing of the output files can be greatly reduced.

XIOS can take advantage of the parallel I/O functionality of NetCDF4\footnote{This requires that your
NetCDF4 library is linked to an HDF5 library that has been correctly compiled (\ie\ with the configure option
\texttt{--enable-parallel})} to have each XIOS server write
to a single output file. This facility is ideal for
small to moderate size configurations but can be problematic with large models due to the large memory
requirements and the inability to use NetCDF4's compression capabilities in this "one\_file" mode.

XIOS2 has the option of using two levels of I/O servers so it may be possible, in some circumstances,
to use a single I/O server at the second level to enable compression.
In many cases though, it is
often more robust to use "multiple\_file" mode (where each XIOS server writes to a separate file) and to
recombine these files as a post-processing step. The
\href{https://sites.nemo-ocean.io/user-guide/tools.html#rebuild-nemo}{REBUILD\_NEMO} tool is provided for this purpose.
As the number of XIOS servers is typically much less than the number of \NEMO\ processes, significantly fewer
output files will be generated compared to the legacy method (which outputs one file per \NEMO\ process), reducing
the overhead of this post-processing step.
For smaller configurations this post-processing step can be avoided entirely, even without a parallel-enabled
NetCDF4 library, by using only one XIOS server.

XIOS3 provides a more versatile approach with the concept of "pools
and services" where different "pools" of XIOS processes can be assigned to perform
different services.  Chief amongst these services are "gatherers" and "writers"
which are similar to the two-level server capabilties of XIOS2, but, crucially, allow
the user better control over the assignment of resources. With care, it is possible
to achieve sustained "one\_file" output even with large models. These capabilities
are relatively new at the time of the 5.0 release, but the approach is documented in
the \href{https://sites.nemo-ocean.io/user-guide/xios3demo.html}{XIOS3 demonstrator}
of the on-line user guide.

%% =================================================================================================
\subsection{Main XIOS configuration file (\textit{iodef.xml})}

The behaviour of XIOS is controlled by settings in external XML configuration files, with settings for different
applications (or components of one) split into separate "contexts". These settings are specified via the
top-level \textit{iodef.xml} file (see for example \path{./cfgs/ORCA2_ICE_PISCES/EXPREF/iodef.xml}).
Basic details on XML syntax and rules can be found in \autoref{subsec:DIA_iom_xml}.

In \NEMO, the \textit{iodef.xml} file typically contains settings for the "xios" context
(controlling the overall functionality of XIOS) and for one or more "nemo" contexts (defining the fields and grids used,
as well as the output files to be generated, by \NEMO\ and any AGRIF child grids).
Further information on these contexts can be found in \autoref{subsec:DIA_iom_xml_tags}.

%% =================================================================================================
\subsubsection{The "xios" context}

"xios" context settings that might commonly be configured are shown in \autoref{tab:DIA_iom_xios_context}.

% NOTE: These are not described in the text, except for using_server
\begin{table}
  \caption["xios" context variables in \textit{iodef.xml}]{"xios" context variables typically used
  in the \textit{iodef.xml} configuration files used by \NEMO}
  \begin{tabularx}{\textwidth}{|lXl|}
    \hline
    variable name                                                           &
    description                                                             &
    example  \\
    \hline
    \hline
    \texttt{info\_level}                                                    &
    verbosity level (0 to 100)                                              &
    0        \\
    \hline
    \texttt{using\_server}                                                  &
    activate attached (false) or detached(true) mode                         &
    true     \\
    \hline
    \texttt{using\_oasis}                                                   &
    XIOS is used with OASIS (true) or not (false)                            &
    false    \\
    \hline
    \texttt{oasis\_codes\_id}                                               &
    [\textbf{XIOS 2 only}] when using oasis, define the identifier of \NEMO\ in the namcouple.
    Note that the identifier of XIOS is "xios.x"                            &
    oceanx   \\
    \hline
  \end{tabularx}
  \label{tab:DIA_iom_xios_context}
\end{table}

%% =================================================================================================
\subsubsection{The "nemo" context}

"nemo" context settings are usually separated into several XML files, each handling a different component of
the configuration. These files are included in \textit{iodef.xml} via a nested set of \texttt{src} directives
(see \autoref{subsec:DIA_iom_xml_nesting}), usually via an intermediate file \textit{context\_nemo.xml}.
\eg\ for the ORCA2\_ICE\_PISCES reference configuration, this hierarchy of files can be represented as:

\begin{verbatim}
iodef.xml <----------- <context id="nemo" src="./context_nemo.xml"/>
context_nemo.xml <-+
                   |
                   +-- <field_definition  src="./field_def_nemo-oce.xml"/>
                   +-- <field_definition  src="./field_def_nemo-ice.xml"/>
                   +-- <field_definition  src="./field_def_nemo-pisces.xml"/>
                   |
                   +-- <file_definition   src="./file_def_nemo-oce.xml"/>
                   +-- <file_definition   src="./file_def_nemo-ice.xml"/>
                   +-- <file_definition   src="./file_def_nemo-pisces.xml"/>
                   |
                   +-- <axis_definition   src="./axis_def_nemo.xml"/>
                   |
                   +-- <domain_definition src="./domain_def_nemo.xml"/>
                   |
                   +-- <grid_definition   src="./grid_def_nemo.xml"/>
\end{verbatim}

The purposes and contents of these XML files will be explained further in later sections.

%% =================================================================================================
\subsection{Practical issues}

%% =================================================================================================
\subsubsection{Installation}

As mentioned, XIOS is supported separately and must be downloaded and compiled before it can be used with \NEMO.
See the installation guide on the \href{http://forge.ipsl.jussieu.fr/ioserver/wiki}{XIOS wiki} for help and guidance.
\NEMO\ will then need to link to the compiled XIOS library- see the
\href{https://sites.nemo-ocean.io/user-guide/install.html#download-and-install-the-nemo-code}{NEMO user guide}.

%% =================================================================================================
\subsubsection{Attached or detached mode?}

For both XIOS2 and XIOS3, a key setting in the "xios" context (\textit{iodef.xml}) is:

\xmlline|<variable id="using_server" type="bool"></variable>|

which determines whether or not the server will be used in
\textit{attached mode}
(as a library) [\texttt{.false.}] or in \textit{detached mode}
(as an external executable on N additional, dedicated cpus) [\texttt{.true.}].
The \textit{attached mode} is simpler to use but much less efficient for
massively parallel applications.
The output produced will also depend on the type of each file requested in the
\texttt{file\_definition} sections.  The type can be either ''multiple\_file'' or
''one\_file'' (explained more fully in later sections).

In \textit{attached mode} and if the type of file is ''multiple\_file'',
then each \NEMO\ process will also act as an I/O server and produce its own set of output files.
Superficially, this emulates the standard behaviour of \NEMO\ without XIOS.
However, the subdomain written out by each process does not correspond to
the \forcode{jpi x jpj} domain actually computed by the process (although it may if \forcode{jpni=1}).
Instead each process will have collected and written out a number of complete longitudinal strips.
If the ''one\_file'' option is chosen then all processes will collect their longitudinal strips and
write (in parallel) to a single output file.

In \textit{detached mode} and if the type of file is ''multiple\_file'',
then each stand-alone XIOS process will collect data for a range of complete longitudinal strips and
write to its own set of output files.
If the ''one\_file'' option is chosen then all XIOS processes will collect their longitudinal strips and
write (in parallel) to a single output file.
Note running in detached mode requires launching a Multiple Process Multiple Data (MPMD) parallel job.
The following subsection provides a typical example but the syntax will vary in different MPP environments.

%% =================================================================================================
\subsubsection{Number of cpus used by XIOS in detached mode}

The number of cores used by the XIOS servers is specified when launching the model. This number
should be from \texttildelow 1/10 to \texttildelow 1/50 of the
number of cores dedicated to \NEMO.  Some manufacturers suggest using O($\sqrt{N}$)
dedicated I/O processors for $N$ processors, but this is a general recommendation and not
specific to \NEMO.  It is difficult to provide precise recommendations because the optimal
choice will depend on the particular hardware properties of the target system (parallel
filesystem performance, available memory, memory bandwidth etc.) and the volume and
frequency of data to be created.  Here is an example of using 2 cpus for XIOS servers and 62
cpus for \NEMO\ with \texttt{mpirun}:

\begin{cmds}
  mpirun -np 62 ./nemo.exe : -np 2 ./xios_server.exe
\end{cmds}

%% =================================================================================================
\subsubsection{Add your own outputs}

It is very easy to add your own outputs with XIOS and the \NEMO\ \rou{iom\_put} interface.
Many standard fields and diagnostics are already prepared (\ie, steps 1 to 3 below have been done) and
simply need to be activated by including the required output in a file definition (step 4).
To add new diagnostics, all 4 of the following steps must be taken.

\begin{enumerate}
\item In the \NEMO\ code, add a "\forcode{CALL iom_put( 'identifier', array )}" for the array that is
to be output as a diagnostic. In most cases, this will be in a part of the code which is executed
only once per timestep and after the array has been updated for that timestep.

Adding this call simply exposes the array to the XIOS workflow; whether or not (and at
which frequency) the corresponding diagnostic is actually output by XIOS will be determined by the contents of the
file definition (see step 4).
\item If necessary, add "\forcode{USE iom ! I/O manager library}" to the list of used
modules in the upper part of your module.
\item In the appropriate \path{./cfgs/SHARED/field_def_nemo-....xml} files, add a definition for your diagnostic to the
field definition using the same identifier you used in the  \NEMO\ Fortran code (see
subsequent sections for details of the XML syntax and rules). For example:
\begin{xmllines}
<field_definition>
   <field_group id="grid_T" grid_ref="grid_T_3D"> <!-- T grid -->
      <field id="identifier" long_name="blabla" />
   </field_group>
</field_definition>
\end{xmllines}
This definition must be added to the \texttt{field\_group} whose reference grid (\texttt{grid\_ref}) is consistent
with the size of the array passed to \rou{iom\_put}. The \texttt{grid\_ref} attribute refers to
definitions set in \textit{grid\_def\_nemo.xml} which, in turn, reference domains and axes defined either
in the code (\rou{iom\_set\_domain\_attr} and \rou{iom\_set\_axis\_attr} in \mdl{iom}) or
in the XML configuration files (\textit{domain\_def\_nemo.xml} and \textit{axis\_def\_nemo.xml}).  \eg\ :
\begin{xmllines}
<grid_definition>
  <grid id="grid_T_3D" >
    <domain domain_ref="grid_T" />
    <axis axis_ref="deptht" />
  </grid>
</grid_definition>
\end{xmllines}
Note that if the array passed to \rou{iom\_put} is computed within the Surface Boundary Condition module
(\autoref{chap:SBC}), then the corresponding field definition must be added within the SBC \texttt{field\_group},
\xmlcode{<field_group id="SBC" ...>}. This is because the array is updated every \np{nn_fsbc}{nn\_fsbc} time steps
and the frequency of operations in the SBC \texttt{field\_group} has been defined accordingly
(see \rou{iom\_set\_field\_attr} in \mdl{iom}).
\item Finally, add your field to one or more file definitions defined in \textit{file\_def\_nemo-*.xml}
(each corresponding to an output file- again, see the subsequent sections for XML syntax and rules)
\begin{xmllines}
<file_definition>
   <file_group id="5d" output_freq="5d"  output_level="10" enabled=".TRUE.">  <!-- 5d files -->
      <file id="file1" name_suffix="_grid_T" description="ocean T grid variables" >
         <field field_ref="identifier" />
      </file>
   </file_group>
</file_definition>
\end{xmllines}
\end{enumerate}

%% =================================================================================================
\subsection{XML fundamentals}
\label{subsec:DIA_iom_xml}

This subsection discusses some basic aspects of the XML syntax used by XIOS.
Further information can be found in the XIOS reference and user guides available
\href{https://forge.ipsl.jussieu.fr/ioserver/}{here}.

%% =================================================================================================
\subsubsection{XML basic rules}

XML tags begin with the less-than character ("$<$") and end with the greater-than character ("$>$").
You use tags to mark the start and end of elements, which are the logical units of information in an XML document.
In addition to marking the beginning of an element, XML start tags also provide a place to specify attributes.
An attribute specifies a single property for an element, using a name/value pair, for example:
\xmlcode{<a b="x" c="y" d="z"> ... </a>}.
See \href{http://www.xmlnews.org/docs/xml-basics.html}{here} for more details.

%% =================================================================================================
\subsubsection{XML tags}
\label{subsec:DIA_iom_xml_tags}

The XML tags used by XIOS are organised into 7 families:
\texttt{context}, \texttt{axis}, \texttt{domain}, \texttt{grid}, \texttt{field}, \texttt{file} and \texttt{variable}.
Each tag family has a hierarchy of three scopes (except for \texttt{context}), shown in \autoref{tab:DIA_xml_scope}.

\begin{table}
  \caption[Hierarchy of tag scopes used by XIOS]{Hierarchy of scopes used by tags in the XIOS XML configuration files}
  \begin{tabular*}{\textwidth}{|p{0.15\textwidth}p{0.4\textwidth}p{0.35\textwidth}|}
    \hline
    Scope   & description                                                                 &
                                                                                            example                          \\
    \hline
    \hline
    \texttt{root}    & declaration of the root element that can contain element groups or elements &
                                                                                            \xmlcode{<file_definition ... >} \\
    \hline
    \texttt{group}   & declaration of a group element that can contain element groups or elements  &
                                                                                            \xmlcode{<file_group      ... >} \\
    \hline
    \texttt{element} & declaration of an element that can contain elements                         &
                                                                                            \xmlcode{<file            ... >} \\
    \hline
  \end{tabular*}
  \label{tab:DIA_xml_scope}
\end{table}

Each element may have several attributes.
Some attributes are mandatory, some are optional with a default value, and others are completely optional.
A special attribute, \texttt{id}, is used to identify an element (or a group of elements) and must have a unique
value within each element family.
This attribute is optional, but the corresponding element cannot be referenced if this is not defined.

XIOS "contexts" (definitions and settings for different applications or components of one) are
separated by the \texttt{context} tag.
No interference is possible between 2 different contexts.
Each context has its own calendar and an associated timestep.
The contexts used by \NEMO\ (which can be defined in any order) are shown in \autoref{tab:DIA_xml_contexts}.
The "xios" context uses only 1 tag (\autoref{tab:DIA_xml_tags_xios}), while the other contexts related to \NEMO\
use 5 tags (\autoref{tab:DIA_xml_tags_nemo}).

\begin{table}
  \caption[XIOS contexts used by \NEMO]{XIOS contexts used by \NEMO}
  \begin{tabular}{|p{0.15\textwidth}p{0.4\textwidth}p{0.35\textwidth}|}
    \hline
    context         &	description                                                                &
                                                                                                     example                              \\
    \hline
    \hline
    \texttt{xios}    &	context containing information for XIOS                                    &
                                                                                                     \xmlcode{<context id="xios" ... >}   \\
    \hline
    \texttt{nemo}    &	context containing I/O information for \NEMO\ (mother grid when using AGRIF)  &
                                                                                                     \xmlcode{<context id="nemo" ... >}   \\
    \hline
    \texttt{n\_nemo} &	context containing I/O information for \NEMO\ child grid n (when using AGRIF) &
                                                                                                     \xmlcode{<context id="n_nemo" ... >} \\
    \hline
  \end{tabular}
  \label{tab:DIA_xml_contexts}
\end{table}

\begin{table}
  \caption[XIOS tags used by the xios context]{XIOS tags used by the xios context}
  \begin{tabular}{|p{0.2\textwidth}p{0.35\textwidth}p{0.35\textwidth}|}
    \hline
    context tag                                      &
                                                       description                                      &
                                                                                                          example                              \\
    \hline
    \hline
    \texttt{variable\_definition}                             &
                                                       define variables needed by XIOS.
                                                       This can be seen as a kind of namelist for XIOS. &
                                                                                                          \xmlcode{<variable_definition ... >} \\
    \hline
  \end{tabular}
  \label{tab:DIA_xml_tags_xios}
\end{table}

\begin{table}
  \caption[XIOS tags used by the nemo contexts]{XIOS tags used by the nemo contexts
           (both mother and child grids when using AGRIF)}
  \begin{tabular}{|p{0.2\textwidth}p{0.35\textwidth}p{0.35\textwidth}|}
    \hline
    context tag        &	description                                                               &
                                                                                                       example                            \\
    \hline
    \hline
    \texttt{field\_definition}  &	define all variables that can potentially be outputted                    &
                                                                                                       \xmlcode{<field_definition ... >}  \\
    \hline
    \texttt{file\_definition}   &	define the netcdf files to be created and the variables they will contain &
                                                                                                       \xmlcode{<file_definition ... >}   \\
    \hline
    \texttt{axis\_definition}   &	define vertical axis                                                      &
                                                                                                       \xmlcode{<axis_definition ... >}   \\
    \hline
    \texttt{domain\_definition} &	define the horizontal grids                                               &
                                                                                                       \xmlcode{<domain_definition ... >} \\
    \hline
    \texttt{grid\_definition}   &	define the 2D and 3D grids (association of an axis and a domain)          &
                                                                                                       \xmlcode{<grid_definition ... >}   \\
    \hline
  \end{tabular}
  \label{tab:DIA_xml_tags_nemo}
\end{table}

%% =================================================================================================
\subsubsection{Nesting XML files}
\label{subsec:DIA_iom_xml_nesting}

The main XML file (\textit{iodef.xml}) can be split into different parts to improve its readability.
These other XML files can then be included in the \textit{iodef.xml} file via the \texttt{src} attribute:
\begin{xmllines}
<context id="nemo" src="./context_nemo.xml"/>
\end{xmllines}

\noindent In the \NEMO\ reference configurations, the field, file and grid definitions are typically split
over several XML files in this manner. \eg\ the \textit{context\_nemo.xml} file for AGRIF\_DEMO contains:
\begin{xmllines}
<!-- Fields definition -->
    <field_definition src="./field_def_nemo-oce.xml"      />   <!--  NEMO ocean dynamics               -->
    <field_definition src="./field_def_nemo-ice.xml"      />   <!--  NEMO ocean sea ice                -->
    <field_definition src="./field_def_nemo-pisces.xml"   />   <!--  NEMO ocean biogeochemical         -->
    <field_definition src="./field_def_nemo-innerttrc.xml"/>   <!--  NEMO ocean inert passive tracer   -->

<!-- Files definition -->
    <file_definition src="./file_def_nemo-oce.xml"/>       <!--  NEMO ocean dynamics                   -->
    <file_definition src="./file_def_nemo-ice.xml"/>       <!--  NEMO ocean sea ice                    -->
    <file_definition src="./file_def_nemo-innerttrc.xml"/> <!--  NEMO ocean inert passive tracer       -->

<!-- Grids/domains/axes definition -->
    <axis_definition   src="./axis_def_nemo.xml"/>         <!-- Axis definition -->
    <domain_definition src="./domain_def_nemo.xml"/>       <!-- Domain definition -->
    <grid_definition   src="./grid_def_nemo.xml"/>         <!-- Grids definition -->
\end{xmllines}

%% =================================================================================================
\subsubsection{Use of inheritance}

XML extensively uses the concept of inheritance.
XML has a tree based structure with a parent-child oriented relation: all children inherit attributes from their parent,
and an attribute defined in a child replaces the inherited attribute value.
Note that the special attribute \texttt{id} is never inherited.
\\
\\
Example 1: direct inheritance.

\begin{xmllines}
<field_definition operation="average" >
  <field_group id="grid_T" grid_ref="grid_T_2D"> <!-- T grid -->
     <field id="sst"                    />   <!-- averaged      sst -->
     <field id="sss" operation="instant"/>   <!-- instantaneous sss -->
  </field_group>
</field_definition>
\end{xmllines}

\noindent The field ''sst'' which is part (or a child) of the \texttt{field\_definition} will inherit the
value ''average'' of the attribute ''operation'' from its parent.
Note that a child can overwrite the attribute definition inherited from its parents.
In the example above, the field ''sss'' will for example output instantaneous values instead of average values.
\\
\\
Example 2: inheritance by reference: inherit (and overwrite, if needed) the attributes of a tag you are referring to:


\begin{xmllines}
<field_definition>
   <field_group id="grid_T" grid_ref="grid_T_2D"> <!-- T grid -->
      <field id="sst" long_name="sea surface temperature" />
      <field id="sss" long_name="sea surface salinity"    />
   </field_group>
</field_definition>

<file_definition>
   <file id="myfile" output_freq="1d" />
      <field field_ref="sst"                            />  <!-- default -->
      <field field_ref="sss" long_name="my description" />  <!-- overwritten -->
   </file>
</file_definition>
\end{xmllines}

%% =================================================================================================
\subsubsection{Use of groups}

Groups can be used for 2 purposes.
Firstly, a group can be used to define common attributes to be shared by the elements of
the group through inheritance.
In the following example, we define a group of 2D and 3D fields on the T grid:

\begin{xmllines}
<field_definition>
   <field_group id="grid_T" grid_ref="grid_T_2D">
      <field id="toce" long_name="temperature"             unit="degC" grid_ref="grid_T_3D"/>
      <field id="sst"  long_name="sea surface temperature" unit="degC"                     />
      <field id="sss"  long_name="sea surface salinity"    unit="psu"                      />
      <field id="ssh"  long_name="sea surface height"      unit="m"                        />
   </field_group>
</field_definition>
\end{xmllines}

Most of the fields are 2D, so the 2D grid definition (''grid\_T\_2D'') is used by the group.
Field ''toce'' is 3D, so the 2D grid definition inherited from the group is overwritten by that
of the 3D grid (''grid\_T\_3D''). \\

\noindent Secondly, a group can be used to refer to multiple elements with a single reference.
Several examples of groups of fields are included at the end of the field definition XML configuration files (
\path{./cfgs/SHARED/field_def_nemo-oce.xml},
\path{./cfgs/SHARED/field_def_nemo-pisces.xml} and
\path{./cfgs/SHARED/field_def_nemo-ice.xml} ) .
For example, a shortlist of variables on the U grid:

\begin{xmllines}
<field_definition>
   <field_group id="groupU" >
      <field field_ref="uoce"  />
      <field field_ref="ssu" />
      <field field_ref="utau"  />
   </field_group>
</field_definition>
\end{xmllines}

can be included in a file definition via the \texttt{group\_ref} attribute:

\begin{xmllines}
<file_definition>
   <file id="myfile_U" output_freq="1d" />
      <field_group group_ref="groupU" />
      <field field_ref="uocetr_eff"   />  <!-- add another field -->
   </file>
</file_definition>
\end{xmllines}

%% =================================================================================================
\subsection{Detailed functionalities}

This subsection discusses some of the functionality offered by XIOS, several examples of which can be found within
the files \path{./cfgs/ORCA2_ICE_PISCES/EXPREF/*.xml}.
Again, refer to the XIOS reference and user guides, available \href{https://forge.ipsl.jussieu.fr/ioserver/}{here},
for more information.

%% =================================================================================================
\subsubsection{Define horizontal subdomains/zooms}

Horizontal subdomains ("zooms") are defined through the attributes \texttt{ibegin}, \texttt{jbegin},
\texttt{ni}, \texttt{nj} of the \texttt{zoom\_domain} tag.
This must appear within a \texttt{domain} tag, and must therefore be placed in the domain definition part
of the XML (\ie\ between the \texttt{domain\_definition} tags in \path{./cfgs/SHARED/domain_def_nemo.xml}).

Note that \textbf{\texttt{zoom\_domain} is deprecated in XIOS3} and will eventually be removed;
\texttt{extract\_domain} should be used instead.
XIOS3 still supports the use of \texttt{zoom\_domain}, but will generate warnings stating that this has been renamed
to \texttt{extract\_domain}.

For example, a 5 by 5 box with the bottom left corner at point (10,10) would be defined as:

\begin{xmllines}
<domain_definition>
   <domain id="myzoomT" domain_ref="grid_T">
      <zoom_domain ibegin="10" jbegin="10" ni="5" nj="5" />
   </domain>
</domain_definition>
\end{xmllines}

and would then be used for diagnostic output via the \texttt{domain\_ref} attribute of the \texttt{field} tag family, \eg\:

\begin{xmllines}
<file_definition>
   <file id="myfile_zoom" output_freq="1d" >
      <field field_ref="toce" domain_ref="myzoomT"/>
   </file>
</file_definition>
\end{xmllines}

\noindent However, only \texttt{grid\_ref} or a \texttt{domain\_ref}/\texttt{axis\_ref} pair may be specified, not both.
In the example above, field "toce" has likely been defined with \texttt{grid\_ref="grid\_T\_3D"} in the field definition
XML configuration file. This will be inherited, so we must override \texttt{grid\_ref} instead of \texttt{domain\_ref}
by defining a new grid (a copy of "grid\_T\_3D" in \path{./cfgs/SHARED/grid_def_nemo.xml}):

\begin{xmllines}
<grid_definition>
   <grid id="grid_T_3D_myzoomT">
      <domain domain_ref="myzoomT" />
      <axis axis_ref="deptht" />
   </grid>
</grid_definition>
\end{xmllines}

and then referencing this in the \texttt{field} tag:

\begin{xmllines}
<file_definition>
   <file id="myfile_zoom" output_freq="1d" >
      <field field_ref="toce" grid_ref="grid_T_3D_myzoomT"/>
   </file>
</file_definition>
\end{xmllines}

\noindent Moorings are seen as an extreme case corresponding to a 1 by 1 subdomain.
The Equatorial section, the TAO, RAMA and PIRATA moorings are already defined in the code and
can therefore be used without needing to specify their (i,j) position in the grid.
These predefined zooms can be activated by the use of a specific \texttt{domain\_ref}:
''EqT'', ''EqU'' or ''EqW'' for the equatorial sections and
the mooring position for TAO, RAMA and PIRATA followed by ''T'', \eg\:

\begin{xmllines}
<file_definition>
   <file id="myfile_zoom" output_freq="1d" >
      <field field_ref="sst" domain_ref="0n180wT"/>
   </file>
</file_definition>
\end{xmllines}

\noindent A full list of these section and mooring domains can be found in \path{./cfgs/SHARED/domain_def_nemo.xml}. \\

\noindent As noted in \autoref{sec:DIA_iom}, using ''multiple\_file'' type output will produce one file per XIOS
server with each file containing a different part of the full domain, which may split the subdomain across
several files. In this case, tools like \href{https://sites.nemo-ocean.io/user-guide/tools.html#rebuild-nemo}{REBUILD\_NEMO}
should be used to combine these files.

%% =================================================================================================
\subsubsection{Define vertical zooms}

Vertical zooms are defined through the attributes \texttt{begin} and \texttt{n} of the \texttt{zoom\_axis} tag.
This must appear within an \texttt{axis} tag, and must therefore be placed in the axis definition part
of the XML (\ie\ between the \texttt{axis\_definition} tags in \path{./cfgs/SHARED/axis_def_nemo.xml}).
Note that as for \texttt{zoom\_domain}, \textbf{\texttt{zoom\_axis} is deprecated in XIOS3} and \texttt{extract\_axis}
should be used instead.

For example, a zoom corresponding to the top 300m of the ocean would be defined as:

\begin{xmllines}
<axis_definition>
   <!-- Vertical zoom for a 31-levels ORCA2 grid. For eORCA1 300m corresponds to n=35 -->
   <axis id="deptht300" axis_ref="deptht" >
     <zoom_axis begin="0" n="19" />
   </axis>
</axis_definition>
\end{xmllines}

and would then be used for diagnostic output via the \texttt{axis\_ref} attribute of the \texttt{field} tag family, \eg\:

\begin{xmllines}
<file_definition>
   <file id="myfile_zoom" output_freq="1d" >
      <field field_ref="diag_1d" axis_ref="deptht300"/>
   </file>
</file_definition>
\end{xmllines}

\noindent As noted in the previous section, only \texttt{grid\_ref} or a \texttt{domain\_ref}/\texttt{axis\_ref} pair
may be specified, not both. Therefore in the case of a 3D diagnostic, we must override \texttt{grid\_ref} instead of
\texttt{axis\_ref} by defining a new grid (a copy of grid\_T\_3D in \path{./cfgs/SHARED/grid_def_nemo.xml}):

\begin{xmllines}
<grid_definition>
   <grid id="grid_T_3D_0_300m">
      <domain domain_ref="grid_T" />
      <axis axis_ref="deptht300" />
   </grid>
</grid_definition>
\end{xmllines}

and then referencing this in the \texttt{field} tag:

\begin{xmllines}
<file_definition>
   <file id="myfile_zoom" output_freq="1d" >
      <field field_ref="toce" grid_ref="grid_T_3D_0_300m"/>
   </file>
</file_definition>
\end{xmllines}

%% =================================================================================================
\subsubsection{Changes to the names of output files applied by \NEMO}

The output file names are defined by the attributes \texttt{name} and \texttt{name\_suffix} of the
\texttt{file} tag family. For example:

\begin{xmllines}
<file_definition>
   <file_group id="1d" output_freq="1d" name="myfile_1d" >
      <file id="myfileA" name_suffix="_AAA" > <!-- will create file "myfile_1d_AAA"  -->
      ...
      </file>
      <file id="myfileB" name_suffix="_BBB" > <!-- will create file "myfile_1d_BBB" -->
      ...
      </file>
   </file_group>
</file_definition>
\end{xmllines}

However it is often very convenient to include the name of the experiment,
the output file frequency and the start/end dates of the simulation in the file name,
which are stored either in the namelist or in the XML file.
To achieve this, we added the following rule:
if the \texttt{id} of the \texttt{file} tag is ''fileN'' (where N = 1 to 999 on 1 to 3 digits) or
one of the predefined sections or moorings (see next subsection),
parts of the \texttt{name} and \texttt{name\_suffix} attributes (which can be inherited)
will be automatically replaced if they correspond to any of the placeholders in \autoref{tab:DIA_xios_filename_subs}. \\

\begin{table}
  \caption[File name placeholder strings and their substitutions]{
           Placeholder strings for the names of diagnostic output files generated by XIOS and
           the strings they are substituted for, when the file \texttt{id} has the form ''fileN''}
  \begin{tabularx}{\textwidth}{|lX|}
    \hline
    \centering Placeholder string &
    Automatically replaced by                          \\
    \hline
    \hline
    \centering \texttt{@expname@} &
    The experiment name (from \texttt{cn\_exp} in the namelist) \\
    \hline
    \centering \texttt{@freq@}    &
    Output frequency (from XML attribute \texttt{output\_freq})     \\
    \hline
    \centering \texttt{@startdate@}        &
    Starting date of the simulation (from \texttt{nn\_date0} in the restart or the namelist).
    \newline
    \verb?yyyymmdd?          format                   \\
    \hline
    \centering \texttt{@startdatefull@}    &
    Starting date of the simulation (from \texttt{nn\_date0} in the restart or the namelist).
    \newline
    \verb?yyyymmdd_hh:mm:ss? format                    \\
    \hline
    \centering \texttt{@enddate@}          &
    Ending date of the simulation   (from \texttt{nn\_date0} and \texttt{nn\_itend} in the namelist).
    \newline
    \verb?yyyymmdd?          format                    \\
    \hline
    \centering \texttt{@enddatefull@}      &
    Ending date of the simulation   (from \texttt{nn\_date0} and \texttt{nn\_itend} in the namelist).
    \newline
    \verb?yyyymmdd_hh:mm:ss? format                    \\
    \hline
  \end{tabularx}
  \label{tab:DIA_xios_filename_subs}
\end{table}

\noindent For example,
\begin{xmllines}
<file_definition>
   <file id="file66" name="myfile_@expname@_@startdate@_freq@freq@" output_freq="1d" >
</file_definition>
\end{xmllines}

\noindent with the namelist:
\begin{forlines}
cn_exp    = "ORCA2"
nn_date0  = 19891231
ln_rstart = .false.
\end{forlines}

\noindent will give the following file name radical: \textit{myfile\_ORCA2\_19891231\_freq1d}

%% =================================================================================================
\subsubsection{Other XML attributes set by \NEMO}

The values of some XML attributes (including \texttt{name\_suffix}, discussed in the previous subsection) are
automatically set by the \rou{set\_xmlatt} subroutine in \NEMO\ (\mdl{iom}). These attributes and their values are given
in \autoref{tab:DIA_xios_auto_xml}.
Any definition of these attributes in the XML files will be overwritten; by convention their values are set to
''auto'' (for strings) or ''0000'' (for integers), although this is not necessary.

\begin{table}
  \caption[XML attributes set automatically by \NEMO]{
           XIOS XML attributes that are set automatically by \NEMO, excluding \texttt{name} and \texttt{name\_suffix}}
  \begin{tabular}{|l|c|c|}
    \hline
    Tag family and \texttt{id} affected by automatic definition                                  &
    Attribute name                                                                      &
    Attribute value                                                                     \\
    of some of their attributes                                                         &
                                                                                        &
                                                                                        \\
    \hline
    \hline
    \texttt{field\_definition}                                                          &
    \texttt{freq\_op}                                                                   &
    \np{rn_rdt}{rn\_rdt}                                                                \\
    \hline
    \texttt{field}: SBC, SBC\_scalar, ABL                                               &
    \texttt{freq\_op}                                                                   &
    \np{rn_rdt}{rn\_rdt} $\times$ \np{nn_fsbc}{nn\_fsbc}                                \\
    \hline
    \texttt{field}: trendT\_even                                                        &
    \texttt{freq\_op}                                                                   &
    $2 \times$ \np{rn_rdt}{rn\_rdt}                                                     \\
    \hline
    \texttt{field}: trendT\_odd                                                         &
    \texttt{freq\_op}                                                                   &
    $2 \times$ \np{rn_rdt}{rn\_rdt}                                                     \\
                                                                                        &
    \texttt{freq\_offset}                                                               &
    $-1$                                                                                \\
    \hline
    \texttt{zoom\_domain}: EqT, EqU, EqW                                                 &
    \texttt{jbegin}, \texttt{ni},                                                       &
    set according to the grid                                                           \\
    \hline
    \texttt{zoom\_domain}: TAO, RAMA and PIRATA moorings                                 &
    \texttt{ibegin}, \texttt{jbegin},                                                   &
    set according to the grid                                                           \\
    \hline
  \end{tabular}
  \label{tab:DIA_xios_auto_xml}
\end{table}

%% =================================================================================================
\subsubsection{Advanced use of XIOS functionalities}
\label{subsec:DIA_io_xios_adv}

XIOS can do far more than just gather and write output. Importantly, it can perform computations with the
fields it receives providing opportunities to create derived quantities without burdening the model simulation.
This section provides a few illustrations of the possibilities:

\begin{enumerate}
\item Using algebraic expressions

A new diagnostic can be derived from existing diagnostics, either in the file definition:

\begin{xmllines}
<file_definition>
   <file id="derived_vars" output_freq="1d" >
      <field field_ref="sst"  name="tosK"  unit="degK" > sst + 273.15 </field>
      <field field_ref="taum" name="taum2" unit="N2/m4" long_name="square of wind stress module" > taum * taum </field>
      <field field_ref="qt"   name="stupid_check" > qt - qsr - qns </field>
   </file>
</file_definition>
\end{xmllines}

or in the field definition:

\begin{xmllines}
<field_definition>
   <field id="sst2" field_ref="sst" long_name="square of sea surface temperature" unit="degC2" > sst * sst </field>
</field_definition>
\end{xmllines}

and then referenced in the file definition:

\begin{xmllines}
<file_definition>
   <file id="derived_vars" output_freq="1d" >
      <field field_ref="sst2" > sst2 </field>
   </file>
</file_definition>
\end{xmllines}

% NOTE: Is this still the case?
Note that in this case, simply adding "\xmlcode{<field field_ref="sst2" />}" to the file definition
would not work since "sst2" would not be evaluated.

\item Use of the ``@'' function: example 1, weighted temporal average

The ``@'' function can be used in algebraic expressions to chain temporal operations.
In this example, it is used to output a weighted temporal average of the temperature
(with the time-varying layer thickness as the weight).

The product of the two quantities is first added as a new variable in the field definition:

\begin{xmllines}
<field_definition operation="average" freq_op="1ts">
   <field id="toce_e3t" long_name="temperature * e3t" unit="degC*m" grid_ref="grid_T_3D" >toce * e3t</field>
</field_definition>
\end{xmllines}

The \texttt{operation="average"} and \texttt{freq\_op="1ts"} attributes specify the temporal operation and its
sampling frequency- \texttt{toce} and \texttt{e3t} will be used to calculate \texttt{toce\_e3t} for every timestep,
which will then be averaged over a time period set by the \texttt{output\_freq} or \texttt{freq\_op} attributes
(the latter is given priority) in the file definition. For example:

\begin{xmllines}
<file_definition>
   <file_group id="5d" output_freq="5d"  output_level="10" enabled=".true." >  <!-- 5d files -->
      <file id="file1" name_suffix="_grid_T" description="ocean T grid variables" >
         <!-- 5-day averages  -->
         <field field_ref="toce" />
         <!-- 1-day averages, output once every 5 days -->
         <field field_ref="toce" freq_op="1d" name="toce_1d" />
      </file>
   </file_group>
</file_definition>
\end{xmllines}

To produce a 5-day weighted average, the 5-day average of the weighted temperature
(\texttt{@toce\_e3t}) must be divided by that of the layer thickness (\texttt{@e3t}):

\begin{xmllines}
<file_definition>
   <file_group id="5d" output_freq="5d"  output_level="10" enabled=".true." >  <!-- 5d files -->
      <file id="file1" name_suffix="_grid_T" description="ocean T grid variables" >
         <field field_ref="toce" operation="instant" freq_op="5d" > @toce_e3t / @e3t </field>
      </file>
   </file_group>
</file_definition>
\end{xmllines}

Normally, the \texttt{operation="average"} and \texttt{freq\_op="1ts"} attributes inherited from the field
definition would be overwritten by the \texttt{operation="instant"} and \texttt{freq\_op="5d"} attributes
in the file definition. This would result in instantaneous output (data for one timestep) every 5 days.

The ``@'' function overrides this behaviour so that instead, the temporal operations are applied separately.
Specifically, it indicates that the temporal operation for the adjacent field should be performed before evaluating
the algebraic expression it is part of. This results in 2 chained temporal operations:

- Temporal operation 1: the operation type and sampling frequency are set by the \texttt{operation} and
\texttt{freq\_op} attributes in the field definition, while the temporal period of the operation is set
by the \texttt{freq\_op} attribute in the file definition.

- Temporal operation 2: the operation type, sampling frequency and temporal period are specified
by the \texttt{operation}, \texttt{freq\_op} and \texttt{output\_freq} attributes in the file definition.

For the above thickness-weighted temperature example, the following operations occur in order:

- \texttt{toce\_e3t} is calculated from \texttt{toce} and \texttt{e3t} for every timestep

- 5-day averages of \texttt{toce\_e3t} and \texttt{e3t} are calculated (temporal operation 1)

- The weighted average, \texttt{toce\_e3t / e3t}, is calculated using these 5-day averages

- The instantaneous value of this expression is output every 5 days (temporal operation 2)