\section{Reduced data checks}
\label{sec:red_checks}
% -----------------------------------------------------------------------
\subsection{Overview}
\label{sec:red_checks:overview}
% -----------------------------------------------------------------------
Link to reduced checks:
\begin{itemize}
    \item test
    \item\href{https://docs.google.com/spreadsheets/d/1zvU_XFA1ZOJE111qZKiav7v6ptYveWpDkMHjdhiN06M/edit?gid=541545005#gid=541545005}{RED-ONLINE-NIRPS-HE}
    \item\href{https://docs.google.com/spreadsheets/d/1zvU_XFA1ZOJE111qZKiav7v6ptYveWpDkMHjdhiN06M/edit?gid=279548254#gid=279548254}{RED-ONLINE-NIRPS-HA}
\end{itemize}

Check everyday (for the previous day) after running the manual trigger. 

Do not check todays values (or run the manual trigger for today) 

Check for any columns with a FALSE. 

If there is not test for today and yesterday, please \hyperref[sec:how_to:re_run]{run the red checks again}. 

If something is FALSE, please run the reduced checks again. 

Any FALSES that someone has told you to ignore should be commented here:
\begin{itemize}
    \item \href{https://docs.google.com/spreadsheets/d/1zvU_XFA1ZOJE111qZKiav7v6ptYveWpDkMHjdhiN06M/edit?gid=68390787#gid=68390787}{Comments-NIRPS-HE}
    \item \href{https://docs.google.com/spreadsheets/d/1zvU_XFA1ZOJE111qZKiav7v6ptYveWpDkMHjdhiN06M/edit?gid=346030560#gid=346030560}{Comments-NIRPS-HA}
\end{itemize}


If you enter the wrong night you cannot undo this yourself, please see \hyperref[sec:manual_trigger:undo]{this section} on how to undo this.

% -----------------------------------------------------------------------
\newpage
\subsection{Current tests}
\label{sec:red_checks:current_tests}
% -----------------------------------------------------------------------
These tests are described in detail in the following sections:

\begin{itemize}
    \item \hyperref[sec:red_checks:blank]{BLANK}: Always True if night has been run 
    \item \hyperref[sec:red_checks:has_obsdir]{HAS\_OBSDIR}: Check that manual trigger symlinked observation directories 
    \item \hyperref[sec:red_checks:apero_calib]{APERO\_CALIB}: Check that symlinked directories have all required calibrations for APERO 
    \item \hyperref[sec:red_checks:manual_start]{MANUAL\_START}: Check that the manual trigger was started for this observation directory 
    \item \hyperref[sec:red_checks:manual_end]{MANUAL\_END}: Check that the manual trigger got to the end 
    \item \hyperref[sec:red_checks:apero_start]{APERO\_START}: Check that APERO+LBL processing (inside manual trigger) was started 
    \item \hyperref[sec:red_checks:apero_end]{APERO\_END}: Check that APERO+LBL processing(inside manual trigger) ended 
    \item \hyperref[sec:red_checks:ari_start]{ARI\_START}: Check that LBL (inside manual trigger) was started 
    \item \hyperref[sec:red_checks:ari_end]{ARI\_END}: Check that LBL (inside manual trigger) ended 
    \item \hyperref[sec:red_checks:pixel_shifts]{PIXEL\_SHIFTS}: Check for pixel shifts in pp files 
    \item \hyperref[sec:red_checks:excess_modal]{EXCESS\_MODAL}: Check excess modal noise 
    \item \hyperref[sec:red_checks:low_snr]{LOW\_SNR}: Check for low SNR
    \item \hyperref[sec:red_checks:prev_reduc]{PREV\_REDUC}: Check whether the last 7 days has any raw data that has not been reduced

\end{itemize}

% -----------------------------------------------------------------------
\newpage
\subsection{RED CHECK: BLANK}
\label{sec:red_checks:blank}
% -----------------------------------------------------------------------
\subsubsection{Overview}

This test is there to show that the reduced check ran at least once. Even if every other column is FALSE if the check was run for this date it should be True.

\subsubsection{Requirements}

None.

\subsubsection{What to do}

If FALSE there is a problem. 

Please contact the following immediately:

\begin{contacttable}
    \ccNeil{1} & \eeNeil \\
    \ccLison{} & \eeLison \\
    \ccEtienne{} & \eeEtienne \\
    \hline
\end{contacttable}

% -----------------------------------------------------------------------
\newpage
\subsection{RED CHECK: HAS\_OBSDIR}
\label{sec:red_checks:has_obsdir}
% -----------------------------------------------------------------------
\subsubsection{Overview}

This test will be FALSE if there is no observation directory in the apero profile’s raw data (i.e. the symlinks have not been created).

This test will be FALSE until the manual trigger has been run.

\subsubsection{Requirements}

\begin{itemize}
    \item \lstcode{BLANK = True}
    \item (All raw checks have passed) or (Failures verified as ignorable and are documented)
    \item Manual trigger has been run
\end{itemize}

\subsubsection{What to do}

Please verify that you are not checking today's value. This test should only be tested for yesterday (or earlier) observation dates.

If FALSE please \hyperref[sec:how_to:re_run]{re-run the check} with \lstcode{--test=HAS_OBSDIR}. 

If still FALSE please try \hyperref[sec:manual_trigger]{re-running the manual trigger} with \lstcode{--only_links=True} to make sure we have files sym-linked correctly (followed by re-running the checks).

If still FALSE and all requirements above are satisfied, please contact the following people stating that you have re-run the manual trigger with the \lstcode{--only_links} argument and the HAS\_OBSDIR is still failing.

\begin{contacttable}
    \ccNeil{1} & \eeNeil \\
    \ccLison{} & \eeLison \\
    \ccEtienne{} & \eeEtienne \\
    \hline
\end{contacttable}

% -----------------------------------------------------------------------
\newpage
\subsection{RED CHECK: APERO\_CALIB}
\label{sec:red_checks:apero_calib}
% -----------------------------------------------------------------------
\subsubsection{Overview}

This runs the calibration check from apero\_precheck. 

It essentially checks for the correct number of calibration files (as defined inside APERO).

\subsubsection{Requirements}

\begin{itemize}
    \item \lstcode{BLANK = True}
    \item \lstcode{HAS_OBSDIR = True}
    \item (All raw checks have passed) or (Failures verified as ignorable and are documented)
    \item Manual trigger has been run
\end{itemize}

\subsubsection{What to do}

Please verify that you are not checking today's value. This test should only be tested for yesterday (or earlier) observation dates.

If FALSE please \hyperref[sec:how_to:re_run]{re-run the check} with \lstcode{--test=APERO_CALIB}. 

If you rejected (or someone else) rejected all calibrations for this night then this test is expected to be FALSE - please add a comment on the checks page saying which calibrations were rejected (you can just say `all calibrations were rejected' if this was the case and mention that this was the reason \lstcode{APERO_CALIB} failed.

If still FALSE please try \hyperref[sec:manual_trigger]{re-running the manual trigger} with \lstcode{--only_links=True} to make sure we have files sym-linked correctly (followed by re-running the checks).

Please then \hyperref[sec:how_to:eso_archives]{go to the ESO archive} and produce a full list of calibrations for this observation directory.

If still FALSE and all requirements above are satisfied, please contact the following people stating that you have re-run the manual trigger with the \lstcode{--only_links} argument and the APERO\_CALIB is still failing.

\begin{contacttable}
    \ccNeil{1} & \eeNeil \\
    \ccLison{} & \eeLison \\
    \ccEtienne{} & \eeEtienne \\
    \hline
\end{contacttable}

% -----------------------------------------------------------------------
\newpage
\subsection{RED CHECK: MANUAL\_START}
\label{sec:red_checks:manual_start}
% -----------------------------------------------------------------------
\subsubsection{Overview}

This flags whether the manual trigger has been started at least once for this observation date.

\subsubsection{Requirements}

\begin{itemize}
    \item \lstcode{BLANK = True}
    \item \lstcode{HAS_OBSDIR = True}
    \item (All raw checks have passed) or (Failures verified as ignorable and are documented)
    \item Manual trigger has been run
\end{itemize}

\subsubsection{What to do}

If FALSE please \hyperref[sec:how_to:re_run]{re-run the check} with \lstcode{--test=MANUAL_START}. 

Please try \hyperref[sec:manual_trigger]{re-running the manual trigger} and look for errors in the code running. 

If still FALSE and all requirements above are satisfied, please contact the following people stating that you have re-run the manual trigger and the MANUAL\_START reduced check is still failing.

\begin{contacttable}
    \ccNeil{1} & \eeNeil \\
    \ccLison{} & \eeLison \\
    \ccEtienne{} & \eeEtienne \\
    \hline
\end{contacttable}

% -----------------------------------------------------------------------
\newpage
\subsection{RED CHECK: MANUAL\_END}
\label{sec:red_checks:manual_end}
% -----------------------------------------------------------------------
\subsubsection{Overview}

If FALSE this normally means the manual trigger crashed before finishing completely.

\subsubsection{Requirements}

\begin{itemize}
    \item \lstcode{BLANK = True}
    \item \lstcode{HAS_OBSDIR = True}
    \item \lstcode{MANUAL_START = True}
    \item (All raw checks have passed) or (Failures verified as ignorable and are documented)
    \item Manual trigger has been run
\end{itemize}

\subsubsection{What to do}

If FALSE please \hyperref[sec:how_to:re_run]{re-run the check} with \lstcode{--test=MANUAL\_END}. 

Please try \hyperref[sec:manual_trigger]{re-running the manual trigger} and look for errors in the code running. 

If still FALSE and all requirements above are satisfied, please contact the following people stating that you have re-run the manual trigger and the MANUAL\_END reduced check is still failing.

\begin{contacttable}
    \ccNeil{1} & \eeNeil \\
    \ccLison{} & \eeLison \\
    \ccEtienne{} & \eeEtienne \\
    \hline
\end{contacttable}

% -----------------------------------------------------------------------
\newpage
\subsection{RED CHECK: APERO\_START}
\label{sec:red_checks:apero_start}
% -----------------------------------------------------------------------
\subsubsection{Overview}

If FALSE this normally means the manual trigger has crashed before apero processing could start.

\subsubsection{Requirements}

\begin{itemize}
    \item \lstcode{BLANK = True}
    \item \lstcode{HAS_OBSDIR = True}
    \item \lstcode{MANUAL_START = True}
    \item (All raw checks have passed) or (Failures verified as ignorable and are documented)
    \item Manual trigger has been run
\end{itemize}

\subsubsection{What to do}

If FALSE please \hyperref[sec:how_to:re_run]{re-run the check} with \lstcode{--test=APERO_START}. 

If still FALSE please try \hyperref[sec:manual_trigger]{re-running the manual trigger} with \lstcode{--only_apero_process=True} and look for errors in running code. 

Please then email the following people with the copy and paste of what happened in the manual trigger:

\begin{contacttable}
    \ccNeil{1} & \eeNeil \\
    \ccLison{} & \eeLison \\
    \ccEtienne{} & \eeEtienne \\
    \hline
\end{contacttable}

% -----------------------------------------------------------------------
\newpage
\subsection{RED CHECK: APERO\_END}
\label{sec:red_checks:apero_end}
% -----------------------------------------------------------------------
\subsubsection{Overview}

If FALSE normally means the manual trigger has crashed before apero processing finished.

\subsubsection{Requirements}

\begin{itemize}
    \item \lstcode{BLANK = True}
    \item \lstcode{HAS_OBSDIR = True}
    \item \lstcode{MANUAL_START = True}
    \item \lstcode{APERO_START = True}
    \item (All raw checks have passed) or (Failures verified as ignorable and are documented)
    \item Manual trigger has been run
\end{itemize}

\subsubsection{What to do}

If FALSE please \hyperref[sec:how_to:re_run]{re-run the check} with \lstcode{--test=APERO_END}. 

If still FALSE please try \hyperref[sec:manual_trigger]{re-running the manual trigger} with \lstcode{--only_apero_process=True} and look for errors in running code. 

Please then \hyperref[sec:how_to:find_log]{locate the log file} for \lstpath{apero_processing.py} and email the following people with a copy of the log (and/or the location of the log file on disk):

\begin{contacttable}
    \ccNeil{1} & \eeNeil \\
    \ccLison{} & \eeLison \\
    \ccEtienne{} & \eeEtienne \\
    \hline
\end{contacttable}




% -----------------------------------------------------------------------
\newpage
\subsection{RED CHECK: ARI\_START}
\label{sec:red_checks:ari_start}
% -----------------------------------------------------------------------
\subsubsection{Overview}

If FALSE this normally means the manual trigger has crashed before ARI was started.

\subsubsection{Requirements}

\begin{itemize}
    \item \lstcode{BLANK = True}
    \item \lstcode{HAS_OBSDIR = True}
    \item \lstcode{MANUAL_START = True}
    \item (All raw checks have passed) or (Failures verified as ignorable and are documented)
    \item Manual trigger has been run
\end{itemize}

\subsubsection{What to do}

If FALSE please \hyperref[sec:how_to:re_run]{re-run the check} with \lstcode{--test=ARI_START}. 

If still FALSE please try \hyperref[sec:manual_trigger]{re-running the manual trigger} with \lstcode{--only_ari=True} and look for errors in running code. 

Please then email the following people with the copy and paste of what happened in the manual trigger:

\begin{contacttable}
    \ccNeil{1} & \eeNeil \\
    \ccLison{} & \eeLison \\
    \ccEtienne{} & \eeEtienne \\
    \hline
\end{contacttable}

% -----------------------------------------------------------------------
\newpage
\subsection{RED CHECK: ARI\_END}
\label{sec:red_checks:ari_end}
% -----------------------------------------------------------------------
\subsubsection{Overview}

If FALSE this normally means the manual trigger has crashed before ARI finished.

\subsubsection{Requirements}

\begin{itemize}
    \item \lstcode{BLANK = True}
    \item \lstcode{HAS_OBSDIR = True}
    \item \lstcode{MANUAL_START = True}
    \item \lstcode{ARI_START = True}
    \item (All raw checks have passed) or (Failures verified as ignorable and are documented)
    \item Manual trigger has been run
\end{itemize}

\subsubsection{What to do}

If FALSE please \hyperref[sec:how_to:re_run]{re-run the check} with \lstcode{--test=ARI_END}. 

If still FALSE please try \hyperref[sec:manual_trigger]{re-running the manual trigger} with \lstcode{--only_ari=True} and look for errors in running code. 


Please then \hyperref[sec:how_to:find_log]{locate the log file} for \lstpath{apero_ri.py} and email the following people with a copy of the log (and/or the location of the log file on disk):

\begin{contacttable}
    \ccNeil{1} & \eeNeil \\
    \ccLison{} & \eeLison \\
    \ccEtienne{} & \eeEtienne \\
    \hline
\end{contacttable}

% -----------------------------------------------------------------------
\newpage
\subsection{RED CHECK: PIXEL\_SHIFTS}
\label{sec:red_checks:pixel_shifts}
% -----------------------------------------------------------------------
\subsubsection{Overview}

Checks for pixel shifts in the preprocessed files. 

Specifically checks `DETOFFDX' and `DETOFFDY' header keys to look for any non-zero values 

\subsubsection{Requirements}

\begin{itemize}
    \item \lstcode{BLANK = True}
    \item \lstcode{HAS_OBSDIR = True}
    \item \lstcode{MANUAL_START = True}
    \item \lstcode{APERO_START = True}
    \item \lstcode{APERO_END = True}
    \item (All raw checks have passed) or (Failures verified as ignorable and are documented)
    \item Manual trigger has been run
\end{itemize}

\subsubsection{What to do}

If FALSE please \hyperref[sec:how_to:re_run]{re-run the check} with \lstcode{--test=PIXEL_SHIFTS}. 

If still FALSE please email:

\begin{contacttable}
    \ccEtienne{1} & \eeEtienne \\
    \ccNeil{} & \eeNeil \\
    \ccLison{} & \eeLison \\
    \hline
\end{contacttable}

% -----------------------------------------------------------------------
\newpage
\subsection{RED CHECK: EXCESS\_MODAL}
\label{sec:red_checks:excess_modal}
% -----------------------------------------------------------------------
\subsubsection{Overview}

Test for excess modal noise in telluric stars. It checks for tcorr files of vetted telluric stars and computes the rms of the pixel to pixel vs the rms with a stride a 20 pixels on a sample order in H band. The pixel to pixel rms should be smaller than the rms with a stride of 20 pixels. The pixel-to-pixel samples photon noise while the other one samples the modal noise. The test is passed if the pixel-to-pixel rms is smaller than the modal noise rms by a known threshold.

\subsubsection{Requirements}

\begin{itemize}
    \item \lstcode{BLANK = True}
    \item \lstcode{HAS_OBSDIR = True}
    \item \lstcode{MANUAL_START = True}
    \item \lstcode{APERO_START = True}
    \item \lstcode{APERO_END = True}
    \item (All raw checks have passed) or (Failures verified as ignorable and are documented)
    \item Manual trigger has been run
    \item Hot star observations (should be True always if no hot stars were observed this night)
\end{itemize}

\subsubsection{What to do}

If FALSE please \hyperref[sec:how_to:re_run]{re-run the check} with \lstcode{--test=EXCESS_MODAL}. 

If still FALSE please email:

\begin{contacttable}
    \ccEtienne{1} & \eeEtienne \\
    \ccNeil{} & \eeNeil \\
    \ccLison{} & \eeLison \\
    \hline
\end{contacttable}

% -----------------------------------------------------------------------
\newpage
\subsection{RED CHECK: LOW\_SNR}
\label{sec:red_checks:low_snr}
% -----------------------------------------------------------------------
\subsubsection{Overview}

This test checks whether any science files have \lstcode{SNR < 10} in extracted order 15 and 60.

\subsubsection{Requirements}

\begin{itemize}
    \item \lstcode{BLANK = True}
    \item \lstcode{HAS_OBSDIR = True}
    \item \lstcode{MANUAL_START = True}
    \item \lstcode{APERO_START = True}
    \item \lstcode{APERO_END = True}
    \item (All raw checks have passed) or (Failures verified as ignorable and are documented)
    \item Manual trigger has been run
\end{itemize}

\subsubsection{What to do}

If FALSE please \hyperref[sec:how_to:re_run]{re-run the check} with \lstcode{--test=LOW_SNR}. 

If still FALSE you should check ARI for previous observations of this object.

First check the \href{https://docs.google.com/spreadsheets/d/1aXSy0CgM7l3RfZFDoQLs1i8k3LgATB01Tygw7eiYBMo/edit?usp=sharing}{APERO object flags spreadsheet}. If the object is in this list with the \lstcode{CHECK=LOW_SNR} then you can \hyperref[sec:how_to:override_check]{override the value with the APERO check override code} with \lstcode{--test=LOW_SNR}.

\note{This object is always below or around an SNR of 10}

If the previous observations are always below or around 10, you should bring this up at the next meeting but \hyperref[sec:how_to:override_check]{override the value with the APERO check override code} with \lstcode{--test=LOW_SNR}. Please state that you are overriding as the object (state the object name) is usually around this SNR. If deemed okay remember to add this to the \href{https://docs.google.com/spreadsheets/d/1aXSy0CgM7l3RfZFDoQLs1i8k3LgATB01Tygw7eiYBMo/edit?usp=sharing}{APERO object flags spreadsheet} after the meeting.

If the weather was terrible or another issue was mentioned in the log that could explain the low SNR (\hyperref[sec:how_to:find_observer]{this should be reported in the log here}) you should bring this up at the next meeting but \hyperref[sec:how_to:override_check]{override the value with the APERO check override code} with \lstcode{--test=LOW_SNR}. Please state that you are overriding as the object (state the object name) due to bad weather conditions (or the other reason for the low SNR). Please then \hyperref[sec:how_to:apero_reject]{reject this file} so it is not used in future reductions.

If the previous observations are usually way above 10 and the weather wasn't terrible you should report this to the following people, stating that the SNR was flagged as being well below average (give the object name and the SNR usually found and the SNR for this observation):

\begin{contacttable}
    \ccCurrentObserver{1} & \eeCurrentObserver \\
    \ccTelescope{} & \eeTelescope \\
    \ccDNOS{} & \eeDNOS \\
    \ccNeil{} & \eeNeil \\
    \ccLison{} & \eeLison \\
    \ccFred{} & \eeFred \\
    \ccEtienne{} & \eeEtienne \\
    \ccLucile{} & \eeLucile \\
    \ccRomain{} & \eeRomain \\
    \hline
\end{contacttable}

Please then \hyperref[sec:how_to:apero_reject]{reject this file} so it is not used in future reductions.

% -----------------------------------------------------------------------
\newpage
\subsection{RED CHECK: PREV\_REDUC}
\label{sec:red_checks:prev_reduc}
% -----------------------------------------------------------------------
\subsubsection{Overview}

This test checks whether there are any raw files without any reduced products. 

Specifically, every raw file must have a `pp file' for the previous 7 days. 

Every day after this failure (for up to 7 days) will fail until this is fixed. 


\subsubsection{Requirements}

\begin{itemize}
    \item \lstcode{BLANK = True}
    \item \lstcode{HAS_OBSDIR = True}
    \item \lstcode{MANUAL_START = True}
    \item \lstcode{APERO_START = True}
    \item \lstcode{APERO_END = True}
    \item (All raw checks have passed) or (Failures verified as ignorable and are documented)
    \item Manual trigger has been run
\end{itemize}

\subsubsection{What to do}

If FALSE please \hyperref[sec:how_to:re_run]{re-run the check} with \lstcode{--test=PREV_REDUC}. 

If still FALSE you should get a list of files that are missing. 

Please try \hyperref[sec:manual_trigger]{re-running the manual trigger} on this specific observation directory (seen in the file path). 

If still False after re-running the manual trigger please check the DPR TYPE of the files that are missing. 

This can be done by running the following command: 
\begin{codebox}[label={lst:dfit_fitsort}]{Bash}
\begin{lstlisting}[language=Bash]
dfits {filename} | fitsort dpr.type  
\end{lstlisting}
\end{codebox}

Also \hyperref[sec:how_to:find_log]{please locate the log file} for the preprocessing of this file (\lstpath{apero_preprocess} – this is most likely where the error came from. 

In rare cases the log will not exist, but in most cases it should!

\note{Hint for locating the correct preprocesing log file}

You need to first find the APERO group directory inside the \lstpath{msg/processing/} directory (this will most likely be the most recent APERO group) you can then change into the observation directory, then there may be a log of logs so using a grep works well to locate the log file: 

\begin{codebox}[label={lst:pp_log_hint}]{Bash}
\begin{lstlisting}[language=Bash]
grep -n {file identifier} *preprocess* 
\end{lstlisting}
\end{codebox}

Where \lstcode{file identifier} is the \lstpath{NIRPS-YYYY-MM-DDThh_mm_ss_sss} part of the filename (without \lstpath{_pp... .fits}) 

After re-running the manual trigger, running the dfits command and finding the preprocessing log file for this file/ these files. Please email the list of files (and the log from the reduced check) to the following people:

\begin{contacttable}
    \ccNeil{1} & \eeNeil \\
    \ccEtienne{} & \eeEtienne \\
    \ccLison{} & \eeLison \\
    \hline
\end{contacttable}
