\section{Manual trigger}
\label{sec:manual_trigger}

\subsection{Overview}
\label{sec:manual_trigger:overview}

The manual trigger does the following: 

\begin{enumerate}
    \item Runs raw and reduced checks before and after most stages of the manual trigger 
    \item Creates symlinks from the raw download directory to the apero profile raw directory (depending on the yaml file used) 
    \begin{itemize}
        \item  For NIRPS-HE (online): 
        \begin{itemize}
            \item From \lstpath{/nirps_raw/nirps/raw-data/nirps_he/}
            \item To \lstpath{/cosmos99/nirps/apero-data/nirps_he_online/raw/}
        \end{itemize}
        \item  For NIRPS-HA (online): 
        \begin{itemize}
            \item From \lstpath{/nirps_raw/nirps/raw-data/nirps_ha/}
            \item To \lstpath{/cosmos99/nirps/apero-data/nirps_ha_online/raw/}
        \end{itemize}
        \item  For NIRPS-HE (offline): 
        \begin{itemize}
            \item From \lstpath{/nirps_raw/nirps/raw-data/nirps_he/}
            \item To \lstpath{/cosmos99/nirps/apero-data/nirps_he_offline/raw/}
        \end{itemize}
        \item  For NIRPS-HA (offline): 
        \begin{itemize}
            \item From \lstpath{/nirps_raw/nirps/raw-data/nirps_ha/}
            \item To \lstpath{/cosmos99/nirps/apero-data/nirps_ha_offline/raw/}
        \end{itemize}
    \end{itemize}

    \item Runs \lstcode{apero_processing.py} (with either the \lstcode{online_run.ini} or \lstcode{offline_run.ini} depending on the yaml file used).

    \item Creates or updates an `objects' directory with sym-links to each astrophysical object (using \lstcode{apero_get.py}) 

    \item Runs the apero reduction interface (ARI) - using \lstcode{apero_ri.py}

\end{enumerate}

Note you can stop a manual trigger and `undo` most of its actions. Please see \hyperref[sec:manual_trigger:undo]{this section}.


To run the manual trigger you use the following commands. 

\subsubsection{Step 1/6}

\hyperref[sec:how_to:log_in]{Log in to the UdeM machine}. 

\subsubsection{Step 2/6} 

\hyperref[sec:how_to:use_screen]{Log into the correct screen} (or create one) 

You can check this by using the command: \lstcode{screen -list}.

And look for the appropriately named screen (normally something like \lstcode{nirps_he_online}) 

If the screen is attached please contact: 

\begin{contacttable}
    \ccNeil{1} & \eeNeil \\
    \ccEtienne{} & \eeEtienne \\
    \ccLison{} & \eeLison \\
    \hline
\end{contacttable}


\subsubsection{Step 3/6}

\hyperref[sec:how_to:load_apero_profile]{Activate the conda environment} (if not already activated, if an APERO profile is activated \hyperref[sec:how_to:load_apero_profile]{make sure you check} it is the correct one!) 

\subsubsection{Step 4/6}

Change to the manual trigger directory by typing this command: 
\begin{codebox}[]{Bash}
\begin{lstlisting}[language=Bash]
manual_trigger
\end{lstlisting}
\end{codebox}


\subsubsection{Step 5/6}

Run the manual trigger script:

\begin{codebox}[]{Bash}
\begin{lstlisting}[language=Bash]
manual_trigger.py {yaml_file} --obsdir={OBSDIR}  
\end{lstlisting}
\end{codebox}

Where \lstcode{yaml_file} depends on what you are reducing (normally an online profile, see section \ref{sec:manual_trigger:online_red}). For offline profiles in section \ref{sec:manual_trigger:offline_red}.

Where \lstcode{OBSDIR} is the \lstpath{YYYY-MM-DD} you wish to run, note it can be a comma-separated list, if left out it will run for all nights, if mistyped it will try to run for this night and you’ll have to contact us to sort it out! 

Note unlike the raw and reduced checks you cannot use \lstcode{--yesterday} and \lstcode{--today} (though maybe we will add this in the future if people prefer this option). 

Note in normal operations you should be running for yesterday. Never reduce data for today without permission - observations come in at the end of the night - so observations for `today' are only normally available on disk `tomorrow'.

\subsubsection{Step 6/6}

Last step is to \hyperref[sec:how_to:use_screen]{detach the screen}.  

Please also remember to check back regularly as something may have gone wrong. After the manual trigger is completed, you can re-check the reduced checks, then check the apero errors, then fill out the daily form. 

\subsubsection{Optional arguments}

You can also add other arguments which may be required for how you want to use the manual trigger. 

\note{Only run one step}

You can only run one step by setting one of the following arguments to True (please don’t set multiple) 

\begin{itemize}
    \item To only create the symlinks add the argument: \lstcode{--only_links=True}
    \item To only run apero processing add the argument: \lstcode{--only_apero_process=True}
    \item To only create or update the “objects” directory add the argument: \lstcode{--only_aperoget=True} 
    \item To only run ARI add the argument: \lstcode{--only_ari=True}
\end{itemize}

Note you cannot turn off the checks that happen inside the manual trigger.  


\note{Run a subset of the steps}

You can also run any combination of these with the following (setting them to True and False as needed):  
\begin{itemize}
    \item \lstcode{--links=True} or \lstcode{--links=False}
    \item \lstcode{--apero_process=True} or \lstcode{--apero_process=False}
    \item \lstcode{--get=True} or \lstcode{--get=False}
    \item \lstcode{--ari=True} or \lstcode{--ari=False}
\end{itemize}

Note the \lstcode{--only} commands basically set all other ones of these to False (please do not use both \lstcode{--only} and the command above together – choose one or the other!) 

\note{Run in test mode}

When the manual trigger crashes (or you wish to see what would happen with the arguments you have given) you can run in test mode first. 

This is done by adding the following to the arguments: \lstcode{--test=True}

\note{Run with a different run file}

As mentioned above, by default the manual trigger uses the run file defined in the yaml file given as an argument (linked to an APERO profile, e.g., NIRPS-HE online). 

Sometimes you will need to use another run file (see this section). 

For the manual trigger this can be done by using the argument: \lstcode{--run={RUN FILE}} 

% ---------------------------------------------------------------------------
\subsection{Online reductions}
\label{sec:manual_trigger:online_red}
% ---------------------------------------------------------------------------

The online reductions are the main reduction you will run. They are (in general) run on a nightly bases by the babysitters after all raw checks are either TRUE or dealt with (i.e. someone has said you can ignore any FALSEs and those FALSEs have been commented). 

\begin{itemize}
    \item For NIRPS-HE: 
    \begin{itemize}
        \item The apero profile to use is \lstcode{nirps_he_online}
        \item The manual trigger yaml file is \lstpath{nirps_he_online_udem.yaml}
        \item The \hyperref[sec:raw_checks]{raw checks} page is \href{https://docs.google.com/spreadsheets/d/1zvU_XFA1ZOJE111qZKiav7v6ptYveWpDkMHjdhiN06M/edit?gid=1741354113#gid=1741354113}{RAW-NIRPS-HE}
        \item The \hyperref[sec:red_checks]{reduced checks} page is \href{https://docs.google.com/spreadsheets/d/1zvU_XFA1ZOJE111qZKiav7v6ptYveWpDkMHjdhiN06M/edit?gid=541545005#gid=541545005}{RED-ONLINE-NIRPS-HE}
        \item The \hyperref[sec:how_to:false_check]{check comments} page is \href{https://docs.google.com/spreadsheets/d/1zvU_XFA1ZOJE111qZKiav7v6ptYveWpDkMHjdhiN06M/edit?gid=68390787#gid=68390787}{Comments-NIRPS-HE}
    \end{itemize}
    \item For NIRPS-HA: 
    \begin{itemize}
        \item The apero profile to use is \lstcode{nirps_ha_online}
        \item The manual trigger yaml file is \lstpath{nirps_ha_online_udem.yaml}
        \item The \hyperref[sec:raw_checks]{raw checks} page is \href{https://docs.google.com/spreadsheets/d/1zvU_XFA1ZOJE111qZKiav7v6ptYveWpDkMHjdhiN06M/edit?gid=1017212188#gid=1017212188}{RAW-NIRPS-HA}
        \item The \hyperref[sec:red_checks]{reduced checks} page is \href{https://docs.google.com/spreadsheets/d/1zvU_XFA1ZOJE111qZKiav7v6ptYveWpDkMHjdhiN06M/edit?gid=279548254#gid=279548254}{RED-ONLINE-NIRPS-HA}
        \item The \hyperref[sec:how_to:false_check]{check comments} page is \href{https://docs.google.com/spreadsheets/d/1zvU_XFA1ZOJE111qZKiav7v6ptYveWpDkMHjdhiN06M/edit?gid=346030560#gid=346030560}{Comments-NIRPS-HA}
    \end{itemize}
\end{itemize}

Normally you will be running for yesterday remember to add the argument: \lstcode{--obsdir=YYYY-MM-DD} 

Where \lstpath{YYYY-MM-DD} is yesterday (i.e. \lstpath{2024-12-25}) 

Note if for some reason a night was missed (e.g. you were waiting for a problem to be fixed) you can run multiple nights at once: \lstcode{--obsdir=2024-12-25,2025-12-26}

% ---------------------------------------------------------------------------
\subsection{Offline reductions}
\label{sec:manual_trigger:offline_red}
% ---------------------------------------------------------------------------

The offline reduction has a similar process to the online but with a few more important steps. 

\important{Do not run the offline reductions without being asked to by one of the following people:} 

\begin{contacttable}
    \ccNeil{1} & \eeNeil \\
    \ccEtienne{} & \eeEtienne \\
    \ccLison{} & \eeLison \\
    \hline
\end{contacttable}

\important{Note that you will be asked to reset all data. Please make sure you are in the correct profile – everything is deleted after this point!} 

To run the offline reduction first \hyperref[sec:manual_trigger:online_red]{follow steps 1-6 here}. We start from step 8 below to emphasize that you must do these 6 steps first. 

When the manual trigger has been completed (note this could be 3-4 weeks) then you need to do the following: 

\subsubsection{Step 7/10}

Run the `catch up' yaml file in the manual trigger 

e.g. for NIRPS-HA (once inside the manual trigger directory) 

\begin{codebox}[]{Bash}
\begin{lstlisting}[language=Bash]
python manual_trigger.py nirps_ha_offline_udem_catchup.yaml --since={date}
\end{lstlisting}
\end{codebox}

Where \lstcode{date} should be $\sim$2 days before you ran the original offline reduction (to make sure we have reduced all files with a sensible overlap). 

This will run the manual trigger for all days since the reduction was running – depending how long the offline reduction was running this may take a few days. 

\subsubsection{Step 8/10}

Once the catch-up is done \href{https://forms.gle/PxKL8BerBu1XNoRr7}{please re-fill out the the babysitter form}. 

\subsubsection{Step 9/10}

Verify with the NIRPS Montreal team that the data is good (usually this will be done across several group meetings). 

\subsubsection{Step 10/10}

Warn everyone that NIRPS-HE online or NIRPS-HA online data is about to be erased (maybe give people a day to copy any data they require immediately). 

It is important to let the current NIRPS online babysitter know that they should not reduced any observations for this APERO profile until this step is complete. 

Then and only then run the offline to online script. 

Warning this will delete all online data for the matching APERO profile – and if it goes wrong may lead to having to reset everything and start again leading to no data for 3-4 weeks – so be careful!  

Once this is completed please let everyone know that they can use the NIRPS online data and let the current babysitter know that they will need to run the online APERO profile for this for all dates since the catch-up script was run (so possibly a few days). This can be checked by looking at the reduced data directories: 

\begin{itemize}
    \item For NIRPS-HE online: \lstpath{/cosmos99/nirps/apero-data/nirps_he_online/red/}
    \item For NIRPS-HA online: \lstpath{/cosmos99/nirps/apero-data/nirps_ha_online/red/}
\end{itemize}

Missing directories need to be run by the online babysitter. 

% ---------------------------------------------------------------------------
\subsection{How to undo a manual trigger run}
\label{sec:manual_trigger:undo}
% ---------------------------------------------------------------------------

Sometimes we run the wrong night or just need to stop the manual trigger.

A simple \lstcode{Ctrl+C} isn't usually enough to stop everything the manual trigger wants to do, however you have to be careful doing multiple \lstcode{Ctrl+C} as you could corrupt a database.

To be safe do a \lstcode{Ctrl+C} once and wait to see what message comes up.

If you get a message as follows: 
\begin{bcodebox}[]{Bash}
\begin{lstlisting}[language=Bash, style=nonumbering]
18:06:49.342-!!|LOCSCI[00029]| SIGINT or CTRL-C detected. Exiting
\end{lstlisting}
\end{bcodebox}

Please \important{do not} press Ctrl+C again. And wait for the next line to say something similar to the following:
\begin{bcodebox}[]{Bash}
\begin{lstlisting}[language=Bash, style=nonumbering]
18:06:49.450-@!|LOCSCI[00029]| W[40-003-00005]: Recipe {recipe name} has NOT been successfully completed
18:06:49.450-@!|LOCSCI[00029]|  (59.214 seconds)
\end{lstlisting}
\end{bcodebox}

The manual trigger will then go to the next task which you may \lstcode{Ctrl+C} again, follow the same procedure of waiting if a similar line to above is shown, otherwise continue with more \lstcode{Ctrl+C} \important{one at a time}.

Eventually, you'll be out of the running codes and will have cancelled the manual trigger successfully.

Unfortunately there is a little cleaning up to do after this:
\begin{enumerate}
    \item \hyperref[sec:how_to:apero_remove]{Use apero remove on the night} you cancelled (unless you don't think you should remove all data from the night run - i.e. running a custom run)
    \item \hyperref[sec:manual_trigger:undo:mt_log]{Remove the entries from the manual trigger log}
    \item \hyperref[sec:how_to:re_run]{Re-run the reduced checks} again for this night
\end{enumerate}

\subsubsection{Removing the entries from the manual trigger log}
\label{sec:manual_trigger:undo:mt_log}

\hyperref[sec:how_to:find_log]{Locate the manual trigger log file}

Open it in \lstcode{nano} (or similar text editor of your choice). Find the lines that correspond to your night (they should be at the bottom of the file).
Remove these lines, e.g. for 2024-03-10, NIRPS-HA was run incorrectly, so the following lines need to be removed.

\begin{bcodebox}[]{Bash}
\begin{lstlisting}[language=Bash, style=nonumbering]
2025-03-10T17:33:55.446, nirps ha online, MANUAL_START, 2025-03-09, "None"
2025-03-10T17:34:53.459, nirps ha online, APERO_START, 2025-03-09, "None"
2025-03-10T18:14:52.506, nirps ha online, APERO_END, 2025-03-09, "None"
2025-03-10T18:21:27.136, nirps ha online, ARI_START, 2025-03-09, "None"
2025-03-10T18:28:29.498, nirps ha online, ARI_END, 2025-03-09, "None"
2025-03-10T18:28:45.990, nirps ha online, MANUAL_END, 2025-03-09, "None"
\end{lstlisting}
\end{bcodebox}

After this please make sure you re-run the checks for this night (to update the google sheet).


\subsubsection{Adding a non-real or future night}
\label{sec:manual_trigger:bad_night}

It is possible to add a non-real night (i.e. `2025-01') or a night in the future (i.e. `2095-01-01') these should be removed from two places - however you will not be able to do this.

Please contact the following people with the following instruction on how to correct this.


\begin{contacttable}
    \ccNeil{1} & \eeNeil \\
    \ccEtienne{} & \eeEtienne \\
    \ccLison{} & \eeLison \\
    \hline
\end{contacttable}

Instructions:

\begin{enumerate}
    \item Login to the apero-drs gmail account and remove the offending rows
    \item Go to the apero-checks directory (just type `apero\_check') and find the `local' directory
    \item Remove all backup databases where you removed a row
    \item Re-run the offending checks (with `--yesterday') to re-create the backup files
\end{enumerate}

Note that this can fail if another apero\_check is already running - what will happen is that you'll remove a row and it will come back - so you may have to re-follow these steps again. Once the backup database has been remove this should not be a problem (however you must have removed the row in the google sheet as well).

% ---------------------------------------------------------------------------
\subsection{Who to contact for help with the manual trigger}
\label{sec:manual_trigger:contact}
% ---------------------------------------------------------------------------

First, if you have a problem with the manual trigger, please check \hyperref[sec:troubleshooting]{the troubleshooting section}. 

If your problem is not described there, please email:  

\begin{contacttable}
    \ccNeil{1} & \eeNeil \\
    \ccEtienne{} & \eeEtienne \\
    \ccLison{} & \eeLison \\
    \hline
\end{contacttable}


% ---------------------------------------------------------------------------
\subsection{Offline to Online script}
\label{sec:manual_trigger:offline_to_online}
% ---------------------------------------------------------------------------

This script currently needs rewriting. 

Please contact the following people to know you have reached this step: 

\begin{contacttable}
    \ccNeil{1} & \eeNeil \\
    \ccEtienne{} & \eeEtienne \\
    \ccLison{} & \eeLison \\
    \hline
\end{contacttable}