6.6. Monitor ROCOTO Run

Click here to view full rocoto documentation on GitHub

6.6.1. Using command line

You can use Rocoto commands with arguments to check the status of your experiment.

Start or continue a run:

rocotorun -d /path/to/workflow/database/file -w /path/to/workflow/xml/file

Check the status of the workflow:

rocotostat -d /path/to/workflow/database/file -w /path/to/workflow/xml/file [-c YYYYMMDDCCmm,[YYYYMMDDCCmm,...]] [-t taskname,[taskname,...]] [-s] [-T]

Note

YYYYMMDDCCmm = YearMonthDayCycleMinute …where mm/Minute is ’00’ for all cycles currently.

Check the status of a job:

rocotocheck -d /path/to/workflow/database/file -w /path/to/workflow/xml/file -c YYYYMMDDCCmm -t taskname

Force a task to run (ignores dependencies - USE CAREFULLY!):

rocotoboot -d /path/to/workflow/database/file -w /path/to/workflow/xml/file -c YYYYMMDDCCmm -t taskname

Rerun task(s):

rocotorewind -d /path/to/workflow/database/file -w /path/to/workflow/xml/file -c YYYYMMDDCCmm -t taskname

(If job is currently queued or running rocoto will kill the job. Run rocotorun afterwards to fire off rewound task.)

Set a task to complete (overwrites current state):

rocotocomplete -d /path/to/workflow/database/file -w /path/to/workflow/xml/file -c YYYYMMDDCCmm -t taskname

(Will not kill queued or running job, only update status.)

Several dates and task names may be specified in the same command by adding more -c and -t options. However, lists are not allowed.

6.6.2. Use ROCOTO viewer

An alternative approach is to use a GUI that was designed to assist with monitoring global workflow experiments that use ROCOTO. It can be found under the workflow folder in global-workflow.

6.6.2.1. Usage

./rocoto_viewer.py -d /path/to/workflow/database/file -w /path/to/workflow/xml/file

Note

Note 1: Terminal/window must be wide enough to display all experiment information columns, viewer will complain if not.

Note 2: The viewer requires the full path to the database and xml files if you are not in your EXPDIR when you invoke it.

Note 3: Only TERM=xterm is supported. You may wish to create a shell function to switch automatically if you are in a different terminal:

Bash example:

function rv {
  oldterm=${TERM};
  export TERM='xterm';
  ${PATH_TO_VIEWER}/rocoto_viewer.py $@;
  export TERM=${oldterm};
}

6.6.2.2. What the viewer shows

Sample output from Rocoto viewer

The figure above shows a sample output from a Rocoto viewer for a running experiment. Where:

First column: cycle (YYYYMMDDCCmm, YYYY=year, MM=month, DD=day, CC=cycle hour, mm=minute)

Second column: task name (a “<” symbol indicates a group/meta-task, click “x” when meta-task is selected to expand/collapse)

Third column: job ID from scheduler

Fourth column: job state (QUEUED, RUNNING, SUCCEEDED, FAILED, or DEAD)

Fifth column: exit code (0 if all ended well)

Sixth column: number of tries/attempts to run job (0 when not yet run or just rewound, 1 when run once successfully, 2+ for multiple tries up to max try value where job is considered DEAD)

Seventh column: job duration in seconds

6.6.2.3. How to navigate the viewer

The rocoto viewer accepts both mouse and keyboard inputs. Click “h” for help menu and more options.

Available viewer commands:

c = get information on selected job
r = rewind (rerun) selected job, group, or cycle
R = run rocotorun
b = boot (forcibly run) selected job or group
-> = right arrow key, advance viewer forward to next cycle
<- = left arrow key, advance viewer backward to previous cycle
Q = quit/exit viewer

Advanced features:

Select multiple tasks at once

Click “Enter” on a task to select it, click on other tasks or use the up/down arrows to move to other tasks and click “Enter” to select them as well.

When you next choose “r” for rewinding the pop-up window will now ask if you are sure you want to rewind all those selected tasks.

Rewind entire group or cycle

Group - While group/metatask is collapsed (<) click “r” to rewind whole group/metatask.

Cycle - Use up arrow to move selector up past the first task until the entire left column is highlighted. Click “r” and the entire cycle will be rewound.