Pegasus: Get your Wings: Pegasus Run Controls

Have you ever been in a situation where the run has started and you realize that you needed to add two more workers, or drop a couple of them? In such cases, you wait for the run to complete, make the modifications and then start the run again.

Let us take another scenario, a run is going on and is taking time. You want to know the status that how the workers are performing, how many CPUs are active and so on. Wouldn’t it be great if you could get to know the current run status?

Another rare situation is that a job appears to have hung and you want to terminate the job.  No, do not use CTRL-C, this should be strictly avoided. What should you do?

The good news is that Pegasus has a solution to all of these and more. The solution is Pegasus Run Controls commands which are provided by the go_pegasus file.

The go_pegasus file is used to specify run control commands in Pegasus. This file allows you to do modifications while your run is still going on. Let us understand in detail how it works.

1. You must create this file in the run directory.
2. Pegasus reads the contents of the file and takes appropriate actions.
3. Pegasus prints the contents to the user log and then the file is deleted.
4. You can create go_pegasus file again with new run control commands.

The Pegasus Run Control commands are:

• farm add
• farm drop
• farm status
• shutdown

farm add

The farm add command adds resources at runtime. It adds workers with the given resource specification.

Pegasus is a massively parallel tool. Runtime scales when more CPUs are added. The farm add interactive command can be used to add large number of CPUs as and when they are available to an existing Pegasus run. On the other hand, if CPUs are needed for other time critical jobs, they can be easily freed up by using farm drop.

The syntax of the command is:

farm add resource_specification

Let us understand it with the help of an example.

echo "farm add mode:local, mp:1, mt:2" > go_pegasus

INFO: Status: Runset 99% complete as of 2021-05-22 23:42:43 Elapsed: 00:36:51 CPU time[s]: 1033

Worker: Active CPUs  Total Mem[MB]  Active Mem[MB]  CPU time[s]

      1:    0 /   4           6309              75         1033

INFO:    Detected 'go_pegasus' file, executing commands from the file...
INFO:    Received add worker request with spec "mode:local, mp:1, mt:2".
INFO:    Worker 2 started on host sjf-dsgssvp-35 with 2 cpus

In the example above, one worker is added with 2-CPUs. The status is printed and the go_pegasus file is deleted.

Here is another farm add example where Pegasus acquires resources on the fly from LSF queue:

farm add mode:lsf, mp:1, mt:36,  dp_options:"-q pegasus"

This adds one worker with 36-CPUs.

farm drop 

The farm drop command removes resources at runtime. The workers to be dropped are specified with their IDs. Use this option when there is a long ‘tail’. The command does not drop the workers immediately. It drops the workers after the scheduled engines finish the execution. The syntax of the command is:

farm drop workerid1 [ workerid2  workerid3 ...]

The following example drops workers 3 and 4:

farm drop 3 4

farm status

The farm status command prints the current status of all the workers. The syntax of the command is:

farm status

For example, when you add farm status command in the go_pegasus file, the worker status is printed as:

Shutdown

Use the shutdown command when a Pegasus job appears to have hung. For example, when only 1 or 2 CPUs are active and CPU time increment is 1 or 2 seconds per 1-minute interval. Instead of killing the run, you should use the shutdown command.

This prints useful debug information to the internal pegasus.log and exits the run. This information in pegasus.log is encrypted. So, after the run is stopped, you can contact Cadence Support who can decrypt and debug the issue.  

To use the command, write shutdown in the go_pegasus file

With this last command, I will also shut down. Hope you liked the blog.

Special thanks to Dibyendu Goswami (Digo) who is the subject matter expert of this blog.

About Pegasus Get Your Wings

Pegasus: Get your Wings is a blog series to showcase the capabilities of Pegasus and to familiarize you with its notable features.