How to pause a Monte-Carlo simulation run and release license

Dear Svilen,

Di you happen to read the On-line support article at URL:

https://support.cadence.com/apex/ArticleAttachmentPortal?id=a1Od0000000vbbQEAQ&pageName=ArticleContent

The process for suspending a simulation set is explained as well as the process for resuming its operation. Right click on the progress bar of the job as shown in Figure 1 and you are provided the option to suspend the job. As shown in Figure 2, the job progress bar will then appear yellow and indicate it is suspended.

Shawn

Figure 1

This process is explained

Thanks, Shawn.

This is a little different than what I asked. I didn't ask for suspending the whole MC run. Say I run a MC with 10 points, or 10 iteration, whatever you want to call it. I would like to suspend the run of point 1, say, but keep all other 9 point still running. And then I want to resume point 1 from where it was stopped.

Thanks, Shawn.

This is a little different than what I asked. I didn't ask for suspending the whole MC run. Say I run a MC with 10 points, or 10 iteration, whatever you want to call it. I would like to suspend the run of point 1, say, but keep all other 9 point still running. And then I want to resume point 1 from where it was stopped.

If using LSCS, you can (on the Monte Carlo options) control the Points per job in the Advanced - Show All Options section. If you change this from group automatically, you can then set this to be "1". If you then run with 10 jobs in your job policy, you can just suspend one of them.

Andrew 

I am running ICRP not LSCS for the job that has already started. I didn't get it though. If I set 1 point per job,won't this start just one MC point, not 10, like in a parallel run?

Andrew, here is a slightly different but still related question. If I RMB click on the test name in assembler and go to Environment, I see the "Check in License When Suspended" box unchecked by default. How can I have it checked by default? What should I write in the .cdsenv for this to happen?

If using ICRP, this is not controllable - the points per run is always "group automatically". What that means is that ADE will take the total number of Monte Carlo points/iterations and multiply by the number of corners, and then divide that by the number of jobs in ADE and then run Spectre with blocks of monte carlo with that number of points (if that number is greater than the requested number of Monte Carlo points, it will run that number in each run). In essence it starts a Monte Carlo analysis within the simulator with numruns=N - and they are run sequentially. You cannot then stop individual Monte Carlo points - you can only stop the entire block that is running.

With LSCS you can choose how many Monte Carlo points are grouped together. So if you run 10 points, with number of points per job being 1, and have your job setup to run 5 jobs, then each simulation will run a single Monte Carlo point at a time, with 5 simulations in parallel - but it will work through and run all 10 Monte Carlo points (so roughly speaking each "job" will run two simulations).

For the check in license when suspended and the cdsenv to control this, that's covered in the post from Shawn above (which references an article I wrote on Cadence Online Support).

Andrew

Thanks, Andrew. I think I sort of understand it now. However, here are few observations.

As I said, I was running ICRP. My setup was one MC run with 10  points with number of jobs in parallel in the job setup equal to 3. If I understand correctly what you explained, the blocks which the MC run will handle for sequential simulations is 3 and then 1 single point remains. I was expecting to see 3 points run in parallel at the beginning of the simulation, then when these 3 finish another 3 will star, followed by the last 3 and at the end the single one remaining. Instead, it simulates the first 2 blocks of 3, in total 6 points and then switches to 2 blocks of 2 points for the last remaining 4 points. It looks it changes the size of the blocks on the go.

The other thing I noticed was that if I check that box for "Check in License When Suspended" in the Environment options and if I get the PID of one of the running MC points and then do in the terminal from which I started cadence

kill -SIGTSTP PID

the simulation with that PID is suspended and the license released.

Then if I do kill -SIGCONT PID something interesting happens. The assembler continues to show that simulation as suspended, but if I check the spectre.out file, it is actually running from the place it was suspended and also I can see that the license is checked out again. I didn't wait till the end of the simulation, so I am not sure if it will show the results of the expressions I have set since it was showing as suspended after it was restarted.

I ahve put in my .cdsenv

spectre.envOpts   lsusp   boolean   t
spectre.envOpts lsuspEnvOverridesState  boolean t

but it didn't work. The message in the CIW said these options were not loaded.

Some questions:

1. Which IC sub-version are you using?
2. Are you doing the kill on the spectre process, or the ICRP process?
3. What is the error you get in the CIW when you have those entries in your .cdsenv ?

Regards,

Andrew

Andrew Beckett said:

If using ICRP, this is not controllable - the points per run is always "group automatically".

However, there is a workaround: https://support.cadence.com/apex/ArticleAttachmentPortal?id=a1Od0000003rGJJEA2 

Frank Wiedmann said:

However, there is a workaround

Of course - thanks Frank. I was just trying to keep things simple ;-)

Hi Andrew,

1. I have ICADVM18.1-64b
2. for the kill I get the pid from the spectre.log file, at the top of the file
3. it is not really an error, but a warning. It says:
*Warning* envSetVal: Can't set the value of variable 'lsuspEnvOverridesState' in tool[.partition]. 'spectre.envOpts' - it has not been registered.

Thanks
Svilen

OK, you truncated the version number so didn't give the sub version. I know however if it's ICADVM18.1 and not ICADVM20.1 then it must be between the base and ISR13 (with .500.13 at the end). The lsuspEnvOverridesState was added in ISR17 (so with .500.17 on the end) as outlined in the article mentioned above.

I'd have thought it would be better to suspend the ICRP by using Right Mouse->Suspend over the icon in the Run Summary, and then that passes it onto Spectre. You can then Resume in the same place - no need to look up process ids. That said, ADE should really detect when the process was resumed - but maybe it was broken in the older version y you're using.

Andrew

I am sorry about missing the last part. It is ICADVM18.1-64b.500.9. So yes, looks like old.