Reporting simulation time for suspended simulations

DB,

Which version are you using? I just used ctrl-C to kill the job, and got:

Spectre received SIGINT.

Saving the states into file at t=266.853 us...

Killed by user.

Aggregate audit (3:57:55 PM, Tue Jun 16, 2020):
Time used: CPU = 2.83 s, elapsed = 2.89 s, util. = 98.1%.
Time spent in licensing: elapsed = 51.9 ms.
Peak memory used = 53.3 Mbytes.
Simulation started at: 3:57:52 PM, Tue Jun 16, 2020, ended at: 3:57:55 PM, Tue Jun
16, 2020, with elapsed time (wall clock): 2.89 s.
spectre completes with 0 errors, 0 warnings, and 5 notices.

How are you killing the job?

Andrew.

DB,

Which version are you using? I just used ctrl-C to kill the job, and got:

Spectre received SIGINT.

Saving the states into file at t=266.853 us...

Killed by user.

Aggregate audit (3:57:55 PM, Tue Jun 16, 2020):
Time used: CPU = 2.83 s, elapsed = 2.89 s, util. = 98.1%.
Time spent in licensing: elapsed = 51.9 ms.
Peak memory used = 53.3 Mbytes.
Simulation started at: 3:57:52 PM, Tue Jun 16, 2020, ended at: 3:57:55 PM, Tue Jun
16, 2020, with elapsed time (wall clock): 2.89 s.
spectre completes with 0 errors, 0 warnings, and 5 notices.

How are you killing the job?

Andrew.

Hi Andrew

I am using ctrl-C to kill the job. I am using SPECTRE191.

Is there something needs to specified for spectre options?

Thanks
DB

I tried with both SPECTRE 18.1 and 19.1 (latest hot fixes of each, but I'd be surprised if that made a difference). It was a transient analysis too. No special options (just spectre file.scs or spectre +aps file.scs). What are you running?

Andrew.

Dear Andrew and debaabed,

Using spectre Version 18.1.0.556.isr13, and issuing an LSF kill command on one of a set of Assembler simulations that was in progress, I also verified that the resulting simulation spectre.out file contains the elapsed time of the interrupted simulation:

...Spectre received SIGINT sent from PID: 133413 with UID: 0.
Process hierarchy of PID 133413 :
    PID        COMMAND
    133413     /tools/lsf/Current/10.1/linux2.6-glibc2.3-x86_64/etc/sbatchd
    118432     /tools/lsf/Current/10.1/linux2.6-glibc2.3-x86_64/etc/sbatchd

Saving the states into file at t=1.19042 us...

Killed by user.

Aggregate audit (4:16:30 PM, Tue Jun 16, 2020):
Time used: CPU = 32.1 s, elapsed = 32.2 s, util. = 99.5%.
Time spent in licensing: elapsed = 92.5 ms.
Peak memory used = 125 Mbytes.
Simulation started at: 4:15:58 PM, Tue Jun 16, 2020, ended at: 4:16:30 PM, Tue Jun 16, 2020, with elapsed time (wall clock): 32.2 s.
spectre completes with 0 errors, 2 warnings, and 3 notices.

I was curious as I have a shell script that compiles all the simulation times for an Assembler/ADE-XL simulation run to create a histogram and wanted to verify it handled a simulation ending prematurely correctly.

Shawn

Hi Andrew and Shawn

I am running a transient simulation.

The command line is as below

/cad/tools/cadence/SPECTRE191/tools.lnx86/bin/spectre -64 -E +lqt 0 \
+lsuspend +mt=4 +preset=cx  \
input.spe =log \
./LOGS/input.spelog

Thanks

DB

Dear debated,

I updated my environment to a version of 19.1 to allow me to run spectrex and used a spectre input file " input.scs". The run line, which I called "testx" is shown below and mimics what I believe is your run line:

$ cat testx
spectre -64 -E +lqt 0 \
+lsuspend +mt=4 +preset=cx \
input.scs =log \
../psf/input.spelog
$

I issued a Control-C to interrupt the interactive simulation and printed the last 20 lines of the output log and also include the version I am using:

$ testx
^C
Memory fault(coredump)
$ tail -20 ../psf/input.spelog
iprobe = 18.18 % of total equations
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

During simulation, the CPU load for active processors is :
0 (91.4 %) 1 (5.8 %)
Total: 97.3%

Spectre received SIGINT.

Saving the states into file at t=5.34581 us...

Killed by user.

Aggregate audit (12:19:13 AM, Wed Jun 17, 2020):
Time used: CPU = 127 s (2m 6.8s), elapsed = 135 s (2m 14.6s), util. = 94.2%.
Time spent in licensing: elapsed = 7.5 s, percentage of total = 5.57%.
Peak memory used = 115 Mbytes.
Simulation started at: 12:16:58 AM, Wed Jun 17, 2020, ended at: 12:19:13 AM, Wed Jun 17, 2020, with elapsed time (wall clock): 135 s (2m 14.6s).
spectre completes with 0 errors, 4 warnings, and 9 notices.

$ grep -e "Vers" ../psf/input.spelog
Version 19.1.0.348.isr6 64bit -- 1 May 2020

Hence, I am still seeing the elapsed time measurement in the output log. It might be insightful if you would attach your output log of the interrupted simulation. I am wondering if simulating your netlist is producing some other type of simulation error and hence the process termination sequence is different than the processes Andrew and I are observing - if that makes any sense!

Shawn

Dear debaabed,

My apologies for the error in my prior post - my spell checker decided your userid should be "debated" in lieu of your correct userid of debaabed. Sorry!

Shawn

Hi Shawn and Andrew,

With ctrl-<C> I am able to get the simulation time log at the end of the file.

But with ctrl-<Z> to suspend the simulation and then finally killing it, doesn't print the simulation time. I miscommunicated this earlier on this therad.

Why doesn't second method work?

Thanks

Debasish

Hi Debasish,

When you've suspended the simulation (using signal TSTP, i.e. ctrl-Z), then it goes into a particular handler to suspend the license usage. If you then kill it after that, it's still in this suspend mode - otherwise it would have to reclaim the license and then exit through the normal path.

This strikes me as a fairly unusual requirement - to suspend the simulation, then kill it, and expect it to report on what it's done so far. If you really want it, you can resume the simulation then kill it.

Most customers would not want the "kill" to have to reclaim the license to run through the normal flow to write the state file and so on - because a) that would temporarily consume the license, and potentially they may have to wait for it to be available.

Andrew.

Hi Andrew,

Thanks for the explanation. 

The scenario is such that multiple long simulations are going on, and to temporary release the licences, few long sims are suspended. And in some cases, killed afterwards. But looks like the correct as you described is way to first bring the job to foreground, and then use ctrl-<C> to log the simulation time.

Thanks

DB