Voltus-FI: How to automate the creation of selfheating-aware EM/IR text reports after every run

Hi Henry,

If your emir.conf is complete, you should be able to get all text reports once APS simulation has finished. Can you paste the contents of your emir.conf here?

Regards,
Saloni

Hi Saloni,

This is the content of the emir.conf file, please check if it is correct:

net name=[IDUT.*] analysis=[vmax vavg imax iavg iavgabs irms]
spf aliasterm = "ppode_ulvt_mac 1=S 2=G"
spf aliasterm = "pch_ulvt_mac 1=d 2=g 3=s"
spf aliasterm = "pch_mpodeulvt_mac 1=d 2=g 3=s"
spf aliasterm = "npode_ulvt_mac 1=S 2=G"
spf aliasterm = "nch_ulvt_mac 1=d 2=g 3=s"
spf aliasterm = "nch_mpodeulvt_mac 1=d 2=g 3=s"
time [128p 200p]
solver method=[direct]
emirutil autorun=[true]
emirutil print_em_report=[net=all_nets type=acrms_sh filename=acrms_sh.txt]
emirutil print_em_report=[net=all_nets type=avg_sh filename=avg_sh.txt]
emirutil print_thermal_report=[type=metal filename=thermal.txt]
net name=[IDUT.*] analysis=[selfheating]

Regards,

Henry

Hi Henry,

I don't see a reason why the runDir and runName entered in GUI should not work in the batch-mode. The files _qrc_tech_dir and _save_layers should be present in the extraction database.

Have you re-run the flow after setting the env var CDS_MMSIM_VOLTUSFI_ROOT? Does Voltus-Fi version look different in vfi.log now?

Regarding QRC shapeserver error, I've seen it occur mostly due to an issue in extraction database, but I am surprised you see this error only when running the flow batch-mode (assuming you have sync'ed the versions now). Do you have the below setting in Quantus CCL?

output_setup \
	-keep_temporary_files "true"

At this stage, it might be worth getting in touch with Cadence customer support so someone can review your setup via a Webex session.

Regards,
Saloni

Hi Saloni,

Thanks for your help. After getting further support from Cadence support AE, I have the correct setup to get the VFI batch run going after spectre simulation.

In the Spectre netlist/ directory, I noticed that the self-heating XML files (e.g. input.dtemp0.xml) are always created after the batch mode run started (i.e. vfi.log timestamp ~1min earlier than shXMLFile timestamp). This causes the XML file not found by the batch mode run. Is there a way to fix this, or we suppose to use the XML file from the prior rather than the current run?

Hi,

This doesn't sound normal. Can you please let me know the following:

1. What Spectre version is being used?

2. What ICADV12 version is being used?

3. Finally, please paste the contents of vfi.log.

Regards,

Saloni

Certainly. Here are the info:

Spectre version:  mmsim15.10.isr6
ICADV12 version:  ICADV12.3 isr15
VFI version: ICADV12.3 isr15

vfi.log:

INFO: Starting Voltus-Fi Batch Mode Application.

Command run is:
******************************************************************
/tools/cds/ICADV12.30.isr15.lnx86/tools.lnx86/sev/bin/64bit/emirreport -c /work/ms_scratch_kel/hchan/simulation/my_sandbox_ems
he/tb_clkgen_half_slice_fo1_emsheshe/maestro_hch/results/maestro/Interactive.73/1/my_sandbox_emshe:tb_clkgen_half_slice_fo1_em
sheshe:1/netlist/emir.conf -db /work/ms_scratch_kel/hchan/simulation/my_sandbox_emshe/tb_clkgen_half_slice_fo1_emsheshe/maestr
o_hch/results/maestro/Interactive.73/1/my_sandbox_emshe:tb_clkgen_half_slice_fo1_emsheshe:1/netlist/../psf/input.emir0_bin -ou
tdir /work/ms_scratch_kel/hchan/simulation/my_sandbox_emshe/tb_clkgen_half_slice_fo1_emsheshe/maestro_hch/results/maestro/Inte
ractive.73/1/my_sandbox_emshe:tb_clkgen_half_slice_fo1_emsheshe:1/netlist/../psf/
******************************************************************
Voltus-Fi version being used is as follows:
******************************************************************
 Voltus-Fi batch mode application (64-bit) - High Capacity Power IR/EM - v06.17-e469_1 ((09/17/2017 17:31:56))
 -----------------------------------------------------------------
 Copyright (c)  Cadence Design Systems, Inc.  1996 - 2014
******************************************************************

INFO: loading MMSIM results
INFO: loading MMSIM all net results
Info: Reading Self Heating TMI File: /work/ms_scratch_kel/hchan/simulation/my_sandbox_emshe/tb_clkgen_half_slice_fo1_emsheshe/
maestro_hch/results/maestro/Interactive.73/1/my_sandbox_emshe:tb_clkgen_half_slice_fo1_emsheshe:1/netlist/input.dtemp0.xml
FATAL: Unexpected error : Could not open file : /work/ms_scratch_kel/hchan/simulation/my_sandbox_emshe/tb_clkgen_half_slice_fo1_emsheshe/maestro_hch/results/maestro/Interactive.73/1/my_sandbox_emshe:tb_clkgen_half_slice_fo1_emsheshe:1/netlist/input.dtemp0.xml


Process Info :
  User time          = 0 seconds
  System time        = 0 seconds
  Children user time = 4 seconds
  Children sys time  = 3 seconds
  Virtual mem. size  = 327,905,280 bytes
  Total alloc size   = 0 bytes
  In use alloc size  = 0 bytes
  Page fault w/o IO  = 24,708
  Page fault w/  IO  = 157
  No. of swaps       = 0
  Page size          = 4,096 bytes
  Real Time(HH:MM:SS)= 00:00:24
  Peak Memory Usage  = 100392 kB


INFO: Ending Voltus-Fi Batch Mode Application.

netlist > ls -ltr
total 40374
-rw-r--r--  1 hchan eng      352 Mar 19 11:15 spectre.sim
-rw-r--r--  1 hchan eng       29 Mar 19 11:15 spectre.inp
drwxr-xr-x  2 hchan eng        0 Mar 19 11:15 raw/
-rw-r--r--  1 hchan eng      176 Mar 19 11:15 control
drwxr-xr-x  2 hchan eng     1271 Mar 19 11:15 amap/
-rw-r--r--  1 hchan eng      223 May 30 09:03 netlistHeader
-rw-r--r--  1 hchan eng        0 May 30 09:03 netlistFooter
-rw-r--r--  1 hchan eng    38233 May 30 09:03 netlist
drwxr-xr-x  2 hchan eng       25 May 30 09:03 map/
drwxr-xr-x 10 hchan eng      257 May 30 09:03 ihnl/
drwxr-xr-x  2 hchan eng        0 May 30 14:57 statAlters/
-rw-r--r--  1 hchan eng      524 May 30 14:58 si.env
-rw-r--r--  1 hchan eng       86 May 30 14:58 si.foregnd.log
-rw-r--r--  1 hchan eng    39796 May 30 14:58 input.scs
-rw-r--r--  1 hchan eng     1509 May 30 14:58 emir.conf
-rwxr-xr-x  1 hchan eng      179 May 30 14:58 runSimulation*
-rwxr-xr-x  1 hchan eng      607 May 30 14:58 runGenricJob9*
lrwxrwxrwx  1 hchan eng       30 May 30 14:58 cds.lib -> /proj/ms_kel/hchan/16n/cds.lib
-rw-r--r--  1 hchan eng  4901040 May 30 15:03 spectre.ic
-rw-r--r--  1 hchan eng  4806374 May 30 15:09 spectre.fc
-rw-r--r--  1 hchan eng   261413 May 30 15:10 summary.rpt
-rw-r--r--  1 hchan eng     3800 May 30 15:10 vfi.log
-rw-r--r--  1 hchan eng      591 May 30 15:11 input.tmiage0_106096.100021
-rw-r--r--  1 hchan eng 20290408 May 30 15:11 input.tmiage0_106096.100021.dat
-rw-r--r--  1 hchan eng  1347085 May 30 15:11 input.dtemp0.xml
-rw-r--r--  1 hchan eng    63173 May 30 15:11 input.dtemp0
-rw-r--r--  1 hchan eng       17 May 30 15:11 input.tmiage

Certainly. Here are the info:

Spectre version:  mmsim15.10.isr6

ICADV12  (and Voltus-FI) version: ICADV12.3 isr15

netlist/ directory and vfi.log file:

Thanks, it appears to be a bug in MMSIM where it is invoking emirreport before *.xml is available on the disk. Your MMSIM version is very old (more than 2 years), so can you please move to a newer version and check if that solves the issue?

Going back to the older posts in this thread, I notice you were using Spectre171 at some point, so that will be a good version to try.

Regards,

Saloni

Hi Saloni,

Thanks for the info.

I tried mmsim17.10.isr2 and it behaves similarly (could you confirm?). Meanwhile, Spectre171 is not fully setup in our methodology. I'll check with our infrastructure team if it can be enabled.

Henry

Hi Saloni,

Our "mmsim17" is actually Spectre v17. Would you mind let us know since which Spectre version has this bug been resolved? We would look into installing this or newer version.

Henry

Hi Henry,

It isn't a known bug and I know a lot of our customers are running the flow the way you want to, but this is the first time I've heard of the issue. This is why I suspected an older Spectre version to be the reason. But if you see the same problem with Spectre171 as well, then the issue is somewhere else. Have you tried running the simulation on a different machine, or a different testcase?

Regards,

Saloni

Hi Saloni,

I tried running on various remote machines and locally, with 3 testcases, and got similar behavior. I also experimented to replace emirreport with a shell script that delays the call until .xml files are written, with an ample 5min timeout, and it ended in the timeout. So clearly, the emirreport call is not a spun-off process, but part of the post-simulation flow.

It seems that indeed there is an ordering problem internally, the xml files generation step starts only after the emirreport call returns. Would you be able to confirm this, or there are other users not having this issue?

Regards,

Henry

Hi Saloni,

I tried running on various remote machines and locally, with 3 testcases, and got similar behavior. I also experimented to replace emirreport with a shell script that delays the call until .xml files are written, with an ample 5min timeout, and it ended in the timeout. So clearly, the emirreport call is not a spun-off process, but part of the post-simulation flow.

It seems that indeed there is an ordering problem internally, the xml files generation step starts only after the emirreport call returns. Would you be able to confirm this, or there are other users not having this issue?

Regards,

Henry

Hi Henry,

I don't see the ordering problem, so if it's a bug, then we'll need to determine what's triggering it. I just ran the SHE flow now to test Spectre 171 ISR2, and I can see in the log a message confirming that *.xml file has been generated right after which emirreport is called. These messages show the timestamps too.

Initial condition solution time: CPU = 3 ms, elapsed = 3.13997 ms.
Intrinsic tran analysis time:    CPU = 46.993 ms, elapsed = 55.31 ms.
TMI results xml file .../results/maestro/Interactive.3/3/testLib:metal_res_sim:1/netlist/input.tmideg0.xml generated at Mon Jun  4 03:42:11 2018

        Time window lookup file of EMIR binary files '../psf//input.bin_time_window' is generated.

Calling emirreport at Mon Jun  4 03:42:11 2018

Can you check what your Spectre log says?

Regards,

Saloni

Hi Saloni,

Thanks for the information. In my spectre log, I don't have the line "TMI results xml file ..." before:

Spectre (R) Circuit Simulator
Version 17.1.0.160.isr2 64bit
. . .
Initial condition solution time: CPU = 34.8797 s, elapsed = 17.675 s.
Intrinsic tran analysis time:    CPU = 3.58198 ks, elapsed = 404.355 s.

Time window lookup file of EMIR binary files '../psf//input.bin_time_window' is generated.

Calling emirreport at Mon Jun  4 08:12:54 2018
Creating EMIR report
Warning from spectre during transient analysis `tran'.
    WARNING (EMIR-2603): Failed to run command line: '/tools/cds/ICADV12.30.isr15.lnx86/bin/emirreport [...]. Can not generate EMIR report with emirreport.
. . .
Opening the PSFASCII file ../psf/subckts.info.subckts ...
TMI results xml file /work/ms_scratch_kel/hchan/simulation/my_sandbox_emshe/tb_clkgen_half_slice_fo1_emsheshe/maestro_hch/results/maestro/Interactive.85/1/my_sandbox_emshe:tb_clkgen_half_slice_fo1_emsheshe:1/netlist/input.tmideg0.xml generated at Mon Jun  4 08:14:26 2018
. . .

Do you know what settings did I miss?

In Simulation > Reliability > Setup,  TMI analysis mode is enabled. Simulator mode: Spectre native, Enabled self-heating, Enable aging, Aging type: TMI Aging.

In emir.conf, selfheating options are specified:   

net name=[IDUT.*] analysis=[vmax vavg iavg irms selfheating]
emirutil shEMEffect=[beolT]
emirutil cds.lib=/proj/ms_kel/hchan/16n/cds.lib
emirutil view=[lib=my_sandbox_emshe cell=clkgen_half_slice_fo1 view=layout]
emirutil shXMLFile=/work/ms_scratch_kel/hchan/simulation/my_sandbox_emshe/tb_clkgen_half_slice_fo1_emsheshe/maestro_hch/results/maestro/Interactive.84/1/my_sandbox_emshe:tb_clkgen_half_slice_fo1_emsheshe:1/netlist/input.dtemp0.xml
emirutil qrc_output=[runDir=/home/hchan/pmcVeriPhy/my_sandbox_emshe/clkgen_half_slice_fo1/EXT_QRC/RCC_SPICE/tmp runName=clkgen_half_slice_fo1]
emirutil shParamFile=/home/liblib/16n/techdata/v1_0_beta/voltus_Fi/param.sh
emirutil techfile=/home/liblib/16n/techdata/v1_0_beta/voltus_Fi/qrcTechFile
emirutil print_em_report=[net=all_nets type=acrms_sh filename=acrms_sh.txt]
emirutil print_em_report=[net=all_nets type=avg_sh filename=avg_sh.txt]
emirutil print_thermal_report=[type=metal filename=thermal.txt]
emirutil coupledSH=true

By any chance the simulation results in psfascii format would delay the writing of the XML file? In the spectre log, after emirreport call, I have a series of messages reading info from psf/ and then write out data files, including the xml file:

Creating EMIR report
finalTimeOP: writing operating point information to rawfile.
Opening the PSF file ../psf/finalTimeOP.info ...
modelParameter: writing model parameter values to rawfile.
Opening the PSF file ../psf/modelParameter.info ...
element: writing instance parameter values to rawfile.
Opening the PSF file ../psf/element.info ...
outputParameter: writing output parameter values to rawfile.
Opening the PSF file ../psf/outputParameter.info ...
designParamVals: writing netlist parameters to rawfile.
Opening the PSFASCII file ../psf/designParamVals.info ...
primitives: writing primitives to rawfile.
Opening the PSFASCII file ../psf/primitives.info.primitives ...
subckts: writing subcircuits to rawfile.
Opening the PSFASCII file ../psf/subckts.info.subckts ...
TMI results xml file /work/ms_scratch_kel/hchan/simulation/my_sandbox_emshe/tb_clkgen_half_slice_fo1_emsheshe/maestro_hch/results/maestro/Interactive.88/1/my_sandbox_emshe:tb_clkgen_half_slice_fo1_emsheshe:1/netlist/input.tmideg0.xml generated at Mon Jun  4 09:28:09 2018

Hi Henry,

Thanks for sharing, I think I understand the problem. You have missed enabling Reliability Analyses from maestro view.

When running the SHE flow, you should typically see 3 simulations running: fresh, stress and she. Correspondingly, the point numbers in maestro database are 1, 2, 3. Fresh simulation is optional, so the minimum simulations/points you'd get are 2. Out of these simulations, the last one is the SHE run where emirreport should be launched.

In your log excerpts, I see all messages in point '1' e.g. Interactive.85/1, while the SHE simulation should have been Interactive.2 or .3. This gave me the clue that the setup isn't correct.

Here's a snapshot of how reliability analysis should be enabled, and how the results will look like. Let me know if this helps.

HI Saloni,

Thanks for the detailed information. I enabled TMI in the ADE explorer menu Simulation > Reliability > setup, but indeed didn't enable TMI under Reliability Analysis setup > options > Reliability. I did exactly what your screenshots showed, but the execution order of writing emirreport and then xml file did not change in the SHE run. I tried

Fresh+Stress+SHE tests

Fresh+SHE tests

Fresh Test + stress file input.tmiage0 from prior run + SHE test

Enable TMI self-heating only

Enable TMI self-heating and aging

All tests used Spectre 17.1 isr2, ICADV12.3 isr15, ended with xml file not found, because it was written after the vfi.log finished.

I noticed that all my runs created the input.dtemp0.xml, not not input.tmiage0.xml file shown in your log.

Regards,

Henry

Hi Henry,

'input.dtemp0.xml' created in SHE run is read by Voltus-Fi for the analysis, so it seems the file is getting generated for you, but later than expected.

I'll need to look at your setup environment to be able to assist further as the behaviour you are seeing is indeed bizarre. Can I request you to contact Cadence customer support once again now? If it turns out to be setup issue, we should plug that gap so a user does not run into this problem. Or if it's actually a software issue, then the AE can try to reproduce it internally and help R&D fix it.

Thanks,

Saloni