Strange scaling with multithreading in APS simulations

 Hi,

 I wonder if anybody has seen anything like we observe for some of our typical anslog APS simulations when experimenting with different multitreading options on different hosts.

 Usually, when the multithreading settings (manual selection in ADE-XL) is increased from 1 to x CPUs, the total elapsed time increases somewhat with each CPU added, and the CPU time being consumed (as shown close to the end in the "spectre.out" file) stays in the same order of magnitude (maybe in a 10%-20% range). However, for some of our circuits, this does not apply - instead, when e.g. going from 3 CPUs to 4 CPUs, both CPU time and elapsed time suddenly skyrocket (CPU time e.g. doubles). For those examples, a two CPU run maybe be faster (in elapsed time) than an eight CPU run. The "runtime explosion point" is not always the same - sometimes it's between 3 and 4 CPUs, sometimes later, and sometimes it's not there at all.

Naturally, we are looking into hardware differences on our hosts, as also the behavior of different hosts (on different  sites, but also sometimes on the same site)  differs. However, so far we have failed to observe anything similar with any other tools  (from any EDA vendor).

We have looked into the influence of data location (locao disk on local server, somwhere else), processor affinity settings (and yes, I know that this is not something we want to use in real operations), different hardware (the older Westmere Xeons seem to have less trouble than the later Sandy Bridge or even Ivy Bridge models), total load on the server (in one situation, runtime improved significantly when there was other load on the same server!) and other factors, but so far we have not ben able to find a pattern behind this.

Of course we are also suspecting a tool bug, but the Cadence VCAD team (in Germany) we are in contact with has not been able to identify anything like this even after contacting Cadence R&D. They have also checked the correctness of our simulation setup and options.

 As this effect doubles the simulation runtimefore some circuit, we need to follow up on tha.Thus, in parallel to continue working with our Cadence VACD suppor, I'd like to learn if any user user might have seen anything like this before.

Our Cadence software:

o virtuoso 615.500-16

o MMSIM  Spectre Version 13.1.0.087.isr1 32bit -- 11 Nov 2013

Hardware:

o Dell R720 with E5-2690, 2.90GHz , 128GB/256GB RAM

Best regards,

Manfred