Virtuoso ADE Explorer/Assembler problem with load balancing system (type SGE)

Dear All,

I have come across problem when trying to run jobs over load balancing system of SGE type while using Virtuoso ADE Explorer and/or Assembler (IC_6.1.7.715 aka IC6.1.7-64b.500.15). I have no problems when it comes to older tools like ADE-XL. I had spent some time to debug it and by exploring inside of the farm node I have found out that :

1) if spectre job to be excuted is of single point (no parameter scans etc), then it does it on the farm node via axlGenerixGlobalLauncher script which calls runSimulation script - and all goes ok.

2) if job is of multipoints then as usual it uses runICRPxx script that launches nograph virtuoso which in normal conditions should start simulator via runSimulation script. However.. instead - it tries to launch simulator back on the submission host (!) via cdsRemshClient call. Here is the output of 'ps' command run over farm node farm036:

[root@farm036 ~]# ps uaxwwf | grep user
root 21420 0.0 0.0 9044 664 pts/0 S+ 17:34 0:00 \_ grep --color=auto user
user 17608 0.0 0.0 22076 1712 ? Ss 17:31 0:00 \_ -sh /opt/sge/default/spool/farm036/job_scripts/42280
user 17649 0.0 0.0 9528 1344 ? S 17:31 0:00 \_ /bin/sh /projects/PDK/SigmaDelta/V1.0/workAreas/user/.tmp_user/.cmddir0/runICRP0
user 17688 3.7 0.7 1452752 467976 ? S 17:31 0:05 \_ /eda/cadence/2017-18/RHELx86/IC_6.1.7.715/tools.lnx86/dfII/bin/64bit/virtuoso -noautostart -mpssession virtuoso11005 -mpshost work053 -davinciService DaVinciService_11005_1519057891 -log /projects/PDK/SigmaDelta/V1.0/workAreas/user/logs_user/logs0/Job0.log -licenseLockFileName /projects/PDK/SigmaDelta/V1.0/workAreas/user/.tmp_user/.work053_11005 -nograph -nostdin -axlChildIdFlag 0
user 21410 0.0 0.0 2460 708 ? S 17:34 0:00 \_ /eda/cadence/2017-18/RHELx86/IC_6.1.7.715/tools/bin/32bit/cdsRemshClient work053 -cdsPath tools/bin/cdsServIpc -s farm036 -c 47686 -n 1 -r 0 -x cd /simulation/user/ResDir/testWB/testInverterTB/maestro/results/maestro/ExplorerRun.0/1/testWB:testInverterTB:1/netlist ; ./runSimulation
user 21418 0.0 0.0 0 0 ? Z 17:34 0:00 \_ [cdsRemote.scr] <defunct>

where work053 is the submission host (where main ADE Explorer is run) and farm036 is the farm node.

so far I have cut .cdsinit init to the minimum and tried:

;; forcing distributed mode to NOT copy files into exec machine:
envSetVal("asimenv.distributed" "copyMode" 'boolean nil)
;; disabling interactive ADE Assembler and Explorer modes:
envSetVal("maestro.simulation" "interactiveE" 'boolean nil)
envSetVal("maestro.simulation" "interactiveA" 'boolean nil)
;; forcing to use ssh in remote shell calls:
envSetVal("asimenv.distributed" "remoteShell" 'string "ssh")
;; helpers for debugging ADE-XL/Assebler :
(when (axlIsICRPProcess)
_axlDebugTimeFlag = t
(sstatus errsetTrace t)
_stacktrace = 50
;_axlCallbkDebugFlag = t ;uncomment this if debugging callbacks
)

;; make stdout and stderr visible
(envSetVal "adexl.icrpStartup" "showJobStdout" 'boolean t)
(envSetVal "adexl.icrpStartup" "showJobStderr" 'boolean t)

and set env. variable AXL_PEJM_DEBUG=3 as suggested in one of trouble shutting articles on the KB.

As I had mentioned above - I have no problem with job executions over SGE farm while using ADE-XL tool.

Any ideas ?

Wojciech