Stats

  • Locked Locked
  • Replies 4
  • Subscribers 144
  • Views 14298
  • Members are here 0
This discussion has been locked.
You can no longer post new replies to this discussion. If you have a question you can start a new discussion

running multiple ocean simulations from a single host

Wonyoung

Hello,

I had problems running multiple ocean simulations from a single host under IC5.1.4. Several of the simulations would crash, but the problem was solved by running only 1 ocean simulation per host. (related post: www.cadence.com/.../18294.aspx).

I tried this with IC6 to see if that changes anything. We have a variety of hosts with 4-12 cores each, and I ran 1 ocean simulation per core (4-12 simulations per host). I used the same command "ocean -log CDS.log.mylogfile -nograph -restore myoceanfile.ocn", and used different log file and .ocn file names across simulations. The ocean simulations crashed again, but with a slightly different message (attached below). Again, the problem didn't appear when I ran only 1 simulation per host.

Could anyone help me solve this problem and let me run multiple ocean simulations per host? It just doesn't seem right that I can run only 1 ocean simulation per host when there's a ton of multi-core hosts out there.

Thank you very much.

X connection to vasuki.eecs.harvard.edu:80.0 host broken (explicit kill or server shutdown)
\e IO Error 32 (Broken pipe) on Display "vasuki.eecs.harvard.edu:80.0"
\e Aborting due to fatal X IO error.
\o INFO (DB-170018): (dbPanic) dumping stack
\o virtuoso:dbPanic+197()
\o libvirtuos_sh.so:voDoAbortFuncs+37()
\o libvirtuos_sh.so:voSigAbort+167()
\o :__kernel_sigreturn+0()
\o libc.so.6:abort+257()
\o virtuoso:gliXIOErrorHandler+325()
\o libX11.so.6:_XIOError+45()
\o libX11.so.6:libm.so.6/PLT+2159027()
\o libX11.so.6:XFlush+49()
\o virtuoso:_Z12hiiSetCursorm+277()
\o virtuoso:hiiDiceCursor+47()
\o virtuoso:hiGetUserAbort+158()
\o libvirtuos_sh.so:voGetUserAbort+34()
\o libcdsIlIpc_sh.so:ipcxSleep+175()
\o libil_sh.so:iliVcodeEval2+21063()
\o libil_sh.so:ilxSkillEval2+216()
\o libil_sh.so:iliLoadPort+1431()
\o libil_sh.so:iliVcodeEval2+26932()
\o libil_sh.so:ilxSkillEval2+216()
\o libil_sh.so:iliEvalString+611()
\o libil_sh.so:_ilErrSetString+235()
\o libil_sh.so:ilErrSetString2+60()
\o virtuoso:hiMainLoop+404()
\o virtuoso:_ZN3rde15startHiMainLoopEv+2619()
\o virtuoso:Tcl_Main+781()
\o virtuoso:_ZN17TclCommandAdaptor8mainLoopEiPPcPKcRKSt6vectorI9ctuStringSaIS5_EEPFvvE+591()
\o virtuoso:_Z11finaleEntryiPPcPKcbP16finaleAppHarness+3885()
\o virtuoso:_Z12rdeStartMainiPPc+61()
\o virtuoso:main+997()
\o virtuoso has encountered a fatal internal application error and will now exit.
\o A crash report file for this error can be found at: /tmp/crashReport_031511_1758_IC6.1.4.500.1_wonyoung_vasuki.eecs.harvard.edu.log

  •  Hi Wonyoung,

    Please enter a Service Request - I could not find a similar looking stack trace in our problem database after a quick search, so it is probably best to file an SR with Customer Support.  I can see that you are at a US University, so if you are not one of the contacts who is able to file an SR, please ask a professor or other contact.

    Best regards,

    Lawrence.

  • Hi Lawrence,

    I am experiencing the exact same problem with the new AMS utility 'runams' that is available since IC 5.1.41 ISR 138, IC 6.1.3 ISR 14.

    I run concurrent runams commands and i see some kind of contention happening because I am randomly getting the following error for some runs:

    Running :  runams -lib $LIBNAME -cell $CELLNAME -view $VIEWNAME -netlist all -rundir . -cdslib $CDS_WORKAREA/setup/dfII/$PROJECT.cds.lib
    STDERR: IO Error 32 (Broken pipe) on Display "clvclo10:80.0"
    Aborting due to fatal X IO error.
    Exit code: 6


    In my example, when I run 12 runams calls in parallel using separate directories with separate logfiles and separate CDS.log files (on different topcells),  I get the 2 or 3 random fails (different cell each time).  If I run the same 12 runams calls serially, they all execute to completion without any problem.  I suspect that runams is falling victim the same problem as described by the user since it is basically a wrapper to virtuoso.

    I have a testcase to provide to R&D if that helps ...


    cheers

    Nicolas

  • We can recommend to customers to use a grid computing solution to dispatch their ocean/runams jobs on different hosts. In my case, I introduced LSF calls into my simplan to randomize the hosts, and it solved my problem by greatly reducing the likelihood of contention.

  • Nicolas - you should contact customer support so that this can be investigated.

Community Guidelines

The Cadence Design Communities support Cadence users and technologists interacting to exchange ideas, news, technical information, and best practices to solve problems and get the most from Cadence technology. The community is open to everyone, and to provide the most value, we require participants to follow our Community Guidelines that facilitate a quality exchange of ideas and information. By accessing, contributing, using or downloading any materials from the site, you agree to be bound by the full Community Guidelines.