I run atof("9.12") in my CIW, it will return 9.11999999999999
And I run virtuoso -nocdsinit, it will return 9.12
What will impact that?
This setting controls the display precision:
you can test the setting using status(fullPrecision). The documentation states this:
If t, unformatted print functions (print, println, printlev) print floating point numbers in full precision (usually 16 digits); otherwise, the default is about 7 digits of precision.
Thanks a lot Lawrence!!
What if I want it to return "9.12"?
sstatus(fullPrecision nil)atof("9.12")=> 9.12You didn't state what fullPrecision is currently set to, but I expect that it was set to 't', so setting it to 'nil' should give you the above result.
Thanks a lot!~
I was working on a script that compared coordinates and the print out showed the coords were the same but, if i did a minus of the two coords I would not get zero. I would get 1e-18.
Does the sstatus(fullPrecision nil) fix this problem?