What setting will impact atof() output

Hi,

This setting controls the display precision:

sstatus(fullPrecision t)

you can test the setting using status(fullPrecision).  The documentation states this:

 If t, unformatted print functions (print, println, printlev) print floating point numbers in full precision (usually 16 digits); otherwise, the default is about 7 digits of precision.

Best regards,

Lawrence.

Thanks a lot Lawrence!!

What if I want it to return "9.12"?

sstatus(fullPrecision nil)
atof("9.12")
=> 9.12

You didn't state what fullPrecision is currently set to, but I expect that it was set to 't', so setting it to 'nil' should give you the above result.

Regards,

Lawrence.

Works well.

Thanks a lot!~

I was working on a script that compared coordinates and the print out showed the coords were the same but, if i did a minus of the two coords I would not get zero. I would get 1e-18.

Does the sstatus(fullPrecision nil) fix this problem?

No, it won't. That simply means that the default print representation when numbers are printed in the CIW (e.g using println) will show 16 digits rather than 7. The storage of the numbers is unaffected.

If the numbers both come from the database then they would be exactly the same (the same rounding error for both) - it's only if they've been computed differently. You may need to subtract them and check if the difference is less than (say) a tenth of the database unit (so if DBUperUU is 1000, your database resolution is 0.001 um). So do abs(x1-x2)<0.0001 rather than using ==.

Andrew

I'm not sure, but as Andrew has repeatedly stated on this forum, regardless of the language (SKILL, Python, C etc.) one should not perform comparisons of floating point numbers the way that you might for strings or integers.

Try using the nearlyEqual() function to check for equality within a tolerance.

Thanks Andrew and skillUser for the info

Thanks Andrew and skillUser for the info