Stats

  • Locked Locked
  • Replies 1
  • Subscribers 129
  • Views 16099
  • Members are here 0
This discussion has been locked.
You can no longer post new replies to this discussion. If you have a question you can start a new discussion

About floor()

bobbygang

Hi,
I find a strange thing.
When I input
floor(0.1/0.1)
it gives me 1
but if I input
floor((1-0.9)/0.1)
it gives me 0

Why?
How to avoid this situation?

Regards,
Bobby

 

 

Parents
  • Andrew Beckett

    Bobby,

    This is due to floating point rounding errors, found in pretty much every computer language. The trouble is that 0.1 is actually a recurring fraction when represented in binary - so when you do the 1-0.9 you end up with a different answer (very slightly) than 0.1 - due to truncation of the binary digits right at the end. It's similar to the fact that in decimal, if you represented your calculations to 4 decimal places, 3.0/3.0 would be 1.0 whereas if you did (1.0/3.0)*3.0 you'd end up with 0.9999. This can be seen:

    a=1-0.9 => 0.1
    a-0.1 => -2.775558e-17
    sstatus(fullPrecision t)
    a => 0.09999999999999998

    because a is slightly less than 0.1, dividing by 0.1 gives an answer just less than 1, so floor rounds down to 0.

    The answer is to use round(), or to add a small fudge-factor before applying round.

    You might want to read What Every Computer Scientist Should Know About Floating Point Arithmetic

    Regards,

    Andrew.

Reply
  • Andrew Beckett

    Bobby,

    This is due to floating point rounding errors, found in pretty much every computer language. The trouble is that 0.1 is actually a recurring fraction when represented in binary - so when you do the 1-0.9 you end up with a different answer (very slightly) than 0.1 - due to truncation of the binary digits right at the end. It's similar to the fact that in decimal, if you represented your calculations to 4 decimal places, 3.0/3.0 would be 1.0 whereas if you did (1.0/3.0)*3.0 you'd end up with 0.9999. This can be seen:

    a=1-0.9 => 0.1
    a-0.1 => -2.775558e-17
    sstatus(fullPrecision t)
    a => 0.09999999999999998

    because a is slightly less than 0.1, dividing by 0.1 gives an answer just less than 1, so floor rounds down to 0.

    The answer is to use round(), or to add a small fudge-factor before applying round.

    You might want to read What Every Computer Scientist Should Know About Floating Point Arithmetic

    Regards,

    Andrew.

Children
No Data

Community Guidelines

The Cadence Design Communities support Cadence users and technologists interacting to exchange ideas, news, technical information, and best practices to solve problems and get the most from Cadence technology. The community is open to everyone, and to provide the most value, we require participants to follow our Community Guidelines that facilitate a quality exchange of ideas and information. By accessing, contributing, using or downloading any materials from the site, you agree to be bound by the full Community Guidelines.