Equal numbers do not equal?

Martin,

The subtraction is causing small floating point rounding errors - this is explained in numerous posts on the internet (not just about SKILL - it happens anywhere where IEEE floating point representation is used).  This is because the numbers are represented with a 53 bit binary mantissa, and so numbers that look simple in decimal (such as 0.1), these are recurring binary fractions (in the same way that 1/3 in decimal is 0.33333 recurring). That recurring binary digit then gets truncated at the 53rd bit, and manipulation of these numbers can lead to a loss of precision at the end of the sequence, leading to tiny differences when certain operations are performed.

Put simply, you should never use == (equal) to compare floating point numbers (unless you know they are from the same source). So for example, you can compare x or y coordinates from the database because they will be converted the same way every time. However, if you manipulate these in some way then there's no guarantee they will not suffer from differences in loss of precision (the subtractions in your case).

If you know the precision, you can do:

precision=cv~>DBUperUU ; or you could give it a fixed number
foreach(mapcar intSize removeListDuplicates(foreach(mapcar size openSizeList round(size*precision))) intSize/precision)

Since the numbers are coming from the database, and that internally has integer precision, converting them back into integers and de-duping them first is a safe way to handle this.

Andrew

Andrew,

Got it. Thanks for you explaination. 

BR,

Martin