TRUNC function Error

[asjex]

He're's my data:

98.9000000000 (location Register!N1006)

I'm using this formula:

=TRUNC(Register!N1006, 2)

This should truncate the number to 98.90, but I get 98.89. What gives?

Screen grab - if this works

I'm using Excel 2010 on Windows XP.

 

[pgc01]

He're's my data:

98.9000000000 (location Register!N1006)

Hi
Welcome to the board

Do you mean that you've confirmed that that's the data or that's just what you are seeing?

Please confirm. In another cell write:

=(Register!N1006=98.9)

and check if the result of the formula is TRUE.

 

[asjex]

Thank you for your help!

The formula was FALSE. I brought it out to 10 decimals, and apparently that wasn't enough - I had to go to 13 to see that the number was actually

98.899999999999700000000000000000

So, follow up question, then.

I have a formula
=O105+N104

O105 is
-480.000000000000000000000000000000
and N104 is
541.900000000000000000000000000000
yet it equals
61.899999999999700000000000000000


????????????????????????????????????

- Aaron

 

[pgc01]

Aaron, I'm glad it helped.

As for you new question, small rounding errors are to be expected as the result of conversions base 2 <-> base 10.

Please read my answer in post #3 in this thread:

http://www.mrexcel.com/forum/showthread.php?t=454140

and if you want to dig deeper follow the link therein.

 

[vlady]

just an additional

you can use this in trunc

=TRUNC(O105+N104,2)

 

[asjex]

The "set precision as displayed" is magic for me, thanks!

Otherwise, I'll have to use ROUND on ALL my calculations!

Thanks for pointing me to the KB article. Perfect.

Thanks again!

- Aaron

 

[pgc01]

I'm glad it helped.

Remark: "Set precision as displayed" is very handy in some cases but be sure you understand that all that is not displayed is really lost, there's no going back.

Notice ms warning:

Note Using the Precision as displayed option can have cumulative calculation effects that can make your data increasingly inaccurate over time. Use this option only if you are certain that the displayed precision will maintain the accuracy of your data.

http://office.microsoft.com/en-us/excel-help/set-rounding-precision-HA010218870.aspx