topic Re: Scary calculation in OS X.4 Calculator in Collaboration with other software

Scary calculation in OS X.4 Calculator

Ignacio Azpiazu — Tue, 13 Jun 2006 16:18:22 GMT

Trying to figure out why this happens and if it can have any impact on CAD. In the computers and systems I've been able to check out:

OS X.3.# you go to the Calculator, type 200000 * 1.12 and get 224000.
OS X.4, same thing, you get a weird 224000.000000000029.

(In View, precision needs to be set at 11 or higher.)

The widget calculator (not sure how it handles precision) and Excel at 16 decimals seem to be doing OK.

Re: Scary calculation in OS X.4 Calculator

Anonymous — Tue, 13 Jun 2006 18:08:37 GMT

Precision errors are common, but usually inconsequential. Remember that the decimal values are being translated to binary and there are limits to how much overhead programmers want to take on. I wouldn't worry about it. In the real world accuracy past 4 to 5 places is rarely needed and past 11 is pretty much unknown.

Re: Scary calculation in OS X.4 Calculator

TomWaltz — Tue, 13 Jun 2006 18:41:02 GMT

If I can find a contractor that can build to that accuracy, I'm betting he's a different kind of alien than the president likes to talk about.

Re: Scary calculation in OS X.4 Calculator

Thomas Holm — Tue, 13 Jun 2006 21:34:30 GMT

Ignacio wrote:
OS X.3.# you go to the Calculator, type 200000 * 1.12 and get 224000.
OS X.4, same thing, you get a weird 224000.000000000029.

It's a known problem with the Calculator in OSX.4. I think there was another one of the same kind in OSX.3, but I don't remember the figures. It's not a hardware or system problem. Therefore, other calculators work.
I guess Apple don't think people use the Calculator much. At least not enough to give thsi problem enough priority. I think it's embarrassing for them.

Re: Scary calculation in OS X.4 Calculator

Ignacio Azpiazu — Wed, 14 Jun 2006 21:11:37 GMT

Thomas wrote:
It's a known problem with the Calculator in OSX.4.

Phew, thanks.