Re: Half O.T.(Keypad Functions On Sharp) [rounding by Casio vs. HP]

On Sat, 08 Mar 2008 18:26:29 -0600, S.C. wrote:

My 12-digit Casio fx-260 can't decide when to round and when to
truncate in the display. It truncates whenever |result|<1
and rounds otherwise.

Is that the only explanation?

For example, when I try 2/3 on it, it displays:

0.666666666

Then I add 1 to the answer and it shows:

1.666666667

Subtracting 1 returns back to the 0.666666666 earlier.

Similarly, pi returns 3.141592654, but subtracting 3 from that leaves
0.141592653

What you are expecting is "round in the display,
to the number of significant digits actually displayed"
(as the HP48/49/50 series generally does),
but it appears that your Casio first rounds an internal
12-digit result to a constant 10-digits,
and then displays as much of that as it can,
dropping off the last digit when "0." is prefixed as an afterthought.

Thus, 2/3 when rounded to ten mantissa digits is .6666666667
but if prefixing "0." pushes the last digit off the display,
then you would see only 0.666666666

However, 2/3 plus 1, rounded to ten digits, would display 1.666666667

pi minus 3, when rounded to ten digits, is .1415926536
but if prefixing "0." pushes the last digit off the display,
then you would see only 0.141592653

But pi itself, rounded to 10 digits, is 3.141592654
(calculators using 11-digit internal mantissas
tend to use 3.1415926536 as an internal value for pi,
or 3.14159265359 for 12-digit internal mantissas).

The HP48/49/50 series calculators display pi as 3.14159265359
but there is a lot more to what goes on internally,
not only because they internally calculate every numerical function
to 15 digits (with 5-digit exponent) and then deliver the rounded
12-digit result (with 3-digit exponent) to be stored, rounding *that* again,
for display purposes only, if fewer digits are actually shown,
but also because they analyze functions a bit more deeply,
to deliver more accurate results than most common 12-digit calculators.

For example, in the HP48/49/50 series, RAD 3.14159265359 SIN
results in -2.06761537357E-13, adding which to the original argument
(on paper, to get the exact sum) yields
3.141592653589793238462643, which is the accurate value of pi
to 25 significant digits -- any calculator which can do that
is effectively using an internal value of at least that precision for pi
(what does your Casio get?)

The thread "HP4x internal digi accuracy" (March 21-24,2007)
suggests that doing the above in SysRPL produces pi to 31 digits,
and other cases illustrating more careful and thorough numerical workmanship:
http://groups.google.com/group/comp.sys.hp48/browse_thread/thread/16ea6ddfb49e34eb/

[r->] [OFF]
.

Relevant Pages

  • Re: Cobol Myth Busters
    ... than just TRUNC (wich has 3 flavors on IBM zSeries). ... using 18 digits: packed decimal is 74% faster than binary ... packed decimal is 60% slower than binary ... DISPLAY compared to packed decimal ...
    (comp.lang.cobol)
  • Re: FAQ 4.2 Why is int() broken?
    ... that's over 400 times faster. ... Calculators can be slow - they do only primitive ... It also displays all digits (unless ... told to display a rounded result), so that wouldn't make a difference ...
    (comp.lang.perl.misc)
  • Re: FPGA Journal Article
    ... 10 decimal digits fixed-point display, ... Above 1 MHz limited by time base accuracy, below 1 MHz by display ... but clutter up the design. ... A 20 times more accurate time base would cost <$20 extra. ...
    (comp.arch.fpga)
  • Re: Price Data Formatting
    ... When you load in a datafile of a stock or commodity, ... only 3 digits following the decimal rather than 4. ... So when I display prices in my application, I want to use the SAME format as ... It doesn't fit well in my textboxes for display. ...
    (microsoft.public.vb.general.discussion)
  • Re: Bally 14 digit display repair (Atlantis and others) 6803 game
    ... Perhaps one of the digit driver transistors are blown. ... The bad thing about these is the you will have to bend the display down in ... The Millions place has all segments lit all the time. ... a couple other segments on other digits that do not light up as well. ...
    (rec.games.pinball)