Floating Point , Wide zero etc

Hi
I have separated this explanation out of the previous thread for clarity.
Pick Basic may not be the language of choice for

calculating a lunar modules trajectory, it is however excellent for Fra Luca
Pacioli's double entry book-keeping which has an

absolute requirement for arithmetic accuracy.
There are three things that affect this accuracy in a Pick program. They
are ICONV/OCONV, Precision and Floating point

arithmetic.
A short example demonstrates the first two.

001 EXECUTE "CLEAR-FILE DATA TEST"
002 OPEN '','TEST' TO TESTF ELSE CRT "BOOM NO TEST";STOP
003 PI = 22/7
004 FOR A = 1 TO 4
005 REC = "MD":A
006 REC<2> = ICONV(PI,REC)
007 REC<3> = OCONV(REC<2>,REC)
008 WRITE REC ON TESTF,A
009 NEXT A
010 REC = "VANILLA"
011 REC<2> = PI
012 WRITE REC ON TESTF,"VANILLA"
013 EXECUTE "SORT TEST A1 A2 A3"


:RUN MMBP TRY
Page 1 TEST 06:10:05 04 Jul 2009

TEST...... A1................... A2................... A3...................

1 MD1 31 3.1
2 MD2 314 3.14
3 MD3 3143 3.143
4 MD4 31428 3.1428
VANILLA VANILLA 3.1428

[405] 5 items listed out of 5 items.
The default precision of 4 is used. If we now insert a PRECISION 6
statement
Page 1 TEST 06:20:11 04 Jul 2009

TEST...... A1................... A2................... A3...................

1 MD1 31 3.1
2 MD2 314 3.14
3 MD3 3143 3.143
4 MD4 31429 3.1429
VANILLA VANILLA 3.142857

[405] 5 items listed out of 5 items.
We have now introduced an accounting anomaly, the same code has produced a
different answer in raw data mode amd a slightly

different answer in MD4. The latter is very important to me as oil pricing
is done at 4 decimal places and the tax office

require 6 places for GST calculation. All of which have to be rounded to
decimal 2 for the final invoice value.
The simple rule is never store unconverted data for accounting. My
preference is to use ICONV for all dates and accounting

data; whatever one does, and it is normally decimal 2 for invoice values, be
consistent.

The next step is Floating Point, an area fraught with danger that can
produce the infamous wide zero and has some known

anomalies in the basic algorithms. Please use it for actuarial calculations
of the "value" of derivatives but steer well

clear for normal accounting.
Floating Point forces the use of large binary numbers for the calculations.
If we consider a normal file with a 2 decimal component there are 100
possible values for that component. All well and good

in decimal but there is no direct binary equivalent for many of the values
0.01 + 0.02 + 0.03 is fine but if we take
o.66
..10101000111101011100001010001111 the binary approximation converts back to
..659999999916181
and with
o.77
..11000101000111101011100001010001 the binary approximation converts back to
..769999999785796

it is not difficult to see how over a large data set a simple addition can
rapidly introduce a decimal component slightly

smaller than anticipated.
Say we are expecting nnnnnnn.13 and we have nnnnnnnn.1297 All that has
happened is that the default PRECISION of 4 has taken

a number that looks something like nnnnnnnnn.1297399999etc and given us the
precise figure requested. If we had converted

the figure with ICONV(RESULT,MD2) we would have what we required.
The simple solution is always store results in the converted format and to
avoid anomalies creeping in over a large data set

always do the conversion at each step of the calculation
Peter McMurray


.

Relevant Pages

  • Re: demonic numbers !
    ... > corresponding calculation of error intervals. ... Then they're either not doing enough math for rounding errors to ... accumulate or they're not using a wide enough range of magnitudes. ... They don't need more than 53 bits of precision to store those values. ...
    (comp.lang.lisp)
  • =?Utf-8?Q?Re:_I_don=C2=B4t_view_the_value_right?= =?Utf-8?Q?_after_six_decimal_place_in_Exce
    ... You might consider setting the option Tools> Options> Calculation> ... Precision As Displayed. ... final result, not intermediated calculations (subexpressions). ... the workbook first because some changes cannot be undone by simply ...
    (microsoft.public.excel.misc)
  • Re: ROUNDING RESULT OF CALCULATION UP OR DOWN
    ... In one of the calculation rules we had to round to a whole year. ... I keep repeating we need decimal arithmetic in Excel, ... | ??>> the limits of precision? ...
    (microsoft.public.excel.misc)
  • Re: Problem with a small "exercise" code using compiler xlf
    ... going from REAL to DOUBLE PRECISION solved part of the ... In the case of problems like this, negative square roots when ... in the intermediate calculation gives a negative square root. ... one could choose one rounding mode for one part ...
    (comp.lang.fortran)
  • Re: For Andrew Breen, plasma
    ... physicists use slide rules and everything. ... you can get almost all the precision you want! ... calculation with it, but have forgotten the usage of the log-log scales ... I had a couple of K&E 20 inch rules. ...
    (sci.military.naval)