Re: Decimals

Evan wrote on Fri, 28 Aug 2009 12:27:47 -0700:

On Fri, 28 Aug 2009 11:27:07 -0700, Evan Kirshenbaum
<kirshenbaum@xxxxxxxxxx> wrote:

Whether I read it as "point six two five" or "five eighths"
would probably depend on what I believed the intent of the
writer to be with respect to the precision. If I just see a
number like .625, I'm more likely to assume that the
tolerance is ±.0626 than that it's ±.0005.

Standard practice is that the tolerance is one unit of the
last place unless otherwise specified, so for .625 it would
be +/- .001, for .62 it would be +/- .01, etc.

Standard it may be, but my experience is that in a lot of
domains that "otherwise specified" is implied from the data or
from experience. And I was a little imprecise with "precision"
and "tolerance". If I measure something as .671, there's no
way I'm going to write it down as .6 or .68. It's .7 or .69.
The precision, I was taught, is half of the smallest unit
unless otherwise specified.

But I think that that's one of the things that makes people
resist using decimals for many things. If I'm building a
bookcase, getting it to the nearest eighth of an inch is
probably fine. But I can't specify the eighths exactly as
decimals without implying that I need it to the nearest
thousandth of an inch.

My normal practice in quoting scientific results was to give the probable error, or sometimes standard deviation, in parentheses. Thus I might quote a result as 1.234(12), indicating that the standard deviation was 0.012. (I did indicate which value I was using since probable error and standard deviation are not the same thing.).

--

James Silverton
Potomac, Maryland

Email, with obvious alterations: not.jim.silverton.at.verizon.not

.

Relevant Pages

  • Re: prePEP: Decimal data type
    ... You fought against limited precision, ... arbitrarily) coerced into a decimal-fraction before operating ... of "unbounded precision" to come up with real use cases for them, ... > with a divfunction that lets you specify the number of places ...
    (comp.lang.python)
  • Re: Oracle-Number werden tlw. als Zahl/Double erstellt
    ... you can also specify a precision (total number of digits) and scale. ... der wohl in unterschiedlichen Versionen unterschiedliche DDL-Anweisungen für ein Number-Feld ohne jegliche Dimensionierung erstellt hat. ...
    (microsoft.public.de.access.clientserver)
  • Re: Monetary calculations in CL
    ... things like yield rates that specify a number of places of precision and ... round the 13th (by implication do not employ precision to more than the ... calculations can easily consume 20 or more significant digits. ... of places of significance and the rounding policy are supplied. ...
    (comp.lang.lisp)
  • Re: Gentler Decimal Floating-Point
    ... or really to specify decimal arithmetic. ... that the range, precision, or scale factor is specified in decimal ... precision in decimal digits for binary arithmetic. ... The language rules specify how those conversions are to be ...
    (comp.arch.arithmetic)
  • Re: SELECTED_REAL_KIND clarification plse
    ... >> relationship between the arguments and the bytes of storage used. ... this will produce a REAL with at least as much precision as IEEE ... >> precision vs exponent range issues for reals, ... eg. if I specify SELECTED_INT_KIND, ...
    (comp.lang.fortran)
Loading