float storage and usage



Greetings.

I need to convert some columns of type numeric(12, 0) to hold floating point
information scale and precision I can't determine in advance (customer data
can vary wildly) so I wanted to use the datatype that offers the maximum scale
and precision..

I'm targeting sql server 2005 systems (not 2000).

It seems my choices are real and float, and the docs seem to indicate that
float offers with widest ranges.

I'm trying out using the 'float' for the new data type as the default
precision is said to be 53.. Does this mean the total number of digits is up
to 53?

I don't know if there is anything else I need to take into account since these
two columns are part of a primary key, and I supposed, therefore, are indexed.

thanks
Jeff Kish
.



Relevant Pages

  • Re: Reciprocal square root (Was: Re: If It Were Easy...)
    ... of about 20 cycles: ... The latency per iteration can reduced a bit by regrouping, ... except in for the float precision function which uses ...
    (comp.arch)
  • Re: Losing Precision from FLOAT in DBD::Informix
    ... The versions of CSDK and IDS are also not ... STSM, Informix Database Engineering, IBM Data Management ... >> are declared FLOAT. ... >> - where and why is the precision lost? ...
    (perl.dbi.users)
  • Re: Reciprocal square root (Was: Re: If It Were Easy...)
    ... The worst case is on core2 where it takes 81 cycles ... Using double precision costs 8 cycles. ... I don't see how we can get better than 0.9 ULP without doing at least one NR step with double precision. ... this also reduces accuracy to below float precision. ...
    (comp.arch)
  • Re: Reciprocal square root (Was: Re: If It Were Easy...)
    ... by discarding all my versions that convert from float ... slower in float precision. ... cycles throughput and 56 cycles latency on core2 (its latency has ...
    (comp.arch)
  • Re: converting float to double
    ... >> insists that I read the stock prices as float. ... >> Since everywhere else the system uses double to hold these prices, ... A 64 bit integer will correctly model currency to 18 digits (with ... calcuations in 110 digits of precision (so that things like interest ...
    (comp.lang.c)