Re: Arithmetic operation bug

On Friday 24 July 2009, Manoj Chourasia wrote:
|v1="128.015"
|puts((v1.to_f*1000).to_i).to_s // giving 128014 instead of 128015
|
|Why the the result is rounding up to 128014 but the correct result
|should be 128015.

I'm not an expert, but I think because when you write 128.015, the number you
actually get is 128.0149999999999863575... (because 128.015 can't be
represented as a number with a finite number of decimal digits in binary, just
like, for example, 1/17 can't be repsented with a finite decimal number in
decimal). You usually don't see this because by default only a small number of
decimal digits are shown, taking rounding into account. You can obtain a
greater number of digits using, for example, format:

puts format("%.20f", 128.015)
=> 128.01499999999998635758

Now, when you multiply the number by 1000, what you actually get is:
128014.99999999998544808477. And, since Float#to_i truncates the number (it
doesn't round it), you get the number 128014.

I hope this helps

Stefano

.

Relevant Pages

  • Re: Deviation of double
    ... int iDigits); ... I believe you're confused about rounding. ... round to 90.62 if only two decimal digits are required. ... Why are you expecting 90.625? ...
    (comp.lang.cpp)
  • Re: This calculation is just wrong / computer cant count!
    ... BCD also has the advantage of providing more significant decimal ... arithemtic, AS IMPLEMENTED ON THE PENTIUM FPU, as demonstrated by my Floating Point ... If you can't understand that using binary to represent decimal digits ... properly rounding. ...
    (microsoft.public.vc.mfc)
  • Re: Rounding error when doing float-to-integer
    ... Agree with Glenn here. ... significant figures" step. ... The rounding is to 24 binary digits. ... The "7 decimal digits" bit is only a rough approximation that allows ...
    (comp.lang.fortran)
  • Re: Pixel Mania
    ... infinite number of decimal digits following the decimal point. ... truncated number that VB rounds to 15 displayed digits is an integer power ... Rounding in situations one doesn't expect. ... Most people would expect it to round first IMO. ...
    (microsoft.public.vb.general.discussion)
  • Re: round not rounding to 0 places
    ... a value to zero decimal places using the following format, ... Round a number to a given precision in decimal digits (default 0 ...
    (comp.lang.python)