Re: printf and style g

Mark Dickinson <dickinsm@xxxxxxxxx> writes:

On Jun 20, 1:00 am, Tim Rentsch <t...@xxxxxxxxxxxxxxxxxxx> wrote:
Vincent Lefevre <vincent+n...@xxxxxxxxxx> writes:
The standard says for fprintf (7.19.6.1):

g,G A double argument representing a floating-point number is
converted in style f or e (or in style F or E in the case of
a G conversion specifier), depending on the value converted
and the precision. Let P equal the precision if nonzero, 6
if the precision is omitted, or 1 if the precision is zero.
Then, if a conversion with style E would have an exponent
of X:
- if P > X >= -4, the conversion is with style f (or F) and
precision P-(X+1).
- otherwise, the conversion is with style e (or E) and
precision P-1.

but it doesn't explicitly say what precision to choose for "if
a conversion with style E would have an exponent of X" (this is
important since X depends on the precision). It seems logical
to choose P-1 (as in the "otherwise" case), but shouldn't the
standard make this clear?

I found this question very confusing. First, the quoted clause
serves to define X.

That's the point. It doesn't serve to define X, because the
value of X can depend on what precision was used for the
style E conversion.

For clarity, I'll refer to the conversion in the quoted
clause as the 'would have' conversion, to distinguish
it from the eventual intended g-style conversion.

Example: support the value to be formatted is 99.75.
What's X? A 'would have' conversion using style E
and precision 1 would give '1.0E+2', giving X = 2.
A 'would have' conversion using style E and precision
2 would give '9.98E+1' (or possibly 9.97E+1 on some
systems), giving X = 1.

[...]

Can you give an example format and value to be output that
illustrates the ambiguity you think is there?

Suppose we want to format 99.75 (again) with g-style
formatting and a precision of 2. Again, as above, X
depends on the precision used for the 'would give'
conversion: if we use precision 1 for that, then X is
2, and the result of the 'g' formatting (after stripping
trailing zeros as per the standard) is "1e+2". If we use
precision 2 or higher, then X is 1 and the result of the
'g' formatting is "100". The second result is clearly
wrong, since it ends up giving 3 significant digits
rather than 2.

Now consider g-style formatting of 99.75 with P = 3.
If we use precision 2 for the 'would give' format, we
get X = 1 and the result of the 'g' formatting is "99.8".
But if we use precision 1 for the 'would give' format
then X = 2 and the formatted result is "100". Again,
the second result is wrong: the result of rounding
99.75 to 3 significant figures should be 99.8, however
one chooses to express it.

So the value of X *does* depend on the precision used,
and the precision used should thus be specified. A
'would have' precision of P-1 is the correct one to use,
since this has the effect of rounding to a decimal number
with exactly P significant digits, which is the intent of
'g' formatting in the first place.

I'm 72.3% sure that I'm not misunderstanding Vincent's
point here; I hope he'll step in and correct me if I am.

Quite. I see the point now; thank you for the clarification.
(I also see that Vincent has responded, and am posting my
more complete response there.)
.