Re: Confidence Intervals for Bridge Simulations (long)

On Oct 31, 5:44 am, paulh...@xxxxxxxx wrote:
On Oct 31, 1:42 am, Charles Brenner <cbren...@xxxxxxxxxxxx> wrote:



On Oct 30, 9:54 pm, paulh...@xxxxxxxx wrote:

(4) Calculate the lower and upper bounds of the confidence interval, p minus or plus three times ssd

In warning against careless interpretation of simulation results
without taking into account the realities of sampling statistics,
beware of overstepping in the opposite direction.

The approximation that all probabilities within a 3 standard deviation
confidence interval are equally probable and outcomes outside that
range are impossible -- that's cookbook statistics.

More trials do give more reliable conclusions. But since the effect is
obviously gradual, it must be that even small differences -- so-called
"statistically insignificant" differences -- among small numbers of
trials mean something. If after a grand total of one simulation 3NT
looks better than 2NT, that's evidence that 3NT is really better.
Grant me some sort of prior probability distribution assumption as to
how good each contract might be, and I can compute the strength of
that evidence. (And without some such assumption, confidence intervals
imply nothing.)

Charles

Of course all outcomes within the confidence interval are not equally
probable, the sample mean is the single most likely value for the
population mean, and values toward the edge of the interval are far
less likely. My point is that posters are habitually acting as though
they can draw conclusions from a simulation of 100 hands, by looking
only at the sample mean, ignoring variability.

They can draw conclusions, and some might over-draw. But making a
technical-sounding argument with artificial statistics and buried
assumptions isn't the right rebuttal.

I don't think such
simulations give evidence of any practical value.

"any"? That's an extreme statement. I like better the comments of
Danny & Ranier.

As for your requirement of an assumption about the prior probability
distribution, doesn't the Central Limit Theorem allow us to infer the
population mean from a sample without knowing the population
distribution?

Only in the limit. With only a finite sample you do not know offhand
whether you have reasonably well approached the limit or not.

To understand my point about the need for some prior distribution
knowledge, consider this example: Out of 100 trials, the event X
occurs 1 time -- sample frequency 0.01. Now, at the 99% confidence
level, that's consistent with a population frequency as low as
1/10000. (An event whose population frequency is that low has a 1%
chance to be observed at least once out of 100 trials.) Do you feel
like betting -- I'll give you even money, good sport that I am -- that
X will occur again among the next 10000 trials? Well, you shouldn't
until you find out at least a little bit about what I'm sampling.
Turns out, I'm dealing bridge hands. My event X is some particular
bridge deal, and it won't occur again for eons. Beware of cookbook
statistics.

However, my reason for raising the issue wasn't to point out that the
confidence interval may not necessarily be expected to contain the
true population frequency, but rather to point out that IF we have the
kind of prior distribution information to make confidence intervals
meaningful, then we can instead use it to make a more meaningful
likelihood ratio statement about the meaning of the evidence -- and
that works even with small samples.

Charles

.