Re: results too good?


berniecr@xxxxxxxxxxxx wrote:
Hey, have you guys ever held back results because they are too good?

Not until I understood why they were good.

I am working with neural networks, which are a random process.

Artificial neural networks are not a random process.

However, ANNs are, typically, trained using algorithms with one or
more randomization steps.

After
optimizing their variables and archetecture, I did leave 14% out cross
validation three times.

Insufficient detail.
Regression or classification?
How many inputs and outputs?
Size of data set?
How did you select inputs and architecture?
What training algorithm was used?
Train/Val/Test splits for this?

In this process, about 14% of the datapoints
are used for the test set, a new nn built, and evaluated.

Training to convergence?

Why did you choose 14%? What do you have against the traditional
5 or 10-fold cross-validation?

then the
next 14% is done. I repeated the whole process 3 times, so each member
was predicted 3 and only 3 times. Averaging the three times,

Probably insufficient. If you believe that a 14% test set size is
sufficient,
Use 7-fold cross-validation where regression MSE or classification
error
rate is averaged over the 7 'folds' so that all of the data is used
both for
training and testing in an unbiased way. Search in comp.ai.neural-nets
using cross-validation and XVAL for details.

and
building the regression model between predicted and observed values
yields varrying results. Repeating this whole algorithm repeatedly
gets me final results of varrying quality, although almost all of them
are "pretty good." A couple of time, i get pretty much ideal results,
with an almost perfect line.

I assume you have a regression net and you are referring to the
plot of output vs target.

There were a couple points off, but not
by much, only 0.5 (range 5-12). Those points are expected to be off
because the molecules they are associated with aren't well represented.

So what results should I publish? They are all valid results, but the
better ones might be perceived as faked. It's just that the better
ones were luckily enough able to get closer to the global minimum that
NNs try to reach.

should I try increasing the repetitions to something like 20?

Yes. Run 7-fold XVAL 3 times. Report average and standard deviation
of MSE for each of the 3 folds and for the the total 21 trials.

This
would probably get more reproducible results, but wouldn't the
increased averaging smoothen my line even more?

You'll have to explain to which line you refer.

You could plot MSE vs trial and superimpose the mean and
mean +/- 2*stdev/sqrt(m) lines for trials 1-7, 8-14,15-21 (m=7) and
(different color) 1-21 (m=21). That would tend to convince me.

Hope this helps.

Greg

.

Relevant Pages

  • Re: Regression analysis -- how-to?
    ... > were it not for the fact that, the last time I asked an algorithm ... Do you want to learn about regression? ... > (linear, quadratic, logarithmic, exponential, power and inverse) and at ... > least two types of linear correlation coefficient (rank and product ...
    (comp.lang.pascal.delphi.misc)
  • Re: Help on Partial Least Squares Aalgorithm
    ... > The PLS algorithm does regression on some ... > more than one target variables and give me the regression coefficent ... Now there are ways to rewrite the PLS algorithm using an eigenvector ...
    (sci.stat.math)
  • Re: Analysis of gas mode MM interferometer operation using standard SR formulae.
    ... turn averaging procedure does absolutely nothing to reveal the ... So I would say the full algorithm with the averaging is fairly ... any significant excess over its expected level? ... ALWAYS results in a periodic half-rotation curve, ...
    (sci.physics.relativity)
  • Re: Help on Partial Least Squares Aalgorithm
    ... Paige Miller wrote: ... >> Q are the kx1 regression coefficents. ... >> more than one target variables and give me the regression coefficent ... > even more calculations than the NIPALS or SIMPLS algorithm anyway. ...
    (sci.stat.math)
  • Re: initialization time?
    ... I want a time that fairly represents one algorithm versus another, ... I've been taking the median of five iterations, ... the same answer as throwing out the first two and averaging. ... your code floats into real ram pushing out other applications. ...
    (comp.lang.java.help)