Re: Proof that neural nets work


John Moody <john.atwell.moody@xxxxxxxxx> wrote in message
news:1148427679.017307.93260@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

The first time I read through your recent answers, I thought you
hadn't
gotten the point, but as I got into it I started understanding more of
what
you are saying. In short, we weren't on the same wavelength about
pseudo-random sequences, but you trumped me on the issue of what
you need to know to predict share prices.

Well, of course, it's all relative to me...it's a question of just how
PRECISELY you want to predict share prices. My opinion is that
you can do better than chance if you work at it correctly, but you'll
NEVER be able to PERFECTLY predict future prices with 100%
accuracy.

I see the problem as being like "predicting" the weather; you really
can't "predict" it, but you can intelligently "forecast" it with enough
precision to reasonably know when to plant crops, carry an umbrella,
plan a short vacation, turn off your furnance for the summer...

The theoretical reasons for this are basically covered by "chaos
theory", another branch of mathematics with its own peculiar
monomania...but can a neural net handle a "chaotic" system?

It's getting confusing who wrote what so I'll put names in

Is your newsreader having some quoting difficulties? For some
reason mine does with certain posts, not sure why...

William:
Ah, so would it be fair to say that if there is an input to stock prices
that you are not using as an input to the neural net, the neural net will
not be able to predict future prices?

John:
I agree with this.
It is actually worse than that. In the case of a pseudo random
sequence like RSA you can give a neural net the input 1,2,3,4,5 of the
calculation, and the output f(1), f(2), f(3), f(4), f(5)... is
predictable, but not by a neural network (or any other algorithm that
can fit on a computer that can actually be made) even when the input is
given to the algorithm.

A neural net with 3 stages can be trained to learn any truth
table...but all the possible inputs have to be exercised. If you really
have a general truth table with 128 bits of input (the size of an RSA
key) the training interval would take 2^128 machine cycles (at least!)
which is more time than anyone has.

Damn!

In real life, for stock prices, you have to know what relationship you
are actually looking for, then the neural net can find it, verify it,
quantify it.

So to sum up, you MIGHT be able to prove/disprove simple "TA"
and "charting" systems that only require a time series of price-only
data as a single input...EXCEPT, what's the next problem? What
have "TA" people told you about the effectiveness of their systems?

John:
So if the
green curve it produces is matching the blue curve, this means, it
HAS
found a mathematical relationship.

William:
Well, did it? Has anybody done this (knowing that I did this almost
twenty years ago, and many others claim to have done it)? If you, or
anybody did it, how closely did the "blue curve" (actual values) match
the "green curve" (predicted values)? How much price/volume data
did you feed it, what was the period of the data?

John:
The neural network I am talking about is not going to be any
different than the one you already used, if it used backpropogation and
an arctan transition function.

Well, I guess, it was a long time ago, and it was just a "black box" to
me, didn't really know much about what was going on "under the hood".

If you did a lot of back verification then you would have
already constructed what I am calling the 'blue' and 'green' curve.

I would just run a correlation coefficient against the predicted and
actual results.

I am not an expert at all, in my own experience, I got a little excited
when the screenshot (which predicted XOM) ended up matching what XOM
later did. Then there is the post you read about IBM.

Everybody gets a "little excited" when their "system" seems to be
working. At some point in time, EVERY system SEEMS to be
working...for a while, for some stocks. Then unfortunately most
of them stop working, and many times turn into giant losers.

This is the true nature of the market...or rather, this is the nature
of people trying desperately to "beat the market", which of course,
IS the market...

John:
Well, I am starting from knowing little about the stock market, going
back to that big drop in
Microsoft a few weeks ago, and trying to see what tickers I can load
alongside Microsoft that
match the graph.

William:
It sounds like you are using price-only data for a collection of
"related" stocks for a relatively short period of time to attempt to
predict the price of one stock. I'm actually not sure, even though
you are kind of saying it in plain English, probably because my
mindset is such that I can't comprehend that particular "experiment"
at this point in my life (twenty years ago I might have understood).

John:
Yes, price and volume data only. It is possible to enter indices in
(interest rates etc) and the downloader will load them, I haven't put
that into the documentation as I wasn't sure if it is just an extra
complication.
Interest rates change so slowly .... perhaps, though I should document
how to download and input bond yields.

They can and have changed at different rates at different times in
history. There's an old saying in the stock market, "Don't fight the
Fed", meaning "don't try to make money in the stock market if
interest rates are going up". This is because interest income
instruments like bonds, money markets, are competitive for
investor dollars, and if rates are high people tend to put their
money in the "safer" interest income instruments rather than the
"risky" stock market.

Even if rising interest rates don't much affect some short-term
"pattern" that a particular stock is trading, the market is nothing
but discontinuous, and those rising rates, coupled with several
other factors that people decide to "ignore" can suddenly change
the whole game and completely wreck any predictions made
without taking them into account.

That's why I've tracked interest rates in one way or another
for decades...even as part of pretty simple 15-point "indicator"
system, I feel that tracking them and trading appropriately has
made me money that I wouldn't have had otherwise...

William:
But let's start with price data alone, and look at it from some other
angles. If we look at daily data (or really any period of data), we
"discover" that there IS a simple mathematical relationship, a
STATISTICAL relationship, between today's data and yesterday's
data, because if we plot a histogram of the change between the
two days, we get something sort of resembling a "normal curve",
with a lot of data points clustering around a very small change,
and very few for a large change.

John:
OK I'm with you so far.

William:
Likewise, from chaos theory, our first pass at "attractor
reconstruction"
(n vs. n-1), shows a little fuzzy but easily discernable "attraction"
of
today's stock prices to yesterday's stock prices. But just plain old
logic and common sense and any amount of experience says that
of course traders base their bids and asks in the market on the most
recent sales prices, just like people believe that if their next door
neighbor sold their house for $1,000,000, THEY should also get
$1,000,000...and they'll stick to that unless there are other factors
that cause or allow them to raise or lower the price.

John:
This is making perfect sense to me so far...

William:
Good, now we've proven something about the stock market: you
don't really have to have any particular knowledge about the specifics
of the market, just some "common sense", to begin to analyze the
market.

We've covered one big motivation of sellers in a market, to get at
least the prevailing market price for their good or service. Other
"common sense" motivations: how much do you NEED money rather
than the good or service, how much money did YOU pay for the
good and service (your "cost basis"), etc.


Now, as a BUYER of something in any market, what motivates
you to buy? Well, first, you have to have the MONEY to buy, then
if you are buying an "investment", which "investment" do you choose?
What does "common sense" tell you?

Note that EVERYTHING you NEED to predict the market
comes out of the motivations described above for buyers and
sellers, which is just based on "common sense".

The question is: if you are feeding a neural net price-only stock data,
are you giving it everything it NEEDS to make a prediction?
I would of course ask the same question of any "TA" or "FA"
fanatic that claims that "FA" or "TA" respectively is a waste of
time...

John:
OK I'm starting to understand what you're saying. You are saying,
understanding, the structure of markets is more important....

Yeah, I was kind of getting there...

It is not about saying,
when the volume of such and such share goes up, while the price of such
and such goes down, this predicts some third share will go up. You
are saying, over the long term, you can look and say things like,
'hey, that company's approach is going to be obsolete once search
engines become more popular.'

Nah, wasn't talking about THAT at all. THAT'S a WHOLE different
thing. THAT'S "industry insight", and if you've got a good bead on
THAT, coupled with some general market and accounting knowledge,
you can make more money with THAT in the stock market than with
any cockamammie TA, FA, neural net strategy imaginable.

It's also a correspondingly rare talent, but one that everybody aspires
to in the market. It's really what the market is supposed to be all about,
but because very few people can CONSISTENTLY demonstrate
industry insight most market participants become embittered crazy
losers trying all kinds of whacky "systems" before finally deciding
"THE MARKET IS NUTZ!!!"

Clearly a computer program cannot
think like that....unless you explicitly tell it to.

Hmmm, how could a computer "think" like that? I think we're
entering the realm of science fiction here...

There is a sort of defense (if I were defending use of neural
networks):
that one might notice, people's perceptions affect the market in
complicated ways, and a neural network would notice an effect that
hits all shares of a given type or hits certain combinations first,
even
without understanding the nature or cause of that effect.

This is exactly what "TA" and "charting" people like to say. You
don't need to know what a company makes, the past price action
of the stock tells you everything you need to know about the future
price of the stock.

But basically you are right, a neural network has the intelligence of
an insect, and is not a muse by any means.

Oh, good, you touched back down on planet Earth again, I was
worried we'd lost you...

William:
Now, as a practical matter, after running the neural net on the
same amount of data as was required to come to the above
conclusions, can it actually provide a more profitable prediction
than from what we learned (or already knew) above? If not,
then we're actually a step behind with the neural net, because
we haven't LEARNED anything, due to the "black box" nature
of the predictions...

John:
Yes, it might be (marginally) more profitable to have a computer
helping you to notice and quantify trends and effects you already know
about.
But if it sees something, it cannot tell you what it is.

I have read, some finance companies hire groups of people to sort of
vote on what the connections should be, so there is a real close
connection between human thought and the neural coefficients then. Just
picking up the connections via training, is going to be like you say,
not an educational
experience for the user....

Yeah, which is why I DO like to do LOTS of statistical back-testing
on known strategies and variable using hundred-plus years of data. If
something works or doesn't work, then I know EXACTLY what works
or doesn't work...

John(responding to the other point)
But, sometimes it does a lot better...throughout historical time it is
able to match tomorrow's price knowing
todays prices and volumes.

William:
The term "sometimes" is troubling to me...replace that with a
correlation coefficient and some confidence intervals and I'd
be a happy camper.

John:
Some people do that. In some sense, though can't you agree
it's fair to have someone look at the back verification data as a
graph,
and ask 'would you be happy with that?'

Oh sure, that's not my problem, the problem is the word "sometimes".
My general problem is with the appearance of "data chucking", where
"sometimes" you get a good result, and "sometimes" you get a bad
result, so overall, you're really not getting much of a result at all, but
that point kind of gets lost.

What really kind of gets lost, of course, is that little thing called "the
scientific method", where you keep and statistically analyze ALL the
data from ALL the experiments...

I mean.... a confidence
interval
in percentage terms....is kind of like a digital watch. The exact info
you
want is there.....but for a human being making an intuitive choice a
graph
is nice. BUt I get your point here.

I mentioned somewhere about having to use the statistical data to
go ahead and develop an optimal strategy with "money management";
I can't do that by eyeballing a couple of lines on a graph.

We could produce a confidence interval....um......trying to think.
What would the null hypothesis be.....that the graph matches by
random chance?

Uh, yeah, that's the basic idea, always.

Forget "confidence intervals". Look, for me the drill is simple as pie.
Get thirty years of daily data, or more. Perform correlation coefficient
of the predicted versus the actual signed daily percentage change for
the last 15 years at least, or more, if you want.

If the result is around 0, you got nuttin'. Results between 0.6 and 1.0
DEMAND further reseach and analysis and the eventual development
of a killer trading system. Hell, results between -0.6 and -1.0
DEMAND the same attention, to develop the fabled "Stockbuster"
system...

You're the
statistician,
how about some help here?

Well, I'm not MUCH of a statistician, but my simple suggestion is above.


John:
I am sort of proud of what I just wrote there, because I think I may
have really nailed your
question. Let me know if not, I think it was a great question anyway.

William:
Well, not really,

John: I think I was addressing a limited point: when you said
the last price (the 100,000 dollar house eg) is the most reliable
indicator of tomorrow's price, but can a net do more, I was saying,
a net will in fact repeat the last price, when you train it on current
data
and then move the 'days' to a later date. And then it does better in
historical time, and it is up to the user to decide how much better and
what is the meaning of that. I think we both understood that technical
answer
anyway.

Yeah, it just boils down to the same simple test for me: does it work
or not? If it works sufficiently well, I could care less HOW it works...

William:
The point that a naked human mind has limited powers to calculate
the odds of a game is well-taken, quite obviously true, and has been
proven many times. Yet it is that first "common sense" notion that
is rejected by most market participants; take a poll here, and you'll
find that most think that "the human mind is capable of perceiving
subtle patterns that no computer could see".

This may indeed be true, but it is also true that the human mind
can perceive subtle patterns, like fluffy dogs in clouds, that don't
exist at all...

John: Wow that is really well put!!! Maybe a justification for a
neural
network would be to keep a person honest. They make a decision,
based on intuition. Then they say, can I get this dumb machine to see
it?

That's what I use back-testing and statistical analysis for.

William:
Hell, I've thought about digging out my old neural net
software and running some tests on the data sets I currently use for
market "predictions" and see how closely the numbers match up
against my money-flow model algorithms...

John:
We're curious to learn about your algorithms.

William:
It's a top-down model that basically goes from broad monetary
aggregates->"savings"->investment class->stock market->stock
sectors->individual stocks. Each step includes some amount of
"common sense" forecasts about future money flows, and a dynamic
summation analysis of previous actual changes and forecasts,
looking for "discrepancies", which may be either the result of
bad data (yuck) or (hopefully) what I call "potential", which
is the difference between where an aggregate or stock price
"should be" and where it actually is, generally due to the
"informational/trading friction" inherent in all markets.

The easiest way to describe "potential" is the classic example
of an "undervalued" stock, where the company is growing its
revenues and earnings nicely but the stock is languishing well
below "fair value". Another upstream example would be if
interest rates are rising, but instead of taking the safe returns
of a money market, with yields getting close to perceived
long-term stock market returns, people are still investing
instead in the stock market.

John: that sounds v competent.

William:
Since I am carefully summing the forecasted, current, and previous
flows at each level down to the thousands of individual stocks, I
can chose the stocks with the greatest "potential" at a some
semi-arbitrary
cutoff point (for years I've basically refused to invest in stocks
unless
I could "predict" they would go up at least 100% a year) without
making the "mistakes" of ignoring large-scale monetary aggregate
changes, sector rotation, etc.

John: OK

William:
Generally, I move in at certain points when "trend" starts to move in
the direction of "potential"...

John:
It sort of sounds like you are doing a lot of deeper analysis
than I envisioned. It is hard to think how a single computer program
could be helpful....

Well, there are programs, and there are programs. What I described
is just one "module" of a much larger program, or I can compile it as
a stand-alone program. In any event, it takes a substantial amount of
time to run, because it has to analyze fundamental and price/volume
data for thousands of stocks to choose the "top 100".

I wish I knew more about what you mean about
'flows.' Is it possible to calculate flows of currency, instead of
merely
for example individual volumes?

Sure. Recently here I mentioned that there was $300billion
more in money market funds a few years ago than there is now.
That $300billion "flowed" out of money markets and "flowed"
somewhere else...where? Well, since I'm tracking ALL these
large monetary aggregates, I KNOW, because that money
DID show up somewhere else...and since the average investor
DOESN'T track it, they have no idea it was ever there in
the first place, or think it was $800BILLIONTRILLION
MORE ON THE PROVERBIAL "SIDELINES", based
on...well, based on nothing...

I find volumes frustrating because
they refer to currency flowing in both directions (so a volume is not
as
good as what a mathematician would call a flow).

Ah yes, I and some others use some "proprietary" techniques to
transform "volume" into "flows". "Flows" at the stock asset class
level are reported to the Fed.

Curious if you can
enlighten me with some more details here, like an example, of flows
going down to individual shares which you described (sorry if I'm
using the wrong words here).

Alan Greenspan institutes an "easy money policy" in 1998 to ostensibly
avoid a world-wide economic collapse, economy/jobs boom, people
make use of 401(k) plans to invest in stock mutual funds (and a lot
more people start individual stock investing), stock mutual funds
either allocate their purchases towards tech/Internet stocks "because
that's where the action is" or the mutual funds that receive the most
inflows are the ones that specialize in tech/Internet stocks, Juniper
Networks reports "unexpected" phenomenal market uptake of
its backbone Internet routers, Juniper goes up a hundred-fold in
a few months and has a 4000 P/E ratio...

Is that what you were looking for? Of course, there was lot of
fun trading the other side of that mountain too...the good old days,
I wish they'd come back...

---
William Ernest Reid
Post count: 368



.