Re: Can operant conditioning account for all learning?

Curt Welch wrote:
Tim Tyler <seemysig@xxxxxxxxxxxxxx> wrote:

Of course *now* we have cultural inheritance - and the
"trial and error in each generation" argument no longer
applies. But the current situation is not relevant when
considering the genetic complexity of the human brain.

Once the brain developed enough power to implement language, cultural
inheritance took off. I like looking at that event as the opening up of a
new channel for the communication of hardware design from organism to
organism. Before that really took off, the only way organism design was
passed from person to person was through DNA encoded messages. Each
organism only had channels to two other organism, and they only got one
message from each at birth. It a very slow and limited communication
channel for design features.

But with the addition of a learning brain, and language skills, humans in
effect became "software programmable". The design of their brain could be
changed in real time by instructions received through language. Not only
could we change far faster, we could now receive software updates for
survival from any member of the species, not just our two parents. It's
not as much that humans "created" culturally inheritance as evolution
created it. And once it created it, the speed of evolution exploded
exponentially. More things have changed, and are changing faster, on the
earth since this update to evolution than at any point in the history of
the planet. And it's all due to the learning brain which opened up this
faster channel for the transport of design features.

Yes, sure, but this is all totally irrelevant to the topic
it arose out of - which was whether the human brain has
a lot of genetic complexity, or whether it's basically
equivalent to some simple learning algorithm.

Recent events have not had time to make much difference
to the genetics of the human brain. And the human brain,
is essentially little more that a chimpanzee brain, with
some extra "add more neocortex here" instructions added.

So while it might be fun to digress into the wonders of
cultural evolution, it seems like a digression from the
matter at hand.

I don't happen to believe the brain has all that many adaptations to
the basic learning hardware for each domain it deals with. It believe
the generic learning system is far more powerful than most people
grasp.
The neocortex looks pretty repetitive before it is properly wired up -
and it does some fairly spectacular stuff. But maybe we need to
reproduce all the more structured old bits before we can get off the
ground in the first place.

Others think that as well. It seems to be a recuring thought from John
Casey. There's no doubt that we get a leg-up from extra features hard
wired into us at birth - after all, we could starve or choke to death if we
had to learn to eat, and swallow instead of having much of that built into
us. But there are no dount many other features that give us a leg up.

However, intelligence is not about where you start. It's about how good
your are at learning. It's the speed at which you are able to improve your
behavior on your own.

Well, this /is/ the hypothesis we are discussing. The idea
I am presenting is that this is not true. That where
you start may give you a /big/ leg up, that would be hard to
get through learining alone, through lack of appropriate
training data.

Millions of ancestors died to produce the genetic
configuration of newborn babies - and the pattern of their
deaths produced a complex pattern of information, that
we are all born with. There /may/ be shortcuts that will
make it easier to get hold of the required information
a second time - especially since we can see an existing
route - but it still might be terribly difficult.

This information was originally acquired from the
environment by a dramatically different kind of
adaptive learning mechanism (if that perspective
is helpful at all), using signals like death as
a reinforcing mechanism. It may be difficult to
obtain it from the environment in other ways,
through lack of appropriate reinforcing training
data. And if so, building a "simple learning
algorithm" may be an approach that never gets
far off the ground.

Babies don't exactly start out with much of anything. They are dead dumb.

In your opinion. Babies start out with lots of DNA - which is
very complicated, and we don't understand very much of it.

I don't think it's at all significant how much of leg up evolution gave us.

Right - but is this a rational position?

Our robots can start out totally stupid, and only be at most 1 year behind
humans. So instead of taking 12 years to equal the behavior skills of a 12
year old, it will take the robot 13 years. Not realy important.

However, this /might/ be totally wrong. The "training data" needed
to acquire the inherited material was supplied in the form of ancestor
deaths - and it is not obvious that it is available in other ways.
Without appropriate training data and reinforcement, learning
fails to take place.

Going from perceptions to degree of goal attainment isn't easy.
Nor is allocating rewards to actions with delayed or distributed
consequences easy.

Yes, finding simple solutions to those problems is the key to solve this.

Of course, the brain allocates rewards to the behaviours that
it thinks led to them - at least in part - by using high level
intelligence and a causal analysis of the events that led to
the reward. So, to /eventually/ duplicate that, we should be
careful to /avoid/ wiring up all the reward systems up at
too low a level.

On the other hand we need to avoid giving agents /too/ easy
access to their own reward systems, or they will simply find
ways to snort up all their reward in one big lump.

There are many algorithms in the reinforcement learning work that give us
solutions to those problems - none of which apply directly to the domain
humans work in however. Finding algorithms for the high dimension
non-Markov real time sensory space which the brain works in is _the_
problem of AI as I see it.

Right. But at the moment, our learning algorithms are not up
to scratch, and there's the whole "initial brain architecture"
problem which you are not considering to be significant.

Retinal cells don't just wire themselves into motion and
edge detectors accidentally - or as a result of exposure to
the evironment. And that's just the sensory tip of a huge
"brain ontology" iceberg - which you do not seem to be
considering to be much of a problem, but which /is/ quite
likely to be a rather large problem - of which the
"reward" issues above are simply one facet.

A brain's main function typically involves constructing a simulation of
its environment, and considering the consequences of its possible
actions in terms of reaching its goals.

The brain's approach to this problem is pretty odd. It hacks
together a simulation out of heuristics and exceptions based
heavily on its own past experiences. That's not how e.g.
Deep Blue worked at all - and it may well be worth considering
other approaches to building simulated worlds for AIs - e.g.
by building in what we know about the laws of physics in the
AI's world.

Well, no that's not really what the brain does at all in my view.

The "constructing a simulation of its environment, and considering
the consequences of its possible actions in terms of reaching its
goals" is decision theory, based on an economic analysis, known as "expected utility maximisation". The idea stems from the John von Neumann and Oskar Morgenstern work on microeconomics.

> That's what it looks like it's doing when we watch what our own brain seems to be
doing. That the behavior it produces which we can remember, and talk to
ourselves about. We are aware of the fact that we made a mental plan to do
things, and we are aware of working out the actions in our head before
doing something. But to belive that top level behavior is what the brain
is doing inside, or how the brain works, is a standard fallacy of AI in my
view.

It's a functional description. Most organisms that are
working properly act like economic agents with a defined
goal - namely propagating their genes. They are not
necessarily very /rational/ economic agents, but they
do their best - since failure to behave as such an agent
means you have behaviour patterns that burn up your
resources without providing benefits in terms of your
goals - and evolution does not take kindly to organisms
with severe problems in such areas.

Deep Blue, for instance, worked just like this.

It had a chess simulation, it considered the
possible consequences of its actions, and
predicted their consequences in a simulated
chess universe, and then made the move that
maximised its chances of attaing its goal of
winning the game.

It didn't learn at all, I believe. Its designers
had built the best "chess-physics" simulation they
could manage into it at birth - so there only
thing to learn about was the opponent, which
I believe it didn't bother adapting to at all.

Humans are poorer approximations to the ideal
of a rational economic agent:

Their simulations are appalling, and their
rational behaviour is totally illogical. Yet
microeconomics is pretty clear that this is
the attractor towards which any intelligent
goal-driven system is eventually drawn. If
humans are a poor approximation of this
ideal, so much worse for them.

This analysis suggests that any differences arise
because humans are primitive intelligences - not
because the microeconomics and decision theory
are not the appropriate model for intelligent,
goal driven agents in a complex environment.

The brain selects behaviors based on accumulated statistical
information about the correlations between those behaviors
> and rewards.

I don't have a problem with this description. It's
on a different level from mine, though. Yours is
more of a nuts-and-bolts description.

The bran performs it's "odd" behaviors of acting out behaviors a
head of time (aka planing) because that's a behavior that it's
statistical system as shown to be advantageous to the creation
> of future rewards.

Indeed. That is quite compatible with the idea of agents
as imperfect approximations to rational economic agents.

Rational economic agents consider their possible actions,
consider their consequences on the basis of what they know,
choose the actions which best meet their goals - and then
iterate: e.g.:

``In order to make a decision in the world, you must first
have clearly specified goals. Then you have to identify
the possible actions you have to choose between. For each
of those possible actions you have to consider the
consequences. The consequences won’t just be the immediate
consequences, but you also look down the line and see what
future ramifications might follow from your action. Then
you choose that action which is most likely, in your
assessment, to meet your goals. After acting, you update
your world model based on what the world actually does. In
this way you are continually learning from your experiences.''

- http://www.acceleratingfuture.com/people-blog/?p=221

Brains have crummy physics simulations. Their idea of
predicting the actions of other minds is to pretend that
they are like you, with some tweaks based on past
interactions. And their rationality circuits are
pretty fried - humans exude an aura of biases -
as though they are steaming piles of irrationality.

But this just means we are crappy intelligent
agents born of evolution. The last of the
undesigned minds that will ever exist - perhaps.
We meet our goals miserably. Our failures to act
like rational economic agents necassarily mean we
burn up our resources without providing benefits
in terms of our goals. Future organisms will
attain their goals more successfully. They will
act more like rational economic agents would.
--
__________
|im |yler http://timtyler.org/ tim@xxxxxxxxxxx Remove lock to reply.
.

Relevant Pages

  • Re: The Future
    ... going on" in the human brain which current physical theories cannot ... machine from functioning as a quantum computer and duplicating it. ... But even if we program the AIs to like humans, ...
    (comp.lang.cobol)
  • Re: The Future
    ... on" in the human brain which current physical theories cannot explain. ... from functioning as a quantum computer and duplicating it. ... But even if we program the AIs to like humans, ...
    (comp.lang.cobol)
  • Re: Qualia Question
    ... >> working on the reverse engineering problem. ... In the "everything is physical" model, the mind and the brain are the same ... And that's a simple PC designed by humans. ...
    (comp.ai.philosophy)
  • Re: Could anyone explain this please?
    ... their blood - the brain is an oxygen glutton so.... ... HUMANS have A1 and A2. ... in Negroes, rarely in whites, never in yellows. ...
    (sci.anthropology)
  • Re: Scientists learn why some men may be more interested in football than sex
    ... An official diagnosis is: "schizotypal personality disorder ... brain were activated. ... that of goals being scored," the researchers conclude. ...
    (soc.men)