Re: face-selective region

"feedbackdroids" <feedbackdroids@xxxxxxxxx> wrote:
Curt Welch wrote:

.... [continuation of response to prior post] ..........


From there, you have to add a reinforcement learning system to map the
current state (i.e., the stimulus signals), to behavior. So somewhere
in the link from the state defined by the activity in the cortex to
behavior, the brain must be implementing a reinforcement learning
system that strengthens behaviors based on rewards and weakens
behaviors based on punishments.


H does have his associative memory as the 2nd major part of his
hierarchical scheme. As he mentioned, the power of the overall system
comes from having utterly huge amounts of such memory. This of course
deals with the so-called "common-sense" problem. How can a human
successfully function in the range of environments it functions in, and
escape the usual brittleness problem of most AIs? The primary answer is
to have a lot of memory with stores for many different real-world
situations, skills, etc. This is part of the solution, but the weak
part of H's model is tying everything together.

Yeah, he's got a clear understanding of how associative memory is able to
tell us the "answer" for any new situation. It picks the "memory" which
most closely matches the current situation. He just didn't answer how the
system knows _what_ to do in the situation.

Also, as you indicate, there does need to be some sort of goal-system
overlaid on the rest of it which indicates which items to store and
which items to ignore. Towards the end of the book, H does talk about
applications for his model, but these are somewhat limited apps, such
as air-traffic control, so the goals are really set ahead of time, and
the domain of the learning situation limited, by the external user.

Yeah, I'm in that part of the book now. He tends to focus on the
perception side of the problem and talks about how a memory prediction
system would correctly understand what it was seeing. And he talks about
the obvious advantage of being able to watch a video screen and understand
what you were looking at. But he never answers the question of how you
train the system to produce the behavior you want to to produce - such as
calling the police when you see a crime. At best, he makes reference to
how you can tech it by repetitive example but that's not enough to explain
why the machine would want to mimic your examples instead of just punching
you in the face.

There are a lot of ideas in the book. H indicates that learning can
conceivably take place at "every" level, but this doesn't seem very
true for biolgocial hierarchies.As I've indicated many times,
plasticity seems to increase as you go up the levels, with more
hard-wiring at the lower levels.

Yeah, well, I'm with H on that one as you know. :)

I think you typically see less variation at the lower levels because much
of what is being learned at the lower levels is actually the
characteristics of the sensory systems themselves (which is the same for
all animals of the same species), and then the physical characteristics of
the universe (space/time/gravity etc) (which is the same for everyone).
Not until you get to the higher levels does the system have to start
learning more local effects like culture and local environment so not until
you get to higher levels would you expect to find more variation in what
gets learned.

I don't pretend that I know much of anything about the real physiology so I
don't know exactly what evidence you are seeing that makes you say there is
less plasticity at the lower levels, but if your prime evidence is that
there is less variation at the lower levels, the above argument answers
that.

Regards which behavior is to be ultimately produced, well, H's model
is as yet incomplete. It's basically a framework for a model, with
many details needing to be filled in. Likewise, I thought it was a
little too weak on temporal aspects, production of multiple
sequential operations, etc. I should probably review these aspects of
H.

His idea of how it generates output sequences is that it performs the
inverse of the sensory system. So what needs to happen is that if you
wanted to create a sequence like what was required for the Gettysburg
Address, the high level system needs to first activate the cells which
represent the concept of the Gettysburg Address. Those would activate
and stay active for the entire processes. Those in turn would trigger
the next lower concepts in the sequence, like "Four".


Yeah, I get this. Inverse hierarchy. Divergence of signals going down
the motor hierarchy, analogous to convergence of signals going up the
snesory hierarchy. That's very short-term. You still need a good way to
deal with longer-term activity. Even going around the entire outer
loops [ie, internal comps + external behavior], the loop activity is
very fast, under a second or so. You still need a good way to direct
actions over the course of many seconds, and even minutes.

With feedback systems, one thing leads to another, which leads to
another, on and on, so everything is done on the context of the
previous, but you still need some long-term directional control, else
you'd simply be responding constantly to immediate contingencies.
Brooks' reactiveness extended a bit with STM. It's not clear to me from
H's model how you get more long-term planning into the mix.

Yeah, and reinforcement learning is long term memory. So it's the
reinforcement learning system that answers how you can have a memory that
goes back to yesterday. It answers how you can set a goal on Monday to do
your food shopping on Friday after work and actually remember to do it.

The details of you you implement it to allow this to happen is not obvious,
but at least it gives one answer to the question of how and why it happens.

And reinforcement learning is a long term feedback loop.

That would stay active for as
long as you were saying or writing "Four", and would in turn generate
the next lower level sequence of signals needed to produce the "Four"
behavior. The "Score" behavior would be triggered by the combination of
context signals "Gettysburg Address" and "Four", and maybe even the the
lower level signals for the trailing behavior of producing the "Four"
sound. So, the only temporal signalling needed in the hardware is very
short term - just to create the correct delay from the end of "Four" to
the beginning of "Score". So the hardware which activates the "Score"
internal state only needs a short term temporal relationship to the
context that exists just before we would say "Score" when trying to
generate the entire sequence known to us as the Gettysburg Address.

Each step of a long sequence is generated by the hardware for each step
recognizing that it was now time to become active based on all the
context signals active in the cortex.

Where Jeff seems to see this as the motor cortex doing some inverse of
what the sensory cortex is doing, it doesn't in fact have to do that at
all. It can do the same thing just by acting as another sensory cortex
watching what outputs the lower brain is sending to the muscles. By
doing that, the motor cortex "knows" in effect, the current context of
the motor behavior.


Yes, feedback in the local motor pathways, combined with feedback from
outside around the longer sensory loop hierarchies.


The motor cortex is in effect recognizing that we are currently
reciting the Gettysburg address. And using that fact, each next
behavior in the sequence can be triggered, based on how far we have
currently advanced in the processes. So the motor cortex is driving
behavior, but it's driving it by telling the rest of the brain where we
are in the processes at each moment.


This is still relatively short-term. One thing leads to another. The
previous activity goes around the feedback loops [internal and also
external], is filtered, and then input to the associative memory
system, and new associated outputs are generated. That's fairly
straight-forward. That's the power of associative memory combined with
systems for abstraction and feedback. The output of the memory system
ultimately becomes the next input to itself.

However, you still need the value/goal system to oversee the entire
process. We're not simply reactive beings. There's more to life than
that. This seems to be missing from H's model at present. There is one
tiny reference to goal-directed behavior on p 158.


Rememer H had the hippocampus at the "top" of his hierarchy, but I
think this was just a wild shot in the dark, and wasn't too
compelling.

Yeah, I wasn't either.


Reading some other stuff [I'll start another thread on this, regards
hierarchy and dynamics, from book called The Primate Visual System, ed
Kaas + Collins, 2004 .... good book, you should take a look], the idea
of the HC at the top of the hierarchy probably comes from van Essen's
diagrams of the 30+ visual areas, with HC on top.

However, I think H misinterpreted this. The diagram shows the HC on
top, because essentially the only visual area it connects to is ER
[enterorhinal], which is shown on the diagram as being atop the
hierarchy of visual areas. The HC is not shown to connect with any of
the 30 other visual areas. However, if true, to me this means the main
control of STM is at the the top, rather than the HC being some kind of
"executive" module.

IOW, I interpret the system as the 30+ areas mainly being hierarchical
filters, and STM doesn't factor in significantly until later [ER level,
which is up approx 13 levels into the hierarchy].

I see most of H's idea of a memory/predictive system to be what creates STM
effects. If you have a system that can recognize a cat, and predict that
it's still there, even after you turn your back to it, then that is a STM
memory system that allows you to "know" there's still a cat there 2 seconds
after you can no longer see it. It's just the predictive memory system
that's in effect saying, "I expect with high probability to see more
"cat-patterns" in the sensory data soon". That's basically the system he's
talking about. It's a temporally sensitive pattern matching hardware which
picks up pattern clues not just from the present, but from the recent past
as well. It knows there were cat-patterns in the data 5 seconds ago so
it's predicting there's a higher probability to see more cat-patterns right
now than to see elephant-patterns. This is what allows the associate
memory to see only a small corner of the ear of the cat and quickly
classify it as cat-ear because the odds of that pattern being cat-ear
instead of something else is much higher because of the fact it saw the
rest of the cat 5 seconds ago.

For the classification system to use knowledge of what happened in the
recent past to make more accurate current classifications it must have some
form of short term memory in it.

The answer to where long term memory comes from is the changes that are
made to the hardware to implement reinforcement learning.

I suspect the correct answer is that the hippocampus is the controller
in the reinforcement learning system. It's the hardware which
implements reinforcement learning. And the reason Jeff couldn't figure
out what it's purpose was is because Jeff hasn't yet grasped that there
must be a reinforcement learning system at work to explain how the
brain actually shapes behavior, instead of just shaping knowledge -
which is what his half of the equation answers.


Yes.


However, with such a massively-interconnected system, you might not
need a top-level executive at all. Rather, recurrent activity in the
feedback loops keeps driving one thing to the next.

Right, as long as you have some type of temporal system that can active
signal X some short but fixed period of time after signals Y and Z
happens, then you have the foundation for a sequence generator.
Whether that delay is created with a recurrent feedback loop or other
system is not important.

The output of each
process is the trigger to the next one in sequence,

Exactly.

but at the same
time, the overall activity is continually being refined by the same
recurrent activity. The loop activity is continually stimulating
output from the associative memory, which enters back into the loop
activity.

But to create our type of structured behavior, you also need signals to
define long term context - such as the signal that activates, and stays
active, the entire time you are reciting the Gettysburg address.


Yes, as I discovered above, what seems to be a major shortcoming of H's
model.

This is
needed to allow you to say Four Score in that context, and Four Five,
when in the context of reciting your numbers. Without a hierarchy of
context signals to guide the sequence generation, you would be forced
to always follow the word "four" with the same behavior. So you could
learn to say 4 5, or four score, but not both.

And that is exactly what the neocortex is doing for us. It creates a
hierarchy of context signals not otherwise present in the raw sensory
data. Where as there is no single signal in the raw sensory data to let
the behavior generation system know there is a cat in front of us, the
cortex learns to spot cats and gives us that "cat" signal so we can
generate different output sequences when there is a cat in front of us
than when there is a dog in front of us.


Well, in real brains, we have parallel pathways from subcortical
centers including the amygdala and other limbic areas which factor in
emotions and other goals/values. Missing from H.

Yeah, he makes clear reference that an actual human brain is much more than
just the cortex and gives connections to the lower brain like you mention
as examples. To create a human like robot, with emotions, you would have
to spend all the time to reverse engineer all that lower brain stuff. I of
course agree in principle with that, however, the prime cause of emotions
in humans is the value system created by the reinforcement learning
algorithm. He thinks he could build useful systems without emotions, but I
think that without a value system, you can't build anything very useful at
all because you won't be able to train it to do what you want it to do.
Though how we express our emotions and how they change from day to day
might be heavily regulated by lower brain functions, the need for basic
emotions which is the value system created by reinforcement learning, is a
requirement for AI.

The nice thing about the H framework is that it's easy to add parallel
systems that can have totally different architectures [as the
subcortical and limbic areas have], and wire them [both as inputs and
outputs] into appropriate points in the hierachical framework.

Eg, sample ongoing activity in general, do some calculations, and then
feed either excitatory or inhibitory inputs back into the hierarchy at
strategic points. Such pathways are found in the real brain, but not in
H's current model.

Yeah, but he knows the brain is actually more complex and the book is only
an attempt to identify the correct top level framework so that the details
of the brain can be filled out in a top down fashion instead of the
bottom-up fashion most research on the brain has been done on. And as
such, I think he's on the right track and basically right about most of
what he says. He's just missing the reinforcement learning framework.

Yeah, my point was that, this is so because the overall connection
scheme is dictated by the DNA. Learning doesn't work in a vacuum. It
works in the context of a system highly-structured genetics and refined
by evolution.

Right.

--
Curt Welch http://CurtWelch.Com/
curt@xxxxxxxx http://NewsReader.Com/
.

Relevant Pages

  • Re: face-selective region
    ... comes from having utterly huge amounts of such memory. ... the cortex and other areas of brain. ... the motor hierarchy, analogous to convergence of signals going up the ... and reinforcement learning is long term memory. ...
    (comp.ai.philosophy)
  • Re: Syntax and robot behavior
    ... now because what I have to say below about reinforcement learning touches ... of the brain accessible only indirectly, ... entire neocortex is made up of the same fundamental hardware structure ... Yes, but a computer memory can, though structurally amorphous, ...
    (comp.robotics.misc)
  • Re: simulating xilinx block ram with modelsim
    ... The test bench should initialize memory ... attributes of the BLKRAM to initialize values. ... well of course i checked my input signals and i applie values to them. ...
    (comp.arch.fpga)
  • Re: Carl Shapiro: Where is the Win32 port of CMUCL?
    ... John Thingstad wrote: have to encode environmental differences between the child and the parent process as parameters: I/O redirection, current directory, environment variables, ignored signals, memory limits, scheduling priority etc. ... But forking a process to implement multithreading has the problem that if process1 frees some chunk of heap memory, because it's done with it, the memory lives on in process2, even though that *thread* might not really need any of those heap contents. ... Plus, on Unix, you can have an extra thread to synchronously handle signals, so you don't have to worry about async signals all the time. ... Of course if you execand set close-on-exec flags, most of the above isn't a problem, but as I said it's cheaper because it's only a single system call instead of two, and you can send whole data structures between threads for free. ...
    (comp.lang.lisp)
  • Re: will C# be perfomant enough for the next problem (HW programming)???
    ... "The interface is basically a signal router that can redirect the signals ... between the memory chips and 303's internal cpu and the signals between the ... will I be able to do it without delays ??? ... Memory: 64 measures x 7 TRACKS. ...
    (microsoft.public.dotnet.languages.csharp)