Re: Ben G on reinforcement-learning and the wirehead problem

On Jun 19, 11:50 am, c...@xxxxxxxx (Curt Welch) wrote:

When I wave my hand around it is not making jerking
motions 5 times a second like an eye saccade. It's
moving smoothly and continuously though a 3D space.
The various muscles in my body are likely making
various jerking motions as they control and correct
the motion but the corrections in all the muscles
combined are happening at rates far in excess of 5
times per second.

You jumped from eye saccades to arm movements when I
was talking about temporal illusions of eye movements
to illustrate how you need to actually do experiments
to know what the brain really does! I never suggested
arms move in 5 jerks per second.

You have looked at these experiments and seem to be
concluding that the brain is "making decisions" at
some low rate of something like 5 times per second
and that's just not justified by the type of
experiments you are talking about.

I define making A decision as: the observable action.

When the button is pressed THE decision has been made.

You seem to be talking about lots of little decisions
taking place in the black box. I am talking about
what we actually observe, an output. If someone picks
up a cake we say they made a decision to pick up the
cake, a single event, we do not say they have just made
millions of internal "decisions" to pick up a cake.
If a rat presses a lever we say the rat decided to
press a lever we don't say many decisions resulted
in a lever press which is how I think you see it.

When we record the time delays between the stimulus
and the reaction we find peaks of about 0.24 seconds.
How are we to explain this preference and avoidance
of reaction times? Doesn't make sense if you think
of stimulus responses being continuous in which case
the response would have taken place at any time after
the stimulus.


John, I'm not talking introspection. I'm talking
PHYSICS. It's impossible to build a robot to perform
such behaviors if all you do is update their actuators
5 times per second. Do you honestly think you can
build a robot with arms and hands and make it play
the piano as accurate as a human can by using a
control process that sends new commands to the arms
and fingers only 5 times per second?


As I indicated above, this is another example of the
personal way you use words. If someone presses a
button or lifts an arm that is ONE decision. It is
not about the millions of pulses involved to carry
out that ONE decision. Although I would hasten to
add I am not suggesting a particular part of the
brain makes that decision. It is in some sense a
group decision.

As indicated before decisions take place at preferred
intervals of time after the stimulus indicating an
oscillation is involved just as in a computer program.
If the game character doesn't move in this screen
frame it has to wait for the next screen frame, no
matter how many "decisions" are taking place between
frames.


You do understand that the complex control program
that drives the arms and legs to perform a task such
as play the piano was learned right? Which means the
power of our generic behavior learning hardware has
to have the resolution needed to make a robot with
10 fingers play the piano as accurate as a human right?

Indeed we learn to play a piano. At the start we make
decisions as to what key to press. This is all "recorded"
for playback later. You are confusing this playback
with the decision making process that controls both the
learning and the execution of the process.


I don't have to do an experiment to know that humans
have the ability to use their arms and legs in parallel
for different tasks. Just look at someone walking, or
a drummer playing the drums, or anyone playing a some
sport game.

Current programs in current computers are serial right?
They can control multiple outputs at what appears to be
at the same time. You DO have to do experiments. You
cannot detect the difference between multitasking and
parallel processing when the multitasking is too fast
for your sensory input to handle.


The brain is a parallel control system because it's got
lots of different body parts to control.

The brain is indeed a parallel control system but not all
tasks are possible by this parallel control system in
which case it has to resort to a serial process. Only
experiments on different tasks can tease them apart.


The networks that create this behaviors are fully
interconnected.

What experiments have made you believe this?

I can drive and talk on the phone at the same time, but I
didn't not say I can drive just as well while talking on
the phone nor talk on the phone just as well while driving.

I think there is confusion as to what we mean by a parallel
brain. You see it as one bag of units. I see it as a collection
of useful modules, including modules for controlling modules.
The parts within modules are not all fully interconnected and
the modules are only -potentially- fully connectable if that
is required. Think of the object modules in OOP. It makes
sense that one module doesn't have full or uncontrolled access
to the parts of another module.

The problem is, you start with a belief about how the brain
is wired and then draw conclusions about the brain based on
those beliefs rather than experimental evidence.

Yes, we have a reaction time that is limited by the speed
at which information can flow though the brain. Duh.

But that wasn't the point. Duh. It is the different amounts
of time for different tasks that is the point. It is used to
work things out.

From another post Curt wrote:
John believe humans are far more complex than just generic
learning systems and that to get close to human behaviour
we will need a lot more in the way of innate support
hardware (but is normally fairly vague as to what the
required extra hardware actually does).

The evidence is that the brain's learning system has a lot of
innate hardware at its disposal, which makes evolutionary sense
and is separate as to what is possible with learning machines.

I see no issue with investigating the possibility of a system
showing learning behavior with just raw input. My personal
view is the learning system of the human brain doesn't have
to learn about the raw input as evolution has provided networks
that do that automatically. So although I agree that we learn
ALL our high level behaviors I don't agree that we learn them
out of the raw data.

And it is my view that the few low level innate behaviors made
from raw data would be hard to learn compared with the millions
of possible high level humans behaviors.

I also think the human learning system is primed to learn from
the social environment which is itself evolving. Much of what
we know is taken not from our own learning efforts but from
the accumulated knowledge of this social system. Without this
social system we probably wouldn't be much smarter than a chimp
no matter how much raw data we had available.


JC
.