Re: Existential risks

casey wrote:
On Apr 28, 1:37 pm, c...@xxxxxxxx (Curt Welch) wrote:
...

You are making the mistake of thinking that _we_ (the brain
/ the AI) is in control. We are not. We never were. We are
stupid ass reinforcement learning machines that are total
slaves to the hardware in us which determines our prime
motivations. We don't control those motivations, and as a
result, we don't control what is going to happen to us.

This is just totally wrong. It's based on the false
assumption that "intelligence" = "a desire to survive".
It's based on the assumption that since humans use our
intelligence to do all this complex stuff to help us
survive, that all intelligence will naturally work the
same way. I don't believe that.

I believe "intelligence" = "reinforcement learning machine".
Reinforcement learning machines have no desire to survive.
They only have the innate desire to produce behaviors which
will maximize their reward signal. The behavior the
intelligent machine ends up producing will be whatever
it takes to make the reward system produce more rewards.

+--------+ +---------------+ +-------------+
| reward | | Reinforcement |--->| |
+--->| |---->| Learning | | environment |
| | system | | Machine |<-+-| |
| +--------+ +---------------+ | +-------------+
| |
+<-------------------------------------+


So, essentially you see two connected systems where the states
of the reward system can effect the states of the RLM system
but not vice versa.

But in Real Life the RLM _does_ affect the reward system.

[...]


--
wolf k.
.