Re: A tricky problem about Process.wait and popen


First of all please don't post the same message several times.

uncutstone wrote:
I think I already have understanded all of these. But a understanding
doesn't mean a solution.

Let me give the problem I want to solve.

There is a parent process and a fixed number of child processes. Parent
process is responsible for forking a fixed number of child processes
using IO.popen and then wait for any child process exits. Whenever a
child process complete its task , it will write the result in pipe and
signal the parent process "mission complete" and then exits. When
parent get the signal, it will read correspondant pipe to get result
and then creating a new child process .

Try the implementation I suggested. The IO created by IO.popen will not signal EOF before the child terminated. So just use read or readlines to read the pipe's contents until you get everything (= process terminated). You can then evaluate it as you see fit. (see the suggested code I posted earlier).

If the result returned by the child process is small, pipe doesn't get
full, it works well.

You must not rely on this. Increasing the pipe's size is not an option as you still have a limit. Your process is responsible for reading the pipe in order to prevent the writer from blocking.

But if the result is big and pipe gets full, the child get blocked and
signal cannot send to parent process.

Let me make it simple.
1.Multiple child process want to return result to parent process via
pipe.
2. Parent need a mechanism similar to unix select() to get notification
from child process when it completes its task.

The code is something like this:

main.rb
processNum = 10
pipes = Hash.new
numOfProcessRunning = 0
loop do
apipe = IO.popen("ruby child.rb","w+")
cpid = apipe.gets.strip.to_i
pipes[cpid] = apipe
numOfProcessRunning += 1
if numOfProcessRunning >= processNum
cpid = Process.wait
apipe = pipes[cpid]
apipe.each { |line|
#.... , get the result
}
apipe.close
pipes.delete(cpid)
numOfProcessRunning -= 1
end
end

This looks overly complex. You could do something like this:


PARALLEL = 10

threads = (1..PARALLEL).map do |i|
Thread.new i do |count|
IO.popen("ruby -e 'puts \"child #{count}\"'") do |io|
res = io.read
puts "Child #{count} returned #{res.inspect}"
end
end
end

puts "Started"

threads.each {|th| th.join}

puts "Terminated"


child.rb:
puts Process.pid
$stdout.flush
#..... , do something, then
puts result


Btw, if your child processes are Ruby processes you can use fork with a block:


PARALLEL = 10

threads = (1..PARALLEL).map do |i|
Thread.new i do |count|
read, write = IO.pipe

cpid = fork do
read.close
write.puts "I'm child #{count}"
write.close
end

write.close
res = read.read
puts "Child #{count} with PID #{cpid} returned #{res.inspect}"
end
end

puts "Started"

threads.each {|th| th.join}

puts "Terminated"


If you need to communicate more complex information between child and parent you can use Marshal on the pipe:


PARALLEL = 10

threads = (1..PARALLEL).map do |i|
Thread.new i do |count|
read, write = IO.pipe

cpid = fork do
# in child process
read.close
Marshal.dump({:result => "I'm child #{count}"}, write)
write.close
end

# in parent process
write.close
res = Marshal.load(read)
puts "Child #{count} with PID #{cpid} returned #{res.inspect}"
end
end

puts "Started"

threads.each {|th| th.join}

puts "Terminated"


These examples with explicitly created pipe work the same as the popen approach, i.e. the parent thread blocks until the pipe reaches EOF which in turn happens when the child exits.

Kind regards

robert
.

Relevant Pages

  • Re: fork and pipe
    ... In summary, the pipe(2) call creates a single, unidirectional pipe, ... The parent process calls pipeto open two fds. ... the parent process calls forkto spawn a child process ... the child process inherits all open fds from the parent process, ...
    (comp.unix.programmer)
  • Re: A tricky problem about Process.wait and popen
    ... There is a parent process and a fixed number of child processes. ... using IO.popen and then wait for any child process exits. ... it will read correspondant pipe to get result ...
    (comp.lang.ruby)
  • Re: How to time out a forked command but still see output?
    ... Again, if the program hangs and there's a timeout, then the handler ... closes the pipe and launchs the program in the ... be aware the child process will ... terminate with a SIGPIPE as soon as it tries to write to the closed ...
    (comp.lang.perl.misc)
  • Re: Pipes on Wince (I have them), and device auto-loading.
    ... appropriate paths for each child process. ... the device nature of the pipe from the user, ... like to somehow have the device auto deactivate when the last ... In PipeLib there is a thread that waits ...
    (microsoft.public.windowsce.embedded)
  • Re: Strange Windows warning with IO.popen: The process tried to write to a nonexistent pipe
    ... premature termination of them (in the child process spawn by ruby). ... how do we fix it Luis? ... The process tried to write to a nonexistent pipe. ...
    (comp.lang.ruby)