Re: ORBACUS narrow hangs
- From: Enrico Fichtner <enrico.fichtner@xxxxxxxx>
- Date: Mon, 11 Feb 2008 23:21:35 +0100
ciaran.mchale@xxxxxxxx schrieb:
On 9 Feb, 08:51, Enrico Fichtner <enrico.ficht...@xxxxxxxx> wrote:[clients pass a callback object reference to a register() operation on
a server. This normally seems to work okay, but under stress testing
client applications hang when calling _narrow() for an object in the
server]
It is difficult to debug programs remotely, but I suspect the problem
is deadlock between the clients and the server. It might happen as
follows (I'm taking guesses at some implementation details of your
system).
1. The server is multi-threaded and uses a thread pool of, say, size
10 to process incoming requests.
2. The register() operation is defined to be "oneway".
3. In the register() operation, the server invokes back on the
callback object, perhaps to test
that the callback object is contactable.
4. In the client, you do something like the following:
// basic CORBA initialization
orb = ...
rootPoa = ...
poaMgr = rootPoa->thePOAManager();
// create callback object and register it with server
callbackPoa = rootPoa->create_POA(...);
callbackSv = new CallbackSv(...);
callbackPoa->activate_object(callbackSv);
callbackObj = callbackSv->_this();
serveObj = ...
serverObj->register(callbackObj);
// more initialization that invokes more operations on the server
Server = T::_narrow(obj);
// go into an event loop so we can process callback invocations
POAMgr->activate();
orb->run();
If your architecture is similar to what I have described above then
the problem is that you are calling POAMgr->activate() AFTER serverObj-
register(). The CORBA runtime system in the client application willnot dispatch any incoming request until after you call POAMgr-activate(); if the client application has a single-threadedconcurrency model then you may also have to call orb->run() for
incoming requests to be dispatched. The default state of a POA manager
is "holding" which means that incoming requests will be queued up. It
is only when you call POAMgr->activate() to put the POA manager into
the active state that the POA manager will allow incoming requests
(and any previously queued requests) to be dispatched.
Within the body of register(), the server is trying to invoke on the
client's callback object. The thread in the server trying to do this
will block until the client has invoked POAMgr->activate(). If the
register() operation was a normal (twoway) operation then you would
have gotten deadlock even with just one client trying to register with
the server. However, if, as I suspect, the register() operation is
"oneway" then deadlock is avoided if you have only a small number of
clients invoking register at the same time. This is because a few
threads, equal to the number of clients concurrently invoking
register(), in the server will be temporarily blocked but there will
still be another few threads in the server's thread pool to process
more incoming requests. However, if the number of clients concurrently
invoking register() on the server equals or exceeds the number of
threads in the server's thread pool, then the all the threads in the
server's thread pool will be blocked waiting for
the client to call POAMgr->activate(). Unfortunately, the clients
won't invoke that until after they have invoked some more operations
on the server (note: under some conditions, _narrow() might make a
remote call to the server). These invocations from clients to the
server cannot be dispatched by the server because the server's thread
pool is currently exhausted. The result is deadlock.
As I said near the start of this posting, I am taking a few guesses at
the architecture of your client-server application, so some of the
details in my explanation might not fit with your actual architecture,
but the symptoms you describe certainly do sound like deadlock.
Regards,
Ciaran.
Thanks for the clarification. Even though I couldn't explain it to myself in such a detailed way, I had that feeling that it might be caused by such a locking (patt) situation. However, strangely it doesn't change it's behavior if a spend a huge amount of threads to the pool (e.g. 100) but still run only about 20 clients. What I also noticed, if I start the clients "by hand", client for client, the blocking effect is way less then if I start them all at once through a script.
Best Regards,
Enrico
.
- References:
- ORBACUS narrow hangs
- From: Enrico Fichtner
- Re: ORBACUS narrow hangs
- From: ciaran . mchale
- ORBACUS narrow hangs
- Prev by Date: Re: ORBACUS narrow hangs
- Next by Date: Example CORBA server that is also a Client to another server
- Previous by thread: Re: ORBACUS narrow hangs
- Next by thread: Example CORBA server that is also a Client to another server
- Index(es):
Relevant Pages
|
