Re: network driver + replenishment
- From: gvarndell <gvarndell@xxxxxxxxxxx>
- Date: Fri, 18 Jul 2008 04:15:48 -0700 (PDT)
On Jul 17, 4:45 pm, noiset...@xxxxxxxxx wrote:
On Jul 17, 11:47 am, gvarndell <gvarnd...@xxxxxxxxxxx> wrote:
On Jul 17, 10:32 am, MB <madhu....@xxxxxxxxx> wrote:
Hi
I need a good sugestions about replenishing the buffers in network
driver wrt to performance.
I have a model in which, network driver receive a interrupt and ISR
will add a function to net job Q. Later, this netjob funtion pull the
packet from the network hw and give it to upper layer. Once upper
layer finish processing, it will again give back the buffer to driver,
now, which will be replenish back to network.
I've never seen a driver that works exactly that way -- in fact, it's
unlikely a driver *could* work exactly that way.
Typically, the ISR queues the netjob and leaves.
The netjob then takes the buffer away from the hardware, 'loans' it to
the stack, and immediately gives the hardware a new buffer.
I've always coded my drivers so that the hardware gets a new buffer
right in the ISR.
This reduces, about as much as possible, the likelihood of the
hardware not having a buffer when it needs one.
I've seen many network drivers that try to do all the ISR work at
netjob level and fail miserably when the network spikes with a big
flurry of very small packets.
More often than not, such drivers lock up so badly that the target has
to be reset.
You can defer passing the filled buffer up to the stack (do it in a
netjob) if you want, but I usually do that in the ISR too.
HTH
GV
I'm sorry, but that's wrong. Bear in mind that while you're executing
in ISR context in VxWorks, it's impossible for tasks to run. (Other
ISRs might run depending on the circumstances, but tasks won't.) The
longer you spend in ISR context, the longer you prevent tasks from
running again, even high priority tasks. Now consider what happens
with a typical ethernet controller that uses a DMA descriptor ring:
there might only be one frame waiting to be processed, or there might
be several. In fact, if there's a large amount of inbound traffic, you
could have hundreds or thousands of frames to process. You can't
process all of them at ISR context: that would prevent tasks from
running for an unknown amount of time. There are many VxWorks
customers who would not tolerate that.
If you want to see an example of where this can be a problem, set up a
VxWorks target with a driver that does its buffer handling at ISR
level and blast it with a heavy stream of UDP traffic. Then go to the
target shell and try typing a few commands. You may find that the
shell is sluggish and slow to respond while the traffic is present,
and then becomes fast again when you stop it. This is not supposed to
happen: the shell task runs at a higher priority (1) than tNetTask
(50) so it should always have priority: no matter how high the traffic
load, the shell should always remain responsive. This is the nature of
priority based scheduling in VxWorks. (Note that proper design
dictates that tasks that have high priority also be well behaved: the
high priority allows the task to run when it needs to, but it should
not spend a huge amount of time hogging the CPU, or other tasks will
suffer.)
The general rule is in fact to acknowledge/cancel and mask interrupts
in the ISR, and schedule all the other work to be done at task level
(in tNetTask). This is how most currently shipping drivers are
written. As you correctly noted, the driver will "loan" buffers
containing received frames to the stack, and it has to replace those
buffers with fresh ones. My approach is to attempt to allocate a
replacement buffer first: if that succeeds, then you can swap the
fresh buffer with the one to be loaned, and loan out the dirty buffer
to the stack. If that fails, then the driver leaves the dirty buffer
where it is and resets the DMA descriptor so that it can be re-used. I
also increment an error counter to indicate that a frame has been
discarded. This allows the hardware to continue receiving frames. Once
the out-of-buffers condition passes (it's assumed that running out of
RX buffers is a transient condition), the swap-and-loan process can
resume as usual.
Note that the ethernet chip itself may drop frames if all of the
descriptors in the RX DMA ring fill up. This is called an RX overrun
error. The driver should recover from such errors and not hang or
crash. This condition can arise if there is a lot of network traffic
to receive and tNetTask doesn't get a chance to run before the NIC
consumes all of the RX descriptors. This can happen if there are tasks
in the system running at a higher priority than tNetTask which are
hogging the CPU, or if interrupts are locked out for too long
(intLock()/intUnlock()), or if another driver's ISR is running too
long and preventing tasks from running.
It's also important that the driver implement a fair use policy so
that it does not monopolize tNetTask, especially in the RX path. The
typical strategy is that once the RX handler netJob has been scheduled
into tNetTask, it will continue processing frames from the hardware
until there are none left (the receiver goes idle). This isn't a good
approach though: if the interface is flooded with RX traffic, it might
never go idle, and you will prevent tNetTask from handling work from
any other interfaces in the system. The correct way to handle this is
to only process at most X number of frames at a time, and then break
out of the processing loop and call netJobAdd() to schedule the RX
handler to run again later. Among other things, this also allows the
TX cleanup path of the driver to run in tNetTask too. Note that you
should only re-enable RX interrupts after you have finished all the RX
work you intend to do. Keeping the interrupts disabled while the RX
handler is running prevents unnecessary context switches: there is no
need for the hardware to trigger an interrupt and run the ISR again
while the task level handler is still running: if new RX traffic
arrives, the handler should pick it up without needing to be prodded
again. Reducing the number of interrupts that fire helps the overall
performance of the system.
Now, I realize that the netBufLib API is in fact designed so that it
can be used in ISRs (and documented as such), and there were drivers
in the past that worked that way. However I think most of them were
older BSD-style drivers that were ported to VxWorks along with the
original BSD-based TCP/IP stack. The old style BSD drivers also used
"top half/bottom half" design where it was considered ok to do
processing in the interrupt handler. But this was because the BSD
kernel didn't really give you an alternative: there was no other
context to defer to. Technically, you could create a separate process
context, however this was not practical because the context switching
overhead was too high. With VxWorks, context switching is supposed to
be lightweight since VxWorks uses tasks, not full blown processes.
Also, there is nothing about the "defer to tNetTask" approach in and
of itself that should lead to instability. Of the dozens of drivers
that I've written that use this model, all of them hold up very well
under load, and I've pounded the snot out of them with various torture
tests (netperf, Smartbits). If a driver that uses this model does hang
under load (or worse, causes the whole target to hang), it's not
because the design is wrong: it's because the driver is buggy as an
anthill and needs to be fixed.
For the record, Windows drivers also use the same model: NDIS drivers
have an ISR routine and a handler routine. The ISR runs at device
interrupt level and _only_ acks/cancels and masks interrupts. The
handler does the rest of the work. (The difference though is that in
Windows, the handler runs at DISPATCH_LEVEL, which is not the same as
task context in VxWorks. But it's still a different context from the
ISR.)
-Bill
Wow, you must be a good typer, it would have taken me hours to tap out
such a lengthy reply. ;-)
Anyway, I wanted to respond to just a few key points.
1) responsiveness of the target shell during network storms has never
been something I concerned myself with.
2) if the driver uses netjobs to pre-fetch network buffers so the ISR
can *quickly* access them without calling netBufLib directly, then
the ISR can actually do less work by replacing the buffer at ISR
level. If, for whatever reason, the supply of fresh buffers becomes
exhausted, the ISR then falls back to simply re-using the buffers (as
you mentioned) it already has and loosing the packet that was in it.
3) Code for reliably recovering from receiver shutdowns due to no
buffer is often difficult to write and debug, which increases the cost
and development time. Most vxWorks users will not tolerate that if
they're paying you to do a driver them. If it's easier to ensure that
the receiver never shuts down, and I say it is, *and* the method by
which you accomplish that makes a higher performance driver, then why
not do it?.
4) disabling receiver interrupts and relying on a netjob to re-enable
them is a bad idea. What if a user is furiously typing 'll' over and
over again at the target shell? ;-) (that was a joke)
It's obvious you've been in the trenches with this stuff and you
certainly helped the OP with this post.
I would just suggest that some of what you wrote is Wind River dogma
which has shaky foundation.
I've debugged and written a few network drivers myself.
Regards,
GV
.
- Follow-Ups:
- Re: network driver + replenishment
- From: noisetube
- Re: network driver + replenishment
- References:
- network driver + replenishment
- From: MB
- Re: network driver + replenishment
- From: gvarndell
- Re: network driver + replenishment
- From: noisetube
- network driver + replenishment
- Prev by Date: Help - VxWorks Launcher not starting (Tornado/Vxworks 5.5.x)
- Next by Date: Re: Why calling kernelInit two times.
- Previous by thread: Re: network driver + replenishment
- Next by thread: Re: network driver + replenishment
- Index(es):
Relevant Pages
|
