Re: Ring buffer overflow and LAN91C111 ethernet driver

On Apr 1, 9:19 pm, sant...@xxxxxxxxx wrote:
Hi Bill,

Thanks for your kind reply.
My network driver is the modified version of Linux SMC91111.c.
Some of the modified things are,
1. During each receive interrupt, my receivehandle routine is called
through netjobadd. And ISR routine is as you explained above(i.e
masking).
2. End driver with mux interface support.
3. using clusters for data transmission.

For example, if the chip receives a burst of 10 ethernet frames, you only >need to do a netJobAdd() once...

When i debug my source code,
1. On each receive interrupt my ISR calls netJobAdd through which my
handlereceive routine runs at tNetTask.
For example, when i pinged 60000 bytes data from PC to the target
deivce,
for each ping, netJobAdd function is being called around 42 times.

Well, that could be a problem. Remember what I said: you don't want to
keep calling netJobAdd() when you already know tNetTask has been
scheduled to do some work, and you should mask off the RX interrupt
until you know the task level RX handler has completed. For the SMSC
91C111, which requires programmed I/O, the logic would be something
like this:

void smscRx
(
SMSC_DRV_CTRL * pDrvCtrl;
)
{

/* Check to see if packets are waiting in the chip's SRAM. */

while (chipRxRamNotEmpty(pDrvCtrl) == TRUE)
{
/* Allocate an mBlk tuple */
pMblk = netTupleGet (pDrvCtrl->endObj.pNetPool, ... );
/* Find out the size of the current packet */
pktlen = getSizeOfPkt (pDrvCtrl);
/* Copy the frame data from the SRAM into the mBlk */
copyPktFromChip(pDrvCtrl, pMblk->m_data, pktLen);
/* Set up the required parts of the mBlk header */
pMblk->m_len = pMblk->m_pkthdr.len = pktlen;
pMblk->m_flags = M_PKTHDR|M_EXT;
/* Give the frame to the MUX. */
END_RCV_RTN_CALL (&pDrvCtrl->endObj, pMblk);
/*
* Tell the chip we've completed handling this packet. This
will
* tell the 91C111 to that the SRAM currently occupied by this
* frame is now free to be used by subsequent frames. It
should
* also advance the chip's packet pointer to the next frame.
*/
releasePktMemoryInChip (pDrvCtrl);
}

pDrvCtrl->rxPending = FALSE;
reEnableRxInterrupts (pDrvCtrl);

return;
}

void smscInt
(
SMC_DRV_CTRL * pDrvCtrl;
)
{

status = readInterruptStatusRegister (pDrvCtrl);

if (pDrvCtrl->rxPending == FALSE && status & SMSC_RX_EVENTS)
{
pDrvCtrl->rxPending == TRUE;
maskOffRxInterrupts (pDrvCtrl);
netJobAdd ((FUNCPTR *)smscRx, (int)pDrvCtrl, 0, 0, 0, 0);
}

return;
}

Note what happens here: as soon as we get an RX interrupt, we fix it
so that a) no more RX interrupts can occur, and b) we set rxPending to
TRUE. (In theory, just turning off the RX interrupt should suffice; we
use the rxPending flag as an extra safety mearsure.) This should
guarantee that we don't call netJobAdd() again until the smscRx()
routine has finished grabbing as many frames from the RX SRAM as
possible. This prevents you from submitting too many netJobAdd()s.

Basically, you want to avoid the case where you call netJobAdd(), and
then another interrupt occurs before tNetTask even gets to run,
causing the ISR to call netJobAdd() again. It's okay if you do:

ISR-> netJobAdd(smscRx(), ...);
tNetTask -> smscRx();
ISR-> netJobAdd(smscRx(), ...);
tNetTask -> smscRx();
ISR-> netJobAdd(smscRx(), ...);
tNetTask -> smscRx();
ISR-> netJobAdd(smscRx(), ...);
tNetTask -> smscRx();
ISR-> netJobAdd(smscRx(), ...);
tNetTask -> smscRx();
ISR-> netJobAdd(smscRx(), ...);
tNetTask -> smscRx();
ISR-> netJobAdd(smscRx(), ...);
tNetTask -> smscRx();

It is _not_ ok if you end up with:

ISR-> netJobAdd(smscRx(), ...);
ISR-> netJobAdd(smscRx(), ...);
ISR-> netJobAdd(smscRx(), ...);
ISR-> netJobAdd(smscRx(), ...);
ISR-> netJobAdd(smscRx(), ...);
ISR-> netJobAdd(smscRx(), ...);
ISR-> netJobAdd(smscRx(), ...);
ISR-> netJobAdd(smscRx(), ...);
ISR-> netJobAdd(smscRx(), ...);
ISR-> netJobAdd(smscRx(), ...);
ISR-> netJobAdd(smscRx(), ...);
tNetTask -> smscRx();

This is bad, because you could end up calling netJobAdd() too many
times, at which point you will overflow the job ring. Once that
happens, you will start losing events: subsequent netJobAdd() calls
will fail, and other interrupts may go unserviced.

And regarding my actual problem, i.e.( "netjobadd:ring buffer
overflow" message stops occuring after sometime).

Below are few observations during ringbuffer overflow error while
pinging 60000 bytes(netjobadd ring buffer size is set to 40),

Remember what I said in my previous e-mail: the SMSC 91C111 only has
8K of internal RAM. When I worked with this chip (also on an SH
board), I found that the CPU could not drain the SRAM fast enough at
100Mbps speeds. RX overrun errors were very common. You can recover
from the RX errors, but you will still be dropping frames. This
problem did not occur at 10Mbps speeds, so I rigged the driver to
default to always negotiate a 10Mbps link.

1. Ring buffer overflow message occurs around 85 times.
2. netClusterGet() function returns NULL.
3. pClBuf->pClNext pointer becomes NULL in clusterGet function.
4. Number of free clusters member variable "clNumFree" goes to 0.
5. When stopped pinging, number of restored clusters i.e clNumFree
shows only 43.
i.e Actuall Number of clusters - number of ring buffer overflow
messages = clNumFree
=>128 - 85 = 43
6. mbufShow, netStackSysPoolShow shows many clusters are still free.
7. when i restart the target system, all returns to normal state.

First of all, forget about calling netMBlkGet(), netClBlkGet() and
netClusterGet() separately. Just have your driver create a buffer pool
and call netTupleGet(). It will construct a complete mBlk tuple for
you. (The one benefit to this is that it makes the code a little
simpler.)

Second, one thing that's not clear here is just where you're
allocating your RX buffers from. The fact that you mention
netStackSysPoolShow() suggests to me that you're trying to allocate
them from the TCP/IP stack's own pools. Don't do that. You can have
your driver create its own private buffer pool with netPoolInit().
These mBlks can be loaned to the stack and will be returned when the
stack finishes with them.

Third, you should probably have the driver create a pool with at least
64 mBlk tuples. The number of mBlks available has some bearing on
fragmentation limits: with a fragmented IP packet, the stack will hold
onto all fragments until the whole packet is reassembled. Exactly how
much fragmentation where will be is a function of your particular
application. If you really expect 64K IP packets, then you should
allocate enough buffers to handle 42 fragments (or more, if you expect
a lot of traffic).

Fourth, even in the case where the driver doesn't have enough buffers
to handle extreme fragmentation, it should still recover. That is,
even in the case where netTupleGet() in the smscRX() handler fails, or
in the case of the chip signalling an RX overrun error (all SRAM was
exhausted), the driver should not crash, or fall out of sync with the
chip, or leak buffers. It is expected that you might drop a frame on
receive occasionally. That's ok. But even so, the driver should
continue to work.

As per the above observations, when ring buffer overflow occurs
clusters once used are not getting freed.
In normal operation, i confirmed that clusters are getting freed
continously.

Please tell me or guide me, why the clusters are not getting freed.
Or what are the other things, should i check to know why the clusters
are not getting freed.

My guess is that your driver is providing the TCP/IP stack with only
some of the fragments in your 60000 byte ICMP packet, and the TCP/IP
stack is waiting for the rest to arrive. The TCP/IP code knows there
should be a total of 42 fragments (based on the information in the IP
header of the first packet), so now it is waiting for all 42 fragments
to arrive. But your driver was unable to receive all 42 fragments.
Maybe it only handled 20 of them. The TCP/IP stack must hold onto
those 20 buffers until the last 22 fragments arrive, or until the
reassembly timeout expires.

You can test this to see if my theory is correct:

- Rerun your ping test
- Check the number of free buffers to see if some are missing
- At the VxWorks target shell prompt, do the following:

-> ip_drain()

- Check the number of free buffers again. If the buffers were being
held in the reassembly queue,
calling the ip_drain() function should force the stack to release
them. (ip_drain() is the function
called by the reassembly timer handler; basically, you are fooling
the stack into thinking the
timer has expired early.)

-Bill
.

Relevant Pages

  • [PATCH] [TRIVIAL] Fixing occurrences of "the the "
    ... Registering a driver using platform_driver_probeworks just like ... Free all pages associated with DMA buffers, the buffers and pages lists, and ... * hw - Struct containing variables accessed by shared code ... * This option is used to tune the the maximum retransmission attempts ...
    (Linux-Kernel)
  • Re: [PATCH] [TRIVIAL] Fixing occurrences of "the the "
    ... Registering a driver using platform_driver_probeworks just like ... Free all pages associated with DMA buffers, the buffers and pages lists, and ... * hw - Struct containing variables accessed by shared code ... * This option is used to tune the the maximum retransmission attempts ...
    (Linux-Kernel)
  • Re: DICE II Firewire Driver Settings
    ... usb etc) spend too much time in a DPC call, and the kernel cannot run the ... soundcard driver in time for it to service the audio buffers. ... arguments that people use in the Firewire vs. USB2 wars. ...
    (rec.audio.pro)
  • Re: XP SP2 WDM driver - Sharing Memory between driver and application
    ... Windows 2k/XP/2k3 Filesystem and Driver Consulting ... I'm almost translating my application from> Windows to Linux just because I can't make a WDM driver for my PCI> card... ... > I have a motion control PCI card that can command 2 motors. ... > The only big problem is that I need to share 10 buffers between the> driver and the user app. ...
    (microsoft.public.development.device.drivers)
  • [PATCH] Fix name of Russell King in various comments
    ... Written by Christopher Hoover ... - * Based on fragments of previous driver by Rusell King et al. ... + * Based on fragments of previous driver by Russell King et al. ...
    (Linux-Kernel)
Loading