Re: Degradation of TCP connection

On Jul 24, 9:30 am, noiset...@xxxxxxxxx wrote:
On Jul 23, 2:05 pm, gtd...@xxxxxxxxx wrote:



Yes vxworks does have it's own network data buffers that eventually
can cause problems. I am having a similar issue with vxWorksTCPand
can share what I've been doing to try and diagnose it in the event it
gives you some ideas.

My application requires sending tons of data at very fast rates over a
Gigabit ethernet. Essentially I am trying to send 4,800,000 bytes per
second (this may prove unfeasible). Anyways, what I do is read voltage
samples off an A/D card connected to my Single Board computer via the
PC104-Plus interface, at a rate of 400k samples per second for 6
channels. Each sample has 16 bits, or 2 bytes of data. However, the A/
D card's data buffer can only hold about 64K samples worth of data
before overflowing, so I need to make sure I am taking data off the A/
D card at a constant rate and then sending that data out in large
packets over the ethernet.

After 6000 samples accumulate in the data buffer, I have a PCI
interrupt trip which copies the sample data off the PCI buffer and
into another buffer, then sends a message to my ethernet application
telling it to send the data on the second buffer out over theTCP
socket.

My connection keeps up fine for the first 10 or so interrupt cycles,
then the network write task begins falling behind. After some
troubleshooting, I determined that after sending about 128k bytes out
over theTCPsocket, my problems start. The same holds true when I did
a test with UDP (just to verify that it wasn't the receive side
causing the issue). If I shrink the packets, or send less data at a
time, I still eventually run into the same problem at 128kbytes every
time.

The issue seems to be (just a theory right now) that the vxworks
network data buffer has filled up at this point and needs to free the
memory and re-initialize, so the tNetTask is pending and my network
send call has to wait. The 128k-bytes total seems to match up with the
default network memory block total. I am trying a new method using
something called the zbufSocket Library. This library basically allows
you to send data over a socket without using vxwork's network data
buffer. However, WindRiver took zBuf support out of vxworks version
6.5, so I am wondering whether this is a dead end.

Try this linkhttp://slac.stanford.edu/exp/glast/flight/sw/vxdocs/vxworks/netguide/...
and go to section "4.3.3 Network Memory Pool Configuration". This
covers how the network memory is set up. I am still trying to digest
some of it myself to get a better idea. There does appear to be some
way to diagnose network memory usage.

If I come across something, I'll post a better update.

There are two important points here:

1) You're using VxWorks 6.5, which has a newTCP/IP stack (the IPNET
stack from Interpeak, which Wind River acquired). The documentation
link you posted is for an older version of VxWorks that used a BSD-
derived stack. Some of the information in that documentation is no
longer valid for the new stack, in particular that which pertains to
the stack's internal buffer management: there is no "system pool" and
"data pool" any longer. If you're looking at netPoolShow(), I wouldn't
bother. You can use that to check the ethernet driver's netpool, but
internally IPNET uses a totally different buffer management scheme, so
the netBufLib debugging stuff won't help you.

2) You failed to specify what ethernet controller you're using. I know
that the MPC7447 doesn't include built in networking hardware, so your
board must have some other ethernet chip on it (standalone, or part of
some combined I/O controller device). Please explain what controller
it is, exactly. This is important, because your problem might not be a
general networking issue, but a bug in the ethernet driver. If it
helps, tell us what BSP or single board computer your design is based
on.

And no, you can't use zbufs with the IPNET stack. The major design
difference between IPNET and the BSD-derived stack is that the BSD
code allows internally stored packet data to be fragmented across
multiple buffers (mbufs) while the IPNET stack requires all packets to
fit into a single contiguous buffer. zBufs are a VxWorks-specific
extension to the BSD-derived stack that allow an application supplied
buffer (or buffers) to be directly mated to an mbuf (or mbuf chain)
instead of having to allocate a whole mbuf tuple and copying the data
from the application buffers into the mbuf cluster buffers. The
problem is, sometimes an application may do a large write which has to
be broken up into smaller packets (if an app write()s 64K of data to a
socket, that has to eventually be split up into 1500 byte chunks for
transmission over ethernet). With the BSD mbuf scheme, it's not that
hard to just allocate a bunch of mbufs and set them to point to
different sub-buffers within the bounds of the single large buffer
provided by the application and then chain them together. But because
IPNET requires all packets to fit into a single contiguous buffer, it
doesn't support the ability to chain multiple fragments together. This
means you can't help but copy the data in order to get it formatted
correctly for transmission over the wire, which sort of defeats the
purpose of zero copy buffers. There is talk, however, of finally
implementing scatter/gather within IPNET so that zBufs can be brought
back.

There are arguments for and against both designs. The BSD mbuf-based
design is more flexible and can be more frugal with memory, but the
code is more complex. The IPNET ipnet_packet design is not as
flexible, but the code is simpler, and using its own internal buffer
handling scheme makes it more OS-agnostic, which was one of the IPNET
design requirements. (Personally I prefer the BSD design. Critics
complain that the days when you had to run BSD on a PDP-11 with
minimal memory -- which is what necessitated the more frugal buffer
management scheme in the first place -- are long over, and that I
should learn stop worrying and love large amounts of RAM. I contend
that just because you have a lot of RAM doesn't mean you shouldn't
make frugal use of it, and besides, VxWorks does sometimes have to run
on hardware with minimal memory.)

-Bill

Hi Bill,

Thanks for your thoughtful post.

Here are some stats on the system we're using:
- Curtiss-Wright 124 single-board computer
- The 124 uses an END Ethernet driver, and the physical device is on a
chip called the MV64460 (or Discovery III). I couldn't find any info
on the driver versions. I figure it's linked to our BSP version
number.

You were very helpful to another poster* regarding a problem that you
determined to be a buggy ethernet driver, made by DY-4 Systems. The
original poster from that thread also seemed to be using a Curtiss-
Wright (then DY-4 Systems) single-board computer. From our packet
logs, we found that the MAC address of the single-board computer
resolves to something that starts with DY-4... coincidence? Do you
think that we're running into the same problem as the poster from the
other thread because we're using the same crummy DY-4 ethernet driver?

*
http://groups.google.com/group/comp.os.vxworks/browse_thread/thread/59b0e68445a498c/a0a618f4306f23ad?lnk=st&q=#a0a618f4306f23ad

Thanks for your continued help,
Justin
.

Relevant Pages

  • Re: Degradation of TCP connection
    ... Gigabit ethernet. ... D card's data buffer can only hold about 64K samples worth of data ... link you posted is for an older version of VxWorks that used a BSD- ... but a bug in the ethernet driver. ...
    (comp.os.vxworks)
  • Re: Degradation of TCP connection
    ... After 6000 samples accumulate in the data buffer, ... then the network write task begins falling behind. ... The issue seems to be that the vxworks ... which has a new TCP/IP stack (the IPNET ...
    (comp.os.vxworks)
  • Re: Degradation of TCP connection
    ... Gigabit ethernet. ... D card's data buffer can only hold about 64K samples worth of data ... link you posted is for an older version of VxWorks that used a BSD- ... but a bug in the ethernet driver. ...
    (comp.os.vxworks)
  • Re: Xbox Extender giving network congestion errors
    ... Given the behavioral change you saw with changing your network setup, ... Regarding slow mo playback - next time you see video enter this mode, ... this doesn't sound like buffer depletion; what I would expect you to ... It corrects itself in a really ...
    (microsoft.public.windows.mediacenter)
  • Re: TCPBEUI misbehaving
    ... >> I try to analyse problems on my network. ... The wrkheuristics parameter sets a variety of requester fine-tuning ... 9 buffer small read and write requests until the buffer is full ... 27 force correct open mode when creating files on a core server (reserved ...
    (comp.os.os2.setup.misc)