Re: Degradation of TCP connection

On Jul 24, 2:10 pm, justin.pear...@xxxxxxxxx wrote:
On Jul 24, 9:30 am, noiset...@xxxxxxxxx wrote:



On Jul 23, 2:05 pm, gtd...@xxxxxxxxx wrote:

Yes vxworks does have it's own network data buffers that eventually
can cause problems. I am having a similar issue with vxWorksTCPand
can share what I've been doing to try and diagnose it in the event it
gives you some ideas.

My application requires sending tons of data at very fast rates over a
Gigabit ethernet. Essentially I am trying to send 4,800,000 bytes per
second (this may prove unfeasible). Anyways, what I do is read voltage
samples off an A/D card connected to my Single Board computer via the
PC104-Plus interface, at a rate of 400k samples per second for 6
channels. Each sample has 16 bits, or 2 bytes of data. However, the A/
D card's data buffer can only hold about 64K samples worth of data
before overflowing, so I need to make sure I am taking data off the A/
D card at a constant rate and then sending that data out in large
packets over the ethernet.

After 6000 samples accumulate in the data buffer, I have a PCI
interrupt trip which copies the sample data off the PCI buffer and
into another buffer, then sends a message to my ethernet application
telling it to send the data on the second buffer out over theTCP
socket.

My connection keeps up fine for the first 10 or so interrupt cycles,
then the network write task begins falling behind. After some
troubleshooting, I determined that after sending about 128k bytes out
over theTCPsocket, my problems start. The same holds true when I did
a test with UDP (just to verify that it wasn't the receive side
causing the issue). If I shrink the packets, or send less data at a
time, I still eventually run into the same problem at 128kbytes every
time.

The issue seems to be (just a theory right now) that the vxworks
network data buffer has filled up at this point and needs to free the
memory and re-initialize, so the tNetTask is pending and my network
send call has to wait. The 128k-bytes total seems to match up with the
default network memory block total. I am trying a new method using
something called the zbufSocket Library. This library basically allows
you to send data over a socket without using vxwork's network data
buffer. However, WindRiver took zBuf support out of vxworks version
6.5, so I am wondering whether this is a dead end.

Try this linkhttp://slac.stanford.edu/exp/glast/flight/sw/vxdocs/vxworks/netguide/...
and go to section "4.3.3 Network Memory Pool Configuration". This
covers how the network memory is set up. I am still trying to digest
some of it myself to get a better idea. There does appear to be some
way to diagnose network memory usage.

If I come across something, I'll post a better update.

There are two important points here:

1) You're using VxWorks 6.5, which has a newTCP/IP stack (the IPNET
stack from Interpeak, which Wind River acquired). The documentation
link you posted is for an older version of VxWorks that used a BSD-
derived stack. Some of the information in that documentation is no
longer valid for the new stack, in particular that which pertains to
the stack's internal buffer management: there is no "system pool" and
"data pool" any longer. If you're looking at netPoolShow(), I wouldn't
bother. You can use that to check the ethernet driver's netpool, but
internally IPNET uses a totally different buffer management scheme, so
the netBufLib debugging stuff won't help you.

2) You failed to specify what ethernet controller you're using. I know
that the MPC7447 doesn't include built in networking hardware, so your
board must have some other ethernet chip on it (standalone, or part of
some combined I/O controller device). Please explain what controller
it is, exactly. This is important, because your problem might not be a
general networking issue, but a bug in the ethernet driver. If it
helps, tell us what BSP or single board computer your design is based
on.

And no, you can't use zbufs with the IPNET stack. The major design
difference between IPNET and the BSD-derived stack is that the BSD
code allows internally stored packet data to be fragmented across
multiple buffers (mbufs) while the IPNET stack requires all packets to
fit into a single contiguous buffer. zBufs are a VxWorks-specific
extension to the BSD-derived stack that allow an application supplied
buffer (or buffers) to be directly mated to an mbuf (or mbuf chain)
instead of having to allocate a whole mbuf tuple and copying the data
from the application buffers into the mbuf cluster buffers. The
problem is, sometimes an application may do a large write which has to
be broken up into smaller packets (if an app write()s 64K of data to a
socket, that has to eventually be split up into 1500 byte chunks for
transmission over ethernet). With the BSD mbuf scheme, it's not that
hard to just allocate a bunch of mbufs and set them to point to
different sub-buffers within the bounds of the single large buffer
provided by the application and then chain them together. But because
IPNET requires all packets to fit into a single contiguous buffer, it
doesn't support the ability to chain multiple fragments together. This
means you can't help but copy the data in order to get it formatted
correctly for transmission over the wire, which sort of defeats the
purpose of zero copy buffers. There is talk, however, of finally
implementing scatter/gather within IPNET so that zBufs can be brought
back.

There are arguments for and against both designs. The BSD mbuf-based
design is more flexible and can be more frugal with memory, but the
code is more complex. The IPNET ipnet_packet design is not as
flexible, but the code is simpler, and using its own internal buffer
handling scheme makes it more OS-agnostic, which was one of the IPNET
design requirements. (Personally I prefer the BSD design. Critics
complain that the days when you had to run BSD on a PDP-11 with
minimal memory -- which is what necessitated the more frugal buffer
management scheme in the first place -- are long over, and that I
should learn stop worrying and love large amounts of RAM. I contend
that just because you have a lot of RAM doesn't mean you shouldn't
make frugal use of it, and besides, VxWorks does sometimes have to run
on hardware with minimal memory.)

-Bill

Hi Bill,

Thanks for your thoughtful post.

Here are some stats on the system we're using:
- Curtiss-Wright 124 single-board computer
- The 124 uses an END Ethernet driver, and the physical device is on a
chip called the MV64460 (or Discovery III). I couldn't find any info
on the driver versions. I figure it's linked to our BSP version
number.

You were very helpful to another poster* regarding a problem that you
determined to be a buggy ethernet driver, made by DY-4 Systems. The
original poster from that thread also seemed to be using a Curtiss-
Wright (then DY-4 Systems) single-board computer. From our packet
logs, we found that the MAC address of the single-board computer
resolves to something that starts with DY-4... coincidence? Do you
think that we're running into the same problem as the poster from the
other thread because we're using the same crummy DY-4 ethernet driver?

*http://groups.google.com/group/comp.os.vxworks/browse_thread/thread/5...

Thanks for your continued help,
Justin


No, it's not the same driver.

Drivers usually come from one of two places: either Wind River, or a
3rd party BSP supplier. Wind River supplies drivers for some commonly
available NICs (i.e. the Intel PRO/100 or PRO/1000 PCI cards) and for
some controllers in system-on-chip processors for which they provide
BSPs (i.e. the TSEC ethernet on the Freescale MPC8560). Some board
vendors supply their own BSPs and include their own ethernet drivers
if VxWorks doesn't provide driver support already. (If no driver
exists at all, you can either write one yourself, or pay Wind River
Professional Services to write one for you.)

In the other poster's case, his board had a 10Mbps NatSemi SONIC chip
and VxWorks did include a driver for it (if_sn). Unfortunately, that
driver turned out to be kinda crummy, and didn't hold up well under
load.

In your case, you're probably using the on-board gigabit MACs in the
Discovery III system controller. According to the documentation, it
has 3 gigabit ports. I'm pretty sure the network driver you have was
not written by Wind River, though I don't know if it was done by
Curtiss-Wright or Marvell. If I had to bet, I'd say that at least some
of the code was provided by Marvell.

Looking over your original post, I see that the time that elapses
before the failure is not consistent -- in one case it was 70 hours,
in others 20 and 28 hours. This could be a buffer exhaustion issue,
but to me the variation in reproducibility suggests a race condition
instead. It could be a driver bug, but you'll need to run some more
tests in order to know for sure.

Here are a couple of thoughts:

- When your application and LabView stop communicating, can you still
ping the target from the Windows XP machine? If yes, the ethernet
driver and the IP layer of the stack are still working, at least to
some extent, and it's something at the TCP layer that's gone wrong. If
no, then it could be the driver, or a serious problem in the stack.

- You say that it looks like the VxWorks target stops receiving
traffic from LabView. How did you determine this? I usually check for
receive operation by adding the target shell and the INCLUDE_IFCONFIG
component. At the shell, you can do:

-> ifconfig "motfcc0"
motfcc0 Link type:Ethernet HWaddr 00:04:9f:07:08:09 Queue:none
inet 147.11.46.192 mask 255.255.255.0 broadcast
147.11.46.255
UP RUNNING SIMPLEX BROADCAST MULTICAST
MTU:1500 metric:1 VR:0 ifindex:2
RX packets:24 mcast:7 errors:0 dropped:1
TX packets:6 mcast:0 errors:0
collisions:0 unsupported proto:0
RX bytes:2198 TX bytes:438

value = 0 = 0x0

If you run ifconfig a couple of times and you see the "RX packets" and
"RX bytes" incrementing, then this means the driver is still receiving
frames and passing them into the stack. (The stack maintains the
counters shown by ifconfig, not the driver, so you know the data is
making the transition across the driver/stack boundary.) If this is
the case, it means the receive path is working, but the LabView isn't
getting any response back from the target because the transmit path is
stalled. If you don't see the RX counters incrementing, then the
receiver is actually stuck.

Unfortunately, the TX counters are less useful: they indicate that the
stack sourced packets to the underlying driver, but that doesn't tell
you whether or not the outgoing frames were successfully transmited.
You can test if the driver is still able to send traffic by including
the INCLUDE_PING component in your image. When the target hangs, try
to do:

-> ping "xxx.xxx.xxx.255", 5

from the target shell, where xxx.xxx.xxx is your IP network. This
should (assuming your netmask is 255.255.255.0) cause the target to
send some broadcast packets onto the wire, which you can observe with
Wireshark.

If you see the broadcast packets sent, and the RX counters don't
increment, then the transmit path is working and the receiver is
stalled.

If the broadcast packets don't make it onto the wire, and the RX
counters do increment, then the receive path is working and the
transmitter is stalled.

If the RX counters don't increment and you don't see any broadcast
packets on the wire either, the interface is completely jammed (maybe
it's stopped getting interrupts).

- The board should have at least two ethernet ports (three, if Curtiss-
Wright wired up all 3 MACs). As a test, I would enable a second port
on the target and cable it to another machine. For example, add a
second NIC to your Windows XP host and connect it to the other port on
the target via crossover cable. (Don't cheat by plugging everything
into the same hub/switch/whatever -- ideally, you want the two ports
to be isolated.) Give the second link some dummy IP addresses, like
10.0.0.2 (target) and 10.0.0.1 (Windows XP host). You should be able
to ping the target from your Windows host over both the spare link
(ping 10.0.0.2) and the primary link. Now start your LabView app going
and wait until it fails again. Once it fails, try to ping the target
over both links. If pinging the primary IP address fails, but pinging
the spare IP succeeds, then the problem is almost certainly a driver
bug which has caused the primary interface to become stalled somehow.
(That is, the heavily loaded interface encountered a race condition or
some other error condition from which it did not recover and is now
wedged, while the unloaded spare link is still functional.) If both
links fail to respond to ping, then the problem is more likely a stack
issue. (It could still be a driver issue: each time you send a packet,
the driver takes temporary ownership of it until the TX DMA operation
completes -- if the driver sets up a large TX DMA ring and the
transmitter stalls while it still has ownership of a lot of the
stack's TX buffers, this could prevent the stack from being able to
transmit packets entirely.)

I have found that many vendor supplied drivers follow a pattern. The
companies that make the networking silicon also create an OS-
independent hardware abstraction library for managing the controller.
To make a driver, they port the HAL to the target OS API, and then add
a driver shim over the top of it. This is considered to be more
effective from their perspective since it means that they can support
several OSes with the same piece of core library code. If they find a
bug specifically related to their ethernet hardware, they can then
just patch the HAL code once and fix the bug in all their drivers at
the same time (once they've tested on one OS, they can just recompile
the drivers for all the others to pick up the fix).

This sounds like a great idea, but there are drawbacks. Writing truly
portable code is hard: often the HAL ends up polluted with many
spaghetti #ifdefs to deal with platform differences, which complicates
maintenance. Also, the object code can end up being very large (and
some of it might be dead code that doesn't even apply to your
plarform). If you're targeting Windows or UNIX, you might not care,
but with VxWorks, small footprint is key. And VxWorks has some special
requirements compared to other OSes: sometimes "portable" designs fail
to take those differences into account.

From an OS developer's perspective, the best driver is one that's
small, easy to read, easy to maintain and which makes the best use of
available OS (and network stack) facilities.

Anyway, the point is that I wouldn't be surprised if the driver for
the Discovery III ethernet has a bug lurking in it somewhere. I also
bet a quarter that they didn't provide the source code for it either. :
(

-Bill
.

Relevant Pages

  • Re: Degradation of TCP connection
    ... Gigabit ethernet. ... D card's data buffer can only hold about 64K samples worth of data ... link you posted is for an older version of VxWorks that used a BSD- ... but a bug in the ethernet driver. ...
    (comp.os.vxworks)
  • Re: Degradation of TCP connection
    ... Gigabit ethernet. ... D card's data buffer can only hold about 64K samples worth of data ... then the network write task begins falling behind. ... You're using VxWorks 6.5, which has a newTCP/IP stack (the IPNET ...
    (comp.os.vxworks)
  • [PATCH] macb: RLE and BNA handling
    ... This patch solves two issues we have been experiencing ... The other issue experienced was in the RX part of the driver. ... the driver runs out of unused slots in the RX ring buffer. ... Atmel MACB Ethernet Controller driver ...
    (Linux-Kernel)
  • Re: Ethernet Slowdown Every 20 sec
    ... Maybe the driver is missing an interrupt and thereby using up a buffer, ... Ethernet connection instead, everything works perfectly. ... Obviously I'd like to get rid of this pause in UDP throughput... ...
    (microsoft.public.windowsce.embedded)
  • Re: Difference between synchronous and asynchronous operation/calls (NDISPROT)
    ... The main problem with synchronous operation/calls seems to be the lack of buffers for the driver to store data into. ... Each "read call" supplies the driver with a buffer and optionally a completion routine so that the driver can inform the application when the requested operation is/was done. ... However it seems the current windows/driver design does some copies as well? ...
    (microsoft.public.development.device.drivers)