Re: VxWorks 6.6 - Ethernet interface driver support

On Oct 10, 1:37 pm, ulrik <u.wilh...@xxxxxx> wrote:
On Oct 2, 8:36 pm, noiset...@xxxxxxxxx wrote:

On Oct 2, 7:34 am, ulrik <u.wilh...@xxxxxx> wrote:

On Sep 23, 10:24 pm, noiset...@xxxxxxxxx wrote:

On Sep 23, 9:55 am, ulrik <u.wilh...@xxxxxx> wrote:
[...]

Hello Bill

Many thanks for your suggestions!

I checked the content of the Extended Device Control Register but the
LINK_MODE is already correctly set (10b = Direct Fiber interface using
internal SerDes).

Hm... well then I'm kind of stumped. It sounds like it should be
working right. I don't know why it isn't. Sadly, this is something I
could probably diagnose myself if only I had access to the hardware,
but I don't. :(

A thought that I had was, that it may not be possible to combine
electrical Ethernet interfaces (CCT onboard) with optical Ethernet
interfaces (DSS PMC card), so I wanted to disable the Generic PHY
driver component and check the behavior. However, since excluding that
driver (not quick exclude) would result in removing also the Intel PRO/
1000 VxBus Enhanced  Network Driver component, hints on that these two
components always need to be deployed in combination.

No, you should be able to use all interfaces at the same time. The GEI
driver depends in the genericPhy driver by default, since it's always
needed for copper operation. This is enforced by a .cdf file that's
used by Workbench. The interfaces with TBIMODE set should never select
the genericPhy driver, only the geiTbiPhy driver.

Another thing I tried was to add the following code in the function
"geiReset" (corresponding to how auto negotiation is restarted by the
DSS driver under VxWorks 5.5.1):

Uhm... the code below has a bug:

    if (CSR_READ_4(pDev, GEI_STS) & GEI_STS_TBIMODE)
    {
        CSR_CLRBIT_4(pDev, GEI_CTRL, GEI_CTRL_FD);
        CSR_CLRBIT_4(pDev, GEI_CTRL, GEI_CTRL_SLU);

You read from the TXCW register here:

        tmp = CSR_READ_4(pDev, GEI_TXCW);
        tmp &= ~0x1f;
        tmp |= (0x60|0x80000000);

But you write to the IMC register here:

        CSR_WRITE_4(pDev, GEI_IMC, tmp);

That should have been: CSR_WRITE_4(pDev, GEI_TXCW, tmp);

        CSR_SETBIT_4(pDev, GEI_CTRL, GEI_CTRL_LRST);
        CSR_CLRBIT_4(pDev, GEI_CTRL, GEI_CTRL_LRST);
    }

The result is some additional GIG LED flickering (on all interfaces)
but that is all, links don't come up.

Technically this shouldn't be necessary since this is what the
geiTbiPhy driver already does (see geiTbiPhyModeSet() for the IFM_AUTO
case). It's basically advertising 1000FD and 1000HD capabilities and
enabling autoneg.

Unfortunately, since I don't have one of these cards myself, and since
I can't reproduce the problem with the card that I have, I'm kind of
at a loss for more ideas. It should be working. There may be something
hidden in DSS's driver code. You might look to see if there's any code
that manipulates the pin control (SDP) bits via the GEI_CTRL or
GEI_CTRLEXT register. There may be a magic pin that has to be asserted
to apply power to the optical interfaces. You could also check to see
if there's any code to manipulate the PHY reset bit
(GEI_CTRL_PHY_RST). Normally the VxWorks driver only manipulates this
bit for the 82541 chip.

Note: earlier I had suggested cabling the two ports on the DSS card
together with a fiber patch to see what would happen, but you never
said what happened. (I don't know if you didn't try it or forgot to
report the results.)

-Bill

Best regards

Ulrik

Hello Bill

Correcting the bug that you spotted (thanks) didn’t help, also when
manually restarting the hardware auto negotiation, it fails.

That the auto negotiation has failed seems obvious considering that
the:
- CTRL register bit LRST (Link reset) is set
- CTRL register bits RFCE (Receive Flow Control Enable) and TCFE
(Transmit Flow Control Enable) are cleared
- RXCW register bit RxConfigInvalid (Invalid Symbol during
configuration process) is set
- STATUS register bits FD (Link Full Duplex) and LU (Link Up) are
cleared

For now, I went for a rather pragmatic approach:

- GEITBIPHY driver:

A change was made to the function “geiTbiPhyModeGet” of file
“geiTbiPhy.c” to always return status active, independently of the
result of the hardware auto negotiation.
This had the effect that also the optical interfaces entered the
RUNNING state as indicated by the “ifconfig” command. However, this
was not enough to allow traffic over these interfaces but in addition
a change in the GEI driver was necessary.

- GEI driver:

A change was made to the function “geiMuxConnect” of file
“gei825xxVxbEnd.c”, forcing register bits into the desired state. Code
extract:

/* Clear the CTRL register bit LRST and set the bits RFCE and TFCE */
tmp = CSR_READ_4(pDev, GEI_CTRL);
tmp &= 0xFFFFFFF7;
tmp |= 0x18000000;
CSR_WRITE_4(pDev, GEI_CTRL, tmp);

/* Set the RXCW register to hold the same data as seen under VxWorks
5.5.1 */
CSR_WRITE_4 (pDev, GEI_RXCW, 0xc40041e0);

/* Set the STATUS register bits LU and FD */
tmp = CSR_READ_4(pDev, GEI_STS);
tmp |= 0x00000003;
CSR_WRITE_4(pDev, GEI_STS, tmp);

With this code change, the optical interfaces are operating, being
able to send and receive packets.

I am assuming you added this code after the call to muxDevStart(). Is
this correct?

These last two register updates should be no-ops. Per the Intel
documentation, the RXCW and STATUS registers are read only, so writing
to them should have no effect. (In particular, the link up bit should
be set automatically by the hardware to indicate that a link has been
established.) This means that the only effective change with regards
to how the hardware is configured is to enable RX and TX flow control
in the MAC. I suspect that if you remove the code that tries to write
to RXCW ans STATUS, you'll still get a link.

The value for RXCW which you say is oberved under 5.5.1 seems to
indicate that the link partner to which you have the fiber cards
connected is advertising flow control support (bits 7 and 8 are set).
The flow control bits in the TXCW  register (which control whether or
not  the PRO/1000 advertises flow control support too) are initialized
from the EEPROM. The geiTbiPhy driver preserves them when setting up
autonegotiation, however the GEI driver itself doesn't support it
(which is why it doesn't turn flow control on in the CTRL register).
With my Intel PRO/1000 MF card, the EEPROM sets the TXCW bits on, and
I'm assuming that the EEPROM on your PMC cards is set up the same way.

I think there may be some sort of mismatch between the link partner
and the 82546 because the NIC is advertising flow control support, but
it's not actually used. Technically, since the MAC driver doesn't
support it, it shouldn't be advertising it. Can you please try the
following test for me:

- put back the original copies of gei825xxVxbEnd.c and geiTbiPhy.c
- in geiTbiPhy.c, find the geiTbiPhyInit() routine
- Find the code that says:

    miiBusRead (pDev, pDrvCtrl->miiPhyAddr, GTBI_TXCFG, &miiVal);
    miiVal &= ~(GTBI_TXCFG_1000HD|GTBI_TXCFG_1000FD);
    miiBusWrite (pDev, pDrvCtrl->miiPhyAddr, GTBI_TXCFG, miiVal);

- Change it to:

    miiBusRead (pDev, pDrvCtrl->miiPhyAddr, GTBI_TXCFG, &miiVal);
    miiVal &= ~(GTBI_TXCFG_1000HD|GTBI_TXCFG_1000FD|GTBI_TXCFG_PAUSE);
    miiBusWrite (pDev, pDrvCtrl->miiPhyAddr, GTBI_TXCFG, miiVal);

- Recompile geiTbiPhy and gei825xxVxbEnd and see if you get a link
this time

What this will do is force off the RX and TX pause frame (flow
control) advertisement, which should prevent the link partner from
enabling flow control with this link. This way, the PHY, MAC and link
partner will all agree. Please let me know if this fixes the problem.
I tried this with my PRO/1000 MF card and the one gigE switch I have
available (an old Extreme Networks Summit 5i with GBICs) and it seems
to work ok (of course it worked with the old code too).

It was noted that if packet send/receive is started over both DSS PMC
card pairs (send/receive pair) and one or more cables are unplugged,
the packet send/receive stops on all optical interfaces (each of the
two node boards are equipped with 2 PMC cards and there are two
interfaces per PMC card -> 4 interface pairs).
However, if packet send/receive is started only between one card pair,
packet send/receive is unaffected when unplugging one or both cables
of the other card pair.
Important to note is that this problem is not present when electrical
Ethernet DSS PMC cards are deployed, but they behave as expected: Only
the interface pair for which the cable is removed, the packet send/
receive is interrupted.

This is a little puzzling, but may have to do with the changes you
made to geiTbiPhy. Normally, all interfaces should operate
independently, but it sounds like a link change event on one is
messing up another. I'm hoping that if you restore everything and make
the one change that I suggested above, you won't see this problem
anymore.

As you understand, the current situation is not very satisfying and
I’ll continue to investigate on these issues. Nevertheless, at least I
can run traffic now.

Yeah, I do understand, and I really want to solve this. Please let me
know if my suggested change helps.

Best regards

Ulrik

Hello Bill

Sorry for taking so long to respond!

The code, writing of registers, was added at the end of the
“geiMuxConnect” routine (after the muxDevStart() function call).

The effect of the writing to the registers RXCW and STATUS is outlined
below. The two node boards were equipped with one optical PMC card
each and each of the two interfaces was connected to the corresponding
interface of the other board/card (loop test). The reason why the unit
numbering is not what you may expect is the fact that I force a
certain numbering scheme, so that the interfaces of the upper PMC card
always have the unit numbers 5 and 6 in order to avoid that the
numbering is different when no, one or two PMC cards are attached.

I checked on the content of the STATUS and RXCW registers, at three
different points in time:
1. Register content directly before the write done in the
“geiMuxConnect” routine
2. Register content directly after the write done in the
“geiMuxConnect” routine
3. Register content some seconds later (by calling a routine that
examines the content)

What can be seen is that the write to the STATUS register has no
effect and the write to the RXCW at most a temporary one since it also
changes between point 2 and point 3 in time (without me doing anything
in the mean time).

The values within parentheses show the change in content after I have
removed the write to the STATUS and RXCW registers.

BOARD 1:

unitNumber=5

GEI_STS=dba4
GEI_RXCW=c000000

GEI_STS=dba4
GEI_RXCW=640001e0 (64000000)

GEI_STS=dba7
GEI_RXCW=c40041e0 (c40041e0)

unitNumber=6

GEI_STS=dba0
GEI_RXCW=c000000

GEI_STS=dba0
GEI_RXCW=640001e0 (64000000)

GEI_STS=dba3
GEI_RXCW=c40041e0 (c40041e0)

BOARD 2:

unitNumber=5

GEI_STS=dba4
GEI_RXCW=c000000

GEI_STS=dba4
GEI_RXCW=74000000 (640001e0)

GEI_STS=dba7
GEI_RXCW=c40041e0 (c40041e0)

unitNumber=6

GEI_STS=dba0
GEI_RXCW=c000000

GEI_STS=dba0
GEI_RXCW=74000000 (640001e0)

GEI_STS=dba3
GEI_RXCW=c40041e0 (c40041e0)

Consequently, the writing to the STATUS and RXCW registers was
dropped. However, the writing to the CTRL register is still necessary
since alone the change to the routine “geiTbiPhyInit” that you
proposed doesn’t bring the link to an operating state (seems to have
no effect).

Something else I noted is that the change I earlier did to the file
“geiTbiPy.c”, always flagging for status active in the routine
“geiTbiPhyModeGet”, is no longer needed for bringing the interfaces
into the RUNNING state (indicated by “ifconfig”).

Please note that the strange behavior "unplugging a cable effecting
also other interfaces" still remains.

Many thanks for your support so far and best regards

Ulrik

Hello Bill

“Loop test” may not be the right term but that is how we usually call
it when we connect one “node” board to another and run tests, before
involving any switch or other network elements.

The test setup that I have used to investigate on the fiber optical
ports and get them working was always the same, letting one board talk
to another. Sorry, for not being clear on this in the past.

In the routine “geiInstInit” the routine “vxbNextUnitGet” is normally
called. I changed that, so that the routine “vxbInstUnitSet” is called
with the desired device number, code extract (lower PMC card lower
interface -> gei3, lower PMC card upper interface -> gei4 and so one):

LOCAL void geiInstInit
(
VXB_DEVICE_ID pDev
)
{
struct vxbPciDevice *pPciDev;

pPciDev = (struct vxbPciDevice *) pDev->pBusSpecificDevInfo;

….

else if ( pPciDev->pciBus == 4 && pPciDev->pciDevId == DEVID_82546EB
&& pPciDev->pciFunc == 0 )
{
vxbInstUnitSet(pDev, 4);
}
else if ( pPciDev->pciBus == 4 && pPciDev->pciDevId == DEVID_82546EB
&& pPciDev->pciFunc == 1 )
{
vxbInstUnitSet(pDev, 3);
}
else if ( pPciDev->pciBus == 8 && pPciDev->pciDevId == DEVID_82546EB
&& pPciDev->pciFunc == 0 )
{
vxbInstUnitSet(pDev, 6);
}
else if ( pPciDev->pciBus == 8 && pPciDev->pciDevId == DEVID_82546EB
&& pPciDev->pciFunc == 1 )
{
vxbInstUnitSet(pDev, 5);
}
….

Extensive testing in the past has shown that I can consider this code
change safe and not related to the problem of the optical links not
coming up without the “CTRL register write” workaround.
I changed the above code to swap unit 5 and 6 and 3 and 4 as you
suggested and also reverted back to the original code but without the
“CTRL register write” workaround, the links don’t come up.

Test set-up remarks:
- VxWorks image: GEI driver and GEI TBI driver components included
- The interfaces of a PMC card on one board are connected to the
corresponding interfaces of the corresponding PMC card on the other
board (gei3 –gei3, gei4 – gei4, gei5 – gei5 and gei6 – gei6)
- I am using “ifconfig” and “ping” to check on status and connectivity

I agree with you that a register that is said to be read only also
should be read only. However, it depends on the design what is the
actual case. Nevertheless, what it means is that a write at most would
have some temporary effect since the content of such a registers may
change at any time.

Since, I can live with the current workaround for now, I will have to
put this issue on ice, due to higher prioritized tasks.

Many thanks for you support and ideas

Ulrik
.

Relevant Pages

  • Re: VxWorks 6.6 - Ethernet interface driver support
    ... electrical Ethernet interfaces with optical Ethernet ... interfaces (DSS PMC card), so I wanted to disable the Generic PHY ... driver would result in removing also the Intel PRO/ ... CTRL register bit LRST is set ...
    (comp.os.vxworks)
  • Re: Application can not receive WM_DEVICECHANGE
    ... And the OmniVision driver installation file OVUSB2.INF adds something to ... IoRegisterDeviceInterface to register this AM_KSCATEGORY_CAPTURE, ... interfaces, or for all itnerfaces and then filter the event for the ...
    (microsoft.public.development.device.drivers)
  • Re: VxWorks 6.6 - Ethernet interface driver support
    ... electrical Ethernet interfaces with optical Ethernet ... interfaces (DSS PMC card), so I wanted to disable the Generic PHY ... driver would result in removing also the Intel PRO/ ... But you write to the IMC register here: ...
    (comp.os.vxworks)
  • Re: VxWorks 6.6 - Ethernet interface driver support
    ... I checked the content of the Extended Device Control Register but the ... electrical Ethernet interfaces with optical Ethernet ... interfaces (DSS PMC card), so I wanted to disable the Generic PHY ... driver would result in removing also the Intel PRO/ ...
    (comp.os.vxworks)
  • Re: VxWorks 6.6 - Ethernet interface driver support
    ... Ethernet interface cards, deploying the "Intel PRO/1000 VxBus Enhanced ... Network Driver" in conjunction with the "Generic PHY driver", ... these interfaces. ...   MII_Bus @ 0x005073cc ...
    (comp.os.vxworks)