Jacked network bridges

Gabriel Gunderson gabe at gundy.org
Sat Jan 28 15:12:20 MST 2012


I have an issue with a network setup and have kinda hit a wall. I can
only read so many MAC addresses before going mad.  Can you help?

So, here is my setup (follow if you can):
1) One virtual host
* CentOS 6.2
* Virt via KVM and libvirt
* Supermicro motherboard etc.
* Two built-in NICs -- I think they have Intel chipsets
* Two additional NICs-- also Intel

2) Big-ol' HP switch.
* VLAN - 10 (we'll call this network A)
* VLAN - 20 (and network B)
* VLAN - 100 (and network C)
* All *my* traffic is untagged when it hits the switch.
* All the providers traffic is tagged with the right VLAN.

Networks A & B are from a voice (SIP) provider. They are come to us
via a single cross-connect tying us into the equipment in their rack.
They provide a router for each network that also acts as the default

Network C is my support network and I provide the gateway (with some
custom routes for my VPNs etc.)

Three of the four network interfaces are set up come up on boot, the
last one is not. The three each come up in a bridging configuration
very simular to what's described here (I'll provide exact configs if
it comes to that):

Each of the bridges has STP setup on it (not that it needs it). The
only difference between them is that bridge C gets an IP address via
DHCP (that's the IP that I use to connect to the virtual host with and
get access to the support network).

There is a single CAT6 going from each of the three configured
interfaces on the server to the section of the switch setup for its

At this time, I have 6 virtual guests running. Half of them (let's
call them set X) have interfaces for all 3 networks A, B & C. The
other half only has interfaces for networks B & C (let's call them set
Y) . Each of the guest's interfaces is connected to the bridge on the
virtual host setup for that network.

When any of those guests ping the gateway(s) it looks like this:

virt-guest NIC => virt-host NIC => bridge-x => eth-x => CAT6 => switch
=> tagged with VLAN-x => CAT5 => gateway

Here is what *works*:
All 3 of the virt guests on network A can ping each other as expected.
They can also ping the gateway for that network.

All 6 of the virt guests on network C can ping each other as expected.
They can also ping the gateway for that network.

Here is what *doesn't* work:
On network B, 3 of the virt guests can ping each other and the gateway
(we remember them as set X).  The other 3 (and set Y) can ping each
other but not the gateway for network B and none of the guest in set

This is what I know:
* Network B is jacked :)
* The problem is at layer 2. I know this because I can't arping from
set Y to set X even though they are tied to the same bridge.
* There are no duplicate MAC addresses.
* The whole process of building this setup is fully automated (via
fabric) and can easily be recreated, so it not a human error; it's
more like a misconfiguration (but why set X and not set Y?). Hosts in
set X and Y are all created with the same build process.
* The only thing that's different about them (and I think this *must*
be key) is that set X also has the 3rd interface to the C network.
But, that shouldn't have anything to do with network B not working :/
* I need more sleep.

So, friends, *please* help me figure out what I'm doing wrong. What
additional info can I provide? Thanks!

Humbly yours,

More information about the PLUG mailing list