KVM - Bridging Issues

[Jacob Albretsen]

I've encountered an interesting problem that I've been battling all day.

I have 3 new Dell R420 servers, each with a dual port  Broadcom NetXtreme 
BCM5720 NIC.  My plan is to have em1 be the "maintenance IP" and then use em2 
as a bridge for the VMs I will be running on it.  I am using KVM / libvirtd on 
CentOS 6.3.

/etc/sysconfig/network-scripts/ifcfg-em2:

DEVICE="em2"
HWADDR="90:XX:XX:XX:XX:XX"
NM_CONTROLLED="no"
ONBOOT="yes"
BRIDGE="br2"
UUID="XXX"

/etc/sysconfig/network-scripts/ifcfg-br2:

DEVICE="br2"
TYPE=Bridge
DELAY=0
NM_CONTROLLED="no"
ONBOOT="yes"

The devices come up fine.  "ifconfig" and "brctl show" look normal:

bridge name     bridge id               STP enabled     interfaces
br2             8000.90xxxxxxxxxx       no              em2

When I bring up a VM, the vnet0 also appears.

The issue I am running into is the bridge just flat out does not work.  I know 
em2 itself can work.  I can remove the bridge and just put a static IP (same 
subnet as em1) on it and ping it from other boxes and ping out to other boxes.  
But if I put on the bridge and try to get out through a VM, it goes no where.

I have sysctl.conf such that ipfilter is not used for the bridge traffic.  
Although I did bring down iptables to be sure and there was no change.

net.bridge.bridge-nf-call-arptables = 0
net.bridge.bridge-nf-call-iptables = 0
net.bridge.bridge-nf-call-ip6tables = 0
net.bridge.bridge-nf-filter-vlan-tagged = 0
net.bridge.bridge-nf-filter-pppoe-tagged = 0

I've checked and the modules are loaded.  If I move a VM over to em1 and use a 
macvtap connection, the VMs get out just dandy.

I have a very similar setup on another box and it works just fine.  I even have 
two bridges to two different networks and different VMs point to different 
networks (one network is the same network as the maintenance IP).  The only 
difference is the hardware.  On the one that works, the maintenance IP is on 
em1 (Broadcom NetXtreme II BCM5716) and the two bridges sit on top of a 
different piece of dual port NIC hardware: Intel Corporation 82576 (p1p1 and 
p1p2).

After banging my head all day at this and trying several different things for 
the setup of the bridge on em2, I'm out of ideas.  The only thing I can figure 
is the possibility that since the ones that don't work are on the same 
physical dual port NIC card and the ones that do are not on the same card, 
perhaps there is some sort of hardware limitation on the dual port NIC that 
I've run into.  But I'm finding that a little hard to believe.

Any thoughts or suggestions of things I may have overlooked are welcome.

Thanks!