Sharing Physical NICs with Guest VMs for iSCSI Traffic on Hyper-V

Since Microsoft has introduced support for virtualizing many more workloads with Lync Server 2010 we'll probably begin to see more and more deployments done with a hypervisor. One challenge can be providing shared storage access to a virtual machine guest operating system running on top of a hypervisor such as Hyper-V. An example here would be if a company wants to provide high-availability for the Lync pool back-end running a SQL server cluster, which requires shared storage for the quorum, MSDTC, and data volumes.

It's not uncommon for Hyper-V hosts to be configured in a failover cluster using dedicated NICs for iSCSI access, but how often does a guest VM typically need access to shared storage? Typically not often. Usually if a VM needs access to the SAN you can connect to the LUN from the host and then pass it through to the guest. With a SQL cluster though, both VMs should reside on separate hosts and need direct access to the SAN. In order to accomplish this you need to initiate iSCSI connections from within the guest VM directly to the SAN.

If you have unlimited NICs this is really easy. Just bind a physical NIC (pNIC) connected to the storage network to a Hyper-V virtual switch, create a new virtual NIC (vNIC) on the Hyper-V guest on this switch, and away you go - the guest now has iSCSI network access. For more realistic scenarios you may be limited on your NIC count and have to get creative because you can't dedicate a pNIC to VMs only.

If this is true the first option that comes to mind is to leverage VLAN tagging on a virtual switch already using a pNIC. This way you can tag LAN traffic and storage network traffic separately. The downside to this approach is you're now sharing the same pNIC for both LAN and storage traffic in VMs. The host OS still has a dedicated NIC for iSCSI traffic here. If the LAN traffic starts maxing out the capacity of the adapter you could potentially lose connectivity to the SAN and cause some serious problems for the cluster. This configuration would look like this:

Another option here is to leverage the network adapters already used by the host system for iSCSI access, i.e. the same adapters accessing the SAN and providing high-availability for the guest VMs. That may seem like a poor idea, but I think the alternative of sharing LAN and iSCSI network on the same adapter is far worse. In this configuration the Hyper-V host and the VM guest are both accessing the storage network through the same physical NIC, dedicated purely for storage traffic. A virtual NIC is created on the host and the host traffic also passes through the Hyper-V virtual switch in this scenario. The host and guest sharing a pNIC setup is depicted here:

In order to accomplish this setup you'll want to use the following steps:

  1. Shut down any VMs running on the host OS.
  2. Stop any iSCSI access from the host.
  3. Create a new Hyper-V virtual switch using the pNIC previously dedicated to iSCSI. Be sure to check the box "Allow management operating system to share this network adapter."
  4. Configure the new network adapter on the host OS with the old pNIC IP address and settings.
  5. Reconnect iSCSI targets.
  6. Edit the VM guest and add a new network adapter. Assign the network adapter to the iSCSI network virtual switch.
  7. Configure iSCSI targets from within the guest VM

At this point, all your storage traffic is isolated to the same physical NIC.

Warning: I cannot for the life of me find documentation from Microsoft on whether or not this actually supported. If I were to wager a guess it's on the unsupported side, but probably because of the "We haven't tested this" reason more than the "It doesn't work" reason. Your mileage may vary on your deployment, but this appears to be working just fine. I was a little reluctant to try this thinking the host OS may have iSCSI performance problems through the Hyper-V virtual switch, but all seems well so far.

Broadcom NIC Teaming and Hyper-V on Server 2008 R2

The short of this is if you’re trying to use NIC teaming for the virtual adapter on Server 2008 R2 save yourself the headache, pony up a few extra dollars and buy Intel NICs.  The Broadcoms have a bug in the driver that prevents  this from working correctly on Server 2008 R2 Hyper-V when using a team for the Hyper-V virtual switch. Per the Broadcom driver release notes this is supposed to be a supported configured now, but it does not work correctly. There are two scenarios so far where I’ve been able to reproduce the problem:

  • VM guest has a static MAC assigned and is running on a VM host. Shut down the VM, assign it a dynamic MAC and start it again on the same host. You’ll find it has no network connectivity.

  • VM guest is running on VM Host A with a dynamic MAC. Live Migrate the VM guest to Host B. It has network connectivity at this point, but if you restart the VM on the opposite host you’ll find it receives a new MAC and no longer has network connectivity.

Take a look at this diagram (only showing NICs relevant to Hyper-V) and you’ll see what the setup is that causes the issue. We have 2 Broadcom NICs on Dell R710’s each connected to a different physical switch to protect against a port, NIC, or switch failure. They are teamed in an Active/Passive configuration. No load balancing or link aggregation going on here. The virtual adapter composed of the two team members is then passed through as a virtual switch to Hyper-V and it is not shared with the host operating system. The host itself has a team for its own management and for the Live Migration network, which I’ll point both work flawlessly - the issue here is purely related to Broadcom’s teaming through a Hyper-V virtual switch.


Say I have a VM running on Host A where the NIC team has a hypothetical MAC called MAC A. When it boots up, it receives a dynamic MAC address we'll call MAC C from Host A’s pool. If you try to ping the VM guest’s IP and then look at your ARP table you’ll see something like:

Internet Address Physical Address Type MAC A Dynamic

This is because the NIC team is responsible for answering requests on behalf of the VM. When the NIC team receives traffic for the VM’s IP it will accept it, and then pass it along to the Hyper-V virtual switch. If you were to take a packet trace off the NIC you’ll see the team has modified the Layer 2 destination address to be MAC C, the dynamic MAC the VM got when it booted. This is how the teaming is supposed to work.

Now say I migrate the VM to Host B (where the NIC team has a MAC called MAC B) via Live or Quick migration. The VM retains connectivity and if you take a look at your MAC table you’ll now see something like:

Internet Address Physical Address Type MAC B Dynamic

Yup, the MAC for Host B’s NIC team is now answering requests for the VM’s IP. Again, this is how the teaming is supposed to work. Everything is peachy and you might think your clustering is working out great, until you restart the VM.


When the VM restarts, upon booting it receives a new dynamic MAC from Host B’s pool and you’ll find it has no network connectivity. Your ARP table hasn’t changed (it shouldn’t, the same team is still responsible for the VM), but the guest has been effectively dropped. When I pulled out a packet trace what I noticed was the team was still receiving traffic for the VM’s IP, which ruled out a switching problem, but it was still modifying the packets and sending them to MAC C. When in fact, now the VM has restarted it has MAC D. The problem is that it seems somebody (the driver) forgot to notice the VM has a new MAC and is sending packets to the wrong destination, so the VM never receives any traffic.


I found that toggling the NIC team within the host actually fixes the problem. If you simply disable the virtual team adapter and then re-enable it the VM will instantly get its connectivity back so it seems that during the startup process the team reads the VM MACs it’s supposed to service. I would think this is something it should be doing constantly to prevent this exact issue, but for now it looks like it’s done only at initialization.

The most practical workaround I’ve found so far is to just set static MAC addresses on the VMs within the Hyper-V settings. If the VM’s MAC never changes, this problem simply doesn’t exist. So while that defeats the purpose of the dynamic MAC pool on a Hyper-V host it allows the teaming failover to operate properly while you restart VMs and move them between cluster nodes.

I’ve raised the issue with Dell/Broadcom and they agree it’s a driver problem. There is supposedly a driver update due mid-March, but no guarantees this will be addressed in that update. The next update isn’t slated until June which is a long time to wait, hence the recommendation to just use Intel NICs.

Other notes for the inquisitive:

  • Disabling the team and using only a single adapter makes this work properly.
  • Happens with or without all TOE, checksum and RSS features.
  • No VLAN tagging in use.
  • Issue persists when team members are plugged into the same switch.
  • Latest drivers from Dell/Broadcom (12/15/2009) as of this writing.
  • Happens whether teaming is configured before or after Hyper-V role is installed.