Friday, March 13, 2009

Disabling VLAN tagging in Hyper-V on Windows Server 2008 Server Core

 

Disabling VLAN tagging in Hyper-V on Windows Server 2008 Server Core

Last month, one of my colleagues wrote an article to share his experience with changing a setting on Server Core (http://blogs.technet.com/networking/archive/2009/01/08/configuring-advanced-network-card-settings-in-windows-server-2008-server-core.aspx). I had a call with a similar experience where I found myself asking “where do I find that in Server Core?” I’d like to share it with you so you don’t get caught in the same bind.

Hyper-V has the ability to support VLANs (802.1q) for the Host or the Guest Virtual Machines. Within the HyperV Manager, you open the Virtual Network Manager and can add a VLAN tag to a particular virtual network (Figure 1) or Guest (Figure 2):

clip_image002

Figure 1

clip_image004

Figure 2

This is all fine and good when you have the Hyper-V Manager GUI available to you. The question comes into play when you are running Server Core. First of all, to manage Hyper-V on that server, you can run the Hyper-V Manager MMC remotely on another 2008 server or on a Vista box after installing the following download (http://support.microsoft.com/kb/952627). In my case, a customer was using another 2008 server for management.

What went wrong?

The exact problem occurred when the customer accidentally enabled VLAN tagging on the External Network without having a VLAN-capable switch. As soon as that change was made, they lost all communication with the Hyper-V server and all of its guests. Fortunately, they had an ILO connection to the server so we were still able to get on the box, but the question then was, how do we remove the VLAN tag from the virtual switch without Hyper-V Manager?

The solution

The solution is best understood if you navigate the registry and look at some of the registry keys created for the Virtual Switches. If you navigate to HKLM\SYSTEM\CCS\Services\VMSMP\Parameters, you’ll see some interesting keys there (Figure 3). These keys show all of the NICs on the host server (NicList) and all of the Virtual Networks that have been created (SwitchList).

clip_image006

Figure 3

Now that you know where the information is stored, it’s actually really logical how to find the External Network and remove the VLAN setting. First, under HKLM\SYSTEM\CCS\Services\VMSMP\Parameters\NicList, open the individual GUIDs and examine the "FriendlyName" value until you find the name of the External Virtual network. The results from my lab machine can be seen below for my network called “MSNET” (Figure 4):

clip_image008

Figure 4

Once you find the right virtual network, make note of the GUIDs under SwitchName and PortName in that network. Next, expand HKLM\SYSTEM\CCS\Services\VMSMP\Parameters\SwitchList and find and expand the GUID referenced in SwitchName above. Under that switch GUID, find the PortName GUID referenced above. This is the port in the virtual switch that the host is connected to. I’ve selected that port in Figure 5 below, so you can see the values for that port.

clip_image010

Figure 5

You will see a DWORD called AccessVlanId. Change the value to 0 your Hyper-V box and reboot. Once it comes back up, the VLAN tags will be removed and you should be able to communicate with the server.

Conclusion

VLAN support is a great feature of Hyper-V, but one that should be implemented slowly and thoughtfully if never done before. As always, you shouldn’t go messing with the registry unless you absolutely need to and be sure to test in the lab before rolling out any changes to your production network. You’ll thank me later.

One great article I found on VLAN support in Hyper-V that I wanted to share can be found here:

http://blogs.msdn.com/adamfazio/archive/2008/11/14/understanding-hyper-v-vlans.aspx

- Michael Rendino

Microsoft Enterprise Networking Team : Disabling VLAN tagging in Hyper-V on Windows Server 2008 Server Core

No comments:

Blog Archive