I encountered a strange problem so I thought I’d post about it.
We upgraded many VMware 4.1 ESX nodes to 5.0, and for the most part it went fine. However after the upgrade, our wireless users couldn’t authenticate on the internal network, but the guest network still worked fine. We were using a variety of Cisco access-points and they had worked perfectly fine with RADIUS auth before the upgrade. I tracked down the error, the RADIUS server (now called Network Policy Server in Windows 2008) was throwing error 266. This error would indicate it thought it was receiving a malformed packet! We were using the E1000 NIC driver on the VM’s that hosted the NPS/RADIUS server, and had heard that it could cause problems.
I created a new VM with the VMXNET3 driver, and copied the NPS/RADIUS settings over to this new server, and set the access-points to authenticate to it instead. This worked fine, so I changed the NIC driver on the old VM’s to VMXNET3, but this did not seem to solve the problem. For now I just created new VM’s with the VMXNET3 driver and they are working fine, I am not sure what is different or why this doesn’t quite work properly, but if encounter this problem hopefully this fixes it for you.