1. Introduction

I’m not sure whether this applies to other versions of VMware Player as well, as I haven’t tried any othery, but starting with either some version of the software or some Linux kernel version, the recompilation of the VMware Player kernel modules would fail after a kernel update and a reboot with the new kernel. In this article I’ll got into great detail about the issue (far more that what you need to just fix it), so if you don’t care about “reasons” and just wish to apply the fix, please go ahead and read only 2., then hop to the end of the article at 6.! Those who wish to understand the whole thing can keep reading after 2..

My exact software versions look like this at the moment, so for those, the problem definitely exists:

$ uname -or
2.6.32-696.23.1.el6.x86_64 GNU/Linux

$ vmware-installer -l | grep vmware-player
vmware-player        7.0.0.2305329

2. The issue

After updating your kernel and rebooting like normal, once you restart VMware Player, it’ll ask for a recompilation of its kernel modules for network, block device and other kinds of support to be usable with your updated kernel:

VMware Player 7 asking to recompile the kernel modules

That’d be fine of course, if it wouldn’t suddenly fall on its face like this:

The network kernel module compilation failing on CentOS 6.x

Of course, without network support, the services won’t start, and VMware Player will refuse to even work at all. Wouldn’t make much sense anyway, as pretty much every virtual machine will require network support. So, let’s look into what’s going wrong!

3. A quick, superficial analysis

The VMware Kernel Module Updater is telling us to look at some log file, and that’s what I did (requires root privileges to just look at it by the way). However, it was far less revealing than I’d hoped. The only relevant line to be found was this one:

W110: Failed to build vmnet.  Failed to execute the build command.

Yeah alright, great. That didn’t help much, but at least it gave us the name of the failing module: “vmnet”. Let’s get more detailed output by retrying the module compilation on a terminal instead of with the GUI:

# vmware-modconfig --console --install-all 2>&1 | grep error

That command will launch the same process of compiling the kernel modules, but it’ll show us compiler errors as well. As we’re filtering the output to show errors only, we’ll get this:

/tmp/modconfig-5SHpfd/vmnet-only/netif.c:153:79: error: macro "alloc_netdev" passed 4 arguments, but takes just 3
/tmp/modconfig-5SHpfd/vmnet-only/netif.c:153: error: ‘alloc_netdev’ undeclared (first use in this function)
/tmp/modconfig-5SHpfd/vmnet-only/netif.c:153: error: (Each undeclared identifier is reported only once
/tmp/modconfig-5SHpfd/vmnet-only/netif.c:153: error: for each function it appears in.)

Now that’s actually helpful! We now know that the call of function alloc_netdev() from netif.c, which is a part of vmnet fails. The corresponding VMware kernel module source tarballs can be found in /usr/lib/vmware/modules/source/, which is where we’ll start with the real work.

4. The subsequent in-depth analysis

Enter that directory in a terminal (as root), and you’ll find several source tarballs, one of them being vmnet.tar. Create a backup copy of the tarball for added safety and then unpack it. Something like this:

# cp ./vmnet.tar ./vmnet-stock-backup.tar
# tar -xvf ./vmnet.tar

With that, you’ll get a new subfolder /usr/lib/vmware/modules/source/vmnet-only/.

The errors above showed us that the problem starts in line 153 of netif.c, so open vmnet-only/netif.c in your favorite text editor and jump to said line 153. There you’ll find the following C preprocessor conditional block:

#if LINUX_VERSION_CODE >= KERNEL_VERSION(3, 18, 0) || defined(NET_NAME_USER)
   dev = alloc_netdev(sizeof *netIf, deviceName, NET_NAME_USER, VNetNetIfSetup);
#else
   dev = alloc_netdev(sizeof *netIf, deviceName, VNetNetIfSetup);
#endif

What this does is to compare your currently running Linux kernel version to 3.18.0 and check for the presence of some NET_NAME_USER macro. Depending on that, it calls the alloc_netdev() function with either four or only three parameters, omitting the macro mentioned. Clearly, on CentOS 6.x it’s being called with all four parameters, even though the function expects three on that system, thus breaking the code.

Initially I assumed that the first part of the conditional (the #if #else #endif block) wouldn’t work correctly. So I had a look at the LINUX_VERSION_CODE and KERNEL_VERSION macros. Later I found that to be the wrong place to start looking, but I’ll document my slightly pointless excursion nonetheless, as it proved to be somewhat informative:

4a. The Linux kernel version checking code

Those two macros are defined in /usr/src/kernels/`uname -r`/include/linux/version.h, and look like this on a typical CentOS 6.x system:

#define LINUX_VERSION_CODE 132640
#define KERNEL_VERSION(a,b,c) (((a) << 16) + ((b) << 8) + (c))

The LINUX_VERSION_CODE number is rather interesting. It’s calculated from the Linux version number triplet in a curious way. In my case it’s kernel version 2.6.32, so let a = 2, b = 6 and c = 32. The way to calculate the number would then be 2¹⁶ × a + 2⁸ × b + c. So it’s 2¹⁶ × 2 + 2⁸ × 6 + 32 = 132640. Knowing that, and having looked at the macros, it didn’t make any sense that the code in vmnet-only/netif.c would fail. After a quick search on the web it seemed like that was the usual way to do it too. So time to look at the second part that the code connected to the first via a logical OR (||).

4b. The NET_NAME_USER macro

The check for whether NET_NAME_USER was defined could also trigger the 4-parameter call to alloc_netdev(), so I needed to look at that next. That macro has something to do with network interface naming by user space applications, and it should be defined in the Linux kernel header netdevice.h. On CentOS 6.x that file can be found in /usr/src/kernels/`uname -r`/include/uapi/linux/, and we’ll look at line 170 of it in a text editor, where you can find this:

#define NET_NAME_USER           3       /* provided by user-space */

So it seems to be there alright? But I wanted to really make sure!

4c. Writing a small C program to verify the working of VMwares’ preprocessor conditional

To make sure I wasn’t just seeing things that weren’t really there, I wrote myself a small C program check-nnu+kernelversion.c to check for those macros. It looks like this:

#include <stdio.h>
#include <sys/socket.h>
#include <linux/kernel.h>
#include <linux/version.h>
#include <linux/netdevice.h>
 
int main()
{
#if defined(NET_NAME_USER)
  printf("NET_NAME_USER is \e[1mDEFINED\e[0m in linux/netdevice.h.\n");
#else
  printf("NET_NAME_USER is \e[1mNOT DEFINED\e[0m in linux/netdevice.h.\n");
#endif
 
#if LINUX_VERSION_CODE >= KERNEL_VERSION(3,18,0)
  printf("Kernel version \e[1mGREATER THAN OR EQUAL TO\e[0m 3.18.0.\n");
#else
  printf("Kernel version \e[1mLOWER THAN\e[0m 3.18.0.\n");
#endif
  return 0;
}

You can compile it very easily: $ cc ./check-nnu+kernelversion.c -o check-nnu+kernelversion. And when you run it on CentOS 6.x by executing $ ./check-nnu+kernelversion.o, you’ll likely get exactly this:

Displaying the status of the kernel version and the NET_NAME_USER macro with my “check-nnu+kernelversion” program

Su huh? What? Now wait, let’s make 100% sure, and look at alloc_netdev() as well!

4d.) alloc_netdev()

alloc_netdev() is defined in /usr/src/kernels/`uname -r`/include/linux/netdevice.h, so open that file up, and have a look at lines 2627:2628:

#define alloc_netdev(sizeof_priv, name, setup) \
        alloc_netdev_mq(sizeof_priv, name, setup, 1)

As you can see, it’s defined with three parameters only. The second function is the multiqueue version, that we won’t be using anyway, so that one doesn’t matter to us. So, it’s three parameters in any case, no matter what. The presence of NET_NAME_USER isn’t being taken into account there at all!

So what does that all mean?

5. Conclusion or where I think VMware went wrong

It seems that the VMware developers had assumed that alloc_netdev() would always take that fourth interface naming parameter as soon as we have a Linux kernel newer than or equal to 3.18.0 or if we have NET_NAME_USER defined. But that’s simply not the case!

Even if that macro is defined, the system may still have a version of alloc_netdev() that doesn’t make use of it, maybe because some parts of the kernel code just weren’t backported to CentOS from upstream. That “fourth parameter patch” for alloc_netdev() seems to have been [submitted in 2014], quite some time after the initial release of CentOS 6.x, but long before its current 2.6.32.x kernel versions were rolled out.

Still, CentOS kernels rarely get feature backports, but mostly just security and compatibility fixes, so maybe that’s a reason somehow.

Anyhow, VMwares’ assumption that the presence of NET_NAME_USER would also mean the presence of a past-2014 version of alloc_netdev() in the kernel was flawed, at least on all Red Hat Enterprise Linux 6 systems and its derivatives.

We can either change the OR to an AND in VMwares’ vmnet-only/netif.c, or simply remove the check for NET_NAME_USER in that file altogether. I went for the logical AND variety, although I’m not perfectly sure if this is correct. But I assume it should be, given the circumstantial evidence.

6. The solution

I’ll describe this in a way so that both people who’ve read the in-depth stuff (there’ll be some redundancies for you guys) as well as others who just want to solve this quickly can understand it. First, open up a terminal window, become superuser root and prepare your favorite text editor.

Navigate to /usr/lib/vmware/modules/source/ and backup the file vmnet.tar sitting there, if you haven’t already, for safety and also for reference. Then, unpack it while in /usr/lib/vmware/modules/source/: # tar -xvf ./vmnet.tar. This yields a subfolder vmnet-only/. Open the file vmnet-only/netif.c in your text editor, and look for the following line at line number 153:

#if LINUX_VERSION_CODE >= KERNEL_VERSION(3, 18, 0) || defined(NET_NAME_USER)

Change the || to && (replacing OR with AND) so that the line looks like this (Here I’ve also added a comment on top of it describing the change):

/* Modded version by XIN.at changing || to &&, because some systems may have
 * the NET_NAME_USER macro in linux/netdevice.h while still not having a
 * modern, 4-parameter version of alloc_netdev() in the same header.
 */
#if LINUX_VERSION_CODE >= KERNEL_VERSION(3, 18, 0) && defined(NET_NAME_USER)

Ok, now repack the tarball while still sitting in /usr/lib/vmware/modules/source/ (this will overwrite the existing vmnet.tar):

# tar -cvf vmnet.tar ./vmnet-only

After that, relaunch VMware Player 7 and confirm the recompilation of the kernel modules. If you’ve tried this before, then only the network part should be missing, and building that remaining part should work now:

Successful compilation of the remaining vmnet kernel module

Alright, up and running again! You may now delete /usr/lib/vmware/modules/source/vmnet-only/, but first please make sure you still have a backup copy of the original vmnet.tar first, ’cause you never know!

After that, subsequent updates of the Linux kernel and VMware Player 7 kernel modules will keep working, as VMware Player will keep using the updated /usr/lib/vmware/modules/source/vmnet.tar from here on out.