We have been hearing rumblings about the server operating system and its role being disrupted by virtualization as companies such as VMware push hypervisor and virtualization management software beyond the OS directly to the server. The company highlighted the shift with a series of significant pre-installation announcements during VMworld Europe last week.

VMware will embed its ESX 3i hypervisor in servers from Dell, Fujitsu Siemens, HP and IBM; it scored more specific deals to integrate its hypervisor software with 10 HP ProLiant server models and factory installation on Dell PowerEdge servers.

So what does this mean for OS vendors, and in particular the leading Linux server players Red Hat and Novell?

For its part, Red Hat sees the embedding of VMware and other virtualization software as the next step in the evolution of silicon, but also as a limited early step toward what is possible with a server optimized for virtualization out of the box. In fact, Red Hat laments the fact that a standard virtualization layer seems elusive and calls the latest virtualization pre-installation ‘a rough spot’ along the way. Following its evolution-of-silicon logic, Red Hat says it is hardware vendors such as AMD and Intel that will create that standard virtualization layer and capability. So Red Hat views the latest servers with VMware not as ’servers without an OS,’ but instead as ’servers with a hypervisor.’ Calling for fuller virtualization capability out of the box, Red Hat argues the embedded ESX 3i deals are aimed more at virtualization upgrade paths than revenue.

Still, servers that ship with no OS as opposed to having RHEL pre-installed would seem likely to have an impact on a company whose bread and butter business is pre-installation on servers, wouldn’t it?

Novell sees some upside not only in the proliferation of virtualization, but also in VMware’s ESX Server 3i software, which it describes as easier for customers (32MB footprint with no OS to install or patch), reputedly more secure than non-embedded hypervisors and faster with less code to traverse. Novell indicates VMware may be taking somewhat of a risk, though, since OEMs like HP will look to upsell to their own software to create and manage VMs, which ESX 3i can’t do (it’s a light hypervisor for single-server consolidation). This could also be an opportunity for the OS vendors to take over the creation and management of VMs. Still, Novell concedes that ESX Server 3i and Citrix XenServer OEM Edition could gain traction and greatly marginalize OS-plus-hypervisor offerings. However, the OS is still needed as the guest OS. Novell also realizes it must contend with non-embedded hypervisors from VMware, Citrix, Red Hat and Microsoft.

VMware says its pre-installation deals, which mean servers shipping with virtualization on top of bare metal as opposed to alongside or on top of the OS, are significant in that they will drive proliferation of virtualization across servers shipping today. VMware doesn’t argue the OS is obsolete just yet, though. The company says while its virtualiztion software does some of what the OS has done in the past in terms of resource management and underlying hardware, operating systems are still relevant in their enablement and support of applications.

Seems the Linux vendors think they will remain relevant in virtualiztion, as well. We’ll be watching to see if they can do so.