Memory handling techniques in VMware

               
             The way VMware handles its memory had always amazed me. How can a 4 GB ESXi hypervisor allocate 3 VMs with 2 GB vRAM each? What happens if there is a resource crunch ?

I will try to explain all these questions in this blog.

Let us start from basics. How can we host 3 VMs with 2 GB vRAM each in a 4 GB RAM hypervisor ? The answer is Memory Over commitment. We know that VMware is very much fond with this word ‘Over Commitment’ as it uses this for every physical resource. When we present physical RAM to a VM, it is called virtual RAM (vRAM).

When a VM is provisioned in ESXi with 2 GB, an equivalent swap file will be created by the hypervisor for this VM. This swap file will be stored with .vswp extension in the VM folder or in a location specified by the user provided during VM provisioning. With this swap file, VMware has the flexibility to over commit its resources. When host physical memory is full, Swap file mechanism helps hypervisor to swap out guest physical memory to the swap file which resides in the datastore. The swap file utilization is not the one and only method for handling memory in ESXi.

There are 4 memory handling techniques in ESX/ESXi:

• Transparent Page Sharing (TPS)
• Ballooning
• Memory Compression
• Swapping

Transparent Page Sharing (TPS)

The guest VM’s memory space is divided into 4 KB blocks known as Pages.

TPS is a memory handling method in ESX/ESXi which makes use of two identical pages in host memory. If there are two pages with identical contents in the host memory which are used by two different VMs, one of the pages will be reclaimed and the other will be shared between the VMs. ESXi continuously scans its pages to determine if any shared page is available in the host memory which could be shared with. This is done with the help of a hash value, which is created for each and every page.

Ballooning (vmmemctl)

TPS is a natural method of memory reclamation and doesn’t depend on the memory utilization. If there is a memory crunch, ESXi has other memory reclamation techniques. In these situations, hypervisor will first turn to ballooning option. Ballooning uses the vmware tools in the VM for this process.

One of the beauties of VMware hypervisor is that one VMs is unaware of other VMs running in the same host (In fact they even don’t know that they are VMs). But this sometimes appears to be a curse as the VM will not recognize any resource crunch occurring in the ESXi host or other VMs.

What happens in this scenario ?

Here comes Ballooning for the rescue!!! Ballooning is the technique used by ESXi hosts to let the VMs know about this memory crunch. The hypervisor will initiate ballooning via vmmemctl drivers.

How does ballooning work?

During the time of a memory crunch, hypervisor will inform vmmemctl drivers of VMs. The hypervisor will set a balloon limit which depends on the amount of guest memory to be reclaimed and inflate the balloon. During the time of inflation, the ballooning drivers will make use of the guest physical page and any applications or data present in the guest physical page will be swapped to the guest’s page file (also called as the virtual memory) and this process of swapping is called paging. Please don’t confuse paging with hypervisor swapping which will be discussed later.

By this time you would have understood that ballooning will not work if vmware tools is not installed.

Memory compression and Hypervisor Swapping

Ballooning is the most effective way in reclaiming memory during memory crunch but not the quickest (especially when all your VMs are using their full allotted memory) . If ballooning cannot resolve the memory crunch quickly, hypervisor would seek the help of ‘Swapping’. With this technique, ESXi swaps its memory pages used by VMs to disk. This option is only considered as a last resort, as the action could leave to huge performance impact. Anyway the impact will be lesser if the hypervisors use SSDs.

Memory Compression is a mechanism which is used to reduce the impact of hypervisor swapping. 'Hypervisor Compression' compresses memory pages managed by ESXi, so that the number of pages to be swapped will be reduced thereby reducing the impact.