How to check memory pressure of ESXi

Memory and CPU, both are crucial part of ESXi, sometimes admins over-commit memory of ESXi unintentionally. ESXi has defined threshold to manage memory pressure and uses below memory reclamation techniques.

  • Transparent Page sharing
  • Ballooning
  • Memory compression
  • Swapping
Now, Q :- When will ESXi start memory reclamation techniques ?
Ans: ESXi uses some memory states "High, Clear, Soft, Hard & Low" prior stating memory reclamation.

You can find lot of blogs to know more about above information so, I am not going to cover, what is TPS, ballooning etc. I am writing this blog to share the information, how to check what states and how many times ESXi went for.

memstats is the command used to check memory states.

By default, it shows much more information but use -s option and specify desired columns.


It's a great tool to get better understanding to check whether host is under memory pressure or not.

Snapshot can be nightmare for vmware administrator

Snapshot is the one of best feature of VMware. It is not backup solution but a good option if want to preserve some existing configuration/settings of vm/installed applications. User can take snapshot of vm and make changes if anything goes wrong, roll-back those changes with snapshot, taken prior making  changes. Most of the people uses this as backup and left it as for long time after creation. So, question here is how can be leaving snapshot becomes headache for Administrators.

As you guys are aware, when we create snapshot of vm a delta vmdk disk created and attached with vm. Original vmdk marked as readonly and all changes will be performed at delta vmdk. As Type of delta vmdk disk is thin and can grow upto the size of configured vmdk.

Example:
Size of datastore = 50 GB
Size of vmdk = 30 GB
Size of delta vmdk few MB but can grow upto 30 GB.

Now, suppose vmdk is Thick which already consumed 30 GB space of datastore and delta vmdk can grow up to 30 GB so, total space requirement would be 60 + GB.
Most of the peoples follow 10% threshold of free space. While snapshot created, admin was having enough free space 15GB but if anyhow, vm found as high IOPs, it will easily consume free space and create outage.

If type of vmdk is thin, even outage can't be avoided but can give us more time to full space of datastore.

As a good practice, we should delete snapshot once requirement of its completed.

As a work around, we can also create alert for snapshot usage. If snapshot reaches to configured alert value, alert will be generated and remediate action can be taken.

vNUMA -Virtual NUMA.

vSphere ESXi was taking benefit of NUMA since long time, as ESXi only aware of NUMA node, esxi kernel has the responsibility to place vm right NUMA node. Guest OS wasn't aware but with vNUMA guest OS is also aware of NUMA, it exposes guest OS to physical NUMA. Both guest OS and applications take adavantage of NUMA optimization, which provide performance improvements within VM. vNUMA allows VMs to benefit from NUMA, even if the VM itself is larger than the physical size of the NUMA nodes.


vNUMA comes in picture when you configure vm with more than 8 vCPU and without HotAdd vCPU feature.

Now question is here, how guest OS create virtual NUMA Nodes and how it impact the performance of vm? I have performed some vCPU configuration with VM and below are the results of vNUMA nodes.


CoreInfo is the sysinternal tool, I have used to check the numa configuration at OS level as there is no inbuilt tool in Microsoft OS however, OS is linux, you can use numctl command to check numa at OS level.

Physical host configuration

CPU - 4
Core - 8
Logical Processor with HT - 64

NUMA node - 4 and 1 NUMA node have 8 cores.


1. Virtual machine configured with 1 * 9 vCPU

ESXi kernel has used 2 numa to satisfy vm cpu/memory requirement but at OS level only one numa node is detected.

2. Virtual machine configured with 3 * 4 vCPU

ESXi kernel has given 3 numa to satisfy vm cpu/memory requirement and at OS level 3 numa node is detected.

3. Virtual machine configured with 2 * 9 vCPU

ESXi kernel has given 3 numa to satisfy vm cpu/memory requirement but at OS level only 2 numa node is detected.

It has been observed with above vms configuration, calculation of numa at guest level is directly proportional to socket configured at vm level. What will happen if we configure sockets to vm, more than physical sockets of hosts. Below is the result.

Physical host configuration

CPU - 4
Core - 4
Logical Processor with HT - 32

NUMA node - 4 and 1 NUMA node have 4 cores.

1. Virtual machine configured with 9 * 1 vCPU
vm is placed to 3 numa nodes at esxi kernel and at guest os, 2 numa nodes calculated. 4 NUMA nodes are calculated and 1 numa node is configured with 4 cores. As vm is configured with 9 socket which is more than physical socket of host,  2 numa node is calculated to satisfy 9 socket of vm.

Please correct if you guys found any discrepancies.  :-)

vRo script to delete snapshots of vms listed in text file

vRO script will read text file containing name of VMs, snapshots to be deleted.

To read the text file in vRO script, first of all you need to assign permission to path to which script can read text file. Task is quite simple, you need to append a line containing path along with permission to file named "js-io-rights.conf". Below are the steps.

Modify the js-io-rights.conf file:
  1. vi /etc/vco/app-server/js-io-rights.conf
  2. Press the i key on the keyboard
  3. Copy & paste the following line to the end file:
  4. +rwx /tmp
  5. Press the esc key on the keyboard
  6. Type in :wq! and press the Enter key
  7.  restart vRO server with command "service vco-server restart"
Below are the screenshots would be helpful to create workflow.










 
----------------------------------------Start code of script----------------------------------
var line,tempVM,i,vmname;
var myFile = new FileReader(Path);
myFile.open();
data = myFile.readAll();
var lines = new Array();
var fvms = new Array();
lines = data.split(/\r\n|\n\r/);
System.log(lines);
var vms = VcPlugin.getAllVirtualMachines();
for (line in lines )
{
 tempVM = lines[line];
 //System.log("2nd line " + tempVM);
 for (i in vms)
 {
  vmname = vms[i];
  //System.log(tempVM + " : " + vmname.name);
  if (tempVM.match(vmname.name))
  {
   System.log(tempVM + vms[i].name + " find");
   vm = vmname;
   //System.log(vm.snapshot);
   var allSnaps = new Array();
   var allParents;
   var snapshot = vm.snapshot;
   var rootSnapshotList = snapshot.rootSnapshotList;
   for each (var rsnap in rootSnapshotList)
   {
       getSnapshotsOfVM(rsnap);
   }
      for each (var snap in allSnaps)
   {
       System.log("VM Name: " + snap.vm.name);
          System.log("Snapshot Name: " + snap.name);
          System.log("Snapshot creation time: " + snap.createTime.toLocaleString());
          System.log("Snapshot is old, removing...");
    var task = snap.snapshot.removeSnapshot_Task(false,true);
    var actionResult = System.getModule("com.vmware.library.vc.basic").vim3WaitTaskEnd(task);
   }
  }
 }
}
function getSnapshotsOfVM(snapshotTree)
{
    allSnaps.push(snapshotTree);
    var childTrees = snapshotTree.childSnapshotList;
    if(childTrees != null) {
        for(var index in childTrees) {
            if(childTrees[index] != null) {
                getSnapshotsOfVM(childTrees[index]);
            }
        }
    }
}
 ----------------------------------------End code of script----------------------------------








Code has been modified by me as per mine requirement. It is taken from internet.



PowerCli script to delete snapshot of vms

Add-PSSnapin Vmware.Vimautomation.Core
# Enter your vCenter and credentials
$VIServer = "name of vCenter"

# Connect to vCenter using the variables above
Connect-VIServer -Server $VIServer

#Get Windows Guest machines from input file
$vmguests = Get-Content "C:\csv-files\input.txt"


#Write new config options to VMs
foreach ($vm in $vmguests){
       
        Get-Snapshot -vm $vm | Remove-Snapshot -Confirm:$false      
        
}

# Disconnect from vCenter
Disconnect-VIServer -Server $VIServer -Confirm:$false

PowerCli script to pull LUN id of datastore of cluster

Add-PSSnapin Vmware.Vimautomation.Core
Connect-VIServer “name of vcenter”
$clusters = Get-Cluster
#$host_Cluster = $cluster | %{ Get-VMHost -Location $_ }
foreach($cluster in $clusters)
{
    $host_Cluster = Get-VMHost -Location $cluster
      
foreach($hostServer in $host_Cluster)
{
$esxcli = Get-EsxCli -VMHost $hostServer.Name
$data = $esxcli.storage.core.path.list()
$naa = $esxcli.storage.vmfs.extent.list()
#$data
#$naa
$report=@()
$naa | %{
$D = $_.DeviceName
$dataStore = $_.VolumeName
$ext = $_.ExtentNumber
        $data | %{
        if($_.Device -eq $D)
        {
           $row = "" | select DName, LUN, datastore, extent, Host, Cluster
            $row.Dname = $D
            $row.LUN = $_.LUN
            $row.datastore = $dataStore
            $row.Extent = $ext
            $row.Host = $hostServer.Name
            $row.Cluster = $cluster
        }
                }
           
$report += $row
}
$report | Export-csv -NoTypeInformation "C:\csv-files\datastore_inventory.cvs" -Append
}
}
Disconnect-VIServer -Server $global:DefaultVIServers -Force

vRealize Orchestrator script for RV tool







-----------------------------Code of Script to Pull Information----------------------------


var i,j, vmName;
var contentMessage,logtxtDisk,logtxtNIC,logtxtNetwork;
content = "VM Name,HostName,Cluster,PowerState,RAM,CPU,Core,IP Address,OS \n";
vdiskcontent = "VM Name,vDisk,Path,Size,DataStore \n";
vNetwork = "VM Name,Type,Port group \n";
logtxtHNetwork = "Hostname,vNic,Driver,Address,DevID,PortID,CDP Info \n";
contentMessage = "Hi Team, <br> <br> PFA VMs report. <br> <br> Thanks<br> Manoj";
for ( i in clustername )
{
actionResult = System.getModule("com.vmware.library.vc.cluster").getAllVMsOfCluster(clustername[i]);
                for ( j in actionResult)
                {
                                vmName = actionResult[j].name;
                                var guest = actionResult[j].runtime;
                                var hostname = guest.host.name;
                                var cluster = (actionResult[j].runtime.host).parent;
                                logtext =  vmName + "," + hostname + "," + cluster.name + "," + actionResult[j].runtime.powerState.value + "," + actionResult[j].config.hardware.memoryMB + "," +  actionResult[j].config.hardware.numCPU + "," + actionResult[j].config.hardware.numCoresPerSocket + "," + actionResult[j].guest.ipAddress + "," + actionResult[j].guest.guestFullName + "\n";
                                content = content +  logtext ;
                                //Get Datastores
                                var datastoresProp = new Properties();
                                var diskSizes = new Array();
                                var devices = actionResult[j].config.hardware.device;
                                var l = 0;
                                for (var k in devices)
                                {
                                                if (devices[k] instanceof VcVirtualDisk)
                                                {
                                                                diskSizes[l++] = devices[k].capacityInKB / 1048576;
                                                                var vmdkSize = devices[k].capacityInKB / 1048576;
                                                                var datastore = devices[k].backing.datastore;
                                                                var vmdkName = devices[k].deviceInfo.label;
                                                                var vmdkFName = devices[k].backing.fileName;
                                                                logtxtDisk = vmName + "," + vmdkName + "," + "," + vmdkFName + "," + vmdkSize + "," + datastore.name + "\n";
                                                                vdiskcontent = vdiskcontent + logtxtDisk;                                           
                                                }
                                                if ( devices[k] instanceof VcVirtualVmxnet3)
                                                {
                                                                logtxtNIC = vmName + ",vmxnet3, " + devices[k].backing.deviceName + "\n";
                                                                vNetwork = vNetwork + logtxtNIC;
                                                }
                                                if ( devices[k] instanceof VcVirtualVmxnet2)
                                                {
                                                                logtxtNIC = vmName + ",vmxnet2, " + devices[k].backing.deviceName+ "\n";
                                                                vNetwork = vNetwork + logtxtNIC;
                                                }
                                                if ( devices[k] instanceof VcVirtualPCNet32)
                                                {
                                                                logtxtNIC = vmName + ",PCNet32, " + devices[k].backing.deviceName+ "\n";
                                                                vNetwork = vNetwork + logtxtNIC;
                                                }
                                                if ( devices[k] instanceof VcVirtualE1000)
                                                {
                                                                logtxtNIC = vmName + ",E1000, " + devices[k].backing.deviceName+ "\n";
                                                                vNetwork = vNetwork + logtxtNIC;
                                                }
                                }
                }
}
/*Start - Code to get DataStore and NAA id */
var vdatastoretxt, hostcount, naaid, hname_ds;
var datastore, datastores;
vdatastoretxt = "DataStore,HostCount,HostName,VM Count,URL,Type,MHA,UUID,NAAID \n";
datastores = System.getModule("com.vmware.library.vc.datastore").getAllDatastores();
for each ( datastore in datastores)
{
                try{
                System.log(datastore.name);
                var hosts_ds = datastore.host;
                hname_ds = " ";
                naaid = " ";
                for each (var host_ds in hosts_ds)
                {
                                hname_ds  =  host_ds.key.name + hname_ds;
                }
                if (hosts_ds.length != 0)
                {
                hostname = hosts_ds[0].key;
                for each (var mountedVolume in hostname.configManager.storageSystem.fileSystemVolumeInfo.mountInfo)
                {
                                for each( var volumeExtent in mountedVolume.volume.extent)
                                {
                                                if(mountedVolume.volume.name.match(datastore.name))
                                                {
                                                                var volumeMountinfo = mountedVolume.mountinfo;
                                                                System.log(volumeExtent.diskName);
                                                                naaid = volumeExtent.diskName + naaid;
                                                }             
                                }
                }
                }
                vdatastoretxt  = vdatastoretxt + datastore.name + "," + hosts_ds.length + "," + hname_ds +  "," + datastore.vm.length + "," + datastore.info.url + "," + datastore.summary.type + "," + datastore.summary.multipleHostAccess + "," + datastore.info.vmfs.uuid + "," + naaid + "\n";
                }
                catch (ex)
                {
                  throw "Error"
                }
}
/* End - Datastore & NAA id */

/* Start - Code to Pull Portgroup Information */
var pg_hosts,pg_hostname;
var portgroup,portgroups,vmNics,porttxt;
var promiscuous,ftransmit, macchange;
porttxt = "HostName,Portgroup,VLAN ID,vSwitch,Promiscuous,ForgedTransmits,MacChange,Policy \n";
for ( i in clustername )
{
pg_hosts = System.getModule("com.vmware.library.vc.cluster").getAllHostSystemsOfCluster(clustername[i]);
for each(pg_host in pg_hosts)
{
                pg_hostname = pg_host.name;
                System.log(pg_hostname);
                portgroups = pg_host.configManager.networkSystem.networkInfo.portgroup;
                for each (portgroup in portgroups)
                {
                                if(portgroup.spec.policy.security.allowPromiscuous == "TRUE")
                                {
                                                promiscuous = "True";
                                }
                                else
                                {
                                                promiscuous = "False";
                                }
                                if((portgroup.spec.policy.security.forgedTransmits) = "TRUE")
                                {
                                                ftransmit = "True";
                                }
                                else
                                {
                                                ftransmit = "False";
                                }
                                if((portgroup.spec.policy.security.macChanges) = "TRUE")
                                {
                                                macchng = "True";
                                }
                                else
                                {
                                                macchng = "False";
                                }
                                porttxt = porttxt + pg_hostname + "," + portgroup.spec.name + "," + portgroup.spec.vlanId + "," + portgroup.spec.vswitchName + "," + promiscuous + "," + ftransmit + "," + macchng + "," + portgroup.spec.policy.nicTeaming.policy + "\n";
                System.log(porttxt);
                }
}
}
/* End - Code to Pull Portgroup Information */

/* Start - Code to pull Network information of Host */
var txtNetwork = ";";
var nw_hosts,nw_host,pNic,pnicInfos,pnicInfo,nic,network,networks;
for ( i in clustername )
{
                nw_hosts = System.getModule("com.vmware.library.vc.cluster").getAllHostSystemsOfCluster(clustername[i]);
                for each(nw_host in nw_hosts)
                {
                                pNic = nw_host.configManager.networkSystem.networkInfo.pnic;
                                pnicInfos = nw_host.configManager.networkSystem.queryNetworkHint();
                                for each (pnicInfo in pnicInfos)
                                {
                                                if(pnicInfo.connectedSwitchPort)
                                                {
                                                                for each (nic in pNic)
                                                                {
                                                                                if( nic.device == pnicInfo.device)
                                                                                {
                                                                                                networks = pnicInfo.subnet;
                                                                                                for each ( network in networks)
                                                                                                {
                                                                                                                txtNetwork = network.ipSubnet + "(VLAN:" +  network.vlanId + ")" + txtNetwork;
                                                                                                }
                                                                                                logtxtHNetwork = logtxtHNetwork + nw_host.name + "," + pnicInfo.device +  "," + nic.driver + "," + pnicInfo.connectedSwitchPort.address +  "," + pnicInfo.connectedSwitchPort.devId + "," + pnicInfo.connectedSwitchPort.portId +  "," + txtNetwork + "\n";
                                                                                }
                                                                }
                                                }
                                                else
                                                {
                                                                logtxtHNetwork = logtxtHNetwork + nw_host.name + "," + pnicInfo.device +  ",No Information,No Information,No Information,No Information,No Information \n";
                                                }
                                }
                }
}
/* End - Code to pull Network information of Host */

var message = new EmailMessage();
// Override default settings if and only if input parameter is set
if ( smtpHost != null && smtpHost.length > 0 ){
    message.smtpHost = smtpHost;
}
if ( smtpPort != null && smtpPort > 0 ){
    message.smtpPort = smtpPort;
}
if ( fromName != null && fromName.length > 0){
    message.fromName = fromName;
}
if ( fromAddress != null && fromAddress.length > 0){
    message.fromAddress = fromAddress;
}
message.toAddress = toAddress;
message.subject = subject;
message.ccAddress = ccAddress;
message.addMimePart(contentMessage,"text/html; charset=UTF-8");
//Create Attachment
var fileAttachment = new MimeAttachment();
fileAttachment.name = "vmsReport.csv";
fileAttachment.content = content;
message.addMimePart(fileAttachment,"application/zip");// charset=UTF-8");
fileAttachment.name = "vDiskReport.csv";
fileAttachment.content = vdiskcontent;
message.addMimePart(fileAttachment,"application/zip");// charset=UTF-8");
fileAttachment.name = "vNetwork.csv";
fileAttachment.content = vNetwork;
message.addMimePart(fileAttachment,"application/zip");// charset=UTF-8");
fileAttachment.name = "vDatastore.csv";
fileAttachment.content = vdatastoretxt;
message.addMimePart(fileAttachment,"application/zip");// charset=UTF-8");
fileAttachment.name = "vPortgroup.csv";
fileAttachment.content = porttxt;
message.addMimePart(fileAttachment,"application/zip");// charset=UTF-8");
fileAttachment.name = "vHostNetwork.csv";
fileAttachment.content = logtxtHNetwork;
message.addMimePart(fileAttachment,"application/zip");// charset=UTF-8");

System.log( "sending mail to host: " + message.bccAddress + ":" + message.smtpPort + " with user:" + message.username + ", from:" + message.fromAddress + ", to:" + message.toAddress );
message.sendMessage();

-----------------------------------------End of Script Code----------------------------------



Limit of Scripts:
1. Scsi bus sharing
2. RDM Info.

You are welcome to modify the script.