Log4j vulnerability POC with VMware vCenter.

 

How to create Dashboard in vRealize Operations Manager

Suppose you have a requirement to create a custom dashboard to publish CPU utilization of some set of VMs responsible to serve the application. In my case, I have picked a web application servers.

web-01a -> app-01a -> base-w10

Below are the steps to create a custom dashboard.

1. Create a Custom Group of VMs

2. Create a Views

3. Create a dashboard with the help of custom groups & views.

1. How to create custom group of vms.

Login to vrops and Click on Environment tab.

 

Please wait for sometime so that Health, Risk & Efficiency information will be captured. Below is the screenshot once information captured of custom group named "Webhosting"

Now custom group has been created and information is also captured, next step is to create CPU utilization views for virtual machines.

2. How to create Views in vRealize Operations Manager

Login to vrops and Click on Dashboard, Left side, there is option called Views.

We have completed view creation and it can be seen in Views option of vROPS. Next step is to create dashboad using custom group & views.

3. How to create dashboard.

 

 

Here our dashboard creation completed and we can see the CPU utilization of all three responsible servers for web application hosting. Same way we can create a view for Memory utilization and following dashboard, we can integrate view with dashboard.

Please feel free to correct in case of incorrect information shared. I have used VMWARE HOL LAB to capture the screenshots. Using HOL LAB since long time and can say, these labs are fabolus, well explanation of VMware products.  😃😃😃😃😃😃

How to check memory pressure of ESXi

Memory and CPU, both are crucial part of ESXi, sometimes admins over-commit memory of ESXi unintentionally. ESXi has defined threshold to manage memory pressure and uses below memory reclamation techniques.

• Transparent Page sharing
• Ballooning
• Memory compression
• Swapping

Now, Q :- When will ESXi start memory reclamation techniques ?
Ans: ESXi uses some memory states "High, Clear, Soft, Hard & Low" prior stating memory reclamation.

You can find lot of blogs to know more about above information so, I am not going to cover, what is TPS, ballooning etc. I am writing this blog to share the information, how to check what states and how many times ESXi went for.

memstats is the command used to check memory states.

By default, it shows much more information but use -s option and specify desired columns.

It's a great tool to get better understanding to check whether host is under memory pressure or not.

Snapshot can be nightmare for vmware administrator

Snapshot is the one of best feature of VMware. It is not backup solution but a good option if want to preserve some existing configuration/settings of vm/installed applications. User can take snapshot of vm and make changes if anything goes wrong, roll-back those changes with snapshot, taken prior making  changes. Most of the people uses this as backup and left it as for long time after creation. So, question here is how can be leaving snapshot becomes headache for Administrators.

As you guys are aware, when we create snapshot of vm a delta vmdk disk created and attached with vm. Original vmdk marked as readonly and all changes will be performed at delta vmdk. As Type of delta vmdk disk is thin and can grow upto the size of configured vmdk.

Example:
Size of datastore = 50 GB
Size of vmdk = 30 GB
Size of delta vmdk few MB but can grow upto 30 GB.

Now, suppose vmdk is Thick which already consumed 30 GB space of datastore and delta vmdk can grow up to 30 GB so, total space requirement would be 60 + GB.
Most of the peoples follow 10% threshold of free space. While snapshot created, admin was having enough free space 15GB but if anyhow, vm found as high IOPs, it will easily consume free space and create outage.

If type of vmdk is thin, even outage can't be avoided but can give us more time to full space of datastore.

As a good practice, we should delete snapshot once requirement of its completed.

As a work around, we can also create alert for snapshot usage. If snapshot reaches to configured alert value, alert will be generated and remediate action can be taken.

vNUMA -Virtual NUMA.

vSphere ESXi was taking benefit of NUMA since long time, as ESXi only aware of NUMA node, esxi kernel has the responsibility to place vm right NUMA node. Guest OS wasn't aware but with vNUMA guest OS is also aware of NUMA, it exposes guest OS to physical NUMA. Both guest OS and applications take adavantage of NUMA optimization, which provide performance improvements within VM. vNUMA allows VMs to benefit from NUMA, even if the VM itself is larger than the physical size of the NUMA nodes.


vNUMA comes in picture when you configure vm with more than 8 vCPU and without HotAdd vCPU feature.

Now question is here, how guest OS create virtual NUMA Nodes and how it impact the performance of vm? I have performed some vCPU configuration with VM and below are the results of vNUMA nodes.


CoreInfo is the sysinternal tool, I have used to check the numa configuration at OS level as there is no inbuilt tool in Microsoft OS however, OS is linux, you can use numctl command to check numa at OS level.

Physical host configuration

CPU - 4
Core - 8
Logical Processor with HT - 64

NUMA node - 4 and 1 NUMA node have 8 cores.

1. Virtual machine configured with 1 * 9 vCPU

ESXi kernel has used 2 numa to satisfy vm cpu/memory requirement but at OS level only one numa node is detected.

2. Virtual machine configured with 3 * 4 vCPU

 

ESXi kernel has given 3 numa to satisfy vm cpu/memory requirement and at OS level 3 numa node is detected.

3. Virtual machine configured with 2 * 9 vCPU

ESXi kernel has given 3 numa to satisfy vm cpu/memory requirement but at OS level only 2 numa node is detected.

It has been observed with above vms configuration, calculation of numa at guest level is directly proportional to socket configured at vm level. What will happen if we configure sockets to vm, more than physical sockets of hosts. Below is the result.

Physical host configuration

CPU - 4
Core - 4
Logical Processor with HT - 32

NUMA node - 4 and 1 NUMA node have 4 cores.

1. Virtual machine configured with 9 * 1 vCPU

vm is placed to 3 numa nodes at esxi kernel and at guest os, 2 numa nodes calculated. 4 NUMA nodes are calculated and 1 numa node is configured with 4 cores. As vm is configured with 9 socket which is more than physical socket of host,  2 numa node is calculated to satisfy 9 socket of vm.

Please correct if you guys found any discrepancies.  :-)

vRo script to delete snapshots of vms listed in text file

vRO script will read text file containing name of VMs, snapshots to be deleted.

To read the text file in vRO script, first of all you need to assign permission to path to which script can read text file. Task is quite simple, you need to append a line containing path along with permission to file named "js-io-rights.conf". Below are the steps.

Modify the js-io-rights.conf file:

1. vi /etc/vco/app-server/js-io-rights.conf
2. Press the i key on the keyboard
3. Copy & paste the following line to the end file:
4. +rwx /tmp
5. Press the esc key on the keyboard
6. Type in :wq! and press the Enter key
7.  restart vRO server with command "service vco-server restart"

Below are the screenshots would be helpful to create workflow.

 

----------------------------------------Start code of script----------------------------------

var line,tempVM,i,vmname;
var myFile = new FileReader(Path);
myFile.open();
data = myFile.readAll();
var lines = new Array();
var fvms = new Array();
lines = data.split(/\r\n|\n\r/);
System.log(lines);
var vms = VcPlugin.getAllVirtualMachines();
for (line in lines )
{
 tempVM = lines[line];
 //System.log("2nd line " + tempVM);
 for (i in vms)
 {
  vmname = vms[i];
  //System.log(tempVM + " : " + vmname.name);
  if (tempVM.match(vmname.name))
  {
   System.log(tempVM + vms[i].name + " find");
   vm = vmname;
   //System.log(vm.snapshot);
   var allSnaps = new Array();
   var allParents;
   var snapshot = vm.snapshot;
   var rootSnapshotList = snapshot.rootSnapshotList;
   for each (var rsnap in rootSnapshotList)
   {
       getSnapshotsOfVM(rsnap);
   }
      for each (var snap in allSnaps)
   {
       System.log("VM Name: " + snap.vm.name);
          System.log("Snapshot Name: " + snap.name);
          System.log("Snapshot creation time: " + snap.createTime.toLocaleString());
          System.log("Snapshot is old, removing...");
    var task = snap.snapshot.removeSnapshot_Task(false,true);
    var actionResult = System.getModule("com.vmware.library.vc.basic").vim3WaitTaskEnd(task);
   }
  }
 }
}
function getSnapshotsOfVM(snapshotTree)
{
    allSnaps.push(snapshotTree);
    var childTrees = snapshotTree.childSnapshotList;
    if(childTrees != null) {
        for(var index in childTrees) {
            if(childTrees[index] != null) {
                getSnapshotsOfVM(childTrees[index]);
            }
        }
    }
}

 ----------------------------------------End code of script----------------------------------

Code has been modified by me as per mine requirement. It is taken from internet.

PowerCli script to delete snapshot of vms

Add-PSSnapin Vmware.Vimautomation.Core

# Enter your vCenter and credentials

$VIServer = "name of vCenter"

# Connect to vCenter using the variables above

Connect-VIServer -Server $VIServer

#Get Windows Guest machines from input file

$vmguests = Get-Content "C:\csv-files\input.txt"

#Write new config options to VMs

foreach ($vm in $vmguests){

       

        Get-Snapshot -vm $vm | Remove-Snapshot -Confirm:$false      

        

}

# Disconnect from vCenter

Disconnect-VIServer -Server $VIServer -Confirm:$false