All posts by Ed Grigson

VCAP-DCA Study guide – 6.4 Troubleshooting Storage Performance and Connectivity

VCAP, Virtualisation, VMware, , , , ,

Knowledge

  • Recall vicfg-* commands related to listing storage configuration
  • Recall vSphere 4 storage maximums
  • Identify logs used to troubleshoot storage issues
  • Describe the VMFS file system

Skills and Abilities

  • Use vicfg-* and esxcli to troubleshoot multipathing and PSA?related issues
  • Use vicfg-module to troubleshoot VMkernel storage module configurations
  • Use vicfg-* and esxcli to troubleshoot iSCSI related issues
  • Troubleshoot NFS mounting and permission issues
  • Use esxtop/resxtop and vscsiStats to identify storage performance issues
  • Configure and troubleshoot VMFS datastores using vmkfstools
  • Troubleshoot snapshot and resignaturing issues

Tools

There’s obviously a large overlap between diagnosing performance issues and tuning storage performance, so check section 3.1 in tandem with this objective.

Recall vicfg-* commands related to listing storage configuration

  • vicfg-scsidevs
  • vmkiscsi-tool
  • vicfg-mpath
  • vicfg-iscsi
  • esxcli corestorage | nmp | swiscsi
  • vicfg-nas
  • showmount -e
  • esxtop/resxtop
    • look for CONS/s – this indicates SCSI reservation conflicts and might indicate too many VMs in a LUN. This field isn’t displayed by default (press ‘f’ then ‘f’ again to add it)
  • vscsiStats
  • vmkfstools
  • vicfg-module

Continue reading VCAP-DCA Study guide – 6.4 Troubleshooting Storage Performance and Connectivity →

VCAP-DCA Study notes 6.2 – Troubleshoot CPU and Memory Performance

VCAP, Virtualisation, VMware, , , ,

Knowledge

  • Identify resxtop/esxtop metrics related to memory and CPU
  • Identify vCenter Server Performance Chart metrics related to memory and CPU

Skills and Abilities

  • Troubleshoot ESX/ESXi Host and Virtual Machine CPU performance issues using appropriate metrics
  • Troubleshoot ESX/ESXi Host and Virtual Machine memory performance issues using appropriate metrics
  • Use Hot?Add functionality to resolve identified Virtual Machine CPU and memory performance issues

Tools & learning resources

This is another objective that’s hard to quantify – experience will be the main requirement! There are some great general purpose resources out there;

Note that resxtop (built in to the vMA) does not offer the ‘replay’ mode available in ESX classic. Source: VMworld session MA6580, vMA Tips and Tricks. Continue reading VCAP-DCA Study notes 6.2 – Troubleshoot CPU and Memory Performance →

VCAP-DCA Study guide – 6.3 Troubleshooting Network Performance and Connectivity

VCAP, Virtualisation, VMware, , , , ,

Knowledge

  • Identify virtual switch entries in a Virtual Machine’s configuration file
  • Identify virtual switch entries in the ESX/ESXi Host configuration file
  • Identify CLI commands and tools used to troubleshoot vSphere networking configurations
  • Identify logs used to troubleshoot network issues

Skills and Abilities

  • Utilize net-dvs to troubleshoot vNetwork Distributed Switch configurations
  • Utilize vicfg-* commands to troubleshoot ESX/ESXi network configurations
  • Configure a network packet analyzer in a vSphere environment
  • Troubleshoot Private VLANs
  • Troubleshoot Service Console and vmkernel network configuration issues
  • Troubleshooting related issues
  • Use esxtop/resxtop to identify network performance problems
  • Use CDP and/or network hints to identify connectivity issues
  • Analyze troubleshooting data to determine if the root cause for a given network problem originates in the physical infrastructure or vSphere environment

Tools & learning resources

Identify virtual switch entries in a VMs configuration file

Contains both vSS and vDS entries;

image

In the example VM below it has three vNICs on two separate vDSs. When troubleshooting you may need to coordinate the values here with the net-dvs output on the host;

  • NetworkName will show “” when on a vDS.
  • The .VMX will show the dvPortID, dvPortGroupID and port.connectid used by the VM – all three values can be matched against the net-dvs output and used to check the port configuration details – load balancing, VLAN, packet statistics, security  etc

NOTE: Entries are not grouped together in the .VMX file so check the whole file to ensure you see all relevant entries.

image

Identify virtual switch entries in the ESX/i host configuration file

The host configuration file (same file for both ESX and ESXi);

  • /etc/vmware/esx.conf

Like the .VMX file it contains entries for both switch types although there are only minimal entries for the vDS. Most vDS configuration is held in a separate database and can be viewed using net-dvs (see section 6.3.7).

Command line tools for network troubleshooting

The usual suspects;

  • vicfg-nics
  • vicfg-vmknic
  • vicfg-vswitch (-b) for CDP
  • vicfg-vswif
  • vicfg-route
  • cat /etc/resolv.conf, /etc/hosts
  • net-dvs
  • ping and vmkping

Continue reading VCAP-DCA Study guide – 6.3 Troubleshooting Network Performance and Connectivity →

VCAP-DCA Study guide – 6.5 Troubleshoot vCenter Server and ESX/ESXi Host Management

VCAP, Virtualisation, VMware, , ,

Knowledge

  • Identify CLI commands and tools used to troubleshoot management issues

Skills and Abilities

  • Troubleshoot vCenter Server service and database connection issues
  • Troubleshoot the ESX Service Console firewall
  • Troubleshoot ESX/ESXi server management and connectivity issues
  • Determine the root cause of vSphere management or connectivity issue

Tools

Identify CLI tools used to troubleshoot management issues

  • vicfg-vswitch
  • vicfg-vmknic
  • vicfg-vswif
  • vpxd.exe -s

There are a few more covered later in this objective for restarting management agents on ESX/i hosts. This VMware article on resolution paths is a great place to start learning about troubleshooting.

Troubleshoot vCenter Server service and database connection issues

  • Check the VMware vCenter service is started and the account it’s configured to run as. Check that account isn’t locked out.
  • Start vCentre using vpxd.exe;
    • ‘vpxd.exe –s’ to start it as an application rather than a service. This will show error messages in plain text rather than the cryptic service codes.
    • ‘vpxd.exe –p’ refreshes the password hash used to connect to the database. Used after replacing the default SSL certificates (VMwareKB1003070)
  • How to set SQL as a service dependency – blog post
  • With a lab setup and SQL Express the database often grows to the 4GB limit, at which point the vCenter service will fail. Follow VMwareKB1025914for details of how to cleardown data in the vCenter database.
  • Check the ODBC connectivity using the ‘Test’ button. Check the SQL security http://premier-pharmacy.com/product/eriacta/ logs to see failed authentication attempts.

image

VMwareKB1003979 gives a good overview of the previous processes.

Continue reading VCAP-DCA Study guide – 6.5 Troubleshoot vCenter Server and ESX/ESXi Host Management →

VCAP-DCA Study notes – 3.5 Utilize Advanced vSphere Performance Monitoring Tools

VCAP, Virtualisation, VMware, ,

This is one objective where you definitely have to get hands on – there’s no way you’ll learn esxtop otherwise. Ideally you’ll have a real infrastructure to play with as you want hosts with memory contention, ballooning, swapping, NUMA optimisations etc so you can play with and understand the features.

Knowledge

  • Identify hot keys and fields used with resxtop/esxtop
  • Identify fields used with vscsiStats

Skills and Abilities

  • Configure esxtop/resxtop custom profiles
  • Determine use cases for and apply esxtop/resxtop Interactive, Batch and Replay modes
  • Use vscsiStats to gather storage performance data
  • Use esxtop/resxtop to collect performance data
  • Given esxtop/resxtop output, identify relative performance data for capacity planning purposes

Tools & learning resources

Using resxtop

Two ways of invoking;

  • resxtop –server <esxi host>
  • resxtop –server <vCenter server> –vihost <esxi host>

Continue reading VCAP-DCA Study notes – 3.5 Utilize Advanced vSphere Performance Monitoring Tools →