- Recall vicfg-* commands related to listing storage configuration
- Recall vSphere 4 storage maximums
- Identify logs used to troubleshoot storage issues
- Describe the VMFS file system
Skills and Abilities
- Use vicfg-* and esxcli to troubleshoot multipathing and PSA‐related issues
- Use vicfg-module to troubleshoot VMkernel storage module configurations
- Use vicfg-* and esxcli to troubleshoot iSCSI related issues
- Troubleshoot NFS mounting and permission issues
- Use esxtop/resxtop and vscsiStats to identify storage performance issues
- Configure and troubleshoot VMFS datastores using vmkfstools
- Troubleshoot snapshot and resignaturing issues
- Product Documentation
- ESX Configuration Guide
- ESXi Configuration Guide
- vSphere Command‐Line Interface Installation and Scripting Guide
- vSphere Client
- vicfg-* , esxcli, resxtop/esxtop,vscsiStats, vmkfstools
There’s obviously a large overlap between diagnosing performance issues and tuning storage performance, so check section 3.1 in tandem with this objective.
Recall vicfg-* commands related to listing storage configuration
- esxcli corestorage | nmp | swiscsi
- showmount -e
- look for CONS/s – this indicates SCSI reservation conflicts and might indicate too many VMs in a LUN. This field isn’t displayed by default (press ‘f’ then ‘f’ again to add it)
Identify logs used to troubleshoot storage issues
- /var/log/vmKernel (ESX only)
- vCenter logs
- Use tail -f to watch /var/log/vmkernel to monitor a svMotion
Describe the VMFS file system
Be aware of VMFS alignment – Vaughn Stewart’s latest blogpost
Use vicfg-* and esxcli to troubleshoot multipathing and PSA‐related issues
Refer to section 1.3.
Use vicfg-module to troubleshoot VMkernel storage module configurations
Refer to section 9.1, advanced ESX builds.
Use vicfg-* and esxcli to troubleshoot iSCSI related issues
There are various ways to configure iSCSI at the command line;
- vicfg-iscsi – the most powerful CLI tool as it can configure pretty much anything – targets, host adapters, authentication, etc
- esxcli swiscsi – typically used to configure multipathing for iSCSI. See section 1.1 for details.
- esxcfg-swiscsi – can enable, disable, and query status of iSCSI but not much more.
- esxcfg-hwiscsi – can enable, disable, and query status of iSCSI but not much more.
Things to check;
- Are any LUNs masked? (see section 1.1)
- Which HBAs are available?
vicfg-iscsi –H -l
- Are the expected number of paths available?
- Are the targets correctly configured?
vicfg-iscsi –T –l vmhba33
- Are the LUNs correctly configured?
vicfg-iscsi –L –l vmhba33
- Is authentication correctly configured?
vicfg-iscsi –A –l vmhba33
Troubleshoot NFS mounting and permission issues
There’s a good video on this in the Trainsignal Troubleshooting Course, but in reality there’s not much to check inside VMware – permissions are handled on the storage array or NAS server.
- Use vmkping to ensure the vmKernel has connectivity to the NAS server
- ‘esxcfg-nas –l’ to show configured NFS datastores
- For ESX hosts you may be able to use the SC to diagnose permissions issues;
- Ensure the nfsClient service is enabled on the firewall (esxcfg-firewall –e nfsClient)
- Ensure the service console can see the storage target using ping (the target may be on an isolated network as per best practices)
- Use ‘showmount –e <IP of storage array>’ (showmount –e 192.168.215.33)
Use esxtop/resxtop and vscsiStats to identify storage performance issues
Refer to section 3.5.
Configure and troubleshoot VMFS datastores using vmkfstools
Refer to section 1.2.
Troubleshooting snapshots and resignaturing
vmware-cmd /vmfs/volumes/LocalRAID5/testVM/testVM.vmx hassnapshot
- Ruben Garcia’s excellent post on troubleshooting snapshots
- Monitoring snapshot deletion (VMwareKB1007566)
- Consolidating snapshots – a long but very good article – (VMwareKB1007849)
- Best practices for snapshots in a VMware environment (VMwareKB1025279)
For resignaturing refer to section 1.1.
- TA1394 – Advanced Storage Log Analysis (VMworld 2009 session)
- Eric Sloof’s Advanced Troubleshooting presentation at the Dutch VMUG
- VMware whitepaper on Troubleshooting Performance issues
- Trainsignal’s Troubleshooting for vSphere course
- A combined multivendor post about iSCSI
- Troubleshooting SCSI reservations – VMwareKB1002293, VMwareKB1005009
- Troubleshooting SCSI reservations
- Chad Sakacc’s blogpost about storage performance, and another about VMFS/NFS limits
- VMware article on troubleshooting storage performance
- A good presentation on troubleshooting storage by a VMware engineer