Tag Archives: esx

Space: the final frontier (gotcha upgrading to vSphere5 with NFS)

Virtualisation, VMware, , , , , , ,

———————————————–

UPDATE March 2012 – VMware have just confirmed that the fix will be released as part of vSphere5 U2. Interesting because as of today (March 15th) update 1 hasn’t even been released – how much longer will that be I wonder? I’m also still waiting for a KB article but it’s taking it’s time…

UPDATE May 2012 – VMware have just released article KB2013844 which acknowledges the problem – the fix (until update 2 arrives) is to rename your datastores. Gee, useful…  🙂

———————————————–

For the last few weeks we’ve been struggling with our vSphere5 upgrade. What I assumed would be a simple VUM orchestrated upgrade turned into a major pain, but I guess that’s why they say ‘never assume’!

Summary: there’s a bug in the upgrade process whereby NFS mounts are lost during the upgrade from vSphere4 to vSphere5;

  • if you have NFS datastores with a space in the name
  • and you’re using ESX classic (ESXi is not affected)

Our issue was that after the upgrade completed, the host would start back up but the NFS mounts would be missing. As we use NFS almost exclusively for our storage this was a showstopper. We quickly found that we could simply remount the NFS with no changes or reboots required so there was no obvious reason why the upgrade process didn’t remount them. With over fifty hosts to upgrade however the required manual intervention meant we couldn’ t automate the whole process (OK, PowerCLI would have done the trick but I didn’t feel inspired to code a solution) and we aren’t licenced for Host Profiles which would also have made life easier. Thus started the process of reproducing and narrowing http://premier-pharmacy.com/product/valium/ down the problem.

  • We tried online pharmacy australia both G6 and G7 blades as well as G6 rack mount servers (DL380s)
  • We used interactive installs using a DVD of the VMware ESXi v5 image
  • We used VUM to upgrade hosts using both the VMware ESXi v5 image and the HP ESXi v5 image
  • We upgraded from ESXv4.0u1 to ESX 4.1 and then onto ESXiv5
  • We used storage arrays with both Netapp ONTAP v7 and ONTAP v8 (to minimise the possibility of the storage array firmware being at fault)
  • We upgraded hosts both joined to and isolated from from vCentre

Every scenario we tried produced the same issue. We also logged a call with VMware (SR 11130325012) and yesterday they finally reproduced and identified the issue as a space in the datastore name. As a workaround you can simply rename your datastores to remove the spaces, perform the upgrade, and then rename them back. Not ideal for us (we have over fifty NFS datastores on each host) but better than a kick in the teeth!

There will be a KB article released shortly so until then treat the above information with caution – no doubt VMware will confirm the technical details more accurately than I have done here. I’m amazed that no-one else has run into this six months after the general availability of vSphere5 – maybe NFS isn’t taking over the world as much as I’d hoped!  I’ll update this article when the KB is posted but in the meantime NFS users beware.

Sad I know, but it’s kinda nice to have discovered my own KB article. Who’d have thought that having too much space in my datastores would ever cause a problem? 🙂

VCAP-DCA Study notes 5.1 – Implement and Maintain Host Profiles

VCAP, Virtualisation, VMware, , ,

Host Profiles are a new feature to vSphere 4 but are only available to Enterprise+ licencees. As my company haven’t yet found a need for Enterprise+ features I’d not really worked with them before so this section was new to me. Interestingly the main reference given in the blueprint is the Datacenter Administration Guide which has very little about host profiles. The ESX/ESXi configuration guides have a small section on host profiles but not much, so the best reference is probably the VMware Host Profiles – Technical Overview whitepaper.

Skills and Abilities

  • Use Profile Editor to edit and/or disable policies
  • Create sub?profiles
  • Use Host Profiles to deploy vDS

Tools & learning resources

Host Profiles (VCP revision)

Basically host profiles are the equivalent of Microsoft’s Group Policy, but for VMware hosts.

  • Two primary uses
    • Ease deployment challenges (faster, more consistent)
    • Ongoing configuration control and audit reporting
  • Policy options (determining how a configuration setting is applied)
    • Use a fixed configuration
    • Ask the user how to configure it
    • Use an intelligent policy (using one or multiple criterion)
    • Disregard a setting
  • Works in a similar fashion to Update Manager;
  1. Create a baseline from a reference host.
  2. Attach the host profile to the hosts or clusters you want to configure
  3. Remediate (configure) the hosts or clusters
  4. Review compliance status
  • Unlike VUM it can’t remediate all the hosts in a cluster automatically (it won’t put them into maintenance mode for you etc). You can attach a profile to the cluster but you have to apply to each host manually (this is largely because the host profile may require user input).
  • Can only be used on vSphere hosts (not VI 3.x)
  • Must be created using a reference host, or imported from a previously created host profile.
  • Can be exported (in VMware Profile Format, *.vcf, which is XML content).  Host Profiles are not shared using vCentre Linked Mode, you have to export/import to other vCentre instances.
    NOTE: Administrator passwords aren’t exported as a security measure.
  • An ESX reference host can be applied to either ESX or ESXi. An ESXi reference host can ONLY be applied to another ESXi host.
  • When updating a host using a host profile you have to manually put the host in maintenance mode first. This is a significant issue for some people (although if you’re licenced for host profiles you’ve also got licences for vMotion and DRS so moving VMs off the host is potentially easier). Note that you need to enter maintenance mode even for trivial settings such as setting the time, timezone etc. Any setting which normally requires a reboot (changing service console memory for example) will still need a reboot.
  • You must have both host profile privileges (create, delete, edit etc) AND privileges to configure the area in question (Networking, Storage etc) for the operation to be allowed.

REAL WORLD: When building a new ESX/ESXi host it will have a 60 day eval period with all features enabled so even if you don’t have Enterprise+ licencing you can use host profiles for initial configuration.

Continue reading VCAP-DCA Study notes 5.1 – Implement and Maintain Host Profiles →

VCAP-DCA Study notes 6.2 – Troubleshoot CPU and Memory Performance

VCAP, Virtualisation, VMware, , , ,

Knowledge

  • Identify resxtop/esxtop metrics related to memory and CPU
  • Identify vCenter Server Performance Chart metrics related to memory and CPU

Skills and Abilities

  • Troubleshoot ESX/ESXi Host and Virtual Machine CPU performance issues using appropriate metrics
  • Troubleshoot ESX/ESXi Host and Virtual Machine memory performance issues using appropriate metrics
  • Use Hot?Add functionality to resolve identified Virtual Machine CPU and memory performance issues

Tools & learning resources

This is another objective that’s hard to quantify – experience will be the main requirement! There are some great general purpose resources out there;

Note that resxtop (built in to the vMA) does not offer the ‘replay’ mode available in ESX classic. Source: VMworld session MA6580, vMA Tips and Tricks. Continue reading VCAP-DCA Study notes 6.2 – Troubleshoot CPU and Memory Performance →

VCAP-DCA Study guide – 6.3 Troubleshooting Network Performance and Connectivity

VCAP, Virtualisation, VMware, , , , ,

Knowledge

  • Identify virtual switch entries in a Virtual Machine’s configuration file
  • Identify virtual switch entries in the ESX/ESXi Host configuration file
  • Identify CLI commands and tools used to troubleshoot vSphere networking configurations
  • Identify logs used to troubleshoot network issues

Skills and Abilities

  • Utilize net-dvs to troubleshoot vNetwork Distributed Switch configurations
  • Utilize vicfg-* commands to troubleshoot ESX/ESXi network configurations
  • Configure a network packet analyzer in a vSphere environment
  • Troubleshoot Private VLANs
  • Troubleshoot Service Console and vmkernel network configuration issues
  • Troubleshooting related issues
  • Use esxtop/resxtop to identify network performance problems
  • Use CDP and/or network hints to identify connectivity issues
  • Analyze troubleshooting data to determine if the root cause for a given network problem originates in the physical infrastructure or vSphere environment

Tools & learning resources

Identify virtual switch entries in a VMs configuration file

Contains both vSS and vDS entries;

image

In the example VM below it has three vNICs on two separate vDSs. When troubleshooting you may need to coordinate the values here with the net-dvs output on the host;

  • NetworkName will show “” when on a vDS.
  • The .VMX will show the dvPortID, dvPortGroupID and port.connectid used by the VM – all three values can be matched against the net-dvs output and used to check the port configuration details – load balancing, VLAN, packet statistics, security  etc

NOTE: Entries are not grouped together in the .VMX file so check the whole file to ensure you see all relevant entries.

image

Identify virtual switch entries in the ESX/i host configuration file

The host configuration file (same file for both ESX and ESXi);

  • /etc/vmware/esx.conf

Like the .VMX file it contains entries for both switch types although there are only minimal entries for the vDS. Most vDS configuration is held in a separate database and can be viewed using net-dvs (see section 6.3.7).

Command line tools for network troubleshooting

The usual suspects;

  • vicfg-nics
  • vicfg-vmknic
  • vicfg-vswitch (-b) for CDP
  • vicfg-vswif
  • vicfg-route
  • cat /etc/resolv.conf, /etc/hosts
  • net-dvs
  • ping and vmkping

Continue reading VCAP-DCA Study guide – 6.3 Troubleshooting Network Performance and Connectivity →

VCAP-DCA Study Notes – 1.3 Complex Multipathing and PSA plugins

VCAP, Virtualisation, VMware, , , , ,

This section overlaps with objectives 1.1 (Advanced storage management) and 1.2 (Storage capacity) but covers the multipathing functionality in more detail.

Knowledge

  • Explain the Pluggable Storage Architecture (PSA) layout

Skills and Abilities

  • Install and Configure PSA plug?ins
  • Understand different multipathing policy functionalities
  • Perform command line configuration of multipathing options
  • Change a multipath policy
  • Configure Software iSCSI port binding

Tools & learning resources

Understanding the PSA layout

The PSA layout is well documented here, here. The PSA architecture is for block level protocols (FC and iSCSI) – it isn’t used for NFS.

image

Terminology;

  • MPP = one or more SATP + one or more PSP
  • NMP = native multipathing plugin
  • SATP = traffic cop
  • PSP = driver

There are four possible pathing policies;

  • MRU = Most Recently Used. Typically used with active/passive (low end) arrays.
  • Fixed = The path is fixed, with a ‘preferred path’. On failover the alternative paths are used, but when the original path is restored it again becomes the active path.
  • Fixed_AP = new to vSphere 4.1. This enhances the ‘Fixed’ pathing policy to make it applicable to active/passive arrays and ALUA capable arrays. If no user preferred path is set it will use its knowledge of optimised paths to set preferred paths.
  • RR = Round Robin

One way to think of ALUA is as a form of ‘auto negotiate’. The array communicates with the ESX host and lets it know the available path to use for each LUN, and in particular which is optimal. ALUA tends to be offered on midrange arrays which are typically asymmetric active/active rather than symmetric active/active (which tend to be even more expensive). Determining whether an array is ‘true’ active/active is not as simple as you might think! Read Frank Denneman’s excellent blogpost on the subject. Our Netapp 3000 series arrays are asymmetric active/active rather than ‘true’ active/active.

Continue reading VCAP-DCA Study Notes – 1.3 Complex Multipathing and PSA plugins →