Like many I’m hoping to take the new advanced VMware certification VCAP-DCA later this year. This is my first post in my VCAP-DCA study notes series and covers section 8.3, Administer vSphere with the vMA. The notes are mainly intended as a revision list rather than a tutorial so you should have some knowledge of the topic before starting (there are links at the bottom to help you get started if not). Most of this content has been covered elsewhere previously so kudos to those who spent time discovering it – I’ve simply collected it all together for reference.
I’ve covered the topics on the VCAP-DCA blueprint and included some detail on the new vSphere 4.1 features as you never know when they’ll get included in the exam. The only exceptions are esxcli and vmkfstools which I’ll cover as part of section 1 (managing storage) and section 6 (troubleshooting).
- syslog server (vilogger component)
- centralised scripting repository
- replacement for ESX service console (scripts and third party plugins)
- easier to port service console scripts rather than converting to PowerCLI
- scripts may need amending (new authentication methods etc)
- facilitates move to ESXi
Prerequisites, installation and updating
- ESX host must support 64-bit VMs (Intel EM64T and vT technology)
- ESX 3.5U2 onwards, vCentre 4.0 onwards (2.5 NOT supported)
- 512MB, 1vCPU, 5GB+ disk space
Two deployment methods (download from VMware’s site);
- Import OVF directly from VMware (File -> Import in vCenter)
- Download ZIP file, extract, then Import from local OVF
During install you’ll need to provide network details (DHCP or static) as per any appliance along with a password for the vi-admin account.
NOTE: If you’re installing the appliance online (via .OVF) and you’re behind a firewall the deployment will fail. You need to use a command line utility OVFTOOL (separate download) to specify a proxy;
ovftool –proxy=user:firstname.lastname@example.org http://external-site.com/ovf/package.ovf
- x64 Red-Hat compatible Linux distribution (CentOS 5.3 in latest 4.1 release)
- vSphere CLI + vi-fastpass component (not supplied with standard vCLI installation)
- Syslog component (vilogger)
- SMTP server (for monitoring vMA itself), vSphere SDK for Perl, Java JRE 1.5
- Sample scripts in /usr/bin/
Updating vMA (use VIMA-UPDATE)
- Find available updates;
- Show detailed information about updates (inc packages updated etc);
- sudo vma-update update OR sudo vma-update –b <bulletin>
- NOTE: If you’re behind a proxy you’ll need to configure the /etc/vmware/esxupdate/vimaupdate.conf file to include proxy details. This only allows you to specify an http:// proxy and port – you can’t enter credentials.
- NOTE: You can also update vMA from a local repository (all versions) or an offline bundle (4.1 onwards). See this VMware thread for details.
- NOTE: You can’t upgrade from vMA 1.0 (also known as VIMA) to vMA 4.0. You can check the installed vMA version via the appliance status screen in the VI client (only available when deployed via http (not local) OVF, not updated when using vima-update). You can also check version info via the main config file (see next section for details), or via the splash screen on the appliances console session, or by using ‘cat /etc/vma-release’.
- vma-help – useful information on vMA and vCLI commands
- vima-update (scan | info | update)
- vifp (addserver | listservers | removeserver | rotatepasswords | recoverserver)
- vifptarget (set | clear | display) (vifpinit in v4.0)
- vilogger (enable | disable | list | updatepolicy)
- domainjoin-cli (vSphere 4.1 onwards)
Configure NTP – use chkconfig to enable the service, edit /etc/ntp.conf and /etc/ntp/step-tickers to set servers to use
Configure the timezone and keyboard settings (timestamps are important when used as syslog) – VMware KB1007551. Use UTC (ESXi uses UTC).
|/etc/vma-release||vMA version information|
|/etc/vmware/viconfig/vilogdefaults.xml||Main config file vSphere v4.0|
|/etc/vmware/vMA/vMA.conf||Main config file vSphere v4.1|
|/etc/vmware/esxupdate/vimaupdate.conf||vima-update config file|
|/var/log/vmware/<hostname>/||Location of logfiles collected by vilogger|
- Always use FQDN to specify servers
- It’s perfectly possible to have multiple vMA appliances and for the same servers to be registered via vifp on them all. Likewise you can have multiple vMA’s all collecting logs (via vilogger) from the same hosts.
- VMware recommend one vMA host per 100 ESX servers when using the syslog functionality.
Authentication using Fastpass
This allows unattended authentication for ESX hosts and vCentre. You can then to run tasks against them without further authentication (cron jobs, scripts etc). It’s a two step process;
- Setup the servers you want to authenticate with using VIFP
- Enable the connections using VIFPINIT (for v4.0) or VIFPTARGET (for v4.1)
NOTE: the vifp connections (step 1) are persistent across reboots, but step 2 will need repeating after every logout/reboot.
Step 1 – Enable authentication for the servers
NOTE: When you add a new ESX/i host using Fastpass two users are created locally on the host. These users have no shell defined in /etc/passwd so can’t be used interactively on the host;
- vi-admin (administrator access)
- vi-user (read only)
Step 2 – set target servers
- Security issues
- BAD – Passwords are stored in the credential store in an obfuscated manner (XOR, NOT encrypted. This can be cracked relatively easily. Worse, they’ve available in clear text! Full details here.
- GOOD – The internal password used within the Fastpass system is rotated every week by default.
- GOOD – You can encrypt the filesystem to protect stored credentials. See VMware KB1017669 (vSphere 4.1 onwards only)
- ‘vifp recoverserver’ can be used to fix a corrupted credential store (this can occur when vifp rotatepasswords is interrupted by network issues among other causes). See Troubleshooting section.
Authentication using AD (vMA v4.1 onwards)
You can join vMA to a domain using domainjoin-cli. Once you’ve joined the vMA you can add hosts using the same process as Fastpass but using ‘–authpolicy ad’ parameter.
NOTE: You must reboot vMA after joining the domain.
If you want to execute commands against ESX hosts (either directly or via Virtual Centre using the –vihost parameter) the hosts may also need to be joined to the domain for AD passthru authentication to work (see this VMware communities thread for more). This seems to be a vague area at the moment with quite a few posters experiencing issues.
NOTE: This will produce the errors shown above which can be safely ignored so long as the final SUCCESS appears.
- Once the vMA appliance is joined to a domain you can login to the vMA appliance using your domain credentials (domain\user) instead of vi-admin.
- NOTE: If you join the vMA appliance or ESX hosts to your Microsoft domain they need to be licenced according to the usual Microsoft policy.
- To read more about AD authentication use the links at the bottom of this post.
Using VILogger (syslog server)
Section 6.1 deals with using a syslog server in a virtual infrastructure in more depth, but when using the built in ‘vilogger’ command setting up the vMA as a syslog server is a two step process;
- Enable authentication for the hosts you want to collect logs from using Fastpass.
- Configure and enable logging (all servers, per server or per logfile)
- Turn on logging for all vifp enabled hosts;
- Enable logging for a specific host;
- Display the logging settings;
- Turn off logging for a specific host;
- Update logging configuration;
- Timestamps use UTC on ESXi
- With ESXi prior to 4.1 the vpxa.log files were not sent by default to a syslog server. See VMware KB1017658 for details.
- You can restart the vilogger daemon using ‘service vmware-vilogd restart‘
- To constantly monitor a log you can use ‘tail -f /var/log/vmware/<FQDN of Host>/vpxa.log‘
Managing VMs with vmware-cmd
This command can be used to manage virtual machines, as opposed to most of the vicfg-* tools which work on hosts. Typical uses are for registering VMs to vCenter, taking/removing snapshots and forceably shutting down a failed VM. These operations can be performed more easily using PowerCLI, but it all depends on your requirements – vmware-cmd can be used during a scripted install for instance whereas PowerCLI is unavailable.
The main reference document is VMware’s vSphere Command-Line Interface Installation and Reference Guide (section 8).
Server Operations: vmware-cmd -l vmware-cmd -s register <config_file_path> <datacenter> <resource pool> vmware-cmd -s unregister <config_file_path>
VM Operations: vmware-cmd <cfg> getstate vmware-cmd <cfg> start <powerop_mode> vmware-cmd <cfg> stop <powerop_mode> vmware-cmd <cfg> reset <powerop_mode> vmware-cmd <cfg> suspend <powerop_mode> vmware-cmd <cfg> setguestinfo <variable> <value> vmware-cmd <cfg> getguestinfo <variable> vmware-cmd <cfg> getproductinfo <prodinfo> vmware-cmd <cfg> connectdevice <device_name> vmware-cmd <cfg> disconnectdevice <device_name> vmware-cmd <cfg> getconfigfile vmware-cmd <cfg> getuptime vmware-cmd <cfg> answer vmware-cmd <cfg> gettoolslastactive vmware-cmd <cfg> hassnapshot vmware-cmd <cfg> createsnapshot <name> <description> <quiesce> <memory> vmware-cmd <cfg> revertsnapshot vmware-cmd <cfg> removesnapshots
- edit /etc/resolv.conf, restart networking.
- Edit /etc/sysconfig/network to include domain
Resolving credential store corruption – VMware KB1010178
Whenever the Fastpass authentication is performed an event is logged in vCentre. If the Fastpass authentication is incorrectly configured (stored credentials get out of sync for example) it will try to authentication continuously, generating up to five failure events per second. This can quickly fill the vCentre database as Event data is retained for 180 days by default. This can be changed via the Administration -> vCentre Server Settings -> Database Retention Settings in vCentre
When decommissioning a vMA appliance you should clear down the VIFP server list. This will remove the local vi-admin and vi-user accounts on the ESX/i hosts.
- vMA Homepage at VMware.com
- vFail.net’s VCAP Study guide for vMA, section 8.3
- Bridge the ESX/ESXi Management Gap with VMware vMA (MA6580, VMworld 2010, login required)
- Using AD authentication with vMA (Geeksilver’s blog)
- Using AD authentication with vMA (VirtualGhetto blog)
- Why you should upgrade vMA to v4.1 (VirtualGhetto blog)
- vMA configuration examples