Monthly Archives: September 2012

Online virtualisation labs come of age

With the launch of the new vCloud Suite along with new VMware certification tracks there’s no shortage of technologies to learn so I’ve been building up my home lab in anticipation of some long hours burning the midnight oil. While doing this I’ve been mulling over a simple (I thought) question;

Why buy hardware to build home labs? Can’t we use ‘the cloud’ for our lab requirements?

I spent a while investigating the current marketplace and while some areas are well covered some are just getting started.

A typical IT ‘stack’

As an infrastructure guy I’m interested in the lower half of the IT stack, principally from the hypervisor downwards (I expect that some infrastructure professionals will need to focus on the top part of the stack in the future, but that’s a different post). There are a plenty of cloud services where you can quickly spin up traditional guest OS or application instances (any IaaS/PaaS/SaaS provider, for example Turnkey Linux do some great OSS stuff) but a more limited number that let you provision the lower half of the stack in a virtual lab;

  • At the network layer Cisco’s learning labs offer cloud labs tailored to the Cisco exams (primarily CCNA and CCNP) and are sold as bundles of time per certification track. In October last year Juniper launched the Junosphere Labs, an online environment that you can use for testing or training.
  • For storage EMC provide labs and this year their internal E-Lab is going virtual and a private cloud is in the works (thanks to vSpecialist Burak Uysal for the info). Scott Drummunds has a great post illustrating what these labs offer – it’s pretty impressive (and includes some VMware functionality). These labs let partners test and learn the EMC product portfolio by setting up ‘virtual’ storage arrays and is something that you’d probably struggle to do in most labs. Other storage vendors such as Netapp offer virtual storage appliances (or simulators) but you’ll need to use a separate IaaS service to run them – there’s no public cloud offering.
  • At the hypervisor layer (although more application and guest OS focused) there’s Microsoft’s Technet labs. These have been available for years and for free (are you listening VMware? :-)) and let you play with many of Microsoft’s applications, including Hyper-V, in a live, online lab (Vladan has a good article here, and you can try Windows 2012 labs too). At the latest TechEd2012 conference the labs were made available online for two months afterwards and they were also available at the recent Microsoft Management Summit. As Hyper-V can virtualise itself but can’t run nested VMs the labs are limited to looking at the Hyper-V configuration. I tried these labs and was very impressed – they’re free, easy and quick to use (even if they do require IE).
  • According to this post on Linked-In, HP are also looking at the option of publicly available virtual labs although I couldn’t find any information on what they’ll include.

While not strictly cloud labs (depending on your definition of a cloud service) you could rent space and/or infrastructure in someone else’s datacenter – recently I’ve seen companies start to specialize in offering prebuilt ‘lab’ environments which you can rent for training/testing purposes;

Many large companies will have their own lab facilities and some global companies might offer them internally via private clouds but until recently there were no public cloud services which let you experiment with the hypervisor layer. The well known blogger David Davis had similar thoughts last year and investigated cloud providers who provide ESXi as a VM and was unable to find any. There’s no technical reason why not – vSphere has been able to virtualise itself and run nested VMs for years and although performance might suffer that’s often a secondary concern for a lab environment. It’s also not officially supported but if it’s for training and test/dev rather than production is that a barrier?

Continue reading Online virtualisation labs come of age

Federated login failures – the LSA cache

While working recently on an ADFS federation solution I came across a Microsoft ‘feature’ which doesn’t seem to be well known and which caused me to deliver my project a week late. It often manifests itself via failed logins and affects many products which integrate with AD such as Sharepoint, Office365, OWA, and of course ADFS. This is very much one of those ‘document it here for future reference’ posts but hopefully it’ll help spread the word and maybe save someone else the pain I felt!

To describe how the ‘feature’ affects ADFS you need to understand the communication flow when a federation request is processed. The diagram below (from an MSDN article on using ADFS in Identity solutions) shows a user (the web browser) connecting to a service (the ASP.NET application although it could be almost any app) which uses ADFS federation to determine access;

Communication flow using federated WebSSO

Summarising the steps;

  • The user browses to the web application (step 1)
  • The web app redirects the user to ADFS (step 2,3)
  • ADFS attempts to authenticate the user, usually against Active Directory (step 4)
  • ADFS generates a token (representing the users authentication) which is passed back to the user who then presents it to the app and is given access (steps 5,6,7)

My problem was that while some users were being logged into the web application OK, some were failing and I couldn’t work out why. Diagnosing issues in federation can be tricky as by its nature it often involves multiple parties/companies. The web application company were saying their application worked fine, both redirecting users and processing the returned tokens. The users were entering their credentials and being authenticated against our internal Active Directory. ADFS logs showed that tokens were being generated and sent to the web app. Hmm.

Digging deeper I found that the AD username (the UPN to be precise) being passed into the token generation process within ADFS was occasionally incorrect. The user would type their username into the web form (and be authenticated) but when ADFS tried to generate claims for this user via an LDAP lookup it used an incorrect UPN and hence failed. It seemed as if the Windows authentication process was returning incorrect values to ADFS. This stumped me for a while – how can something as simple and mature as AD authentication go wrong?

Of course it’s not going wrong, its working as designed. It transpires there’s an LSA cache on domain member servers. On occasions where the AD values have changed recently (the default is to cache for 7 days) it can result in the original, rather than the updated, values being returned to the calling application by the AD authentication process. A simple change such as someone getting married and having their AD account updated with their married name could therefore break any dependant applications. Details of this cache can be found in MS KB article 946358, along with the priceless statement “This behaviour may prevent the application from working correctly“. No kidding! This impacted my project more than most because the AD accounts are created programmatically via a web portal and updated later by some scripts. The high rate of change means they’re more susceptible to having old values cached.

This might seem like a niche problem but it also impacts implementations of Sharepoint, OWA, Project server, and Office365 – any product that relies on AD for authentication. These products can be integrated with AD to facilitate single sign on but if you make frequent changes to AD the issues above can occur.

How can I diagnose this issue?

The symptoms will vary between products but thankfully Microsoft have some great documentation on ADFS. The troubleshooting guide details how to enable the advanced ADFS logs via Event Viewer- when you’ve got those check for Event ID 139. The event details shows the actual contents of the authentication token so you can check the UPN and ensure it’s what you expect. If not follow the instructions in the KB article to disable or fine tune the cache retention period on the domain member server (ie the ADFS server, not the AD server).

Further Reading

Understanding the LSA lookup cache

Home labs – the Synology 1512+

I’ve been running a home lab for a few years now and recently I decided it needed a bit of an upgrade. I’ve been looking at the growing trend towards online lab environments but for the time being I made the decision that it’s still cost effective to maintain my own. I need to learn the latest VMware technologies (which requires lab time) and partly because the geek in me wants some new toys. 🙂

Storage was the first thing I needed to address. While I’ve got an Iomega IX2-200 (the two disk version) it’s not really usable as shared storage for a lab due to slow performance (about 17MB/s for read, 13MB/s for writes). If I were a patient man that would be fine for testing but I found myself putting VMs on local disks so I could work quicker which rather defeats the purpose of a lab for HA/DRS etc. I’ve built a home NexentaStor CE server which is feature rich (ZFS, snapshots, dedupe, tiered SSD caching) but I’ve found the configuration and maintenance less than simple and it’s a big, heavy old server (circa 2007) which won’t last much longer. My wishlist included the following;

  • Easy to use – I want to spend my time using it, not configuring and supporting it
  • Small form factor, minimised power consumption
  • Hypervisor friendly – I’d like to play with VMware, Citrix, and Microsoft’s Hyper-V
  • Cloud backup options. I use Dropbox, SugarSync and others and it’d be useful to have built in replication ability.
  • Hook up a USB printer
  • Flexibility to run other tasks – bit torrent, audio/movie streaming, webcams for security etc (which my Iomega also offers)
  • VLAN and aggregated NIC support (both supported by my lab switch, a Cisco SLM2008)
  • Tiered storage/caching (NOT provided by the consumer Synology devices)

My requirements are by no means unique and there were three devices on my shortlist;

I choose Synology for a couple of reasons, primarily because I’ve heard lots of good things about the company from other bloggers (Jason Nash comes to mind) and Synology have a wide range of devices to choose from at different price/performance points. They’re not the cheapest but many people say the software is the best around and having been bitten once with the IX2-200 I figured I’d go upmarket this time. The model I choose was the relatively new DiskStation 1512+, a five bay unit which satisfies most of my requirements with the exception of tiered storage. I was excited when I first read a while ago that some of the Synology units fully support VAAI but not so this particular model according to Synology (the DS412+ has only limited support). I guess it’s always possible that support will find its way into lower end models such as the 1512+ (even if unsupported) at a future date – here’s hoping!

UPDATE Sept 14th 2012 – While both NFS and iSCSI work with vSphere5.0 the 1512+ is only certified by VMware for iSCSI on vSphere 4.1 as of 14th Sept 2012. Previous devices (the 1511+ for example) are listed for both NFS and iSCSI, also with vSphere 4.1. Rather than being incompatible it’s more likely that they just haven’t been tested yet although there are problems with both NFS and iSCSI when using vSphere5.1 and DSM 4.1.

UPDATE Oct 3rd 2012 – Synology have released an update for their DSM software which fixes the compatibility issues with vSphere 5.1 although it’s referred to as ‘improved performance’ in the release notes. I’ve not tested this yet but hopefully it’s all systems go. Good work Synology!

There are some additional features I wasn’t looking for but which will come in useful for a home lab;

  • Syslog server (especially useful with ESXi nowadays)
  • DHCP server
  • CloudStation – ‘Dropbox’ style functionality

Having chosen the unit I then needed to choose the drives to populate it with as the unit doesn’t ship with any. My lab already includes some older disks which I could have reused plus I had two SSDs in the NexentaStor server which I considered cannibalising. After reading this excellent blogpost about choosing disks for NAS devices (and consulting the Synology compatibility list) I went with five WD Red 2TB HDDs as a compromise between space, performance, compatibility, and cost. I missed the introduction of the ‘Red’ range of hard disks that’s targeted at NAS devices and running 24×7 but they get good reviews. This decision means I can keep all three storage devices (Iomega IX2, Nexenta and Synology) online and mess around with advanced features like StorageDRS.

UPDATE Feb 18th 2013 – Tom’s hardware had a look at these WD Red drives and they don’t seem great at high IOps. I’ve not done much benchmarking but maybe worth investigating other options if performance is key.

I bought my Synology from UK based ServersPlus who offered me a great price and free next day shipping too. I was already on their mailing list having come across them on Simon Seagrave’s Techhead.co.uk site – they offer a variety of bundles specifically aimed at VMware home labs (in particular the ML110 G7 bundles are on my wish list and they do a cheaper HP Microserver bundle too) and are worth checking out.

Using the Synology 1512+

Following the setup guide was trivial and I had the NAS up and running on the network in under ten minutes. I formatted my disks using the default Synology Hybrid RAID which offers more flexibility for adding disks and mixing disk types and only has a minimal performance impact. Recent DSM software (v4.0 onwards) has been improved so that the initial format is quick and the longer sector check (which takes many hours) is done in the background, allowing you to start using it much faster.. My first impression was seeing the management software, DSM, which is fantastic! I’m not going to repeat what others have already covered so if you want to know more about the unit and how it performs here’s a great, indepth review.

I enabled the syslog server and was quickly able to get my ESXi hosts logging to it. Time Machine for my MBP took another minute to configure and I’m looking forward to experimenting with CloudStation which offers ‘Dropbox like functionality’ on the Synology.

Chris Wahl’s done some investigation into iSCSI vs NFS performance (although on the Synology DS411 rather than the 1512+) and I found similar results – throughput via iSCSI was roughly half that of NFS. I wondered if I had to enable multiple iSCSI sessions as per this article but doing so didn’t make any difference. All tests were over GB NICs and the Synology has both NICs bonded (2GB LACP);

  • Copying files from my MBP (mixed sizes, 300GB) to the Synology – 50MB/s write
  • Creating a file (using dd in a VM, CentOS 5.4) via an NFS datastore – 40MB/s write
  • Creating a file (using dd in a VM, CentOS 5.4) via an iSCSI datastore – 20MB/s write
  • Creating a thick eager zeroed VMDK on an iSCSI datastore – 75MB/s write

Given Synology’s published figures which claim a possible write speed of 194MB/s these were rather disappointing but they’re initial impressions NOT scientific tests (I also tried a similar methodology to Chris using IO Analyser which also gave me some odd results – average latency over 300ms!) so I’ll update this post once I’ve ironed out the gremlins in my lab.

Tip: make sure you disable the default ‘HDD hibernation’ under the Power settings otherwise you’ll find your lab becoming unresponsive when left for periods of time. VMs don’t like their storage to disappear just because they haven’t used it in a while!

LAST MINUTE UPDATE! Just before I published this post the latest release of DSM, v4.1, was finally made available. DSM 4.1 brings several enhancements and having applied it I can attest that it’s an improvement over an already impressive software suite. Of particular interest to home labs will be the addition of an NTP server, a much improved Resource Monitor which includes IOPS, and an improved mail relay.

Overall I’m really impressed with the Synology unit. It’s been running smoothly for a couple of weeks and the software is definitely a strong point. It’s got a great set of features, good performance, is scalable and might even include VAAI support in the future.

Further Reading

A performance comparison of NAS devices (fantastic site)

Indepth review of the Synology 1512+ (SmallNetBuilder.com)