Tag Archives: federation

Federated login failures – the LSA cache

While working recently on an ADFS federation solution I came across a Microsoft ‘feature’ which doesn’t seem to be well known and which caused me to deliver my project a week late. It often manifests itself via failed logins and affects many products which integrate with AD such as Sharepoint, Office365, OWA, and of course ADFS. This is very much one of those ‘document it here for future reference’ posts but hopefully it’ll help spread the word and maybe save someone else the pain I felt!

To describe how the ‘feature’ affects ADFS you need to understand the communication flow when a federation request is processed. The diagram below (from an MSDN article on using ADFS in Identity solutions) shows a user (the web browser) connecting to a service (the ASP.NET application although it could be almost any app) which uses ADFS federation to determine access;

Communication flow using federated WebSSO

Summarising the steps;

  • The user browses to the web application (step 1)
  • The web app redirects the user to ADFS (step 2,3)
  • ADFS attempts to authenticate the user, usually against Active Directory (step 4)
  • ADFS generates a token (representing the users authentication) which is passed back to the user who then presents it to the app and is given access (steps 5,6,7)

My problem was that while some users were being logged into the web application OK, some were failing and I couldn’t work out why. Diagnosing issues in federation can be tricky as by its nature it often involves multiple parties/companies. The web application company were saying their application worked fine, both redirecting users and processing the returned tokens. The users were entering their credentials and being authenticated against our internal Active Directory. ADFS logs showed that tokens were being generated and sent to the web app. Hmm.

Digging deeper I found that the AD username (the UPN to be precise) being passed into the token generation process within ADFS was occasionally incorrect. The user would type their username into the web form (and be authenticated) but when ADFS tried to generate claims for this user via an LDAP lookup it used an incorrect UPN and hence failed. It seemed as if the Windows authentication process was returning incorrect values to ADFS. This stumped me for a while – how can something as simple and mature as AD authentication go wrong?

Of course it’s not going wrong, its working as designed. It transpires there’s an LSA cache on domain member servers. On occasions where the AD values have changed recently (the default is to cache for 7 days) it can result in the original, rather than the updated, values being returned to the calling application by the AD authentication process. A simple change such as someone getting married and having their AD account updated with their married name could therefore break any dependant applications. Details of this cache can be found in MS KB article 946358, along with the priceless statement “This behaviour may prevent the application from working correctly“. No kidding! This impacted my project more than most because the AD accounts are created programmatically via a web portal and updated later by some scripts. The high rate of change means they’re more susceptible to having old values cached.

This might seem like a niche problem but it also impacts implementations of Sharepoint, OWA, Project server, and Office365 – any product that relies on AD for authentication. These products can be integrated with AD to facilitate single sign on but if you make frequent changes to AD the issues above can occur.

How can I diagnose this issue?

The symptoms will vary between products but thankfully Microsoft have some great documentation on ADFS. The troubleshooting guide details how to enable the advanced ADFS logs via Event Viewer- when you’ve got those check for Event ID 139. The event details shows the actual contents of the authentication token so you can check the UPN and ensure it’s what you expect. If not follow the instructions in the KB article to disable or fine tune the cache retention period on the domain member server (ie the ADFS server, not the AD server).

Further Reading

Understanding the LSA lookup cache

Federated identity and Horizon Application Manager

A recent project at work has required me to implement Microsoft’s Active Directory Federation Services (ADFS) which has been an interesting change from my usual technologies. It’s a mature product (it was released with Windows 2003 and further refined in Windows 2008) designed to allow you to ‘federate’ your Active Directory – in other words to allow third parties to leverage your internal AD in a secure manner. At first I thought this project was a distraction from the skillset I’m working towards (IaaS infrastructure with vCloud Director, View etc) but I’ve since come to realize that federated identity is an essential ingredient in the cloud recipe and one which needs to be understood.

Let me give you an example. My company decided to upgrade an aging training application and for various reasons we outsourced the solution to a third party developer. The idea was that they’d develop (and host) all the training materials and offer it as a service over the web to our customers (SaaS) thus requiring no resource from our internal teams. The only hitch in the plan was the business requirement that the customer, who already has login details for our web portal, should use the same credentials for this remotely hosted training solution. Those credentials are held in an internal AD database so we used ADFS to ‘publish’ them to the third party. Voila! The end users can now login to their training solution unaware that the credentials they enter are authenticated against my AD in the background. The resulting (much simplified!) architecture is shown in the following diagram;

It’s important to realise that there are two distinct actions going on during a login;

  1. Authentication – The AD acts as the identify provider (IdP), making sure the user is who they say they are.
  2. Authorisation – Once authenticated ADFS generates a ‘claim’ which it sends to the third party and this dictates what actions the user can take in the application.

Of course my example is very simplistic but the principle of allowing identities to be shared securely across security boundaries (such as disparate networks and applications) is critical to cloud services. Security is one of the big challenges in the cloud and federation allows you to keep your crown jewels (your user details) secure while still consuming remote services. It’s also important as the number of mobile devices used to access services increases.

Consumer or provider?

The ADFS example above is just one of the many possible scenarios that federated identity must handle. In every federation scenario there is an identity provider (IdP) and a consumer or service provider (SP, sometimes referred to as a relying party). In the the example above my company are the identity provider (our AD holds the identity details) and the consumer is the third party developer who provides a service.

The first choice therefore is whether you’re just going to consume other people’s federated identity services and/or act as an identity provider yourself.

You’ve probably been a consumer of federated identity for a while without even realising it. Everytime you sign into a website using your Twitter or Facebook login (for example) you’re consuming the federated identity service offered by Twitter and Facebook, likewise when you post a comment on a blog which requires a WordPress login. Maybe you’ve logged into a variety of Microsoft services using your Windows Live ID? Same thing.

One of the early commercial attempts at Federated Identity was Microsoft’s Passport which set out to be a universal authentication mechanism for web commerce but security concerns limited it’s adoption and resulted in a proliferation of alternative services (Windows Live ID, Google ID, and Apple ID to name a few well known ones). Here’s a few of the most popular federation protocols in use today;

  • Open-ID (which was formed by Facebook, Google, IBM, Microsoft, PayPal, VeriSign and Yahoo)
  • OAuth (similar to OpenID but used for API delegation, used by Twitter, Salesforce, Google, Facebook etc)
  • SAML (the most widely used federation protocol used by ADFS, Horizon App Mgr, Centrix Workspace and others)
  • WS-Federation (part of the larger WS-* standards)
  • SCIM (the newest and still evolving standard – v1 was ratified in Dec 2011)

So consuming is commonplace but why would you want to become an identity provider and federate your identity out to the world?

You might be reading this article and thinking ‘I don’t offer a service to people on the internet so I’ve no need to provide identity federation’. If all your infrastructure needs are met internally that might be true. What if you want to use public or hybrid cloud? If you want your corporate users to securely use their company login to access SaaS providers like SalesForce.com or Google Apps you’ll need to become an identity provider.

If you’re an internet giant like Google, Microsoft, or Apple you can develop your own identity framework but for everyone else there are frameworks you can quickly ‘bolt on’ to your existing infrastructure which allow you to offer federated services;

The purpose of the above applications varies even though they all provide identity federation. Most include SSO functionality but some are cloud based and others are installed locally (some are deployed via appliances). Some provide ‘application store’ or portal/workspace features which are much like the Citrix access you’re probably familiar with but for both internal and cloud applications.

I was already familiar with the Centrix solution after seeing one of the company founders, Lisa Hammond, give a very good presentation at the recent July 2012 London VMUG. The idea of a converged portal presenting SSO access to all your apps, wherever they reside, is compelling and Centrix has been doing this for quite a while prior to VMware’s entry into the market.

How is this relevant to me as a virtualisation admin?

You’ll have spotted the last entry above, VMware’s Horizon Application Manager. Horizon was released in May 2011 as the first component in the ‘Project Horizon’ vision first previewed at VMworld 2010. It was developed from VMware’s acquisition of TriCipher in August 2010, a company which previously developed a federated identity solution known as MyOneLogin. To quote VMware’s press release at the time;

VMware’s acquisition of TriCipher lets us integrate identity-based security and managed access to applications hosted in the cloud or on-premise. Convenient end-user access to applications on any device with security controls for IT lets customers extend their security and control into public cloud environments.

Earlier this year VMware published a research article on identity, access control, and Horizon which is a great introduction to Horizon and where it fits in the larger ecosystem. VMware would like you to implement Horizon to act as a centralised portal for all your applications whether they reside internally or externally via clouds. Another way to think of it is an ‘app store’ for the enterprise. Michael Letschin (@mletschin) has written a very clear roadmap for the encompassing Project Horizon vision which as a bonus also makes clear where Project Octopus fits in – a great read. For a practical understanding of federation and VDI check out Andre Leibovici’s great article discussing federation in relation to VMware’s View product although he advises that it’s still very much a work in progress.

The principles and terminology of federation (IdP, SP, tokens, relying parties, claim rules etc) are largely the same across all the products listed above so I’m glad that by learning ADFS I’ve actually learnt quite a bit about how Horizon works under the hood.

The bottomline is that if you’re going to use cloud services and you want to avoid a security management nightmare you need to understand federated identity. if you don’t understand your options early on you may find yourself putting in a solution which solves your short term requirements but not your long term goals, and that could lead to implementing multiple solutions – messy!!

This article barely touches the surface of a very complex subject – ADFS is fine if you’re using Microsoft as your on premises identity provider, but what if you use another user directory from Oracle or IBM? What about two factor authentication? What if a third party uses open source Shibboleth and you use ADFS – do they work together? Can you chain authentication systems together and introduce conditional processing? What about multi-tenant clouds and the special challenges they present? Federated authentication is one small part of a wider subject commonly referred to as Identity Management (IDM). I did enough to get our implementation working but it was immediately obvious that it’s a specialised skillset every bit as complex as virtualisation with multiple products, protocols, compatibilities, design choices and pitfalls. I also found it fascinating to see how the various disconnected services are beginning to be ‘hooked’ up to each other using these distributed mechanisms – there’s a long way to go but this is a growth area no doubt.

Further Reading

Here’s a great primer on using ADFS, explained in real world terms (although the concepts apply to any federation)

A good intro to ADFS, particularly when used with Office365

A good video intro to federated identity

Will OpenID transform the enterprise ecosystem?

Federated clouds – Possible?

SCIM: Standards are taking hold in the cloud

Good article on ADFS and the Azure version of AD

Looking at cloud futures – why federation is key to cloud evolution

The essential OAuth primer

Hosted application stores help IT pros wrangle cloud apps

Centrix gave a very good presentation at the July 2012 London VMUG