WIF claims authentication System.ArgumentException

When creating a custom STS authentication provider using the Windows Identity Foundation SDK, the default website the Visual Studio template creates works fine by itself.  However, when integrating with SharePoint, this would error out with this exception when hitting the SharePoint http:///_trust:

Exception of type 'System.ArgumentException' was thrown. Parameter name: encodedValue. 

The trace stack would have SPClaimProviderManager.DecodeClaimFromFormsSuffix(...) as the last method in the call.  It is very sketchy as to what the problem is. 

The problem, as it turns out, is in the form authentication cookie name.  By default ASP.NET web has .ASPXAUTH as the cookie name defined.  For example:

   
     
   

When a SharePoint web application is configured to use FBA or custom authentication provider, it actually expects a forms auth cookie name of "FedAuth".  This can be easily seen using Fiddler. 

When the custom STS web application does a post to http:///_trust, the mismatch in the FBA cookie name resulted in that SharePoint can't find the authentication ticket.  So how to fix this?  the STS web (created by the Visual Sudio WIF SDK template) needs its authentication ticket for the default.aspx to process properly and then post to  to work.  So we can't just remove it, nor can we change its name to "FedAuth", as this would confuse SharePoint.   Changing .ASPXAUTH to some other name has no effect.  

The solution I found, is to expire the STS Web's authentication ticket in default.aspx.  This way, default.aspx processing is not affected, but it won't post the cookie to /_trust.  Add the line in the box into default.aspx.cs:

 

Once this is added, the post request in Fiddle changed from:

 

to:

 

This allows SharePoint's /_trust/ to create its own FBA ticket cookie and redirect to /_layouts/Authenticate.aspx:

And everything from there works fine and user can be logged in normally.

WIF, Claim based authentication, Keyset does not exist

There are many discussions on this often encountered exception when developing STS authentication using Windows Identity Framework in either a custom web application or SharePoint custom authentication provider. 

Exception of type 'System.Security.CryptographicException' was thrown. Keyset does not exist.

As many posters have pointed out, it's because reading the primate key to descrypt the STS message failed.  There are several reason that the private key cannot be read:

• The private is not available in the right place.  For a web application that runs under a service account, it typically reads from the machine's certificate store (Local, My).  If the certificate with the private key is not in the machine store, the application process won't find it.  Typically application can be configured to use the service account's certificate store too.  If that's the case, you must login as the service account, and make sure the certificate is in the Current User store of the service account.
• The private key itself has problems.  Most often in development, it is because the self-generated certificate is not trusted on the server itself.
• The private key is good and in the right place, but the process identity doesn't have read access to it.  Now this is the most common and most difficult cause.  The private keys of the certificates in the Macine Store are stored in the folder C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys on Windows 2008 and Windows 7.  What is confusing is that it's not the whole folder that the process identity needs access to (An admin user can't set permissions on this folder by default anyway), it's the specific private key file that the permission should be granted upon.   Identifying the correct private key file proves to be a tricky thing.  If the server has few certificate related activities, you can guess that the file with the most current date is the one you need.  Otherwise, we need a tool called FindPrivateKey, which is in the WCF Examples download provided by Microsoft: http://msdn.microsoft.com/en-us/library/aa717039.aspx  Then grant at least read permission on the file to the process identity.  The error should go away.