WIF claims authentication System.ArgumentException

When creating a custom STS authentication provider using the Windows Identity Foundation SDK, the default website the Visual Studio template creates works fine by itself.  However, when integrating with SharePoint, this would error out with this exception when hitting the SharePoint http:///_trust:

Exception of type 'System.ArgumentException' was thrown. Parameter name: encodedValue. 

The trace stack would have SPClaimProviderManager.DecodeClaimFromFormsSuffix(...) as the last method in the call.  It is very sketchy as to what the problem is. 

The problem, as it turns out, is in the form authentication cookie name.  By default ASP.NET web has .ASPXAUTH as the cookie name defined.  For example:

   
     
   

When a SharePoint web application is configured to use FBA or custom authentication provider, it actually expects a forms auth cookie name of "FedAuth".  This can be easily seen using Fiddler. 

When the custom STS web application does a post to http:///_trust, the mismatch in the FBA cookie name resulted in that SharePoint can't find the authentication ticket.  So how to fix this?  the STS web (created by the Visual Sudio WIF SDK template) needs its authentication ticket for the default.aspx to process properly and then post to  to work.  So we can't just remove it, nor can we change its name to "FedAuth", as this would confuse SharePoint.   Changing .ASPXAUTH to some other name has no effect.  

The solution I found, is to expire the STS Web's authentication ticket in default.aspx.  This way, default.aspx processing is not affected, but it won't post the cookie to /_trust.  Add the line in the box into default.aspx.cs:

 

Once this is added, the post request in Fiddle changed from:

 

to:

 

This allows SharePoint's /_trust/ to create its own FBA ticket cookie and redirect to /_layouts/Authenticate.aspx:

And everything from there works fine and user can be logged in normally.