F5 Big-IP Load Balanced WCF Services - Update

The post below was some findings from a project related to the authentication between the front and back end services, and F5 configurations.  It closely matches issues identified in my previous post for the SSO server but that was using ntlm/kerberos (I have a feeling that we didn't set up the SPNs for kerberos correctly).  Ultimately I bet that a similar configuration to that below but set up for one-shot-kerberos and using the SPN in the Identity column would have resolved this issue.

tldr;  

Server config:  

Configure the server to accept client certificates using PeerTrust - allows us to authorise the client 

Define the server certificate (the identity of the service - important later on) - allows message security to work 

<behavior name="certificate"> 

<serviceMetadata httpGetEnabled="true" httpsGetEnabled="true" /> 

<serviceDebug includeExceptionDetailInFaults="true" /> 

<serviceCredentials> 

<clientCertificate> 

<authentication certificateValidationMode="PeerTrust" /> 

</clientCertificate> 

<serviceCertificate findValue="<blah>" storeLocation="LocalMachine" x509FindType="FindBySubjectName" /> 

</serviceCredentials> 

</behavior> 

configure the wsHttpBindings to use Message security,  

items highlighted yellow - this is used for the message security and allows load balancing without sticky sessions 

items highlighted green - this is used to authenticate the client using a certificate 

<wsHttpBinding> 

<binding name="certauth" maxReceivedMessageSize="2147483647"> 

<readerQuotas maxStringContentLength="2147483647" maxArrayLength="2147483647" /> 

<security mode="Message"> 

<message algorithmSuite="TripleDesSha256" establishSecurityContext="false" negotiateServiceCredential="false" clientCredentialType="Certificate" /> 

</security> 

</binding> 

</wsHttpBinding> 

Configure the endpoint, basically accept request on <endpoint>/certauth and applies the appropriate message security and authentication 

<endpoint name="certauth" address="certauth" binding="wsHttpBinding" bindingConfiguration="certauth" contract="<contract>" /> 

Client Config: 

Configure the client to accept the server's security configuration using PeerTrust 

Configure the certificate that this client will pass as its credentials 

<endpointBehaviors> 

<behavior name="certificate"> 

<clientCredentials> 

<clientCertificate findValue="<name>" storeLocation="LocalMachine" x509FindType="FindBySubjectName" /> 

<serviceCertificate> 

<authentication certificateValidationMode="PeerTrust" /> 

</serviceCertificate> 

</clientCredentials> 

</behavior> 

</endpointBehaviors> 

Configure the security bindings, must match the service bindings 

<wsHttpBinding> 

<binding name="certauth" maxReceivedMessageSize="2147483647"> 

<readerQuotas maxStringContentLength="2147483647" maxArrayLength="2147483647" /> 

<security mode="Message"> 

<message algorithmSuite="TripleDesSha256" establishSecurityContext="false" negotiateServiceCredential="false" clientCredentialType="Certificate" /> 

</security> 

</binding> 

</wsHttpBinding> 

Configure the endpoint.  Note the /certauth suffix. 

the identity element is essential, it needs to match the serviceCredentials defined on the service.   

<endpoint address="<endpoint>/certauth" binding="wsHttpBinding" bindingConfiguration="certauth" contract="<contract>" behaviorConfiguration="certificate"> 

<identity> 

<certificateReference findValue="<name>" x509FindType="FindBySubjectName" storeLocation="LocalMachine" storeName="TrustedPeople"/> 

</identity> 

</endpoint> 

long version; 

To do this we need to use message security for the following reasons 

• We are using SSL offload, so transport security is not supported (works over SSL only) 
• We are using certificate auth, which is not allowed on TransportWithCredentialOnly 
• We are working across a domain boundary so we cannot use windows authentication 

The second complexity is that we are working in a load balanced environment.  This adds complexity as WCF negotiates a security context to generate the message security before sending the message.  If the Load balancer sends the message to a different server to the one the security context was generated on, then this will cause intermittent security failures. 

The two general solutions to this are to; 

• enable sticky sessions which the F5 then "guarantees" that the security context and message occur on the same server.   
• This did not seem to work - with sticky sessions enabled with the following config (), we still received this errors.   
• If the security context lasted longer than the sticky session timeout, then this could still cause problems, but I don't think this was the case. 
• I have a feeling this was caused by not setting the 'identity' correctly in the WCF config which was a key part of the final working solution, but it should still have worked regardless as both front end servers had success connecting to both back end servers. 
• set wcf establishSecurityContext="false" which basically sets up security for every message, rather than establishing a long term security context, each message would have its own security context applied. 
• this did not work.  I have encountered this before where this was the supposed fix and did not actually resolve the issue, but I thought that it was related to ntlm issues at the time. 
• Sticky sessions and this fix should pretty much have guaranteed the fix as you would never have the sticky session timeout before the security context expired (because it is only ever generated right before the message is sent). 
• This did not work.  I have no explanation for this. 

A final solution that I had previously read which allows a configuration to work without sticky sessions is to set negotiateServiceCredentials="false", however all examples point this to enable "kerberos-one-shot" which allows message security using kerberos if the services are configured with domain users and SPNs.  As we work across domains this wasn't acceptable. 

I was finally able to put a few things together and determined that the use of "establishSecurityContext" along with "negotiateServiceCredentials" should be possible without kerberos if you use certificate security (separate from certificate authentication).  This had partially been done with the  "servicebehaviour" element which defined the serviceCredentials on the server using certificates, and peerTrust for serviceCertificate on the client.  However the next step was to define the serviceIdentity on the client config so that the client knew which certificate to encrypt the message with ( public key) instead of negotiating with the service to determine its certificate before encrypting.   

Setting the Endpoint Identity and setting establishSecurityContext="false" and negotiateServiceCredentials="false" meant that the client now had enough info to encrypt the message without asking the service anything, and the service could decrypt it. 

Note that NONE of these issue related to the authentication certificate not being accepted, it was the "message security" that was the problem, it just happened to require message security so that it could attach the authentication to the payload. 

A side effect of this is that we should now be able to get rid of sticky sessions. 

F5 Big-IP Load Balanced WCF Services

*update* - This follow up may provide more detail on finding a solution 

We have been trying to configure our new F5 Big-IP load balancer for some WCF services and encountered a strange issue. 

The service uses wsHttpBinding with TransportWithMessageCredential over SSL.  The Auth type is Windows.

When disabling either node on the F5, the service worked.  However when both nodes were active, the service failed with an exception:

Exception:
Secure channel cannot be opened because security negotiation with the remote endpoint has failed. This may be due to absent or incorrectly specified EndpointIdentity in the EndpointAddress used to create the channel. Please verify the EndpointIdentity specified or implied by the EndpointAddress correctly identifies the remote endpoint.

Inner Exception:
The request for security token has invalid or malformed elements.

Numerous documentation and blogs highlight that *the* way to support load balancing in WCF is to turn off Security Context Establishment by setting EstablishSecurityContext=false in the binding configuration, or by turning on 'sticky sessions'.
http://ozkary.blogspot.com.au/2010/10/wcf-secure-channel-cannot-be-opened.html
http://msdn.microsoft.com/en-us/library/ms730128.aspx
http://msdn.microsoft.com/en-us/library/vstudio/hh273122(v=vs.100).aspx

We did not want to use sticky sessions; although this did fix the issue the F5 logs showed that load balancing was not working as we wanted.
Unfortunately, we already had EstablishSecurityContext set to false,  so the security negotiation should have been occuring on each request, which meant it should be working. 

After hours of investigating other binding settings, creating test clients, updating the WCFTestTool configurations and generally fumbling around, eventually we went back to reconfiguring the F5.  Although it worked when only one node was active with exactly the same configuration, unless the WCF binding documentation we found *everywhere* was a complete lie it had to be F5.

It was finally traced to the F5 OneConnect (http://support.f5.com/kb/en-us/solutions/public/7000/200/sol7208.html) configuration.  This does some jiggery-pokery to magically pool backend connections to improve performance.  It also seems to break WCF services, at least it broke ours. 

Disabling OneConnect on the F5 application profile resolved the issue immediately.

We now have our load balanced, non-persistent, WCF services behind the shiny F5 working.

As this hadn't come up online that I could find, I can only assume it was related to the combination of TransportWithMessageCredential and Windows as the message credential type.

*edit* So the solution was not as straight forward as we thought, and currently we have reverted back to “sticky sessions” in the F5 to get this to work.  Even with EstablishSecurityContext=false, and OneConnect disabled, the same failure will occur if a single client has two concurrent requests using two separate threads (our clients are web applications) and the F5 routes each connection to a separate service node.

While we investigate further, the short term solution is to use Transport security instead of TransportWithMessageCredential.  As this required a client change, we had to deploy multiple bindings and each client app will upgrade while we use Sticky Sessions on the F5.  Once all clients are on the new binding we can remove the old binding and disable sticky sessions again.

Transport security works for us, but it is not perfect.  It reduces security (SSL only, no message encryption) and reduces flexibility (we can’t for instance switch to client certificate, or username authentication for individual per request auth).

It does however keep the services stable and gives us time to perform a thorough analysis of the problem.

5 Minutes of Android from a .net dev

Introduction

I have been part of a casual group of senior developers working on building a standard development platform for our teams, and have been using it for a few simple trial projects quite successfully.  The original design was to support desktop, web, and Windows Phone development, with the primary focus being the core tiers (data/business) with the presentation layers currently fairly underdeveloped. 

I have produced effective WP7 applications using this framework however, so I figured I would build an Android app to get some insight into how much effort would be involved in supporting multiple native applications.

Backend

I started with a simple design, a ratings system where users can rate, and view the average rating, for a list of items.  As Android doesn’t support SOAP very well, I decided to expose my services via webHttp (REST) endpoints, which works quite well with the ratings model that I am building.

I had a few issues setting up Json support through WCF, for a couple of reasons. 

Firstly I was using DataContract(IsReference=true) to support the data contract serialisation of my entities, which is not supported by DataContractJsonSerializer.  This was required to prevent circular references in my entity design for previous projects.  The solution was to turn this off, and ignore the circular references from my entities using IgnoreDataMember attributes.

As always, as this was within the WCF layer, the errors were woefully inadequate, and it took a while to actually identify this as the issue :/

Secondly, I downloaded and installed cUrl to test my POST/PUT calls, and getting the exact right quotes/escaping strategy for the Json parameter was a pain.  In the end I needed the entire parameter enclosed in single quotes, and each double quote for the key/value pairs needed to be escaped with a standard escape character (\) .

Android

So I now had a RESTful service that android would be able to access quite easily, so on to the android development part.

• Getting the tools
  • This was fairly easy, Google has a very simple guide on how to get the SDK, as well as links to appropriate IDEs
  • The development can be done in a variety IDE’s, including eclipse, but for a change I decided to go with Telerik IntelliJ Idea, which has “built in” android support.
  • It was a bit of a process to download all the components for development.  Once the SDK was installed there were then a number of sub-components to download.
  • And then I needed the right Java Versions, with android only working on Java 1.6 meaning I needed to sign up for an Oracle account to download a legacy jdk.
• Hello World
  • IntelliJ IDEA was good here, creating a new android project was straight-(ish) forward.
  • Much like java projects in general, the code, assets, and resources were scattered all over a number of folders, but at least the build files were all set up and everything built up front.
  • While IDEA has no graphical designer, it does have a preview.  I will need to work out how I can support test data in the designer to make sure things look right.
• Testing
  • Setting up an emulator was very simple, and deploying was automatic when debugging.
  • Debugging was very slow, I would definitely switch to a real device for tracing/debugging in anger.
• Next Steps
  • Web Access
    • Getting a single http request to my REST service was easy
    • note, do not use localhost (yeah i should have known this from my WP7 apps too)
  • Data Binding
    • Not quite there yet, but it is not as simple as I expected – there is no JSON binding support off the bat, and binding list data actually involves creation of arrays of hashmaps.  This was a big surprise.
  • Page Navigation
    • Haven’t looked at this yet
  • UI Design
    • Haven’t looked at this yet

 

Conclusion

I had a lot of reservations on how hard it would be to make a simple android app, but offloading all the logic to the service layer, using RESTful services and designing your application flow for the mobile form factor, it looks like it can be done with fairly minimal effort.

Obviously providing ‘local’ device processing and resources (inputs etc) will complicated this dramatically, but for LOB systems it is definitely feasible.  Providing a native WP7 and Android front-ends is definitely within reach of our application framework.

Autofac and WCF Lifetime Issues

I am building a fairly simple WCF service and decided to use Autofac / Autofac.Integration.Wcf for (admittedly very simple) dependency injection.

I have previously used Unity, with a custom wcf scoped lifetime manager, but I have heard good things about autofac.

Setting up the WCF instance was pretty straight forward following the wiki for a IIS hosted server, and I could immediately use the service.

The next step was to configure the service instance PerCall since I don't want session management, which was also straight forward.

Finally I wanted to register a dependency for the service, which caused some confusion, partly because I was 'doing it wrong', and partly because some of the guides I had seen were wrong.

My registration is shown below, and is based on the Autofac wiki examples
var builder = new ContainerBuilder();
builder.RegisterType<WcfService>();
builder.RegisterType<BusinessService>().InstancePerLifetimeScope();
AutofacHostFactory.Container = builder.Build();
The BusinessService is scoped as InstancePerLifetimeScope, which combined with the AutofacHostFactory definition in the service host markup should generate a new instance per WCF call.

I then called AutofacHostFactory.Container.Resolve() twice in one of my service methods witht he expectation that each call to the service method would create a single new instance of BusinessService, but the two calls in the one method would return the same instance. Unfortunately this was not the case, and a single instance was reused throughout all my service method invokations.

I should have known better and had the BusinessService as a dependency to my service constructor (which was indeed a misdemeanor) BUT, it took a considerable amount of time to find out why the manual service resolution was not respecting the expected Lifetime management. As it turns out, the AutofacHostFactory.Container was not the correct container to use, and now that I think of it, I can understand why, but there is very little description over what happens in the service configuration within Autofac.

Registering the container in the AutofacHostFactory actually creates an Autofac container on the Host Factory itself, which is why the resolution returned a common instance each time, even between calls. Obvious yes, but the code to get the actual container was far less obvious. It wasn't until I saw this 'bug request' that I understood what had happened.

The Autofac service host factory registers a new container lifetime when the service is created, and the only way to get it is to obtain it from the current WCF OperationContext. In Unity this was more explicit because I implemented the Lifetime Manager myself, but in Autofac the Integration.Wcf libraries which things a little more black box, and the lack of documentation was frustrating.

Anyway, as I said, I did it wrong to begin with - adding the BusinessService as a constructor dependency resoved the dependency exactly as I originally expected, but it can definitely be confusing for a developer as they cannot call resolve() on the same container they register it (on the AutofacHostFactory in this case).