F5 Big-IP Load Balanced WCF Services - Update

The post below was some findings from a project related to the authentication between the front and back end services, and F5 configurations.  It closely matches issues identified in my previous post for the SSO server but that was using ntlm/kerberos (I have a feeling that we didn't set up the SPNs for kerberos correctly).  Ultimately I bet that a similar configuration to that below but set up for one-shot-kerberos and using the SPN in the Identity column would have resolved this issue.

tldr;  

Server config:  

Configure the server to accept client certificates using PeerTrust - allows us to authorise the client 

Define the server certificate (the identity of the service - important later on) - allows message security to work 

<behavior name="certificate"> 

<serviceMetadata httpGetEnabled="true" httpsGetEnabled="true" /> 

<serviceDebug includeExceptionDetailInFaults="true" /> 

<serviceCredentials> 

<clientCertificate> 

<authentication certificateValidationMode="PeerTrust" /> 

</clientCertificate> 

<serviceCertificate findValue="<blah>" storeLocation="LocalMachine" x509FindType="FindBySubjectName" /> 

</serviceCredentials> 

</behavior> 

configure the wsHttpBindings to use Message security,  

items highlighted yellow - this is used for the message security and allows load balancing without sticky sessions 

items highlighted green - this is used to authenticate the client using a certificate 

<wsHttpBinding> 

<binding name="certauth" maxReceivedMessageSize="2147483647"> 

<readerQuotas maxStringContentLength="2147483647" maxArrayLength="2147483647" /> 

<security mode="Message"> 

<message algorithmSuite="TripleDesSha256" establishSecurityContext="false" negotiateServiceCredential="false" clientCredentialType="Certificate" /> 

</security> 

</binding> 

</wsHttpBinding> 

Configure the endpoint, basically accept request on <endpoint>/certauth and applies the appropriate message security and authentication 

<endpoint name="certauth" address="certauth" binding="wsHttpBinding" bindingConfiguration="certauth" contract="<contract>" /> 

Client Config: 

Configure the client to accept the server's security configuration using PeerTrust 

Configure the certificate that this client will pass as its credentials 

<endpointBehaviors> 

<behavior name="certificate"> 

<clientCredentials> 

<clientCertificate findValue="<name>" storeLocation="LocalMachine" x509FindType="FindBySubjectName" /> 

<serviceCertificate> 

<authentication certificateValidationMode="PeerTrust" /> 

</serviceCertificate> 

</clientCredentials> 

</behavior> 

</endpointBehaviors> 

Configure the security bindings, must match the service bindings 

<wsHttpBinding> 

<binding name="certauth" maxReceivedMessageSize="2147483647"> 

<readerQuotas maxStringContentLength="2147483647" maxArrayLength="2147483647" /> 

<security mode="Message"> 

<message algorithmSuite="TripleDesSha256" establishSecurityContext="false" negotiateServiceCredential="false" clientCredentialType="Certificate" /> 

</security> 

</binding> 

</wsHttpBinding> 

Configure the endpoint.  Note the /certauth suffix. 

the identity element is essential, it needs to match the serviceCredentials defined on the service.   

<endpoint address="<endpoint>/certauth" binding="wsHttpBinding" bindingConfiguration="certauth" contract="<contract>" behaviorConfiguration="certificate"> 

<identity> 

<certificateReference findValue="<name>" x509FindType="FindBySubjectName" storeLocation="LocalMachine" storeName="TrustedPeople"/> 

</identity> 

</endpoint> 

long version; 

To do this we need to use message security for the following reasons 

• We are using SSL offload, so transport security is not supported (works over SSL only) 
• We are using certificate auth, which is not allowed on TransportWithCredentialOnly 
• We are working across a domain boundary so we cannot use windows authentication 

The second complexity is that we are working in a load balanced environment.  This adds complexity as WCF negotiates a security context to generate the message security before sending the message.  If the Load balancer sends the message to a different server to the one the security context was generated on, then this will cause intermittent security failures. 

The two general solutions to this are to; 

• enable sticky sessions which the F5 then "guarantees" that the security context and message occur on the same server.   
• This did not seem to work - with sticky sessions enabled with the following config (), we still received this errors.   
• If the security context lasted longer than the sticky session timeout, then this could still cause problems, but I don't think this was the case. 
• I have a feeling this was caused by not setting the 'identity' correctly in the WCF config which was a key part of the final working solution, but it should still have worked regardless as both front end servers had success connecting to both back end servers. 
• set wcf establishSecurityContext="false" which basically sets up security for every message, rather than establishing a long term security context, each message would have its own security context applied. 
• this did not work.  I have encountered this before where this was the supposed fix and did not actually resolve the issue, but I thought that it was related to ntlm issues at the time. 
• Sticky sessions and this fix should pretty much have guaranteed the fix as you would never have the sticky session timeout before the security context expired (because it is only ever generated right before the message is sent). 
• This did not work.  I have no explanation for this. 

A final solution that I had previously read which allows a configuration to work without sticky sessions is to set negotiateServiceCredentials="false", however all examples point this to enable "kerberos-one-shot" which allows message security using kerberos if the services are configured with domain users and SPNs.  As we work across domains this wasn't acceptable. 

I was finally able to put a few things together and determined that the use of "establishSecurityContext" along with "negotiateServiceCredentials" should be possible without kerberos if you use certificate security (separate from certificate authentication).  This had partially been done with the  "servicebehaviour" element which defined the serviceCredentials on the server using certificates, and peerTrust for serviceCertificate on the client.  However the next step was to define the serviceIdentity on the client config so that the client knew which certificate to encrypt the message with ( public key) instead of negotiating with the service to determine its certificate before encrypting.   

Setting the Endpoint Identity and setting establishSecurityContext="false" and negotiateServiceCredentials="false" meant that the client now had enough info to encrypt the message without asking the service anything, and the service could decrypt it. 

Note that NONE of these issue related to the authentication certificate not being accepted, it was the "message security" that was the problem, it just happened to require message security so that it could attach the authentication to the payload. 

A side effect of this is that we should now be able to get rid of sticky sessions. 

The Dependency Zoo

This is shamelessly pulled from here: https://nblumhardt.com/2010/01/the-relationship-zoo/ - read it, it is good.  Much better than this.

Quite some time ago I was faced with a problem that I wasn't quite sure how to solve, and one of the dev's I most respect sent me the link above.  While the link wasn't the answer, it helped me formulate the answer I needed by making me think more critically about my dependencies.

Prior to that point dependency injection to me was very useful, but I didn't really how limited my understanding was.  I thought I had a good grasp of why IoC was important, and how to use DI, but I hadn't realised how simplistic my view was.

The problem in my case wasn't that I needed an B, but I needed a specific B and I would only know which B until runtime.  This didn't match the way I thought of Dependency Injection and I couldn't figure out a non-ugly solution.

What this article did was really make me think about how I consider dependencies. 

The article notes that there are a number of common types of relationships between classes:

"I need a B"

"I might need a B"

"I need all the Bs"

"I need to create a B"

"I need to know something about B before I will use it"

All of these have a direct representation in Autofac (and many other DI containers, though the last is not so common), this was a direct result of the author thinking about these relationships before writing Autofac.

Usually the solution is "use the service locator 'anti-pattern'".  I won't argue the point, but it is widely considered an anti-pattern unless it fits a very narrow category.  This does fall within that category, but I didn't think it was an elegant solution to my problem.  I also didn't think of it at the time.  Essentially the service locator pattern is you take the DI container as the dependency and ask it for what you need (usually in a switch statement).

The second most common solution to my problem is to take the "I need to create a B", or Func<X,Y,B> where B is the type we want and X and Y are the parameters that we use to create a B.  This pretty much matches the Factory pattern, where you have a class which creates the objects you want.  In this case autofac is smart enough that if you take a dependency on Func<X,B> and you have an B that takes X as a parameter, it will give you the constructor function of X without having to write a specific "Factory" class. 

The problem in this instance is that it doesn't really solve my problem for two reasons, 1 is that the type of A was common across all my B's, so Autofac wouldn't know which B to give me; I still needed a Factory class.  It does makes my dependee cleaner, but the factory was still riddled with switch statements trying to get the right B.  It is a very useful type of dependency, but not the one I wanted in this instance.

The next most common solution is to take all of the Bs and loop through them asking them if they are the right one.  I've seen this as the "right way to not use a service locator for that one instance where it is usually used" but I'm not particularly convinced. http://blog.ploeh.dk/2011/09/19/MessageDispatchingwithoutServiceLocation/ explains this is a bit more detail.  It works, but it is really inefficient - especially if X is expensive to create.  It is also ugly.

"Expensive to create" might not always be an issue, but if you are creating every B and discarding all but one, every time you ask for an B, that's messy.  This is where Lazy<B> comes in.  Lazy is built into C# (and autofac knows about it) so that you don't actually get a B, you get something that can give you an B when you need it.

This means that if you *might* need an B, then go with Lazy<B> and you will only create/get one when you need it.  It's not always necessary, but if you have a complex dependency hierarchy, or a very intensive constructor, it can make a big difference.  Unfortunately I still needed all of the B's to determine which one I needed, and because I had Lazy<B>'s I couldn't ask it if this was the right one without type inspection/reflection.  Note: be careful of the distinction between Lazy<B> and Owned<B> - I won't cover it here, but it is important. 

So far we have covered the top 4 in the list.  They are all important, very useful to know, but they didn't solve my problem.  The last on the list is the most obscure, and possibly the least widely used, but it is something I love (and have manually implemented in other DI containers that doesn't have it).

Autofac calls this Meta<B,X> where X is the type of an object that is attached to B at the time you register B.  You can then ask the dependency about its X, without creating a B.

Using this we can have lots of B's, and when registering my B's I tell it to attach an X to it.  When I want the right B, I ask it for the IEnumerable<Meta<X,Lazy<B>>> and loop through the list until I find the B that has the X I am looking for.  Autofac does this automatically, all I need to do is tell autofac the X when I register each B.

I can also create a Func<B> factory that does this, cleaning up my dependee a little more - the most common use case for this pattern is the command dispatcher pattern where it doesn't really add any value to abstract this however (since finding the right B for a given ICommand is the whole point of the command dispatcher). 

Some examples of the Metadata you want to attach to a method are:

• The type of ICommand the IHandler handles - this is the one I use the most
• A 'weight' or 'order'  - if we have 10 B's that might be the one I need, I get the highest priority one.
• A 'delegation' pattern, I need to get the approval workflow for invoices > $500.  If this changes to >$1000 in 6 months all I need to do is update the registration of the IApprovalWorkflows to set the new limit.
• A 'feature toggle' or A/B testing - you may define a number of Xs and want to pick a different one based on some criteria.

I don’t expect anyone to come out of this thinking that these examples are the best way to do things, but what I would like you to come away with is a greater appreciation of what it means to have a dependency, and to use appropriate patterns and abstractions instead of just saying "I need a B" ( or new-ing something up).