EF Deleted Items Issue

I noticed an issue in one of my service methods whereby a record I deleted showed up in a subsequent query within a single unit of work.

The example code is

int orderId = order.OrderID;
_orderRepository.delete(order);
Order newOrder = _orderRepository.getAll(x=>x.orderID == orderId);

The above example is a bit contrived, there's a fair bit more that goes on but this code highlights the issue.

Now that I know what is going on this is realtively straight forward, but it is a bit counterintuitive when starting out.

The problem was in my repository, where I was using the DbContext DbSet property for each entity directly, instead of the DbSet.Local property. The difference between the two is that the root DbSet property contains all elements in their modified state (e.g. it contains the deleted order, with an updated state of Deleted), while the Local DbSet property (which is an IObservable of the root DbSet) has the entities in their 'current state' so if you delete an entity from the context it is removed from the Local DbSet.

I say this is counterintuitive because the only way to identify whether an item is deleted or not is through the root context Entry() method, you cannot base a query on the DbSet to exclude deleted items.

The solution is however fairly simple. Since I am using a Unit of Work pattern on the context, and my service methods are a single unit of work, I can use the Local DbSet for my repository actions without any issues down the line with disconnected or orphan entities, and I can do this without any modifications to my service.

So where all my repository queries used to use the code below as the base for all repository queries
IQueryable query = _set.AsQueryable(); //_set is the appropriate DbSet for the entity T in the context
I now simply base all my queries off
IQueryable query = _set.Local.AsQueryable();

Now deleted items should not show up in my list of queries. I hope - I haven't had a chance to actually test it just yet.

*edit*

Well, that was short lived - it seems as though using the Local context only works on previously loaded data, for instance you do a load, then delete an entity, then a load from the local context will not show the deleted item.

This is incredible frustrating as it means I need to know under what scenario I am 'loading' data in order to choose the right context to load from, and means I need to front-load all the entities I will be working with, then use the local context from that point on.

I am seriously thinking of switching to nHibernate over this one.

*edit 2*

I have identified a possible solution, but I am concerned by performance implications

When performing a query on my context, I can use Linq to query the state of each entity in the resulting query and further filter the results.

query.Where(whereClause).ToList().Where(x=> ((DbContext)_context).Entry(x).State != System.Data.EntityState.Deleted ).ToList();

The two performance issues with this are

a) I need to 'ToList()' the query and then apply the state filter (otherwise EF will attempt to apply the filter to the SQL query, which it can't do). This is not ideal, but not critical, and I may be able to force the first Where() to resolve the EF part in another way to avoid the extra list creation.

and b) queries that return a large number of results will be impacted (potentially severely) since each entity will be inspected individually against the context change manager.

So perhaps an nHibernate implementation could wait if this does what I want it to without critical performance implications.

The difference between bad code and good code

A colleague asked for some advice today on a project that he inherited (which I am extending with a separate module incidentally).

The issue was related to the usage of Entity Framework in the code that he had to maintain, and he needed some advice on how to proceed.  The problem was that the service layer was calling the repository multiple times, but each repository method was wrapped in a separate unit of work.
e.g.

public void DeleteEntity(int entityID)
{
    using (var context = EntityContext())
    {
        var entity = context.Postings.SingleOrDefault(p => p.entityID == entityID);
        context.Entities.DeleteObject(entity);
        context.SaveChanges();
    }
}
and
public Entity GetPosting(int entityID)

{
    using (var context = EntityContext())
    {
        return context.Entities.FirstOrDefault(p => p.entityID == entityID);
    }
}

This caused two problems for the developer, who needed to perform a complex action in his service that referenced multiple repository calls.

1. He had no control over the transactional scope for the repository methods
2. Each operation was on a separate EF context, so the service could not load and entity, edit it, and then save the changes (unless the repository was designed for disconnected entities, which it wasn't).

From a maintainability and testability point of view this was also a very poor design, as the repository methods created instances of dependency object (the service method also created instances of the repositories, making the services inherently untestable).


The version of this design that I implemented for my component follows a similar service/repository/entity pattern, but is implemented in a far more testable and robust manner.

The first improvement over the legacy design is in the dependency management
My service accepts a context and all required repositories in the constructor, and my repositories accepts a context, which allows for improved maintainability (all dependencies are described) and testability (all dependencies can be mocked).  This also allows us to use dependency injection/IoC to create our object instances.

The second improvement was in the Unit of Work design.
Rather than have each repository method as a single unit of work, the service methods are the units of work, so any action within the service uses the same context (as it is passed as a dependency to the repositories that the service uses), and each service call acts as a Unit of Work, calling SaveChanges at the end of the service to ensure that the changes act under a single transaction.
There are limitations to this design (your public service methods become an atomic transaction and you should not call other public methods from within another method) but for simplicity and maintainability it is a pretty good solution.

Below is a simple example of the design I am using, preserving maintainability, testability, and predictability.  I'm not saying it is necessarily the best code around, but it solves a number of issues that I often see in other developers code.

public class HydrantService
{
  public HydrantService(HydrantsSqlServer context, EFRepository<Hydrant> hydrantRepository, EFRepository<WorkOrder> workOrderRepository, EFRepository<HydrantStatus> hydrantStatusRepository)
  {
    _context = context;
    _hydrantRepository = hydrantRepository;
    _workOrderRepository = workOrderRepository;
    _hydrantStatusRepository = hydrantStatusRepository;
  }
  public void createFaultRecord(WorkOrder order)
  {
    HydrantStatus status = _hydrantStatusRepository.GetSingle<HydrantStatus>(x => x.StatusCode == "Fault"); //_context.HydrantStatuses.Where(x => x.StatusCode == "Fault").FirstOrDefault();
    order.Hydrant.HydrantStatus = status;
    _workOrderRepository.Add(order);
    _context.SaveChanges();
  }
}

  public class EFRepository<T>
 {
 public EFRepository(IDbContext context)
 {
    _context = context;
  }   public virtual ICollection GetAll()

  {
    IQueryable query = _context.Set();
    return query.ToList();
  }
}

Unit Testing, Moq, EF, and Repositories

　
Well, I have just started a small (8-12 week / 1 resource) project using an unfinished version of our in-house framework for some parts of it. In the process I want to ensure that I integrate some key design patterns (null object, repository, and unit of work) and full unit testing on the service implementation. This will hopefully help alleviate the pain of working with DotNetNuke, cross-application dependencies, and webforms.

So my first step was to property expose services from the dependent application as this is a major point of failure in other systems that use this application, which was pretty straight forward as the application design is not too bad. As this is a shared dependency on the DotNetNuke instance, I did not need to expose this as a WCF service, but could easily change it in the future if necessary. The new service interface will help prevent changes in the core application from breaking the dependent application, as any changes will be reflected as build failures in the service class, highlighting this to the developers and ensuring they either make the change to not break the interface, or let all consumers of this service know there is a breaking update and plan appropriate changes. This is a key issue encountered when services and application references are not well defined, and has caused a number of deployment issues at my current client.

The guts of this post however is to discuss my plan for unit testing, and how I had to rethink my previous statement of going ‘repository-less’. I previously discussed the removal of the repository from the framework and using the DbSet functionality in the EF context as the repository pattern. This worked really well, until I decided to do some unit tests.
I decided to use a mocking library in my unit tests specifically to ensure I was performing appropriately isolated tests, and to reduce the impact of managing test data. I had previously looked at Moles (Microsoft stubbing tool), but it always seemed so cumbersome and confusing, so I picked up Moq instead. I really like the Moq usage pattern, and so I thought it would be a good fit.

So, the plan was to use Moq to create mocks of the repository functions that act in predictable and repeatable ways, which means we can run the service and test that the service behaves as we expect.

An example is given below – in this example I created a service to get a list of ‘stations’ from the dependent application. Since I am testing my service, I want to Mock the dependent application service to act predictably, so I can ensure that my service acts the way I want it to (we are not performing end-to-end integration testing, so we don’t want to rely on the dependent application succeeding or failing at this point)

//when we call ‘GetStations’ with a parameter of 0, our mocked service throws an exception – I know the dependent service reacts in this way, so I can ensure this is integrated in my test
_samsServiceMoq.Setup(x => x.GetStations(0)).Throws();
//when we call ‘GetStations’ with a parameter of -1, our mocked service returns no results
_samsServiceMoq.Setup(x => x.GetStations(-1)).Returns(new List());
//when we call ‘GetStations’ with a parameter of 1, our mocked service returns a list with one item in it
_samsServiceMoq.Setup(x => x.GetStations(1)).Returns(new List() { new Unit(){ UnitID = "100" } });
　
UserService target = new
UserService(_samsServiceMoq.Object); //create an instance of my service, and pass in the mocked dependent service
List actual1;
List actual2;
List actual3;
actual1 = target.GetStations(-1); //execute the service method with the specified parameter
actual2 = target.GetStations(1); //execute the service method with the specified parameter
actual3 = target.GetStations(0); //execute the service method with the specified parameter
_samsServiceMoq.VerifyAll();
//check whether the mocked service methods were called in the execution of our tests – this is useful to ensure that your service method is calling the expected mocked method with the expected parameters.
//check the results from the service to ensure they match what you expect (based on the response from the mocked service)
Assert.IsNotNull(actual1);
Assert.IsNotNull(actual2);
Assert.IsNotNull(actual3);
Assert.AreEqual(0, actual1.Count);
Assert.AreEqual(1, actual2.Count);
Assert.AreEqual("100", actual2[0].UnitID);
Assert.AreEqual(0, actual3.Count);

The above example shows how you can configure a test without worrying about the dependent services, so you can test only the functionality in your service. You will also note that the service itself needs to be designed so that all dependencies are passed to the service, instead of created in the service (this is a key point in ensuring testability of components, all dependencies must be passed to the object). If we did not do this, we could never mock the dependent service, which means we would need to set up the test to ensure the dependent service responds appropriately (configure the dependency, and know/configure sample data that the dependency will respond to).
This works really well, I can test my (admittedly very simple) service without caring about configuring the dependent service. However doing the same thing on an EF repository instead of the dependent service does not work so well. The code below should work, but doesn’t due to limitations in EF/C#/Moq.

_hydrantContextMoq.Setup(x=>x.Hydrants).Returns(_hydrantDbSetMoq.Object);
_hydrantDbSetMoq.Setup(x => x.ToList()).Returns(new List() { new Hydrant() });
HydrantService service = new HydrantService(_hydrantContextMoq.Object);
List actual;
actual = service.GetHydrantList();
_hydrantContextMoq.VerifyAll();
_hydrantDbSetMoq.VerifyAll();
Assert.IsTrue(actual.Count == 1);

Here I am mocking my DbContext to return a mocked IDbSet, and mocking the IDbSet.ToList() to return a list of Hydrants with 1 item. This way I can test my service so that calling getHydrantList on my service returns the single length list. Unfortunately, IDbSet.ToList() is not a mockable method (it is actually an extension method) which means it is not possible to set up a mock for this method. Since my service is using this method, I cannot test my service in isolation of the database.

This is where the Repository comes in. Instead of using the IDbSet.ToList() directly, I would use a Repository GetAll() method which abstracts the call to the underlying DbSet method. As the repository is just another dependency on the service, we can mock this instead of the EF IDbSet, and hence have an appropriately testable service. We will also then have the ability to ensure that the repository supports the null object pattern, so a call to the IDbSet that may return null (such as a find() with an invalid key) can return an appropriate null object to the service, so the service, and all clients, know it will never receive a null as the result of a service operation.

So, big backtrack on the framework repository, and big kudos to Moq for making testing easier (at least for my simple examples so far).