Pages

Wednesday, February 15, 2012

Late Binded Parameterized Tests in NUnit

Recently i overtook a medium sized .Net project which was completely devoid of any sort of test effort. Given the task of drastically improving the quality of the system, development of a test suit for the project was imperative. Great I thought, since I'd been wanting to experiment with a more structured test approach in .Net since toying around with the Grails approach to the subject in the last year or so.

Recently I'd been messing around with GSpec in Grails which I'd been fairly satisfied with using. In more general terms I think the specification based testing frameworks may be on to something. After investigating specification based testing frameworks in .NET for a while i discovered numerous frameworks were available... most notably NSpec and MSpec.

On second thought though the project required intense refactoring and introduction of many other technologies such as migrating from LinqToSql to Fluent NHibernate and introduction of StructureMap. Also the development team was recently assembled and aspiring in nature. So i decided to take the safe path for now and use a more standard testing framework of the old paradigm. Not really having any specific sympathies in this respect i decided to use the tried and true NUnit framework.

Upon writing my first unit tests I quickly became irritated by a sneaking efficiency related desire of copy pasting various aspects of one test case to the next. I sat back and pondered where this irritating behavior seemed to be stemming from. It quickly became evident that i was copy pasting my subject under test in order to perform various test parametrizations. At this time i remembered i had been fooling around with a new feature in NUnit a few years back... test parameterizations. What better time than now to explore parameterized tests and maybe discover a more structured testing approach.

After googling intensely for the next hour or so I came to the conclusion that i needed to use the TestCaseSource attribute in order to supply my test case with parameterizations. So I came up with something like the following:

[TestFixture]
public partial class SomeServiceUnitTest
{
    [SetUp]
    public void SetUp()
    {
        ObjectFactory.Initialize(initializer => initializer.AddRegistry<UnitTestRegistry>());
        ObjectFactory.GetInstance<IUnitOfWork>().BeginTransaction();
    }

    [TearDown]
    public void TearDown()
    {
        ObjectFactory.GetInstance<IUnitOfWork>().RollBack().Dispose();
        ObjectFactory.ResetDefaults();
    }

    [Test]
    [TestCaseSource(typeof(SomeTestData))]
    public decimal SomeTest(SomeType argA, SomeOtherType argB)
    {
        var someService = ObjectFactory.GetInstance<ISomeService>();
        return someService.DoSomething(argA, argB);
    }
}
public class SomeTestData
{
    public static IEnumerable TestCases
    {
        get
        {
            yield return new TestCaseData(null, null)
                .Throws(typeof(ArgumentNullException);
            yield return new TestCaseData(new SomeType(), new SomeOtherType())
                .Throws(typeof(NullReferenceException));
        }
    }
}

Relatively satisfied with the result i sat back and fired the test case off expecting everything in the world to be good.... unfortunately this was not the case. The test case failed miserably because i had expected the SetUp method in my test fixture would be run prior to retrieving test case parameterizations from the SomeTestData class. Why do my unit tests depend on dependency injection? Well because i rely on a practice similar to the Grails way of implementing unit tests in which an SQLite in-memory database is used during unit testing so that mocking of basic repository functionality can be avoided during unit testing (which is an immense time saver!).

After googling around for another hour or so and pondering the issue i came up with a theoretical solution to the problem... late binded parameterized tests. Basically a delegate will instead be yielded for each parametrization which when invoked will return a number of out parameters which can then be fed to the subject under test. In this way it should be possible to defer setup of the parametrizations til after the Setup method in the test fixture has been called.

After successfully implementing a proof of concept i set out to generalize the idea, this is what i came up with. First off, the modified late binded parameterized test now looks like:

using Some.Namespace.NUnit;
using NUnit.Framework;

namespace Some.Namespace
{
    [TestFixture]
    public partial class SomeServiceUnitTest
    {
        [SetUp]
        public void SetUp()
        {
            ObjectFactory.Initialize(initializer => initializer.AddRegistry<UnitTestRegistry>());
            ObjectFactory.GetInstance<IUnitOfWork>().BeginTransaction();
        }

        [TearDown]
        public void TearDown()
        {
            ObjectFactory.GetInstance<IUnitOfWork>().RollBack().Dispose();
            ObjectFactory.ResetDefaults();
        }

        [Test]
        [LateBindedTestCaseSource(typeof(SomeTestData))]
        public decimal SomeTest(LateBindedParameterization<SomeType, SomeOtherType> arg)
        {
            var someService = ObjectFactory.GetInstance<ISomeService>();

            SomeType argA;
            SomeOtherType argB;
            arg.Parameterization.Invoke(out argA, out argB);

            return someService.DoSomething(argA, argB);
        }
    }
}
public class SomeTestData : LateBindedParameterizationSource<SomeType, SomeOtherType>
{
    [TestCaseParameterization(typeof(ArgumentNullException))]
    public void A(out SomeType argA, out SomeOtherType argB)
    {
        argA = null;
        argB = null;
    }

    [TestCaseParameterization(typeof(NullReferenceException))]
    public void B(out SomeType argA, out SomeOtherType argB)
    {
        argA = new SomeType();
        argB = new SomeOtherType();
    }
}

Which is not much different then previously. Now the test case receives an object as argument which is simply a wrapper around a delgate. The test case must then invoke the delegate in order to retrieve the input arguments which can then be fed to the subject under test. I decided to use a different convention for the data source, namely methods instead of a static IEnumerable getter property, which in my humble opinion is much more readable.

The underlying mechanism which enables the above late binded parameterized test is heavily dependent on a big bucket of reflection. The concept generalization is implemented in four classes:

  • LateBindedTestCaseSourceAttribute
  • LateBindedParameterizationSource
  • TestCaseParameterizationAttribute
  • LateBindedParameterization

And here is the implementation:

using System;
using NUnit.Framework;

namespace Some.Namespace.NUnit
{
    public class LateBindedTestCaseSourceAttribute : TestCaseSourceAttribute
    {
        public LateBindedTestCaseSourceAttribute(Type sourceType)
            : base(sourceType, "Data")
        {
        }
    }
}
using System;
using System.Collections.Generic;
using System.Reflection;
using NUnit.Framework;

namespace Some.Namespace.NUnit
{
    public class LateBindedParameterizationSource<T1>
    {
        public IEnumerable<TestCaseData> Data
        {
            get
            {
                foreach (var parameterization in Parameterizer.ResolveParameterizations(GetType()))
                {
                    var testCaseData =
                        new TestCaseData(new LateBindedParameterization<T1>((Parameterization<T1>)Delegate.CreateDelegate(typeof(Parameterization<T1>), this, parameterization.Value)))
                            .SetName(parameterization.Value.Name);

                    yield return Parameterizer.FinishParameterizations(parameterization.Key, testCaseData);
                }
            }
        }
    }

    public class LateBindedParameterizationSource<T1, T2>
    {
        public IEnumerable<TestCaseData> Data
        {
            get
            {
                foreach (var parameterization in Parameterizer.ResolveParameterizations(GetType()))
                {
                    var testCaseData =
                        new TestCaseData(new LateBindedParameterization<T1, T2>((Parameterization<T1, T2>)Delegate.CreateDelegate(typeof(Parameterization<T1, T2>), this, parameterization.Value)))
                            .SetName(parameterization.Value.Name);

                    yield return Parameterizer.FinishParameterizations(parameterization.Key, testCaseData);
                }
            }
        }
    }

    public class LateBindedParameterizationSource<T1, T2, T3>
    {
        public IEnumerable<TestCaseData> Data
        {
            get
            {
                foreach (var parameterization in Parameterizer.ResolveParameterizations(GetType()))
                {
                    var testCaseData =
                        new TestCaseData(new LateBindedParameterization<T1, T2, T3>((Parameterization<T1, T2, T3>)Delegate.CreateDelegate(typeof(Parameterization<T1, T2, T3>), this, parameterization.Value)))
                            .SetName(parameterization.Value.Name);

                    yield return Parameterizer.FinishParameterizations(parameterization.Key, testCaseData);
                }
            }
        }
    }

    public class LateBindedParameterizationSource<T1, T2, T3, T4>
    {
        public IEnumerable<TestCaseData> Data
        {
            get
            {
                foreach (var parameterization in Parameterizer.ResolveParameterizations(GetType()))
                {
                    var testCaseData =
                        new TestCaseData(new LateBindedParameterization<T1, T2, T3, T4>((Parameterization<T1, T2, T3, T4>)Delegate.CreateDelegate(typeof(Parameterization<T1, T2, T3, T4>), this, parameterization.Value)))
                            .SetName(parameterization.Value.Name);

                    yield return Parameterizer.FinishParameterizations(parameterization.Key, testCaseData);
                }
            }
        }
    }

    public class LateBindedParameterizationSource<T1, T2, T3, T4, T5>
    {
        public IEnumerable<TestCaseData> Data
        {
            get
            {
                foreach (var parameterization in Parameterizer.ResolveParameterizations(GetType()))
                {
                    var testCaseData =
                        new TestCaseData(new LateBindedParameterization<T1, T2, T3, T4, T5>((Parameterization<T1, T2, T3, T4, T5>)Delegate.CreateDelegate(typeof(Parameterization<T1, T2, T3, T4, T5>), this, parameterization.Value)))
                            .SetName(parameterization.Value.Name);

                    yield return Parameterizer.FinishParameterizations(parameterization.Key, testCaseData);
                }
            }
        }
    }

    public static class Parameterizer
    {
        public static IEnumerable<KeyValuePair<TestCaseParameterizationAttribute, MethodInfo>> ResolveParameterizations(Type type)
        {
            foreach (var method in type.GetMethods())
            {
                var testCaseMethodAttribute = Attribute.GetCustomAttribute(method, typeof(TestCaseParameterizationAttribute), false) as TestCaseParameterizationAttribute;

                if (testCaseMethodAttribute != null)
                    yield return new KeyValuePair<TestCaseParameterizationAttribute, MethodInfo>(testCaseMethodAttribute, method);
            }
        }

        public static TestCaseData FinishParameterizations(TestCaseParameterizationAttribute attribute, TestCaseData data)
        {
            switch (attribute.Expectation)
            {
                case TestCaseMethodExpectation.ReturnValue:
                    data.Returns(attribute.Returns);
                    break;
                case TestCaseMethodExpectation.Exception:
                    data.Throws(attribute.Throws);
                    break;
            }

            if (attribute.Ignored())
                data.Ignore(attribute.IgnoreReason);

            return data;
        }
    }
}
using System;

namespace Some.Namespace.NUnit
{
    public enum TestCaseMethodExpectation
    {
        ReturnValue,
        Exception
    }

    public class TestCaseParameterizationAttribute : Attribute
    {
        private readonly object returns;
        private readonly Type throws;
        private readonly string ignoreReason;

        private readonly TestCaseMethodExpectation expectation;

        public TestCaseMethodExpectation Expectation
        {
            get { return expectation; }
        }

        public object Returns
        {
            get { return returns; }
        }

        public string IgnoreReason
        {
            get { return ignoreReason; }
        }

        public Type Throws
        {
            get { return throws; }
        }

        public TestCaseParameterizationAttribute(object returns)
        {
            this.returns = returns;
            expectation = TestCaseMethodExpectation.ReturnValue;
        }

        public TestCaseParameterizationAttribute(object returns, string ignoreReason)
        {
            this.returns = returns;
            this.ignoreReason = ignoreReason;
            expectation = TestCaseMethodExpectation.ReturnValue;
        }

        public TestCaseParameterizationAttribute(Type throws)
        {
            this.throws = throws;
            expectation = TestCaseMethodExpectation.Exception;
        }

        public TestCaseParameterizationAttribute(Type throws, string ignoreReason)
        {
            this.throws = throws;
            this.ignoreReason = ignoreReason;
            expectation = TestCaseMethodExpectation.Exception;
        }

        public bool Ignored()
        {
            return ignoreReason != null;
        }
    }
}
namespace Some.Namespace.NUnit
{
    public delegate void Parameterization<T1>(out T1 p1);
    public delegate void Parameterization<T1, T2>(out T1 p1, out T2 p2);
    public delegate void Parameterization<T1, T2, T3>(out T1 p1, out T2 p2, out T3 p3);
    public delegate void Parameterization<T1, T2, T3, T4>(out T1 p1, out T2 p2, out T3 p3, out T4 p4);
    public delegate void Parameterization<T1, T2, T3, T4, T5>(out T1 p1, out T2 p2, out T3 p3, out T4 p, out T5 p5);

    public class LateBindedParameterization<T1>
    {
        public Parameterization<T1> Parameterization { get; private set; }

        public LateBindedParameterization(Parameterization<T1> parameterization)
        {
            Parameterization = parameterization;
        }
    }

    public class LateBindedParameterization<T1, T2>
    {
        public Parameterization<T1, T2> Parameterization { get; private set; }

        public LateBindedParameterization(Parameterization<T1, T2> parameterization)
        {
            Parameterization = parameterization;
        }
    }

    public class LateBindedParameterization<T1, T2, T3>
    {
        public Parameterization<T1, T2, T3> Parameterization { get; private set; }

        public LateBindedParameterization(Parameterization<T1, T2, T3> parameterization)
        {
            Parameterization = parameterization;
        }
    }

    public class LateBindedParameterization<T1, T2, T3, T4>
    {
        public Parameterization<T1, T2, T3, T4> Parameterization { get; private set; }

        public LateBindedParameterization(Parameterization<T1, T2, T3, T4> parameterization)
        {
            Parameterization = parameterization;
        }
    }

    public class LateBindedParameterization<T1, T2, T3, T4, T5>
    {
        public Parameterization<T1, T2, T3, T4, T5> Parameterization { get; private set; }

        public LateBindedParameterization(Parameterization<T1, T2, T3, T4, T5> parameterization)
        {
            Parameterization = parameterization;
        }
    }
}

The above late binded test parameterization implemenation for NUnit is built on top of the most recent version of NUnit. An improvement would be to incorporate it directly into the inner workings of NUnit. Once incorporated directly into NUnit it would be possible to hide all of the implementation details so that the test case method would simply be fed the actual arguments for the subject under test. The modified NUnit framework would simply call the delegate, pull the arguments and pass them on to the test case method.

Provided i discover some unanticipated spare time in the upcoming months i may attempt to pull the NUnit source and implement the above... the famous last words i expect.

0 comments:

Post a Comment