Posts /

The Way We Assert

05 Nov 2017

Look at the following example of the proper way to do assertions in NUnit:

int[] array = new int[] { 1, 2, 3 };
Assert.That(array, Has.Exactly(1).EqualTo(3));
Assert.That(array, Has.Exactly(2).GreaterThan(1));
Assert.That(array, Has.Exactly(3).LessThan(100));

This is how the introductory code for xUnit looks like:

Assert.Equal(4, Add(2, 2));

Wouldn’t it be wonderful if C# possessed some kind of facility that would allow us to evaluate stuff into true or false?

Like, expressions and LINQ?

using System.Linq;

Assert(a.Count(e => e == 3) == 1);
Assert(a.Count(e => e > 1) == 2);
Assert(a.Count(e => e < 100) == 3);
Assert(4 == Add(2, 2))

Unit test frameworks are great for allowing us to run pieces of code in a coordinated fashion. Programming languages are great for evaluating pieces of code. Isn’t it meaningless to take away their core competency?

If you are not satisfied with how your language evaluate expressions, you probably should change your language. Are you sure though that the look of Equal(a, b) is better than a == b?

And if you like a == b, but also like semantics of xUnit’s Equal better, isn’t it time to fix your objects? In C#, you don’t have structural equality for free, but you can nevertheless do whatever you want when writing == operator implementation. That would align equality behavior throughout your codebase, tests or not.

The only sensible argument for the NUnit kind of fluent interfaces for expressions that comes to my mind is to enable the testing infrastructure to provide error messages better than “assertion failed: what should have been true is false”. xUnit is able to tell you “assertion failed: 5 != 8”. This shortens feedback loop, especially when consuming failed test reports from CI, and so it’s vital for making testing worthwhile.

ExpressionToCode

But this is still C# we are talking about. C# has very nice support for expression trees, so we can have tooling like ExpressionToCode:

PAssert.That(() => a.Count(e => e == 3) == 1)

Such assert in the case of a failure (like for a = [ 1, 2, 3, 3, 2, 1 ]) throws an exception with the following message:

assertion failed

a.Count(e => e == 3) == 1
                   a   →   {1, 2, 3, 3, 2, 1}
a.Count(e => e == 3)   →   2

The tool observes evaluation and captures every subexpression’s value. In this output, we see in what state array came into the process and how the whole left side evaluated.

It’s more concise code-wise than the NUnit way, it provides better debug output and mainly: it doesn’t force us to learn just another DSL.

Proposal for integration with property-based testing

Does it feel strange that we have to wrap the expression into parameterless lambda? It’s necessary for the language to map in onto Expression<Func<bool>> that provides visibility for the evaluating library.

We can take this shortcoming as an opportunity: why not integrate property based testing into the approach and provide values for possible argument automatically?

Assert((int a) => a*a > 0)
assertion failed for a = 0

a*a   →   0

This would be elegant, because the empty-argument-list case we were forced to by peculiarities of C# is now the sensible special case of the rule “check the expression holds for all possible inputs”.

Interpretation of formal arguments of the function as “whatever fits there must preserve truth of the statement” also loosely resembles how mathematical logic treats free variables.

Conclusion

I have a suspicion that some property-based testing toolset already has to have some support for visibility into failed expressions, ExpressionToCode-style. Feel free to write me if you know about anything like this.

Also, if you try to implement code that goes along these lines, write about what happened! I’m curious.


Lukáš Lánský (e-mail)
Return to the list of posts


Share: Twitter Facebook Google+