Rerunning all Unit Tests with Different UI Culture

I was recently made aware that some unit tests for Kentor.AuthServices were failing on non-English computers. To handle that, I set up an Azure VM with Swedish installed and made a special unit test that would run all other tests with different UI cultures.

When I first understood that I had tests that were broken when run on non-English computers I of course felt that it should be fixed. The tests should not only run with other languages to enable developers from other countries. The tests should of course be possible to be run and used to find problems if someone reports errors on computers having a special language installed. There’s quite a few places in the code with string formatting and it can differ with different cultures, causing hard to find problems.

What I did was to write a special unit test that finds all other unit tests in the code and runs them with different UI culture. The unit tests are found using LINQ and Reflection (it’s an awesome combination that’s extremely powerful) and then they are run with reflection.

Finding the Tests

The MsTest framework provides ability to run certain methods before and after the tests, so I had to get hold of those too. The linq query I made finds everything needed.

  • All classes decorated with the [TestClass] attribute.
  • The parameterless constructor for each class to be able to instantiate it.
  • The method decorated with the [ClassInitialize] attribute, if any.
  • The method decorated with the [TestInitialize] attribute, if any.
  • All methods decorated with the [TestMethod] attribute, except those decorated with the [NotReRunnable] attribute.
var testClasses = (from t in Assembly.GetExecutingAssembly().DefinedTypes
    where t.GetCustomAttribute<TestClassAttribute>() != null
    select new
        Constructor = t.GetConstructor(new Type[] { }),
        ClassInit = t.GetMethods().Where(
        m => m.GetCustomAttribute<ClassInitializeAttribute>() != null).SingleOrDefault(),
        TestInit = t.GetMethods().Where(
        m => m.GetCustomAttribute<TestInitializeAttribute>() != null).SingleOrDefault(),
        TestCleanup = t.GetMethods().Where(
        m => m.GetCustomAttribute<TestCleanupAttribute>() != null).SingleOrDefault(),
        ClassCleanup = t.GetMethods().Where(
        m => m.GetCustomAttribute<ClassCleanupAttribute>() != null).SingleOrDefault(),
        TestMethods = t.GetMethods().Where(
        m => m.GetCustomAttribute<TestMethodAttribute>() != null
        && m.GetCustomAttribute<NotReRunnableAttribute>() == null).ToList()

The NotReRunnableAttribute class is a simple empty attribute class that is used to mark those test methods that can’t be run twice. In the Kentor.AuthServices library there is some protection against replay attacks, which means that certain test data cannot be reused without restarting the entire test process. Those methods are marked with the [NotReRunnable] attribute and are excluded from the culture testing. Fortunately, none of those methods contained any exception message content checking.

The test that contains the reruns itself is also marked as [NotReRunnable], to not cause an infinite loop when the test calls itself.

Actually it took some time until I understood that was the reason for the tests taking so long. I was even on the way to write a Stack Overflow question on why the foreach loop would never move on to the next culture. Once I found out (after nearly an hour of debugging) I can’t tell how stupid I felt.

Stopped laughing? Good. Then we’ll move on with how to run the tests.

Running the Tests

The tests are run in a loop, that changes the UI Culture to the specified values and tries them all. This means that if any other test is failing this test will also (falsely) fail, but I can take that. The important thing with this test is to make sure that it works on all cultures.

var cultures = new string[] { "en-US", "sv-SE" };
var originalUICulture = Thread.CurrentThread.CurrentUICulture;
var emtpyObjArray = new object[] { };
  foreach (var culture in cultures)
    Thread.CurrentThread.CurrentUICulture = new CultureInfo(culture);
    foreach (var c in testClasses)
      var instance = c.Constructor.Invoke(emtpyObjArray);
      if (c.ClassInit != null)
        c.ClassInit.Invoke(instance, emtpyObjArray);
      foreach (var m in c.TestMethods)
        if (c.TestInit != null)
          c.TestInit.Invoke(instance, emtpyObjArray);
        m.Invoke(instance, emtpyObjArray);
        if (c.TestCleanup != null)
          c.TestCleanup.Invoke(instance, emtpyObjArray);
      if (c.ClassCleanup != null)
        c.ClassCleanup.Invoke(instance, emtpyObjArray);
  Thread.CurrentThread.CurrentUICulture = originalUICulture;

With the test in place I quickly found what unit tests that were incorrectly culture sensitive and could fix them. In my case, they were all checks on the content of NullArgumentException messages. I still do have to check the contents, to make sure that it is the right parameter that is reported as being null, but instead of checking the entire message I now just check that the message contains the parameter name.

Action a = () => identity.ToSaml2Assertion("foo");
// Old cultural sensitive testing.
a.ShouldThrow<ArgumentNullException>().WithMessage("Value cannot be null.\r\nParameter name: identity");
// New cultural insensitive testing (as I had written it).
// New, smarter cultural insensitive testing (as Albin Sunnanbo supplied 
// in a PR after having read this post).

The entire test is available on GitHub.

Leave a comment

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.