Adding an Overload is a Breaking Change

Adding functionality to a library, without touching the existing code should be safe for clients, shouldn’t it? Unfortunately not, adding another overload to a library can be a breaking change. It might work when the library is updated, but suddenly break when the client is recompiled – even though no code was changed. That’s nasty, isn’t it?

When working with Kentor.AuthServices I’ve had to think through versioning more in detail than I’ve done before. When is the right time to go to 1.0? What is the difference between going from 0.8.2 to 0.8.3 or 0.9.0? When researching I found that the answer to all versioning questions is Semantic Versioning.

A version is on the form Major.Minor.Patch.

  • Major is increased if there are breaking changes. The other numbers are reset.
  • Minor is increased if there is added functionality that is non breaking. The patch number is reset.
  • Patch is increased for backwards compatible bug fixes.

The terms breaking changes and backwards compatibile are keys to the definitions, so to use semantic versioning requires keeping tight control of what changes are breaking and not. In most cases it is quite simple, but there are a few pitfalls that I’ll explore in this post.

The background to this post is that I listened to Scott Meyers’ NDC talk Effective Modern C++ and was reminded on how C++ programmes have to keep track of all kinds of nastiness in the language that might turn into hard to track-down bugs. C# is a lot easier in many ways, with way fewer pitfalls, but sometimes I think we make it to simple to ourselves. C++ developers are often quite good at the language details because they have to. Being a C# developer it is possible to survive for much longer without knowing those details, but being ignorant of them will eventually result in nasty bugs. That eventuality will probably not happen on a lazy Tuesday morning with a lot of time to fix it. It will happen a late Friday afternoon, right before the important production deployment scheduled for the weekend…

As C# developers I think that we should have a look at the C++ community and see what we can learn from them. So let’s dive into some code and see how we can break things.

The Baseline Code

Let me introduce version 1 of my sample library.

public static class Utility
{
  public static void Method(object obj)
  {
    Console.WriteLine("Utility.Method(object)");
  }  
 
  public static void DefaultMethod(int i = 7)
  {
    Console.WriteLine("Utility.DefaultMethod({0})", i);
  }
}

There is also a small client program that uses the library.

public static class Program
{
  public static void Main(string[] args)
  {
    Utility.Method(17);
    Utility.DefaultMethod();
  }
}

When built and run, the program outputs the expected values.

Utility.Method(object)
Utility.DefaultMethod(7)

Updating the Library

Later on, the library is updated.

public static class Utility
{
  public static void Method(object obj)
  {
    Console.WriteLine("Utility.Method(object)");
  }  
 
  public static void Method(int i)
  {
    Console.WriteLine("Utility.Method(int)");
  }
 
  public static void DefaultMethod(int i = 42)
  {
      Console.WriteLine("Utility.DefaultMethod({0})", i);
  }
}

A new overload is added and the default value of the parameter to DefaultMethod is changed. This version is compatible with the old one, so the library dll can be changed without rebuilding the client application. When run, the client application produces the following output.

Utility.Method(object)
Utility.DefaultMethod(7)

It’s exactly the same as before. Updating the library to a new version is not affecting the output of the program. But we’ve unintentionally put ourselves in a very unstable situation.

Rebuilding the Client

Imagine that you have just taken over the maintenance of this program. You have a production installation that is working. You have the source that the program was built from and you have the download link to the latest version of the library. Everything looks in order, so you start setting up your development environment, downloads the library and builds the code. But it’s not producing the same results. Whatever you try is in vain. You can’t build the code and produce the same results. The state of the system when you took over was unstable and cannot easily be reproduced.

What has happened is that an apparently innocent rebuild of identical source code changes the behaviour of the program. If rebuilt, the program now produces the following output.

Utility.Method(int)
Utility.DefaultMethod(42)

A different overload is called and the default value is changed, even though the source code is identical.

The problem is in the compilation and linking against the external library of different versions. The overload resolution and default value handling is managed at compile time and uses the information of the external library version used at the time of compilation. Compiling against a different version gives different results; even though running running with a later version of the library doesn’t.

Compile Time Library Linking

It is during compilation that the overload resolution takes place. It checks the available overloads in the library and then hard wires that selection into the compiled output. It doesn’t matter that the run-time version used offers other overloads, the compiled program knows what overload it should call. It also knows what type conversions need to be done to be able to call that overload. In this case, the int will have to be boxed into an object.

The same is true for the default value. Even though the default value is listed along with the utility method, it isn’t handled that way by the compiler. This is how the compiler reasons.

The program is calling Utility.DefaultMethod() but it’s missing an argument. That’s a syntax error. Ha! I can blow the compilation up in pieces with a syntax error! No, wait, there’s something in the method declaration. Ahh a default value. Ok, I’ll take it. I’ll change the call to Utility.DefaultMethod(7) and then I can compile it.

I’ve always suspected that compilers really like blowing up the compilation with syntax error messages and now I have it confirmed.

The important thing however is that the compiler is effectively changing the call to hard wire the default value into the call site. Even though the compiled library is replaced to one with a new default value that doesn’t matter as the default value is extracted during compilation.

Breaking Changes

Ok, now we’ve seen that overload resolution and default values can turn seemingly innocent changes to dependency nightmares. To avoid getting into trouble there are three simple practices that will save the day.

  1. New overloads and changed default values are breaking changes to library compatibility and requires a new major version.
  2. Using default values should be avoided in public facing APIs.
  3. Always make a complete rebuild and test it before deploying when dependencies are updated. This avoids the scenario above where a rebuild without code changes alters the behaviour of the program.

The complete code for this post is available on github. Run the test.bat file to compile and run the different code versions in the same order as used in this post.

  • Pingback: The Morning Brew - Chris Alcock » The Morning Brew #1649 on 2014-07-11
  • Aaron Dandy on 2014-07-11

    I agree that changing a default value should be considered breaking. I think a great way to handle that problem is to only use default(T) (0 or null) as a default value. Any other situation should be a regular parameter with documentation.

    I disagree with your statement on overloads always being breaking however. Certainly they can be but in some circumstances (Method(int i) is a great example) there really should not be a problem. Sure you could code the Method(object) signature to work correctly while Method(int) formats your primary partition but perhaps it would be better if Method(int) did NOT format C:\ . Instead I expect Method(object) when given an int to behave the same as Method(int) , especially considering that int is derived from object. So again, it can be a breaking change if you really want to do harm but in a sane code-base it does not have to be breaking.

    • Anders Abel on 2014-07-11

      Thank your for your comments. You are right in that new overloads are not always a breaking change, but they can be and that is what I wanted to point out with this post.

      In fact at the start of the post I state that adding another overload to a library can be a breaking change. It doesn’t have to be, but if you add a new overload you should be aware of the situations where they are breaking to be able to avoid them.

      I think you nailed it with the “in a sane code-base” comment. A sane code-base has a robust design that lets you introduce new overloads without ambiguities and in case a more specialised overload is offered it of course should be completely compatible with the general one.

  • Max Battcher on 2014-07-11

    The trick here is the “default value overload” problem: truly new overloads are backwards compatible in the CLR (with small issues in subtle casting problems, but those don’t break existing compiled code and only complicate recompiles…), but the problem with the pretty default-providing syntax in C#/VB is that while these can look like “simply changing the overloads”, these don’t actually create new overloads, they replace the existing overload. You can see this in the generated IL: a defaulted parameter is represented as a normal parameter with an attribute defining the default value, not (as many people would expect) generating multiple overloads for each combination with and without the defaulted parameter.

    So yes, overloads are traditionally backwards compatible, but default parameters are not! Beware when using default parameters in API designs.

    It’s just unfortunately confusing that default parameters look so much like traditional method overloading, and yet under the hood work so differently. (Primarily because default parameters were not designed into the CLR, but as syntactic sugar and metadata in the implementing languages…)

    • Justin on 2014-07-11

      There are cases where adding an overload can be a breaking change even without default values. Consider the following class:

      public class Foo
      {
        public void Method(Guid? id) {}
      }

      Used like this:

      new Foo().Method(null);

      Now version 2 comes out with a new overload:

      public class Foo
      {
        public void Method(Guid? id) {}
        public void Method(int? id) {}
      }

      The compiler can no longer resolve which method to call for null. Granted, this isn’t a great design, but it’s valid.

      • Anders Abel on 2014-07-11

        Thank you for your example of the new overload breaking the build. It is yet another case that shows that new overloads should be handled carefully.

    • Anders Abel on 2014-07-11

      Thank you Max for the comments and details about the default implementation.

      Looking at code written by Microsoft (I’ve mostly read ASP.NET MVC and Entity Framework code) they avoid default parameters and instead use overloads.

      void SomeMethod() { SomeMethod(0); }
      void SomeMethod(int i) { SomeMethod(i, null); }
      void SomeMethod(int i, string name) { // Actual implementation }

      Only the method that has the complete set of arguments has an actual implementation. All the other ones just call the next more advanced, supplying a default value. This brings the default value completely under control of the library and avoids the pitfalls of default values.

  • Albin Sunnanbo on 2014-07-13

    In rare cases bug fixes can also be breaking changes. If it fixes something it changes the observable behavior.
    I have used a library with bugs in it where we created a work around. When we later received a bug fix version of the library our work around broke even though there was no visible API change.

    As a library consumer you should always treat library updates as possible breaking changes and let them go through your regression tests before entering production.

    As a library producer however you should of course works as hard as possible to limit and communicate breaking changes. Both changes on the visible API surface (including thrown exceptions and return values) and observable side effects.
    With that in mind you also realize that it is a virtue to keep your API surface tight and clean from side effects. Or in other words: avoid dependencies and interactions across the API surface.

Software Development is a Job – Coding is a Passion

I'm Anders Abel, a systems architect and developer working for Kentor in Stockholm, Sweden.

profile for Anders Abel at Stack Overflow, Q&A for professional and enthusiast programmers

The complete code for all posts is available on GitHub.

Popular Posts

Archives

Series

Powered by WordPress with the Passion for Coding theme.