Side effects in Unit Testing


Photo by Irvan Smith on Unsplash

Today I want to share my recent unpleasant experience. I believe, most of the developers understand it, because it’s something, that is a part of Unit Testing term definition. But we are all humans and sometimes people make mistakes. In my case, the cost of the mistakes, made by one of the developers on the project was couple of hours of work. I’m talking about side effects in Unit Testing.

I believe, that most of the developers understand this and accept it, when they first time read about Stubs, Mocks and all these instruments, which are provided by testing framework or library. In my case it was popular library SinonJS, but I believe it’s not a super-important fact, because the root cause is not the library itself.

When you are writing a Unit Test you always try to think about several important steps:

  1. Creating Mock objects for all dependencies of tested functionality;
  2. Creating the instance of object, which should be tested;
  3. Creating Stubs for functions, which should not be tested in scope of particular test or have undesirable side effects;
  4. Assertions;
  5. Restoring of system’s state to original state.

The last step is basically the most important and (surprise-surprise) the most risky. Why? Because if you did something wrong on the one of the first three steps – your assertions probably fail or exception will be thrown (because state of the system is different from real production usage, hence unexpected by the application) and you’ll notice it on your local development box with the high chances. But if you did something wrong on the last step tests will not fail. They may pass even on CI machine, everything can be good even for couple of months.

And at some point you can notice, that tests are failing on the CI machine randomly, re-trigger the build “fixes” the problem (but it’s not true). The real deal is that order of tests on CI machine is random most of the time and your tests now are dependent on this order.

Personally I think we should differentiate two types of mistakes:

  • Missed/Hidden global state;
  • Incorrect usage of Stub/Mock.

On my current project we have more than 5000 unit tests for JavaScript functionality, so I was scared a little bit. The first root cause in the list above – is really a tough one. Because in the most cases one developer is not aware about all the functionality, the global states and so on and so forth (my case, by the way). And in my opinion only replication of the same order, which produced an error on the server, debugging it and analyzing the global state can help. Fortunately for me it was the second case. It’s a little bit easier to replicate, understand and fix.

Before we dive into the details of the issue I want to give some background on the interesting functionality of SinonJS – sandbox. Basically it’s some instance, which keeps tracking all the Stubs/Mocks you are creating. That’s very handy, because you can restore the state of the entire system by calling sandbox.restore();. It’s very intuitive and helpful functionality. Let me show you one small code sample, I believe it will be more than enough:

var sandbox = sinon.sandbox.create({
properties: ['spy', 'stub', 'mock', 'server', 'requests'],
useFakeServer: false
});

var testInstance = new TestView();

sandbox.stub(testInstance, 'someBooleanMethod').returns(true);
sandbox.stub(testInstance, 'someIntegerMethod').returns(42);
sandbox.stub(TestView, 'someStaticMethod').returns('Important literal.');

// Act and Assert phases go here.

sandbox.restore();

Of course you’ll not write tests exactly the same way. There are special life cycle events, like setup or teardown which should be used for sandbox setup and restore, etc. But as example it should be fine.

Basically you can see three stubs are created here. It’s important to understand, that use of sandbox is not critical for first and second lines, because stubs are created like own properties of instance and will be destroyed anyway with instance. You can re-write these two lines in a little bit different way:

testInstance.someBooleanMethod = sandbox.stub().returns(true);
testInstance.someIntegerMethod = sandbox.stub().returns(42);

The logic of the test will not change. Moreover, you can replace sandbox.stub() with sinon.stub(). But for the third one things are getting different. And now we are really close to the issue I saw on the CI machine. At some point tests become failing with exception: “Attempted to wrap $.modal.close which is already wrapped” ($.modal is one of the jQuery UI component and it’s not important in particular topic). I don’t want to keep intrigue for too long and show you the code:

// One of the tests:
$.modal.close = sandbox.stub();

// ...
// Another test:
sandbox.stub($.modal, 'close'); // Exception will be thrown by SinonJS

It’s important to understand, what is the difference between first and second usages of sandbox.stub. When you are calling stub function with parameters SinonJS – it stores all information about this stub into the sandbox instance: the object, the original function. Hence when calling sandbox.restore it can restore the original state of all affected objects. When you are calling $.modal.close = sandbox.stub(); that way – sandbox knows nothing about $.modal, about original close function, hence cannot restore state of the object.

Hopefully it’s clear, what the issue was, how to find similar issues in your solution and you find this article worth reading. At the end I want to share with you last interesting fact: the incorrect usage of sandbox.stub was committed more than a half year ago. Tests were working fine for a couple of months and nobody noticed this side effect. That’s why I think, this is really important and worth to consider when you are writing new tests or reviewing someone’s pull requests.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s