Mutate Your Code and Test Your Tests

When 100% code coverage is not enough, there’s Mutant.

I have some crazy friends. One of them used to randomly delete lines from his code and run tests, just to see which ones turned “red.”

“Some people just want to see the world burn”

When you think about it, the practice makes some sense.  Sometimes, we or our teammates end up adding buggy code that goes unnoticed simply because “the tests were green.”  So, in addition to testing your code, you must also make sure that your tests are correct.  In other words, you should also “test your tests.”  The question is how to do this while meeting deadlines and having somewhat of a life.

Well, if you are using Ruby 1.9 and RSpec, I have good news for you. There is a gem, appropriately named Mutant, that “mutates” your code and runs your RSpec tests.

This posts shows you how to run Mutant by way of a short example.

Writing the Code

Suppose we have a Post class and we want the feature to add Comments (comments are strings, in order to keep this example simple). We also want to show a comment count by Post. Here’s what the code would look like,

Writing the Tests

And here’s the corresponding RSpec,

When we run our tests, all of them are “green.” Ant that’s great.

Not only that, but, using a code coverage tool like simplecov, we can verify that we have a 100% code coverage.

And normally, that’s where it ends.

Mutating Our Code

Before running Mutant, there are two main concepts that you need to know, kill and alive.

• kill: Mutant modify a line of code and the test fails.
• live: Mutant modify a line of code and the test passes.

Obviously, we expect tests to fails with mutated code, so a live count of zero is what we should expect.

Now let’s run Mutant and see what happens.

Notice that we got an alive count of 3. This indicates that there is something wrong with our test. Let’s review the Mutant output to figure out what happened.

Reviewing the Code

The first Mutant alive warning shows something like this,

Now we can see the mistake. We are using comments_count as a cache, but we are not testing that this variable is set correctly. This line is executed, so we get 100% coverage. But unfortunately, it not is tested.

Sometimes 100% code coverage is not good enough.

Now we can modify our spec file and add the missing test,

And run Mutant, again,

Now we are OK: we get an alive count of zero and, as expected, the tests fails with the mutated code.

We can relax now knowing that nobody can accidentally add bugs to our code just because “the tests were green.”

Happy Hacking!