Mutate Your Code and Test Your Tests

When 100% code coverage is not enough, there’s Mutant.

I have some crazy friends. One of them used to randomly delete lines from his code and run tests, just to see which ones turned “red.”

“Some people just want to see the world burn”

When you think about it, the practice makes some sense.  Sometimes, we or our teammates end up adding buggy code that goes unnoticed simply because “the tests were green.”  So, in addition to testing your code, you must also make sure that your tests are correct.  In other words, you should also “test your tests.”  The question is how to do this while meeting deadlines and having somewhat of a life.

Well, if you are using Ruby 1.9 and RSpec, I have good news for you. There is a gem, appropriately named Mutant, that “mutates” your code and runs your RSpec tests.

This posts shows you how to run Mutant by way of a short example.

Writing the Code

Suppose we have a Post class and we want the feature to add Comments (comments are strings, in order to keep this example simple). We also want to show a comment count by Post. Here’s what the code would look like,

class Post
  attr_reader :comments, :comments_count

  def initialize
    @comments = []
    @comments_count = 0
  end

  def add_comment(comment)
    @comments << comment
    @comments_count = @comments.size
    self
  end
end

Writing the Tests

And here’s the corresponding RSpec,

describe Post, '#add_comment' do
  subject(:post) { Post.new }
  let(:comment) { "Lorem ipsum dolor sit amet" }

  it "return self" do
    expect(post.add_comment(comment)).to be(post)
  end

  it "add comment to post" do
    post.add_comment(comment)
    expect(post.comments).to include(comment)
  end
end

When we run our tests, all of them are “green.” Ant that’s great.

Not only that, but, using a code coverage tool like simplecov, we can verify that we have a 100% code coverage.

And normally, that’s where it ends.

Mutating Our Code

Before running Mutant, there are two main concepts that you need to know, kill and alive.

• kill: Mutant modify a line of code and the test fails.
• live: Mutant modify a line of code and the test passes.

Obviously, we expect tests to fails with mutated code, so a live count of zero is what we should expect.

Now let’s run Mutant and see what happens.

$ mutant -I lib -r post --rspec-dm2 ::Post#add_comment
Mutant configuration:
Matcher:   #<Mutant::Matcher::ObjectSpace scope_name_pattern=/\APost(\z|::)/>
Filter:    Mutant::Mutation::Filter::ALL
Strategy:  #<Mutant::Strategy::Rspec::DM2>
Subject: Post#add_comment:/Users/omarvargas/Posts/mutant/lib/post.rb:9

...

!! Mutant alive: rspec:Post#initialize:/Users/omarvargas/Posts/mutant/lib/post.rb:4:078b4 !!!
@@ -1,5 +0,4 @@
 def initialize
   @comments = []
-  @comments_count = @comments.size
 end

...

Took: (0.18s)

subjects:   1
mutations:  10
noop_fails: 0
kills:      7
alive:      3
mtime:      1.83s
rtime:      2.09s

Notice that we got an alive count of 3. This indicates that there is something wrong with our test. Let’s review the Mutant output to figure out what happened.

Reviewing the Code

The first Mutant alive warning shows something like this,

!!! Mutant alive: rspec:Post#add_comment:/Users/omarvargas/Posts/mutant/lib/post.rb:9:b6b83 !!!
@@ -1,6 +0,5 @@
 def add_comment(comment)
   ((@comments) << (comment))
-  @comments_count = @comments.size
   self
 end

Now we can see the mistake. We are using comments_count as a cache, but we are not testing that this variable is set correctly. This line is executed, so we get 100% coverage. But unfortunately, it not is tested.

Sometimes 100% code coverage is not good enough.

Now we can modify our spec file and add the missing test,

describe Post, '#add_comment' do
...
  it "set comments count properly" do
    post.add_comment(comment)
    expect(post.comments_count).to be(1)
  end
...
end

And run Mutant, again,

$ mutant -I lib -r post --rspec-dm2 ::Post#add_comment
...

subjects:   1
mutations:  10
noop_fails: 0
kills:      10
alive:      0
mtime:      1.93s
rtime:      2.19s

Now we are OK: we get an alive count of zero and, as expected, the tests fails with the mutated code.

We can relax now knowing that nobody can accidentally add bugs to our code just because “the tests were green.”

Happy Hacking!