Mutate Your Code and Test Your Tests

When 100% code coverage is not enough, there’s Mutant.
I have some crazy friends. One of them used to randomly delete lines from his code and run tests, just to see which ones turned “red.”
“Some people just want to see the world burn”
When you think about it, the practice makes some sense. Sometimes, we or our teammates end up adding buggy code that goes unnoticed simply because “the tests were green.” So, in addition to testing your code, you must also make sure that your tests are correct. In other words, you should also “test your tests.” The question is how to do this while meeting deadlines and having somewhat of a life.
Well, if you are using Ruby 1.9 and RSpec, I have good news for you. There is a gem, appropriately named Mutant, that “mutates” your code and runs your RSpec tests.
This posts shows you how to run Mutant by way of a short example.
Writing the Code
Suppose we have a Post
class and we want the feature to add Comments
(comments are strings, in order to keep this example simple). We also want to show a comment count by Post. Here’s what the code would look like,
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
<span class="k">class</span> <span class="nc">Post</span> <span class="kp">attr_reader</span> <span class="ss">:comments</span><span class="p">,</span> <span class="ss">:comments_count</span> <span class="k">def</span> <span class="nf">initialize</span> <span class="vi">@comments</span> <span class="o">=</span> <span class="o">[]</span> <span class="vi">@comments_count</span> <span class="o">=</span> <span class="mi">0</span> <span class="k">end</span> <span class="k">def</span> <span class="nf">add_comment</span><span class="p">(</span><span class="n">comment</span><span class="p">)</span> <span class="vi">@comments</span> <span class="o"><<</span> <span class="n">comment</span> <span class="vi">@comments_count</span> <span class="o">=</span> <span class="vi">@comments</span><span class="o">.</span><span class="n">size</span> <span class="nb">self</span> <span class="k">end</span> <span class="k">end</span> |
Writing the Tests
And here’s the corresponding RSpec,
1 2 3 4 5 6 7 8 9 10 11 12 13 |
<span class="n">describe</span> <span class="no">Post</span><span class="p">,</span> <span class="s1">'#add_comment'</span> <span class="k">do</span> <span class="n">subject</span><span class="p">(</span><span class="ss">:post</span><span class="p">)</span> <span class="p">{</span> <span class="no">Post</span><span class="o">.</span><span class="n">new</span> <span class="p">}</span> <span class="n">let</span><span class="p">(</span><span class="ss">:comment</span><span class="p">)</span> <span class="p">{</span> <span class="s2">"Lorem ipsum dolor sit amet"</span> <span class="p">}</span> <span class="n">it</span> <span class="s2">"return self"</span> <span class="k">do</span> <span class="n">expect</span><span class="p">(</span><span class="n">post</span><span class="o">.</span><span class="n">add_comment</span><span class="p">(</span><span class="n">comment</span><span class="p">))</span><span class="o">.</span><span class="n">to</span> <span class="n">be</span><span class="p">(</span><span class="n">post</span><span class="p">)</span> <span class="k">end</span> <span class="n">it</span> <span class="s2">"add comment to post"</span> <span class="k">do</span> <span class="n">post</span><span class="o">.</span><span class="n">add_comment</span><span class="p">(</span><span class="n">comment</span><span class="p">)</span> <span class="n">expect</span><span class="p">(</span><span class="n">post</span><span class="o">.</span><span class="n">comments</span><span class="p">)</span><span class="o">.</span><span class="n">to</span> <span class="kp">include</span><span class="p">(</span><span class="n">comment</span><span class="p">)</span> <span class="k">end</span> <span class="k">end</span> |
When we run our tests, all of them are “green.” Ant that’s great.
Not only that, but, using a code coverage tool like simplecov, we can verify that we have a 100% code coverage.
And normally, that’s where it ends.
Mutating Our Code
Before running Mutant, there are two main concepts that you need to know, kill
and alive
.
kill:
Mutant modify a line of code and the test fails.live:
Mutant modify a line of code and the test passes.
Obviously, we expect tests to fails with mutated code, so a live
count of zero is what we should expect.
Now let’s run Mutant and see what happens.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 |
$ mutant -I lib -r post --rspec-dm2 ::Post#add_comment Mutant configuration: Matcher: #<Mutant::Matcher::ObjectSpace scope_name_pattern=/\APost(\z|::)/> Filter: Mutant::Mutation::Filter::ALL Strategy: #<Mutant::Strategy::Rspec::DM2> Subject: Post#add_comment:/Users/omarvargas/Posts/mutant/lib/post.rb:9 ... <span class="gs">!! Mutant alive: rspec:Post#initialize:/Users/omarvargas/Posts/mutant/lib/post.rb:4:078b4 !!!</span> <span class="gu">@@ -1,5 +0,4 @@</span> def initialize @comments = [] <span class="gd">- @comments_count = @comments.size</span> end ... Took: (0.18s) subjects: 1 mutations: 10 noop_fails: 0 kills: 7 alive: 3 mtime: 1.83s rtime: 2.09s |
Notice that we got an alive
count of 3. This indicates that there is something wrong with our test. Let’s review the Mutant output to figure out what happened.
Reviewing the Code
The first Mutant alive
warning shows something like this,
1 2 3 4 5 6 7 |
<span class="gs">!!! Mutant alive: rspec:Post#add_comment:/Users/omarvargas/Posts/mutant/lib/post.rb:9:b6b83 !!!</span> <span class="gu">@@ -1,6 +0,5 @@</span> def add_comment(comment) ((@comments) << (comment)) <span class="gd">- @comments_count = @comments.size</span> self end |
Now we can see the mistake. We are using comments_count
as a cache, but we are not testing that this variable is set correctly. This line is executed, so we get 100% coverage. But unfortunately, it not is tested.
Sometimes 100% code coverage is not good enough.
Now we can modify our spec file and add the missing test,
1 2 3 4 5 6 7 8 |
<span class="n">describe</span> <span class="no">Post</span><span class="p">,</span> <span class="s1">'#add_comment'</span> <span class="k">do</span> <span class="o">.</span><span class="n">.</span><span class="o">.</span> <span class="n">it</span> <span class="s2">"set comments count properly"</span> <span class="k">do</span> <span class="n">post</span><span class="o">.</span><span class="n">add_comment</span><span class="p">(</span><span class="n">comment</span><span class="p">)</span> <span class="n">expect</span><span class="p">(</span><span class="n">post</span><span class="o">.</span><span class="n">comments_count</span><span class="p">)</span><span class="o">.</span><span class="n">to</span> <span class="n">be</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span> <span class="k">end</span> <span class="o">.</span><span class="n">.</span><span class="o">.</span> <span class="k">end</span> |
And run Mutant, again,
1 2 3 4 5 6 7 8 9 10 |
<span class="nv">$ </span>mutant -I lib -r post --rspec-dm2 ::Post#add_comment ... subjects: 1 mutations: 10 noop_fails: 0 kills: 10 alive: 0 mtime: 1.93s rtime: 2.19s |
Now we are OK: we get an alive
count of zero and, as expected, the tests fails with the mutated code.
We can relax now knowing that nobody can accidentally add bugs to our code just because “the tests were green.”
Happy Hacking!