Parameterized unit tests come in handy when you have a simple API that you want to test in the same way with multiple inputs and expected outputs. Instead of creating duplicates of a test with small variations, it’s often nicer to have a single test that runs multiple times with different parameters – a parameterized test. This post examines some simple but problematic ways of writing parameterized tests in Ruby and then explains how the param_test gem can make them more robust.

Assertions in Loops Considered Harmful

Let’s say we’re unit testing a simple method like this one:

# whitespace.rb
# Whether a given string includes any whitespace.
def includes_whitespace?(string)
  not string.match(/\s/).nil?
end

Aside: All code used in this post can be found here.

It will be tempting to write a unit test like this:

# bad_looping_test.rb
require 'whitespace'
require 'test/unit'

class BadLoopingTest < Test::Unit::TestCase
  def test_includes_whitespace
    inputs = ["hello world", "foo bar", "foo", "bar\n"]
    inputs.each do |input|
      assert includes_whitespace? input
    end
  end
end

This can be considered a parametrized test. We’re testing the same assertion against 4 different inputs. But: Don’t do it this way! Putting an assertion inside a loop should almost always be avoided. If the test fails you might not be able to tell at which of the 4 input strings it failed. What’s more, any failure aborts the loop, so subsequent inputs won’t be tested in the same run, hiding further failures until later.

To wit, the above test actually does fail and the output is most unhelpful:

$ ruby -I"test/blog" test/blog/bad_looping_test.rb
# Running tests:

F

Finished tests in 0.000565s.

1) Failure:
 test_includes_whitespace(BadLoopingTest) [test/blog/bad_looping_test.rb:8]:
 Failed assertion, no message given.

1 tests, 3 assertions, 1 failures, 0 errors, 0 skips

As a remedy you could add an explicit failure message to the assertion that includes the input being tested, but since that’s optional and onerous people will often conveniently forget about it.

Parameterized Tests with ActiveSupport’s test Method

ActiveSupport’s unit test extensions add a declarative test method to test cases that allows for a relatively simple way to improve on this:

# better_declarative_test.rb
require 'whitespace'
require 'active_support/test_case'
require 'test/unit'

class BetterDeclarativeTest < ActiveSupport::TestCase
  ["hello world", "foo bar", "foo", "bar\n"].each do |input|
    test "#{input} includes whitespace" do
      assert includes_whitespace? input
    end
  end
end

This is better. It gently nudges you towards writing a good test description that will also help in identifying failures. The test now fails in a more explicit way:

$ ruby -I"test/blog" test/blog/better_declarative_test.rb
# Running tests:

..F.

Finished tests in 0.001183s.

1) Failure:
 test_foo_includes_whitespace(BetterDeclarativeTest) [test/blog/better_declarative_test.rb:8]:
 Failed assertion, no message given.

4 tests, 4 assertions, 1 failures, 0 errors, 0 skips

What actually happened here? ActiveSupport’s test generates a test method named after the test description it was given. Since we called test 4 times, once for each of the 4 inputs, 4 methods were generated. From the name of the failing method test_foo_includes_whitespace we know that the offending input is "foo".

Aside: While the simplicity of xUnit-style unit tests is great and surely one of the reasons it’s so popular across a wide range of programming languages, it’s also a hindrance in this case. The only metadata available about a test is effectively its method name. It would be great to have an explicit test description metadata field without the restrictions of a method name. This is something that newer testing frameworks such as RSpec got right (though I have other issues with RSpec).

Unfortunately, using test this way isn’t a general solution. Some tests written using this pattern might not run at all. The main problem is that method names must be unique within a test case (or any class, really). So if you neglect to include all input parameters in the description or, more subtle, if some of your inputs boil down to identical method names, your test will abort early with a somewhat obscure exception.

As an illustration, consider what happens if we modify the test inputs to be:

# bad_declarative_test.rb
require 'whitespace'
require 'active_support/test_case'
require 'test/unit'

class BadDeclarativeTest < ActiveSupport::TestCase
  ["  ", "\n"].each do |input|
    test "#{input} includes whitespace" do
      assert includes_whitespace? input
    end
  end
end

We have a string consisting of two spaces and a string consisting of a new line as inputs. Running this will yield:

$ ruby -I"test/blog" test/blog/bad_declarative_test.rb
~/.rvm/gems/ruby-1.9.3-p327/gems/activesupport-3.2.11/lib/active_support/testing/declarative.rb:28:in `test': test__includes_whitespace is already defined in BadDeclarativeTest (RuntimeError)
from test/blog/bad_declarative_test.rb:7:in `block in <class:BadDeclarativeTest>'
from test/blog/bad_declarative_test.rb:6:in `each'
from test/blog/bad_declarative_test.rb:6:in `<class:BadDeclarativeTest>'
from test/blog/bad_declarative_test.rb:5:in `<main>'

This failed to even generate the test methods because test collapses all whitespace to a single underscore when converting the description to a method name. Both inputs result in the same method name, so upon trying to generate the second test method, the above exception is raised.

Robust Parameterized Tests with param_test

I was bumping up against these issues often enough that I packaged up a simple solution in the gem param_test. It adds a single class method param_test to ActiveSupport::TestCase that makes parametrized tests as simple as can be, enforces a test description that includes all input parameters and guarantees that the generated method names are unique.

Rewritten using the param_test gem, the test looks like this:

# single_param_test.rb
require 'whitespace'
require 'param_test'
require 'test/unit'

class SingleParamTest < ActiveSupport::TestCase
  param_test "%s includes whitespace",
  ["hello world", "foo bar", "foo", "bar\n"] do |input|
    assert includes_whitespace? input
  end
end

Note that for the test description param_test uses string formatting (as per Ruby’s format/sprintf method), rather than string interpolation (where you type out explicit variable names, as in "#{input}"). For each test the input parameter will be substituted for the %s.

You can have multiple parameters by test, for example if you want to test both outcomes of includes_whitespace? in one go:

# multiple_params_test.rb
require 'whitespace'
require 'param_test'
require 'test/unit'

class MultipleParamsTest < ActiveSupport::TestCase
  param_test "%s includes whitespace is %s", [
    ["hello world", true],
    ["foo bar", true],
    ["foo", false],
  ] do |input, expected|
    assert_equal expected, includes_whitespace?(input)
  end
end

To be a bit more formal, param_test takes 3 arguments:

1. The test description template. Input parameters will be applied to this string in order using Ruby string formatting. Generally you’ll just want to use the %s format sequence which will be replaced with a string version of the parameter.
2. A list of parameters. This can be a simple flat list if there is only one input parameter per test. For multiple parameters per test it can also be a list of lists, where every inner list constitutes the parameter set for one test. All parameter sets must be of the same length.
3. A block making up the actual test body. This becomes the body of every generated test. The block should have as many arguments a there are parameters in each set.

Behind the scenes param_test catches any mistakes and inconsistencies in your input early. Explicit exceptions will be raised if there is a mismatch between the number of input parameters, placeholders in the description template and arguments to the block. If two tests would end up with the same generated method name, uniqueness is still guaranteed by appending a counter to the method name.

Install the gem and try it out!