Interviews as no-code zones

I wrestled with many new challenges during the last three years, as Head of Engineering at Placemeter building a product and an engineering team from scratch, and one of the toughest ones was hiring. I had little experience initially. How do you get good candidates to even apply at an early-stage startup? How do you decide who to interview? I thought at least I knew how to conduct interviews, since I had spent five years at Google where I averaged about one interview per week.

The specific format of the Google software engineer interview during that time (2007 – 2012) was impressed on me quite strongly: The interviewer states a problem (generally some form of algorithmic challenge), asks the candidate to describe possible solutions verbally and then to write out a solution in code on a whiteboard or on paper. One interview had to fit into 45 minutes, so these problems couldn’t be all that involved.

That last part of writing code in an interview is colloquially known as whiteboarding. I like to use the term live coding since it may happen in a code editor or specialized tool rather than on a literal whiteboard.

img_20120622_110044

Because Google was obviously very successful with it and it was part of my routine for so long, I never questioned this interview format. When I found myself in charge of the interview process at a startup, when I actually had to stop and think about the qualities I want to evaluate in a software engineer, I soon realized the limitations of this format, particularly of the live coding aspect.

My main objection to live coding is that it’s a highly artificial activity which does not resemble anything a software engineer does day-to-day. In the real world, code is written on a keyboard with the support of an IDE and StackOverflow. In the real world, code is not written under time pressure, especially not while also being watched over. In the real world, code is not written in small isolated chunks; it almost always has to integrate with an existing large system. Live coding is so unnatural, I think it very weakly predicts how well someone is going to perform at a tech startup.

I’m not claiming this as an original insight. Google has reportedly stopped relying so much on whiteboard coding, with more focus on just problem solving. Other people have raised issues with live coding, for example Laurie Voss in this excellent post on interviewing.

Programming is easy

Apart from the artifice of live coding, there is a deeper problem with placing coding at the core of interviewing: Programming is easy.

Many people bristle at this idea, so let me explain. Here is a non-exhaustive list of things that I think are harder than programming but are often just as important when creating software:

  • Collaborating in a team. Working in a team requires nuanced communication on an ever-growing list of channels (meetings, email, Slack, git, issue trackers and so on). Being part of a team means making compromises, and that may be hard to swallow.
  • Product design and UI design. An engineer is rarely just a voiceless executing arm of a product manager, especially not in a startup. There has to be a feedback loop, with engineers for example reporting early when a feature is too hard to implement or a UI is too complex or underspecified. Developing a sense for product and UI design (or API design, in backend systems) doesn’t come easy to many.
  • Reading code. In any non-trivial codebase you are going to read a lot more code than you’re going to write. The corollary is that every bit of code is read many more times than it is written, so knowing how to organize and write code for readability is also important (and hard).
  • Debugging. The natural state of software is buggy. Reproducing bugs locally is hard, so is mastering debugging and profiling tools. It gets even trickier with production systems in the mix where you have to interpret live data, logging and monitoring output.

I can illustrate this well with my experience as an intern host. I’ve personally mentored about a dozen interns over the years. They were generally Computer Science undergrads, they were very bright and motivated, and they were already good programmers. I believe that interns get the most out of an internship if you put them to work on real features that are going to be launched to real users (no toy projects or internal-only tools). When I say they were good programmers, I mean that I would give them specs for such a feature, they would get to implementing for a couple of weeks or so, and then they would call me over to their computer to proudly demo what they presumed was the complete feature.

None of them initially realized that they may have completed most of the coding — which they were already capable of without much help from me — but they were far from done. Once they merged their code they would discover that it conflicted with somebody else’s ongoing work, and/or it broke an integration test, and/or when running against production data their code couldn’t handle some users’ messy data, and/or QA revealed their UI component can get into an inconsistent state, and/or spurious exceptions started happening when their code was deployed to production. Sorting out these issues would take at least as much time as they initially spent “programming”.

It’s a long way from a local demo of a feature to getting it into the hands of real users. Most of the actual mentoring I ever did happened during this stage of the software lifecycle, between the first local demo and the public launch. Not by accident this is precisely the stage into which most of the hard activities from above fall.

I see the advancement of a junior engineer to a senior engineer mainly as a shift of focus away from programming to those other, harder activities. Organizing a team of engineers for high productivity is about finding ways to emphasize those concerns in the overall development process.

In the no-code zone

If the correlation between live coding skills and actual engineering skills is weak and testing for coding will give you a one-dimensional picture of a candidate anyway — how should you interview engineers? Here are some techniques I found useful.

Take-home coding challenge. Obviously it’s necessary to evaluate coding skills somehow, but the test should be designed to minimize the drawbacks of live coding and to recreate realistic conditions as much as possible. I like to give candidates a coding challenge that a) they do at home in their own preferred development environment, b) has no deadline, c) and is building on some existing code. I think 2 to 4 hours is appropriate, so it takes some effort to design a challenge that is not too trivial but can be completed in that timeframe.

Take-home challenges have become pretty common in tech startups and can be controversial because of the time commitment required. I have never had a candidate refuse a challenge because I frame it like this: We think working in your own environment without a deadline is the best way for you to demonstrate your coding skills and we will not make you write code at any other point during the hiring process.

Various services have sprung up for administering take-home challenges, none of which I’ve tried. A very simple model worked for me: I share a new private GitHub repo with every candidate, containing instructions and the code they have to build on.

Design discussions. Whiteboards are better for drawing than for coding. I find a good way to ground a design discussion is to base it on a real system that a candidate would work on if hired. For example, I would sketch part of a system on the whiteboard and ask the candidate to design some extension to it. During the discussion I try to move from high-level architecture to very specific low-level details, including possibly UI or API design. If the conversation stalls I ask about tradeoffs and compromises. Under what circumstances would you use C++ as opposed to Python? How do your design decisions change if you are constrained by memory or by bandwidth?

Code reading. I have experimented with showing candidates a piece of code and asking concrete questions about what it does and how they could improve on it. Almost everybody I’ve tried this on has struggled with it, underscoring my point that reading code is really hard. I still think this can lead to revealing discussions.

**

Writing code during interviews is appealing because it produces something we can supposedly judge objectively. I think a carefully designed take-home challenge can satisfy this desire for a code artifact more effectively. Even then, code only says so much about how a candidate is going to fare at software engineering, which includes many activities other than programming. This leaves you with mostly design discussions during in-person interviews, the results of which are unfortunately more nuanced, more subjective, and harder to interpret.

Nobody says hiring is easy. Evaluating a person’s potential in a complex job like software engineering, after spending just a few hours with them, is inherently hard. It’s better to acknowledge that than to act as if there are fixed rules that always lead to the right hiring decision. You should be constantly questioning if your process is working.

I’ve rarely met an interviewee who likes live coding, so explicitly renouncing live coding may work to your advantage as a startup competing for scarce talent. I know I will push for that at my next job.

Lessons on Mobile Platforms from an Ancient JavaScript Game

I recently realized that I’ve been continuously programming in JavaScript for about 16 years. That is, I’ve been writing at least little snippets (and sometimes heaps) of JavaScript at least once a month for 16 years. 16 years! That’s insane. That’s a much longer run than I’ve had with any other language, and it doesn’t look like it’s going to end anytime soon. The funny part is that I’ve never been particularly fond of JavaScript as a language, but there is just no way around it as long as I continue do some sort of web development. I suspect that a lot of other developers who came of age in the late 90s have reached similar accidental JavaScript seniority and that would partly explain why by some metrics it’s on track to become the world’s most popular programming language.

Reminiscing about my early exploits in JavaScript, I searched through my backups to unearth a game I wrote in 1998 and included in my very first application for a paid programming job (I got the job). I was shocked to discover that in a modern browser the game still works exactly as intended, with zero modifications to the code. Here it is, in all its original low-res 90s glory: AsteroidMan, a PacMan clone written in JavaScript.

Screen Shot 2013-11-10 at 1.39.29 PM

In 1998, Netscape 4 and Internet Explorer 4 were the browsers at the cutting edge. They didn’t support much manipulation of the DOM yet, but it was possible to dynamically change the value of a form field — and the source of an image, which enabled games like this one based on a grid of images. One other neat technique for client-side apps I remember from that time was to use frames (yeah, frames, not those new-fangled iframes) to keep global JS state in the parent document, which was preserved across page navigation in the child frames. In my backups I also found a LucasArts-style adventure game I wrote that way. No, it’s too embarrassing to post now.

What do we learn from this exercise in JavaScript archaeology? The web is an insanely good application distribution and execution platform. I opened that HTML page from my backups in a browser and it looked and worked exactly as it did 15 years ago. There was zero installation or configuration required. The game does not only work in recent Chrome, Safari and Firefox on my laptop, it runs on my Android phone and on my iPad. I doubt any more traditional client software written 15 years ago would still run in a modern incarnation of the target platform, especially with zero installation and configuration by the user. Imagine a Windows 98 app running on Windows 8 (that’s a brilliant versioning scheme right there, Microsoft). The web platform is singular in how extremely portable and backwards compatible it is.

Here’s what this means for today. In case you hadn’t noticed, there is currently a big trend away from web apps towards native apps, namely iOS and Android apps. I seriously doubt a native iOS or Android app written today will work in a modern iOS/Android environment in, say, 4 years from now. I can guarantee you that neither iOS or Android will even still be around in a recognizable form 10 years from now. As platforms they are very likely a dead end while the web has a proven track record of being very open ended.

Unfortunately the current trend is unlikely to reverse soon because for now those platforms are a very profitable dead end, both for the platform stewards and for the programmers who serve them. If you have only rudimentary Android or iOS skills you should have no problems finding a well-paying job right now. The trend is a bit less profitable or downright onerous for creators of consumer products (that would be me and also my employer) because building natively for both iOS and Android and maybe even Windows Mobile is an expensive time sink.

I don’t expect any innovation from Apple in this regard since they have always been all about user and developer lock-in. But in my mind Google has a huge opportunity here. Google should push to make web apps first-class citizens in Android. Basically, an Android app may be a light-weight wrapper around a web app with some JavaScript APIs to call out to device APIs. This isn’t a novel idea, Phonegap and others have been trying to tack something very similar on top of Android, but it would help immensely if this approach was sanctioned and widely promoted by Google and ideally developed into an open standard. Such a fairly simple extension to the web has a much better shot at a long shelf life and at getting adopted by other mobile environments. It should also dramatically smooth out the pretty steep learning curve for Android development and thus increase Android’s developer mindshare.

The lesson for developers should be obvious: Think twice before you embark on a mobile-first strategy with native apps only. Consider Phonegap or other hybrid frameworks. Especially if you’re building for longevity or for low maintenance cost, for heaven’s sake, just build a web app.

Robolectric 2.2: Some Pages from the Missing Manual

When I have a conversation about Android development with anyone I usually end up complaining about how hard and annoying testing is. I think Google did a huge disservice to the community by neglecting testing infrastructure since as a result there is no culture of testing in Android development. There are some open source efforts to improve the situation though, and the main tool that has been preserving bits of my sanity while coding for Android is Robolectric.

I use it mostly to write unit tests for activity and fragment classes, since doing that with the tools provided by Android is really rather painful (and slow). With the just released Robolectric version (2.2) I feel its API, especially around maintaining the activity lifecycle, has finally evolved to where tests are really easy and intuitive to write. There is some basic documentation for the new API but I haven’t come across good real world examples, so I thought it could be useful to document some patterns I’ve been using.

Testing Activities

My activity tests tend to have the following shape, here for a fictional class MyActivity:

@RunWith(RobolectricTestRunner.class)
public class MyActivityTest {

    private ActivityController controller;
    private MyActivity activity;

    @Before
    public void setUp() {
    	controller = Robolectric.buildActivity(MyActivity.class);
    }

    @After
    public void tearDown() {
    	controller.destroy();
    }

    private void createWithIntent(String myExtra) {
        Intent intent = new Intent(Robolectric.application,
                MyActivity.class);
    	Bundle extras = new Bundle();
    	extras.putString("myExtra", myExtra);
    	intent.putExtras(extras);
    	activity = controller
    		.withIntent(intent)
    		.create()
    		.start()
                .visible()
    		.get();
    }

    @Test
    public void createsAndDestroysActivity() {
    	createWithIntent("foo");
        // Assertions go here
    }

    @Test
    public void pausesAndResumesActivity() {
    	createWithIntent("foo");
    	controller.pause().resume();
        // Assertions go here
    }

    @Test
    public void recreatesActivity() {
    	createWithIntent("foo");
    	activity.recreate();
        // Assertions go here
    }

}

In Robolectric 2.2, ActivityController is now the canonical API (actually, the only API) for this sort of test. It has methods that make the underlying activity go through state changes. To fully create an activity as it would be presented to a user you generally want to call create(), start() and visible() on it. I’m also putting a destroy() call in an @After method to ensure tearing down the activity always works properly.

As a rule, every activity test I write has at least the 3 tests above, covering the most common state transitions – create/destroy, pause/resume, and recreate. You might be surprised how often you initially implement broken recreate logic. On an actual device, a recreate happens for example when the screen rotates, something that’s quite common but easily overlooked in manual testing.

Often I’ll want to write multiple tests that start an activity with different intents. This is hinted at in the createWithIntent() method which takes a customizable extra parameter for the intent. For a very simple class you might omit that and just create the activity without any intent in the setUp() method.

Testing Fragments

For fragment unit tests I am using a base class that can make any fragment go through the lifecycle of a parent activity:

public class FragmentTestCase {

    private static final String FRAGMENT_TAG = "fragment";

    private ActivityController controller;
    private FragmentActivity activity;
    private T fragment;

    /**
     * Adds the fragment to a new blank activity, thereby fully
     * initializing its view.
     */
    public void startFragment(T fragment) {
        this.fragment = fragment;
        controller = Robolectric.buildActivity(FragmentActivity.class);
        activity = controller.create().start().visible().get();
        FragmentManager manager = activity.getFragmentManager();
        manager.beginTransaction()
        		.add(fragment, FRAGMENT_TAG).commit();
    }

    @After
    public void destroyFragment() {
        if (fragment != null) {
            FragmentManager manager = activity.getFragmentManager();
            manager.beginTransaction().remove(fragment).commit();
            fragment = null;
            activity = null;
        }
    }

    public void pauseAndResumeFragment() {
    	controller.pause().resume();
    }

    public T recreateFragment() {
    	activity.recreate();
        // Recreating the activity creates a new instance of the
        // fragment which we need to retrieve by tag.
    	fragment = (T) activity.getFragmentManager()
    			.findFragmentByTag(FRAGMENT_TAG);
    	return fragment;
    }

}

The general idea is that the best way to put a fragment through realistic state transitions is to add it to a real activity and then make that activity change states. Using a blank FragmentActivity works well for this purpose.

The helper methods allow me to very quickly write at least 3 tests covering the most common transitions, same as above for activities. This is an example for a MyFragment class:

@RunWith(RobolectricTestRunner.class)
public class MyFragmentTest extends FragmentTestCase<MyFragment> {

    private MyFragment fragment;

    @Before
    public void setUp() {
        fragment = new MyFragment();
    }

    @Test
    public void createsAndDestroysFragment() {
        startFragment(fragment);
        // Assertions go here
    }

    @Test
    public void pausesAndResumesFragment() {
        startFragment(fragment);
        pauseAndResumeFragment();
        // Assertions go here
    }

    @Test
    public void recreatesFragment() {
        startFragment(fragment);
        fragment = recreateFragment();
        // Assertions go here
    }

}

***

I’ve had pretty solid results with these patterns so far. I’m beta testing an app with real users right now, fully covered with tests like the above, and I haven’t seen any crashes yet (knock on wood). But I’m sure this can be improved upon. I’d be curious to hear how others use Robolectric since I can find so few real world examples out there.

The Test-Infected Programmer’s Android Apothecary

Earlier this week I gave a lightning talk about testing at an Android meetup in New York. Because, on the surface, testing is not the most exciting topic in the world, I tried to spice it up with a snazzy title and a slightly gimmicky structure. The jury is out whether it worked or not (it didn’t draw many laughs), but decide for yourself based on the slides.

Since the slides are not that useful without the accompanying stage show, here’s a very quick summary. The premise is that test infrastructure provided by the Android SDK out of the box has quite a lot of shortfalls and for someone serious about unit testing there are a bunch of 3rd-party libraries that help ease the pain.

These are the libraries I presented:

Robolectric

  • Fixes the problem of having to run tests on an Android emulator, which can significantly improve test execution speed and plays nicer with most Continuous Integration environments
  • Works by replacing some Android core classes with “shadows” which are hand-written fakes, so some caution is warranted because behavior can be subtly different from the real implementations in the Android VM

FEST Android

  • Simplifies assertions about Android objects such as views
  • Based on FEST fluent assertions, a way to make assertions more readable, type-safe and intuitive

ViewSelector

  • The self-promotion part of the talk: This is a library I wrote myself, based on concepts from assert_select in Rails and Android’s own UiSelector
  • Makes assertions about complex Android UIs easier by selecting views with CSS-style selectors
  • Alpha-quality software, in the sense that the API is not completely stable (I’ll have a more in-depth blog post once I’m happy with the API)

Failure Notifications for Rake Tasks on the Heroku Scheduler

For users of Heroku, the Scheduler add-on is a convenient and cheap way to run scheduled batch jobs. However, you get what you pay for: it’s a little bare bones. For example, it doesn’t support any sort of notifications when a job fails, a major issue unless you compulsively check your logs. For Rails apps with scheduled Rake tasks, the exception_notification-rake gem fixes this problem. This post walks you through configuring an app for email notifications about Rake tasks failing on the Heroku Scheduler.

Note: This guide was produced based on Rails 3.2.12. I don’t know whether the same instructions work unmodified for other versions, but I would love to hear from you if you encounter any issues. I’m also assuming that you’re using a fairly standard Rails development setup, using RVM, Bundler etc.

A Failing Task

First things first. Let’s create a failing Rake task that we’re going to use for testing notification delivery. Since you’re already using Rake tasks with the scheduler, add another task to the .rake file where you keep them (most commonly lib/tasks/scheduler.rake but any .rake file in lib/tasks will do):

task :failing_task => :environment do
  puts "Failing task in environment #{Rails.env}..."
  FAIL!
end

Note that the new task depends on the :environment task. This is required for tasks you want notifications for. The Rails environment needs to be loaded because the configuration we’re going to add later happens during Rails initialization. Verify that this indeed does fail by running it:

$ rake failing_task

Which should produce something like this:

Failing task in environment development...
rake aborted!
undefined method `FAIL!' for main:Object
./lib/tasks/scheduler.rake:33:in `block in <top (required)>'
Tasks: TOP => failing_task

Install the Gem

Add the exception_notification-rake gem to your Gemfile:

gem 'exception_notification-rake', '0.0.4'

At the point of writing this, 0.0.4 is the latest version of the gem. You might want to check RubyGems to see if there is a newer version. Now tell Bundler to update your gems by running:

$ bundle update

Configure & Test Notifications in your Development Environment

Before you actually try this on a remote Heroku server where debugging is difficult, it’s a good idea to test everything end-to-end locally. For testing email delivery locally I have found mailcatcher to be very useful. It runs a simple SMTP server that you can query through a browser. Add mailcatcher to your Gemfile:

group :development do
  gem 'mailcatcher', '0.5.10'
end

We need this for development only. Again, you might want to check mailcatcher on RubyGems for the latest version.

Note: When I was adding mailcatcher to an app running on the Heroku Cedar stack, I encountered a weird gem dependency issue that only manifested itself in the production environment on Heroku. An explicit dependency on the “thin” server in my Gemfile (gem 'thin', '~> 1.5.0') fixed it. You might need the same, Heroku recommends thin anyway for production apps.

After you update your gems with bundle update you can run mailcatcher with:

$ mailcatcher -f

Point your browser to http://127.0.0.1:1080 and you should see an empty mailbox.

Configure your development environment to use the now running mailcatcher by adding this to your config/environments/development.rb file:

# 1. Point ActionMailer at mailcatcher
config.action_mailer.delivery_method = :smtp
config.action_mailer.smtp_settings = { :host => "localhost", :port => 1025 }
config.action_mailer.raise_delivery_errors = true

# 2. Base configuration for ExceptionNotifier
config.middleware.use ExceptionNotifier,
  :exception_recipients => %w{exceptions@example.com},
  :ignore_if => lambda { true }

# 3. Enable Rake notifications
ExceptionNotifier::Rake.configure

The first part configures Rails’ ActionMailer (which is used under the hood to actually send mail) for mailcatcher. The second part sets up ExceptionNotifier, the underlying generic notification middleware (see its documentation for more background), and lastly we enable notifications about Rake failures by calling ExceptionNotifier::Rake.configure.

On the off chance that you’re already using ExceptionNotifier in your development environment, you can omit the second part. Note that in this example we are suppressing all notifications other than about Rake failures with the :ignore_if option.

We should be good to go. If you now run failing_task (in your development environment, which should be the default for a locally run task) a notification will pop up in mailcatcher. Run the task with:

$ rake failing_task

Then go to http://127.0.0.1:1080 to check out the email sent. The email includes the name of the Rake task that failed and a stacktrace of the exception thrown.

Email Configuration for your Heroku App

If all of this worked in your development environment, you’re ready to tackle your production environment on Heroku. But first you need to make sure email delivery in general is configured in your Heroku app. One of two cases will apply to you.

1. Your app is already sending email. Great, you don’t need to do anything since ExceptionNotifier will just use the configuration you already have in place. Skip to the next step.

2. Your app isn’t sending email yet (you haven’t explicitly configured anything or installed any email add-ons). If your app’s email needs are limited to failure notifications, then a simple add-on will work for you. I can recommend the SendGrid Starter add-on. It’s free and allows up to 200 emails per day, more than enough for a few failure notifications. To install it run:

$ heroku addons:add sendgrid:starter

If your app is on the Cedar stack at Heroku, you also need to manually configure SendGrid (it should be automatic on the Bamboo stack). The official documentation has all the details, but just adding this to your config/environments/production.rb file should do the job:

config.action_mailer.delivery_method = :smtp
config.action_mailer.smtp_settings = {
  :address => "smtp.sendgrid.net",
  :port => 587,
  :authentication => :plain,
  :user_name => ENV['SENDGRID_USERNAME'],
  :password => ENV['SENDGRID_PASSWORD'],
  :domain => 'heroku.com'
}

Configure & Test Notifications on Heroku

To enable Rake notifications in your production environment we add similar configuration to the config/environments/production.rb file like we did for the development environment:

# 1. Base configuration for ExceptionNotifier
config.middleware.use ExceptionNotifier,
  :sender_address => %{"Scheduler" <some.address@example.com>},
  :exception_recipients => %w{your.email@example.com},
  :ignore_if => lambda { true }

# 2. Enable Rake notifications
ExceptionNotifier::Rake.configure

Again, if you are already using ExceptionNotifier in your production environment, omit the first part.

This time, the concrete values actually matter. Replace the values of the :sender_address and :exception_recipients options with the actual addresses you want to use as sender and recipients (the sender isn’t actually very significant but could be helpful for example for filtering notification emails in your mail client).

It’s now time to commit your changes (the new failing_task as well as Gemfile and configuration changes) and push them to your Heroku app. If that succeeds, you can try out notification delivery by running failing_task on Heroku:

$ heroku run rake failing_task

Hopefully that failed successfully! All the addresses you listed under the :exception_recipients option should have received an email with details on the failure.

That’s all. Any Rake tasks you schedule from now on with the Heroku Scheduler will trigger notification emails if they fail.

Advanced Configuration

There are a few more configuration options that might be of interest, especially if you were already using ExceptionNotifier before. Check out the documentation for exception_notification-rake for details.

Multiple Delegates in Ruby

I found that, out of the box, delegating method calls to multiple objects is not straight-forward in Ruby. There are two modules in the Ruby standard library that cover most delegation use cases: The delegate module lets you easily delegate all methods to another object, and the forwardable module does the same for some explicitly enumerated methods. Both of them operate on a single delegate.

There are good reasons for this. With multiple delegates it’s not exactly obvious what to do with the multiple return values, and the side effects when a delegate mutates method arguments could be a real headache.

I had a use case where these issues didn’t matter. I wanted to capture $stdout but not actually hide anything from it, meaning that I wanted all output to be written to the console (or whatever $stdout amounts to in any given situation) while creating an in-memory copy of it on the side. My idea was to replace $stdout with an object that delegates both to the original $stdout and to a StringIO object.

I came up with this implementation of a MultiDelegator:

require 'delegate'

class MultiDelegator

  def initialize(delegates)
    @delegates = delegates.map do |del|
      SimpleDelegator.new(del)
    end
  end

  def method_missing(m, *args, &block)
    return_values = @delegates.map do |del|
      del.method_missing(m, *args, &block)
    end
    return_values.first
  end
end

This will forward all method calls to the all the delegates in the list passed to the constructor. Arbitrarily but consistently, all return values except for the one from the first delegate are discarded.

I’m wrapping each delegate in a SimpleDelegator (from the builtin delegate module) because I’m lazy and don’t want to copy the code that deals with the actual method forwarding from there (it’s pretty simple though, you can see for yourself).

This implementation is somewhat simplistic and comes with caveats:

  • It bears repeating: If delegated methods mutate their arguments, you’re gonna have a bad time, most likely.
  • This obviously does not support some of the nice-to-have functionality that the builtin Delegator and SimpleDelegator handle well, such as object equality, freezing, marshaling, tainting.

As a usage example, this is how I solved my original output capturing problem:

captured_output = StringIO.new
begin
  $stdout = MultiDelegator.new([$stdout, captured_output])
  # do something that generates output
  # ...
ensure
  $stdout = STDOUT
end
# do something with captured_output
# ...

Side note: I’m aware that replacing the global $stdout will not actually capture everything that you would consider output. It won’t capture output generated by C extensions or subprocesses; see this discussion for more details.

Robust Parameterized Unit Tests in Ruby with param_test

Parameterized unit tests come in handy when you have a simple API that you want to test in the same way with multiple inputs and expected outputs. Instead of creating duplicates of a test with small variations, it’s often nicer to have a single test that runs multiple times with different parameters – a parameterized test. This post examines some simple but problematic ways of writing parameterized tests in Ruby and then explains how the param_test gem can make them more robust.

Assertions in Loops Considered Harmful

Let’s say we’re unit testing a simple method like this one:

# whitespace.rb
# Whether a given string includes any whitespace.
def includes_whitespace?(string)
  not string.match(/\s/).nil?
end

Aside: All code used in this post can be found here.

It will be tempting to write a unit test like this:

# bad_looping_test.rb
require 'whitespace'
require 'test/unit'

class BadLoopingTest < Test::Unit::TestCase
  def test_includes_whitespace
    inputs = ["hello world", "foo bar", "foo", "bar\n"]
    inputs.each do |input|
      assert includes_whitespace? input
    end
  end
end

This can be considered a parametrized test. We’re testing the same assertion against 4 different inputs. But: Don’t do it this way! Putting an assertion inside a loop should almost always be avoided. If the test fails you might not be able to tell at which of the 4 input strings it failed. What’s more, any failure aborts the loop, so subsequent inputs won’t be tested in the same run, hiding further failures until later.

To wit, the above test actually does fail and the output is most unhelpful:

$ ruby -I"test/blog" test/blog/bad_looping_test.rb
# Running tests:

F

Finished tests in 0.000565s.

1) Failure:
 test_includes_whitespace(BadLoopingTest) [test/blog/bad_looping_test.rb:8]:
 Failed assertion, no message given.

1 tests, 3 assertions, 1 failures, 0 errors, 0 skips

As a remedy you could add an explicit failure message to the assertion that includes the input being tested, but since that’s optional and onerous people will often conveniently forget about it.

Parameterized Tests with ActiveSupport’s test Method

ActiveSupport’s unit test extensions add a declarative test method to test cases that allows for a relatively simple way to improve on this:

# better_declarative_test.rb
require 'whitespace'
require 'active_support/test_case'
require 'test/unit'

class BetterDeclarativeTest < ActiveSupport::TestCase
  ["hello world", "foo bar", "foo", "bar\n"].each do |input|
    test "#{input} includes whitespace" do
      assert includes_whitespace? input
    end
  end
end

This is better. It gently nudges you towards writing a good test description that will also help in identifying failures. The test now fails in a more explicit way:

$ ruby -I"test/blog" test/blog/better_declarative_test.rb
# Running tests:

..F.

Finished tests in 0.001183s.

1) Failure:
 test_foo_includes_whitespace(BetterDeclarativeTest) [test/blog/better_declarative_test.rb:8]:
 Failed assertion, no message given.

4 tests, 4 assertions, 1 failures, 0 errors, 0 skips

What actually happened here? ActiveSupport’s test generates a test method named after the test description it was given. Since we called test 4 times, once for each of the 4 inputs, 4 methods were generated. From the name of the failing method test_foo_includes_whitespace we know that the offending input is "foo".

Aside: While the simplicity of xUnit-style unit tests is great and surely one of the reasons it’s so popular across a wide range of programming languages, it’s also a hindrance in this case. The only metadata available about a test is effectively its method name. It would be great to have an explicit test description metadata field without the restrictions of a method name. This is something that newer testing frameworks such as RSpec got right (though I have other issues with RSpec).

Unfortunately, using test this way isn’t a general solution. Some tests written using this pattern might not run at all. The main problem is that method names must be unique within a test case (or any class, really). So if you neglect to include all input parameters in the description or, more subtle, if some of your inputs boil down to identical method names, your test will abort early with a somewhat obscure exception.

As an illustration, consider what happens if we modify the test inputs to be:

# bad_declarative_test.rb
require 'whitespace'
require 'active_support/test_case'
require 'test/unit'

class BadDeclarativeTest < ActiveSupport::TestCase
  ["  ", "\n"].each do |input|
    test "#{input} includes whitespace" do
      assert includes_whitespace? input
    end
  end
end

We have a string consisting of two spaces and a string consisting of a new line as inputs. Running this will yield:

$ ruby -I"test/blog" test/blog/bad_declarative_test.rb
~/.rvm/gems/ruby-1.9.3-p327/gems/activesupport-3.2.11/lib/active_support/testing/declarative.rb:28:in `test': test__includes_whitespace is already defined in BadDeclarativeTest (RuntimeError)
from test/blog/bad_declarative_test.rb:7:in `block in <class:BadDeclarativeTest>'
from test/blog/bad_declarative_test.rb:6:in `each'
from test/blog/bad_declarative_test.rb:6:in `<class:BadDeclarativeTest>'
from test/blog/bad_declarative_test.rb:5:in `<main>'

This failed to even generate the test methods because test collapses all whitespace to a single underscore when converting the description to a method name. Both inputs result in the same method name, so upon trying to generate the second test method, the above exception is raised.

Robust Parameterized Tests with param_test

I was bumping up against these issues often enough that I packaged up a simple solution in the gem param_test. It adds a single class method param_test to ActiveSupport::TestCase that makes parametrized tests as simple as can be, enforces a test description that includes all input parameters and guarantees that the generated method names are unique.

Rewritten using the param_test gem, the test looks like this:

# single_param_test.rb
require 'whitespace'
require 'param_test'
require 'test/unit'

class SingleParamTest < ActiveSupport::TestCase
  param_test "%s includes whitespace",
  ["hello world", "foo bar", "foo", "bar\n"] do |input|
    assert includes_whitespace? input
  end
end

Note that for the test description param_test uses string formatting (as per Ruby’s format/sprintf method), rather than string interpolation (where you type out explicit variable names, as in "#{input}"). For each test the input parameter will be substituted for the %s.

You can have multiple parameters by test, for example if you want to test both outcomes of includes_whitespace? in one go:

# multiple_params_test.rb
require 'whitespace'
require 'param_test'
require 'test/unit'

class MultipleParamsTest < ActiveSupport::TestCase
  param_test "%s includes whitespace is %s", [
    ["hello world", true],
    ["foo bar", true],
    ["foo", false],
  ] do |input, expected|
    assert_equal expected, includes_whitespace?(input)
  end
end

To be a bit more formal, param_test takes 3 arguments:

  1. The test description template. Input parameters will be applied to this string in order using Ruby string formatting. Generally you’ll just want to use the %s format sequence which will be replaced with a string version of the parameter.
  2. A list of parameters. This can be a simple flat list if there is only one input parameter per test. For multiple parameters per test it can also be a list of lists, where every inner list constitutes the parameter set for one test. All parameter sets must be of the same length.
  3. A block making up the actual test body. This becomes the body of every generated test. The block should have as many arguments a there are parameters in each set.

Behind the scenes param_test catches any mistakes and inconsistencies in your input early. Explicit exceptions will be raised if there is a mismatch between the number of input parameters, placeholders in the description template and arguments to the block. If two tests would end up with the same generated method name, uniqueness is still guaranteed by appending a counter to the method name.

Install the gem and try it out!