For users of Heroku, the Scheduler add-on is a convenient and cheap way to run scheduled batch jobs. However, you get what you pay for: it’s a little bare bones. For example, it doesn’t support any sort of notifications when a job fails, a major issue unless you compulsively check your logs. For Rails apps with scheduled Rake tasks, the exception_notification-rake gem fixes this problem. This post walks you through configuring an app for email notifications about Rake tasks failing on the Heroku Scheduler.

Note: This guide was produced based on Rails 3.2. See the exception_notification-rake documentation for instructions with Rails 4 and 5.

A Failing Task

First things first. Let’s create a failing Rake task that we’re going to use for testing notification delivery. Since you’re already using Rake tasks with the scheduler, add another task to the .rake file where you keep them (most commonly lib/tasks/scheduler.rake but any .rake file in lib/tasks will do):

task :failing_task => :environment do
  puts "Failing task in environment #{Rails.env}..."
  FAIL!
end

Note that the new task depends on the :environment task. This is required for tasks you want notifications for. The Rails environment needs to be loaded because the configuration we’re going to add later happens during Rails initialization. Verify that this indeed does fail by running it:

$ rake failing_task

Which should produce something like this:

Failing task in environment development...
rake aborted!
undefined method `FAIL!' for main:Object
./lib/tasks/scheduler.rake:33:in `block in <top (required)>'
Tasks: TOP => failing_task

Install the Gem

Add the exception_notification-rake gem to your Gemfile:

gem 'exception_notification-rake', '0.0.4'

At the point of writing this, 0.0.4 is the latest version of the gem. You might want to check RubyGems to see if there is a newer version. Now tell Bundler to update your gems by running:

$ bundle update

Configure & Test Notifications in your Development Environment

Before you actually try this on a remote Heroku server where debugging is difficult, it’s a good idea to test everything end-to-end locally. For testing email delivery locally I have found mailcatcher to be very useful. It runs a simple SMTP server that you can query through a browser. Add mailcatcher to your Gemfile:

group :development do
  gem 'mailcatcher', '0.5.10'
end

We need this for development only. Again, you might want to check mailcatcher on RubyGems for the latest version.

Note: When I was adding mailcatcher to an app running on the Heroku Cedar stack, I encountered a weird gem dependency issue that only manifested itself in the production environment on Heroku. An explicit dependency on the “thin” server in my Gemfile (gem 'thin', '~> 1.5.0') fixed it. You might need the same, Heroku recommends thin anyway for production apps.

After you update your gems with bundle update you can run mailcatcher with:

$ mailcatcher -f

Point your browser to http://127.0.0.1:1080 and you should see an empty mailbox.

Configure your development environment to use the now running mailcatcher by adding this to your config/environments/development.rb file:

# 1. Point ActionMailer at mailcatcher
config.action_mailer.delivery_method = :smtp
config.action_mailer.smtp_settings = { :host => "localhost", :port => 1025 }
config.action_mailer.raise_delivery_errors = true

# 2. Base configuration for ExceptionNotifier
config.middleware.use ExceptionNotifier,
  :exception_recipients => %w{exceptions@example.com},
  :ignore_if => lambda { true }

# 3. Enable Rake notifications
ExceptionNotifier::Rake.configure

The first part configures Rails’ ActionMailer (which is used under the hood to actually send mail) for mailcatcher. The second part sets up ExceptionNotifier, the underlying generic notification middleware (see its documentation for more background), and lastly we enable notifications about Rake failures by calling ExceptionNotifier::Rake.configure.

On the off chance that you’re already using ExceptionNotifier in your development environment, you can omit the second part. Note that in this example we are suppressing all notifications other than about Rake failures with the :ignore_if option.

We should be good to go. If you now run failing_task (in your development environment, which should be the default for a locally run task) a notification will pop up in mailcatcher. Run the task with:

$ rake failing_task

Then go to http://127.0.0.1:1080 to check out the email sent. The email includes the name of the Rake task that failed and a stacktrace of the exception thrown.

Email Configuration for your Heroku App

If all of this worked in your development environment, you’re ready to tackle your production environment on Heroku. But first you need to make sure email delivery in general is configured in your Heroku app. One of two cases will apply to you.

1. Your app is already sending email. Great, you don’t need to do anything since ExceptionNotifier will just use the configuration you already have in place. Skip to the next step.

2. Your app isn’t sending email yet (you haven’t explicitly configured anything or installed any email add-ons). If your app’s email needs are limited to failure notifications, then a simple add-on will work for you. I can recommend the SendGrid Starter add-on. It’s free and allows up to 200 emails per day, more than enough for a few failure notifications. To install it run:

$ heroku addons:add sendgrid:starter

If your app is on the Cedar stack at Heroku, you also need to manually configure SendGrid (it should be automatic on the Bamboo stack). The official documentation has all the details, but just adding this to your config/environments/production.rb file should do the job:

config.action_mailer.delivery_method = :smtp
config.action_mailer.smtp_settings = {
  :address => "smtp.sendgrid.net",
  :port => 587,
  :authentication => :plain,
  :user_name => ENV['SENDGRID_USERNAME'],
  :password => ENV['SENDGRID_PASSWORD'],
  :domain => 'heroku.com'
}

Configure & Test Notifications on Heroku

To enable Rake notifications in your production environment we add similar configuration to the config/environments/production.rb file like we did for the development environment:

# 1. Base configuration for ExceptionNotifier
config.middleware.use ExceptionNotifier,
  :sender_address => %{"Scheduler" <some.address@example.com>},
  :exception_recipients => %w{your.email@example.com},
  :ignore_if => lambda { true }

# 2. Enable Rake notifications
ExceptionNotifier::Rake.configure

Again, if you are already using ExceptionNotifier in your production environment, omit the first part.

This time, the concrete values actually matter. Replace the values of the :sender_address and :exception_recipients options with the actual addresses you want to use as sender and recipients (the sender isn’t actually very significant but could be helpful for example for filtering notification emails in your mail client).

It’s now time to commit your changes (the new failing_task as well as Gemfile and configuration changes) and push them to your Heroku app. If that succeeds, you can try out notification delivery by running failing_task on Heroku:

$ heroku run rake failing_task

Hopefully that failed successfully! All the addresses you listed under the :exception_recipients option should have received an email with details on the failure.

That’s all. Any Rake tasks you schedule from now on with the Heroku Scheduler will trigger notification emails if they fail.

Advanced Configuration

There are a few more configuration options that might be of interest, especially if you were already using ExceptionNotifier before. Check out the documentation for exception_notification-rake for details.

I found that, out of the box, delegating method calls to multiple objects is not straight-forward in Ruby. There are two modules in the Ruby standard library that cover most delegation use cases: The delegate module lets you easily delegate all methods to another object, and the forwardable module does the same for some explicitly enumerated methods. Both of them operate on a single delegate.

There are good reasons for this. With multiple delegates it’s not exactly obvious what to do with the multiple return values, and the side effects when a delegate mutates method arguments could be a real headache.

I had a use case where these issues didn’t matter. I wanted to capture $stdout but not actually hide anything from it, meaning that I wanted all output to be written to the console (or whatever $stdout amounts to in any given situation) while creating an in-memory copy of it on the side. My idea was to replace $stdout with an object that delegates both to the original $stdout and to a StringIO object.

I came up with this implementation of a MultiDelegator:

require 'delegate'

class MultiDelegator

  def initialize(delegates)
    @delegates = delegates.map do |del|
      SimpleDelegator.new(del)
    end
  end

  def method_missing(m, *args, &block)
    return_values = @delegates.map do |del|
      del.method_missing(m, *args, &block)
    end
    return_values.first
  end
end

This will forward all method calls to the all the delegates in the list passed to the constructor. Arbitrarily but consistently, all return values except for the one from the first delegate are discarded.

I’m wrapping each delegate in a SimpleDelegator (from the builtin delegate module) because I’m lazy and don’t want to copy the code that deals with the actual method forwarding from there (it’s pretty simple though, you can see for yourself).

This implementation is somewhat simplistic and comes with caveats:

• It bears repeating: If delegated methods mutate their arguments, you’re gonna have a bad time, most likely.
• This obviously does not support some of the nice-to-have functionality that the builtin Delegator and SimpleDelegator handle well, such as object equality, freezing, marshaling, tainting.

As a usage example, this is how I solved my original output capturing problem:

captured_output = StringIO.new
begin
  $stdout = MultiDelegator.new([$stdout, captured_output])
  # do something that generates output
  # ...
ensure
  $stdout = STDOUT
end
# do something with captured_output
# ...

Side note: I’m aware that replacing the global $stdout will not actually capture everything that you would consider output. It won’t capture output generated by C extensions or subprocesses; see this discussion for more details.

Parameterized unit tests come in handy when you have a simple API that you want to test in the same way with multiple inputs and expected outputs. Instead of creating duplicates of a test with small variations, it’s often nicer to have a single test that runs multiple times with different parameters – a parameterized test. This post examines some simple but problematic ways of writing parameterized tests in Ruby and then explains how the param_test gem can make them more robust.

Assertions in Loops Considered Harmful

Let’s say we’re unit testing a simple method like this one:

# whitespace.rb
# Whether a given string includes any whitespace.
def includes_whitespace?(string)
  not string.match(/\s/).nil?
end

Aside: All code used in this post can be found here.

It will be tempting to write a unit test like this:

# bad_looping_test.rb
require 'whitespace'
require 'test/unit'

class BadLoopingTest < Test::Unit::TestCase
  def test_includes_whitespace
    inputs = ["hello world", "foo bar", "foo", "bar\n"]
    inputs.each do |input|
      assert includes_whitespace? input
    end
  end
end

This can be considered a parametrized test. We’re testing the same assertion against 4 different inputs. But: Don’t do it this way! Putting an assertion inside a loop should almost always be avoided. If the test fails you might not be able to tell at which of the 4 input strings it failed. What’s more, any failure aborts the loop, so subsequent inputs won’t be tested in the same run, hiding further failures until later.

To wit, the above test actually does fail and the output is most unhelpful:

$ ruby -I"test/blog" test/blog/bad_looping_test.rb
# Running tests:

F

Finished tests in 0.000565s.

1) Failure:
 test_includes_whitespace(BadLoopingTest) [test/blog/bad_looping_test.rb:8]:
 Failed assertion, no message given.

1 tests, 3 assertions, 1 failures, 0 errors, 0 skips

As a remedy you could add an explicit failure message to the assertion that includes the input being tested, but since that’s optional and onerous people will often conveniently forget about it.

Parameterized Tests with ActiveSupport’s test Method

ActiveSupport’s unit test extensions add a declarative test method to test cases that allows for a relatively simple way to improve on this:

# better_declarative_test.rb
require 'whitespace'
require 'active_support/test_case'
require 'test/unit'

class BetterDeclarativeTest < ActiveSupport::TestCase
  ["hello world", "foo bar", "foo", "bar\n"].each do |input|
    test "#{input} includes whitespace" do
      assert includes_whitespace? input
    end
  end
end

This is better. It gently nudges you towards writing a good test description that will also help in identifying failures. The test now fails in a more explicit way:

$ ruby -I"test/blog" test/blog/better_declarative_test.rb
# Running tests:

..F.

Finished tests in 0.001183s.

1) Failure:
 test_foo_includes_whitespace(BetterDeclarativeTest) [test/blog/better_declarative_test.rb:8]:
 Failed assertion, no message given.

4 tests, 4 assertions, 1 failures, 0 errors, 0 skips

What actually happened here? ActiveSupport’s test generates a test method named after the test description it was given. Since we called test 4 times, once for each of the 4 inputs, 4 methods were generated. From the name of the failing method test_foo_includes_whitespace we know that the offending input is "foo".

Aside: While the simplicity of xUnit-style unit tests is great and surely one of the reasons it’s so popular across a wide range of programming languages, it’s also a hindrance in this case. The only metadata available about a test is effectively its method name. It would be great to have an explicit test description metadata field without the restrictions of a method name. This is something that newer testing frameworks such as RSpec got right (though I have other issues with RSpec).

Unfortunately, using test this way isn’t a general solution. Some tests written using this pattern might not run at all. The main problem is that method names must be unique within a test case (or any class, really). So if you neglect to include all input parameters in the description or, more subtle, if some of your inputs boil down to identical method names, your test will abort early with a somewhat obscure exception.

As an illustration, consider what happens if we modify the test inputs to be:

# bad_declarative_test.rb
require 'whitespace'
require 'active_support/test_case'
require 'test/unit'

class BadDeclarativeTest < ActiveSupport::TestCase
  ["  ", "\n"].each do |input|
    test "#{input} includes whitespace" do
      assert includes_whitespace? input
    end
  end
end

We have a string consisting of two spaces and a string consisting of a new line as inputs. Running this will yield:

$ ruby -I"test/blog" test/blog/bad_declarative_test.rb
~/.rvm/gems/ruby-1.9.3-p327/gems/activesupport-3.2.11/lib/active_support/testing/declarative.rb:28:in `test': test__includes_whitespace is already defined in BadDeclarativeTest (RuntimeError)
from test/blog/bad_declarative_test.rb:7:in `block in <class:BadDeclarativeTest>'
from test/blog/bad_declarative_test.rb:6:in `each'
from test/blog/bad_declarative_test.rb:6:in `<class:BadDeclarativeTest>'
from test/blog/bad_declarative_test.rb:5:in `<main>'

This failed to even generate the test methods because test collapses all whitespace to a single underscore when converting the description to a method name. Both inputs result in the same method name, so upon trying to generate the second test method, the above exception is raised.

Robust Parameterized Tests with param_test

I was bumping up against these issues often enough that I packaged up a simple solution in the gem param_test. It adds a single class method param_test to ActiveSupport::TestCase that makes parametrized tests as simple as can be, enforces a test description that includes all input parameters and guarantees that the generated method names are unique.

Rewritten using the param_test gem, the test looks like this:

# single_param_test.rb
require 'whitespace'
require 'param_test'
require 'test/unit'

class SingleParamTest < ActiveSupport::TestCase
  param_test "%s includes whitespace",
  ["hello world", "foo bar", "foo", "bar\n"] do |input|
    assert includes_whitespace? input
  end
end

Note that for the test description param_test uses string formatting (as per Ruby’s format/sprintf method), rather than string interpolation (where you type out explicit variable names, as in "#{input}"). For each test the input parameter will be substituted for the %s.

You can have multiple parameters by test, for example if you want to test both outcomes of includes_whitespace? in one go:

# multiple_params_test.rb
require 'whitespace'
require 'param_test'
require 'test/unit'

class MultipleParamsTest < ActiveSupport::TestCase
  param_test "%s includes whitespace is %s", [
    ["hello world", true],
    ["foo bar", true],
    ["foo", false],
  ] do |input, expected|
    assert_equal expected, includes_whitespace?(input)
  end
end

To be a bit more formal, param_test takes 3 arguments:

1. The test description template. Input parameters will be applied to this string in order using Ruby string formatting. Generally you’ll just want to use the %s format sequence which will be replaced with a string version of the parameter.
2. A list of parameters. This can be a simple flat list if there is only one input parameter per test. For multiple parameters per test it can also be a list of lists, where every inner list constitutes the parameter set for one test. All parameter sets must be of the same length.
3. A block making up the actual test body. This becomes the body of every generated test. The block should have as many arguments a there are parameters in each set.

Behind the scenes param_test catches any mistakes and inconsistencies in your input early. Explicit exceptions will be raised if there is a mismatch between the number of input parameters, placeholders in the description template and arguments to the block. If two tests would end up with the same generated method name, uniqueness is still guaranteed by appending a counter to the method name.

Install the gem and try it out!