For users of Heroku, the Scheduler add-on is a convenient and cheap way to run scheduled batch jobs. However, you get what you pay for: it’s a little bare bones. For example, it doesn’t support any sort of notifications when a job fails, a major issue unless you compulsively check your logs. For Rails apps with scheduled Rake tasks, the exception_notification-rake gem fixes this problem. This post walks you through configuring an app for email notifications about Rake tasks failing on the Heroku Scheduler.

Note: This guide was produced based on Rails 3.2. See the exception_notification-rake documentation for instructions with Rails 4 and 5.

A Failing Task

First things first. Let’s create a failing Rake task that we’re going to use for testing notification delivery. Since you’re already using Rake tasks with the scheduler, add another task to the .rake file where you keep them (most commonly lib/tasks/scheduler.rake but any .rake file in lib/tasks will do):

task :failing_task => :environment do
  puts "Failing task in environment #{Rails.env}..."
  FAIL!
end

Note that the new task depends on the :environment task. This is required for tasks you want notifications for. The Rails environment needs to be loaded because the configuration we’re going to add later happens during Rails initialization. Verify that this indeed does fail by running it:

$ rake failing_task

Which should produce something like this:

Failing task in environment development...
rake aborted!
undefined method `FAIL!' for main:Object
./lib/tasks/scheduler.rake:33:in `block in <top (required)>'
Tasks: TOP => failing_task

Install the Gem

Add the exception_notification-rake gem to your Gemfile:

gem 'exception_notification-rake', '0.0.4'

At the point of writing this, 0.0.4 is the latest version of the gem. You might want to check RubyGems to see if there is a newer version. Now tell Bundler to update your gems by running:

$ bundle update

Configure & Test Notifications in your Development Environment

Before you actually try this on a remote Heroku server where debugging is difficult, it’s a good idea to test everything end-to-end locally. For testing email delivery locally I have found mailcatcher to be very useful. It runs a simple SMTP server that you can query through a browser. Add mailcatcher to your Gemfile:

group :development do
  gem 'mailcatcher', '0.5.10'
end

We need this for development only. Again, you might want to check mailcatcher on RubyGems for the latest version.

Note: When I was adding mailcatcher to an app running on the Heroku Cedar stack, I encountered a weird gem dependency issue that only manifested itself in the production environment on Heroku. An explicit dependency on the “thin” server in my Gemfile (gem 'thin', '~> 1.5.0') fixed it. You might need the same, Heroku recommends thin anyway for production apps.

After you update your gems with bundle update you can run mailcatcher with:

$ mailcatcher -f

Point your browser to http://127.0.0.1:1080 and you should see an empty mailbox.

Configure your development environment to use the now running mailcatcher by adding this to your config/environments/development.rb file:

# 1. Point ActionMailer at mailcatcher
config.action_mailer.delivery_method = :smtp
config.action_mailer.smtp_settings = { :host => "localhost", :port => 1025 }
config.action_mailer.raise_delivery_errors = true

# 2. Base configuration for ExceptionNotifier
config.middleware.use ExceptionNotifier,
  :exception_recipients => %w{exceptions@example.com},
  :ignore_if => lambda { true }

# 3. Enable Rake notifications
ExceptionNotifier::Rake.configure

The first part configures Rails’ ActionMailer (which is used under the hood to actually send mail) for mailcatcher. The second part sets up ExceptionNotifier, the underlying generic notification middleware (see its documentation for more background), and lastly we enable notifications about Rake failures by calling ExceptionNotifier::Rake.configure.

On the off chance that you’re already using ExceptionNotifier in your development environment, you can omit the second part. Note that in this example we are suppressing all notifications other than about Rake failures with the :ignore_if option.

We should be good to go. If you now run failing_task (in your development environment, which should be the default for a locally run task) a notification will pop up in mailcatcher. Run the task with:

$ rake failing_task

Then go to http://127.0.0.1:1080 to check out the email sent. The email includes the name of the Rake task that failed and a stacktrace of the exception thrown.

Email Configuration for your Heroku App

If all of this worked in your development environment, you’re ready to tackle your production environment on Heroku. But first you need to make sure email delivery in general is configured in your Heroku app. One of two cases will apply to you.

1. Your app is already sending email. Great, you don’t need to do anything since ExceptionNotifier will just use the configuration you already have in place. Skip to the next step.

2. Your app isn’t sending email yet (you haven’t explicitly configured anything or installed any email add-ons). If your app’s email needs are limited to failure notifications, then a simple add-on will work for you. I can recommend the SendGrid Starter add-on. It’s free and allows up to 200 emails per day, more than enough for a few failure notifications. To install it run:

$ heroku addons:add sendgrid:starter

If your app is on the Cedar stack at Heroku, you also need to manually configure SendGrid (it should be automatic on the Bamboo stack). The official documentation has all the details, but just adding this to your config/environments/production.rb file should do the job:

config.action_mailer.delivery_method = :smtp
config.action_mailer.smtp_settings = {
  :address => "smtp.sendgrid.net",
  :port => 587,
  :authentication => :plain,
  :user_name => ENV['SENDGRID_USERNAME'],
  :password => ENV['SENDGRID_PASSWORD'],
  :domain => 'heroku.com'
}

Configure & Test Notifications on Heroku

To enable Rake notifications in your production environment we add similar configuration to the config/environments/production.rb file like we did for the development environment:

# 1. Base configuration for ExceptionNotifier
config.middleware.use ExceptionNotifier,
  :sender_address => %{"Scheduler" <some.address@example.com>},
  :exception_recipients => %w{your.email@example.com},
  :ignore_if => lambda { true }

# 2. Enable Rake notifications
ExceptionNotifier::Rake.configure

Again, if you are already using ExceptionNotifier in your production environment, omit the first part.

This time, the concrete values actually matter. Replace the values of the :sender_address and :exception_recipients options with the actual addresses you want to use as sender and recipients (the sender isn’t actually very significant but could be helpful for example for filtering notification emails in your mail client).

It’s now time to commit your changes (the new failing_task as well as Gemfile and configuration changes) and push them to your Heroku app. If that succeeds, you can try out notification delivery by running failing_task on Heroku:

$ heroku run rake failing_task

Hopefully that failed successfully! All the addresses you listed under the :exception_recipients option should have received an email with details on the failure.

That’s all. Any Rake tasks you schedule from now on with the Heroku Scheduler will trigger notification emails if they fail.

Advanced Configuration

There are a few more configuration options that might be of interest, especially if you were already using ExceptionNotifier before. Check out the documentation for exception_notification-rake for details.

I found that, out of the box, delegating method calls to multiple objects is not straight-forward in Ruby. There are two modules in the Ruby standard library that cover most delegation use cases: The delegate module lets you easily delegate all methods to another object, and the forwardable module does the same for some explicitly enumerated methods. Both of them operate on a single delegate.

There are good reasons for this. With multiple delegates it’s not exactly obvious what to do with the multiple return values, and the side effects when a delegate mutates method arguments could be a real headache.

I had a use case where these issues didn’t matter. I wanted to capture $stdout but not actually hide anything from it, meaning that I wanted all output to be written to the console (or whatever $stdout amounts to in any given situation) while creating an in-memory copy of it on the side. My idea was to replace $stdout with an object that delegates both to the original $stdout and to a StringIO object.

I came up with this implementation of a MultiDelegator:

require 'delegate'

class MultiDelegator

  def initialize(delegates)
    @delegates = delegates.map do |del|
      SimpleDelegator.new(del)
    end
  end

  def method_missing(m, *args, &block)
    return_values = @delegates.map do |del|
      del.method_missing(m, *args, &block)
    end
    return_values.first
  end
end

This will forward all method calls to the all the delegates in the list passed to the constructor. Arbitrarily but consistently, all return values except for the one from the first delegate are discarded.

I’m wrapping each delegate in a SimpleDelegator (from the builtin delegate module) because I’m lazy and don’t want to copy the code that deals with the actual method forwarding from there (it’s pretty simple though, you can see for yourself).

This implementation is somewhat simplistic and comes with caveats:

• It bears repeating: If delegated methods mutate their arguments, you’re gonna have a bad time, most likely.
• This obviously does not support some of the nice-to-have functionality that the builtin Delegator and SimpleDelegator handle well, such as object equality, freezing, marshaling, tainting.

As a usage example, this is how I solved my original output capturing problem:

captured_output = StringIO.new
begin
  $stdout = MultiDelegator.new([$stdout, captured_output])
  # do something that generates output
  # ...
ensure
  $stdout = STDOUT
end
# do something with captured_output
# ...

Side note: I’m aware that replacing the global $stdout will not actually capture everything that you would consider output. It won’t capture output generated by C extensions or subprocesses; see this discussion for more details.