QR code

DynamoDB + Rake + Maven + Rack::Test

  • Odessa, Ukraine
  • comments

Amazon Web Servicesaws Rubyruby testing

In SixNines.io, one of my Ruby pet web apps, I’m using DynamoDB, a NoSQL cloud database by AWS. It works like a charm, but the problem is that it’s not so easy to create an integration test, to make sure my code works together with the “real” DynamoDB server and tables. Let me show you how it was solved. The code is open source and you can see it in the yegor256/sixnines GitHub repo.

How to bootstrap DynamoDB Local

First, you need to use DynamoDB Local, a command line tool created by AWS exactly for the purposes of testing. You need to start it before your integration tests and stop it afterwards.

To make things simpler I suggest you use jcabi-dynamodb-maven-plugin, a Maven plugin that I made a few years ago. You will need to add pom.xml to your repository and start/stop Maven from a Rakefile, just like I’m doing here:

task :dynamo do
  FileUtils.rm_rf('dynamodb-local/target')
  pid = Process.spawn('mvn', 'install', chdir: 'dynamodb-local')
  at_exit do
    `kill -TERM #{pid}`
  end
  begin
    Dynamo.new.aws.describe_table(table_name: 'sn-endpoints')
  rescue Exception => e
    sleep(5)
    retry
  end
end

First, I’m removing dynamodb-local/target, the directory where Maven keeps its temporary files, to make sure we always start from scratch.

Then, I’m starting mvn install, using Process.spawn, as a background process with pid as its process ID (this won’t work in Windows, only Linux/Mac). Then I immediately register an at_exit Ruby hook, which will be executed if Ruby dies for any reason. I’m sure it’s obvious why I have to do that—in order to avoid garbage running in the background after Rake is finished or terminated.

Pay attention, I’m using kill -TERM instead of kill -KILL, in order to give Maven a chance to wrap everything up, terminate DynamoDB Local correctly, close its TCP port and exit.

How to check that it’s running

Next I’m checking the status of sn-endpoints, one of the tables in the DynamoDB Local. It has to be there if the server is up and running. It will be created by jcabi-dynamodb-maven-plugin according to sn-endpoints.json, its JSON configuration.

Most probably the table won’t be ready immediately though, since it takes time to bootstrap Maven, start the server, and create tables there. That’s why, if the exception is thrown, I catch it, wait for 5 seconds and try again. I keep trying many times, until the server is ready. Eventually it will be. It takes about 12-15 seconds on my MacBook, which means 2-3 attempts/exceptions.

How to connect to DynamoDB Local

My classes need to know how to connect to the server during integration tests. In production they need to connect to AWS, in testing they have to know about the DynamoDB Local instance I just started. This is what I have in my class Dynamo, which is responsible for the very connection with DynamoDB. Its decision on where to connect is based on the environment variable RACK_ENV, which is set to "test" in test__helper.rb, which is included by rake/testtask in front of all other tests, thanks to the double underscore in its name.

If the environment variable is set to "test", Dynamo takes the connectivity parameters from the YAML file dynamodb-local/target/dynamo.yml created by maven-resources-plugin:copy-resources. The TCP port of the DynamoDB Local database will be there, as well as the DynamoDB authentication key and secret.

How to run the integration tests

This is the easiest part. I just use my objects the way they are supposed to be used in production and they connect to DynamoDB Local instead of AWS.

I’m using Rack::Test in order to test the entire application, via a set of HTTP calls. For example, here I’m trying to render Jeff’s user account page. Its HTTP response code is supposed to be 200:

require 'test/unit'
require 'rack/test'
class AppTest < Test::Unit::TestCase
  include Rack::Test::Methods
  def app
    Sinatra::Application
  end
  def test_user_account
    header('Cookie', 'sixnines=jeff')
    get('/a')
    assert_equal(200, last_response.status)
  end

##

Now you can run the entire test from the command line. You can see how Rultor runs it while releasing a new version: full log. Also, see how it works in Travis. In a nutshell:

  • You call rake in the command line and Rake starts;
  • Rake attempts to run the default task, which depends on test;
  • Rake attempts to run test, which depends on dynamo;
  • Rake, inside test task, runs mvn install in the background with this pom.xml;
  • Maven unpacks DynamoDB Local installation package;
  • Maven reserves a random TCP port and stores its value into ${dynamo.port};
  • Maven saves ${dynamo.port} and key/secret pair info;
  • Maven starts DynamoDB Local, binding it to the reserved TCP port;
  • Rake waits for DynamoDB Local availability on the reserved port;
  • Rake imports all test classes starting from test__helper.rb;
  • Environment variable RACK_ENV is set to "test";
  • Rack::Test attempts to dispatch a web page;
  • Dynamo loads YAML config from dynamo.yml and connects to DynamoDB Local;
  • Rake stops;
  • Ruby terminates Maven and it stops DynamoDB Local.

That’s it.

sixnines availability badge   GitHub stars