Blog


Nov

Why wait_until was removed from Capybara


The release of Capybara 2.0.0 removed the wait_until method from the API. This seems to have frustrated a few people, so let me explain why this decision was reached and what your options are for the future.

wait_until was removed for several reasons:

  1. It simply isn't necessary for most applications and test suites.
  2. Its existence confuses people into thinking that it is necessary, when in fact it isn't.
  3. It was added at a point where Capybara's auto-waiting was much more primitive than it is now.
  4. It used to be used internally, it no longer is.

Capybara has a had a very important feature since pretty much the beginning which is that it automatically waits for elements to appear or disappear on the page. If you do something like find("#foo"), this will block until an element with id "foo" appears on the page, as will has_content?("bar"), click_link("baz") and most other things you can do with Capybara.

Let's look at a more complicated case:

find("#foo").click_link("baz")

Even if #foo is originally on the page and then removed and replaced with a #foo which contains baz after a short wait, Capybara will still figure this out. Let's make that really clear, Capybara is ridiculously good at waiting for content.

For the most part, this behaviour is completely transparent, and you don't even really have to think about it, because Capybara just does it for you. What trips a lot of people up is that they try to do something like this:

page.find("foo").text.should contain("login failed")

And now they have introduced potential timing issues. text, being just a regular method, which returns a regular string, isn't going to sit around and wait for anything. It will simply return the text as it appears when Capybara gets to this line and call it a day. Now after a long debugging session, our developer has found the timing issue. They now realize that there is a wait_until method in the API, and immediately think that, "hey, this sounds like what I need!"

wait_until do
  page.find("#foo").text.should contain("login failed")
end

Fantastic, it now works! The thing is though, Capybara could have easily figured out how to wait for this content, without you muddying up your specs with tons of explicit calls to wait_until. Our developer could simply have done this:

page.find("#foo").should have_content("login failed")

Or even:

page.should have_selector("#foo", :text => "login failed")

And the problem would have solved itself.

As long as you stick to the Capybara API, and have a basic grasp of how its waiting behaviour works, you should never have to use wait_until explicitly.

Synchronize

Capybara 2 introduces a new method called synchronize. While this method is part of the public API, don't run off and use it just yet. It has a very distinct use case, which is completely different from what you're probably using wait_until for. You will most likely only ever have to call this if you access the low level driver directly through #native. In this case, you might receive errors like Selenium's StaleElementReferenceError. It's these kinds of errors that synchronize prevents.

Assserting on model objects

I am firmly convinced that asserting on the state of the interface is in every way superior to asserting on the state of your model objects in a full-stack test. In some cases, especially if your interface is asynchronous, you might still want to do it though.

This is the only legitimate use case for wait_until I've heard of in Capybara.

Imagine that in the following scenario, "Liked" is shown immediately, through JS, regardless of if the change was actually persisted to the server or not:

click_link("Like")
page.should have_content("Liked")
post.reload.should be_liked

This would cause timing issues, if the AJAX request is slower than the reload. We could have written this as:

click_link("Like")
page.should have_content("Liked")
wait_until { post.reload.liked? }

Where wait_until could be implemented like this: https://gist.github.com/d8da686061f0a59ffdf7

Though I personally would have preferred something like:

click_link("Like")
page.should have_content("Liked")
expect { post.reload.liked? }.to become_true

Which could be implemented like this: https://gist.github.com/4129937

You could have also asserted this through the UI:

click_link("Like")
page.should have_content("Liked")
visit current_path
page.should have_content("Liked")

But it is kind of verbose, and it's also a lot slower, so I can understand why the model test might be preferred.

But then why not just bundle it

If and when you need some kind of behaviour that waits for things, wait_until is a giant big sledgehammer. There are more fine grained, sophisticated tools built into Capybara, and I want you to learn about them, because those are some of the best features of the library. And when the built in tools aren't enough, there are more sophisticated tools available than that clunky hammer. So hopefully the removal of wait_until encourages you to write better tests.

Nov

Simple tricks to clean up your Capybara tests


I'd like to share with you a few simple tricks we use all the time at Elabs to clean up our Capybara tests.

Often we want to find a specific area of the page, and perform some action in that area. With the imminent release of Capybara 2.0, you will probably find yourselves doing this even more, since you will sometimes have to be more specific about which element you want to interact with, since Capybara will raise an error when more than one element can be found.

You are probably familiar with the within method in Capybara, so say you want to click a link in the main menu, you might do it like this:

within "header nav" do
  click_link "Archive"
end

But in Capybara, all methods like click_link and fill_in and so on can be chained, so you could have written the above more succinctly as:

find("header nav").click_link("Archive")

That's better. However, the central idea behind Capybara is that you should test your application the way a regular user would use it; your user knows what the main menu is, but they don't really know anything about that CSS selector.

I once wrote a blog post called "You're cuking it wrong", about best practices when using Cucumber, it ended with the following general rule:

A step description should never contain regexen, CSS or XPath selectors, any kind of code or data structure. It should be easily understood just by reading the description.

While it doesn't make sense to be as strict about this ideal in plain Capybara tests, there is still a point to be made about avoiding low level detail even in these tests.

With that in mind, let's revisit that example. What if we could simply write:

main_menu.click_link("Archive")

That's much more obvious. We clearly communicate the intent, and we don't include any irrelevant details in the test. The implementation is super simple:

module MenuSteps
  def main_menu
    find("header nav")
  end
end

Now you might ask, what if I want to perform a lot of actions in a particular area of the page, this doesn't look so nice:

login_form.fill_in "Email", :with => "jonas@elabs.se"
login_form.fill_in "Password", :with => "capybara"
login_form.click_button "Login"

Thankfully, the within method can actually take as an argument an actual element, not just a selector. So we can write this instead:

within login_form do
  fill_in "Email", :with => "jonas@elabs.se"
  fill_in "Password", :with => "capybara"
  click_button "Login"
end

Concise and clear.

Arguments

Of course, since main_menu and login_form are just methods, we could create similar methods which take arguments. Imagine we wanted to delete a particular comment on our blog. We could write this:

find(".comment", :text => "Worst article ever!").click_on("Remove")

Hopefully you can see where this is going:

comment("Worst article ever").click_on("Remove")

And the implementation:

module CommentSteps
  def comment(text)
    find(".comment", :text => text)
  end
end

Much better!

Assertions

This is a bit specific to RSpec, but if you're using a different test framework, you can adapt this to your needs.

In another test for our blog, we are creating comments, and we want to assert that the comment appears as intended.

page.should have_selector(".comment", :text => "Best article ever!")

Remember our guideline/rule. That selector has to go! It would be fantastic if we could simply write:

page.should have_comment("Best article ever!")

But now we have a problem. We could create an RSpec matcher, but that would actually be a bit cumbersome. It could look like this:

module CommentSteps
  extend RSpec::Matchers::DSL

  matcher :have_comment do |text|
    match_for_should { |node| node.has_selector?(".comment", :text => text) }
    match_for_should_not { |node| node.has_no_selector?(".comment", :text => text) }
  end
end

It's important that you add both match_for_should and match_for_should_not, otherwise Capybara's waiting behaviour doesn't work properly.

This also doesn't give us particularly great error messages. We will get some variation on "expected true to be false", which isn't particular helpful.

There is a much easier way, which works well for simple cases like these:

module CommentSteps
  def have_comment(text)
    have_selector(".comment", :text => text)
  end
end

Same functionality, but easier to understand, shorter, and it even gives us better errors. We are making use of the fact that have_selector just returns an RSpec matcher.

Ensure on

One thing we face often in our tests is that we need to be on a particular page to do something. This is especially important if we have abstracted something into a helper. Consider this:

module SessionSteps
  def login
    visit login_path
    fill_in "Email", :with => "jonas@elabs.se"
    fill_in "Password", :with => "capybara"
    click_button "Login"
  end
end

But what if we already are on the login page when we call login, we'll visit the page again, which wastes valuable execution time. Let's make sure we don't do that:

module SessionSteps
  def login
    visit login_path unless current_path == login_path
    fill_in "Email", :with => "jonas@elabs.se"
    fill_in "Password", :with => "capybara"
    click_button "Login"
  end
end

We immediately spot that this pattern would be sensible to abstract. We have this in almost all of our projects, and we call it ensure_on:

module CommonSteps
  def ensure_on(path)
    visit(path) unless current_path == path
  end
end

Usage should be fairly obvious:

module SessionSteps
  def login
    ensure_on login_path
    fill_in "Email", :with => "jonas@elabs.se"
    fill_in "Password", :with => "capybara"
    click_button "Login"
  end
end

And for our final trick…

Suppose that on our blog, comments are moderated, and when someone posts a comment, we want to check that it appears in the moderation queue:

fill_in("Comment", :with => "Hi there")
click_on("Post")
visit moderation_queue_path
find(".moderation-queue").should have_comment("Hi there")

Of course we should create a moderation_queue method, like we did in the first example. But then we still need to remember that we need to actually visit that page before we fetch the element. Let's abstract that away:

module ModerationQueueSteps
  def moderation_queue
    ensure_on moderation_queue_path
    find(".moderation-queue")
  end
end

Now we can write:

fill_in("Comment", :with => "Hi there")
click_on("Post")
moderation_queue.should have_comment("Hi there")

That reveals our intent much clearer. We have practically created a small little DSL for our tests, just by creating a couple of really simple helper methods.

So that's it, some really simple tricks which allow you to write cleaner, more understandable tests with Capybara.

Mar

Working with time zones in Ruby on Rails


Rails provides great tools for working with time zones but there's still a lot of things that can go wrong. This blog post aims to shed some light on these gotchas and provide solutions to the most common problems.

The one that probably has tricked me the most times is the fact that Rails fools you to believe it got you all covered all the time (pardon the pun). Don't get me wrong. I want Rails to do as much work for me as possible. But I've learnt the hard way that I can't get away with not knowing when and how Rails is helping me. Another gotcha is the fact that you have more time zones in play than you might first believe. Consider the following: db, server, dev machine, system configured, user specific configured and the browser.

Configure your Rails app

So what tools do we have at our disposal as Rails developers? The most important one is the config.time_zone configuration in your config/application.rb file. ActiveRecord will help you convert from and to (which the documentation fails to explain) UTC and the time zone of your choice. This means that if all you're doing is having users post times through a form and use Active Record to persist it you're good to go.

Processing time information

So what about actually doing something with the time information before persisting it? That's when it becomes tricky.

Parsing

When parsing time information it's important to never do it without specifying the time zone. The best way to do this is to use Time.zone.parse (which will use the time zone specified in config.time_zone) instead of just Time.parse (which will use the computer's time zone).

Work with Numerical and ActiveRecord attributes

Method calls like 2.hours.ago uses the time zone you've configured, so use these if you can! The same thing is true for time attributes on ActiveRecord models.

post = Post.first
post.published_at #=> Thu, 22 Mar 2012 00:00:00 CDT -05:00

ActiveRecord fetches the UTC time from the database and converts it to the time zone in config.time_zone for you.

Date vs Time

Time has date information but Date does NOT have time information. Even if you don't think you care you might realize that you do sooner then later. Be safe and use Time (or DateTime if you need support for times very far from the present).

But let's say you're stuck with a Date that you need to treat as a Time, at least make sure to convert it to your configured time zone:

1.day.from_now # => Fri, 03 Mar 2012 22:04:47 JST +09:00
Date.today.to_time_in_current_zone # => Fri, 02 Mar 2012 00:00:00 JST +09:00

Never use:

Date.today.to_time # => 2012-03-02 00:00:00 +0100

Querying

Since Rails know that your time information is stored as UTC in the database it will convert any time you give it to UTC.

Post.where(["posts.publised_at > ?", Time.zone.now])

Just be sure to never construct the query string by hand and always use Time.zone.now as the base and you should be safe.

Working with APIs

Supplying

Building a web API for others to consume? Make sure to always send all time data as UTC (and specify that this is the case).

Time.zone.now.utc.iso8601 #=> "2012-03-16T14:55:33Z"

Read more about why iso8601 is advisable here: http://devblog.avdi.org/2009/10/25/iso8601-dates-in-ruby/

Consuming

When you get the time information from an external API which you don't have control over you simply need to figure out the format and time zone it's sent to you with. Because Time.zone.parse might not work with the format you receive you might need to use:

Time.strptime(time_string, '%Y-%m-%dT%H:%M:%S%z').in_time_zone(Time.zone)

Why there's no #strptime method on Time.zone when there's a #parse beats me. However don't forget to call in_time_zone(Time.zone) on your result!

Working with multiple user time zones

Many systems needs to support users entering and viewing time information in a variety of time zones. To achieve this you need to store each user's time zone (probably just one of the time zone string names found in rake time:zones:all). Then to actually use that time zone the most common pattern is to simply create a private method in your ActionController and run it as an around filter.

around_filter :user_time_zone, :if => :current_user

def user_time_zone(&block)
  Time.use_zone(current_user.time_zone, &block)
end

This will do the same thing as config.time_zone but on a per request basis. I still recommend to change the default config.time_zone to a time zone that is a good default for your users. (Thank you Matt Bridges for pointing out the potential problems with using a before_filter instead of an around_filter.)

Testing

All the above is something that your tests should catch for you. The problem is that you as the user and your computer as the development server happen to reside in the same time zone. This is rarely the case once you push things to production.

Highgroove just released Zonebie, a gem that helps you deal with this. I haven't had time to try it out myself yet, but it looks very promising. If you find this to be overkill, at least make sure that your tests run with Time.zone set to another time zone than the one your development machine is in!

Bug in Time.zone.parse

Jarkko Laine (@jarkko) pointed out that there's currently a bug in Rails that can have Time.zone.parse lose an hour when your system time is in DST (daylight saving time) and your configured time zone isn't. Jarkko has posted an issue on Rails' issue tracker and written a patch to correct the bug. Until the patch has been accepted or if you're running with older versions of Rails the only safe way to avoid this bug is to either monkey patch Rails in your app with Jarkko's fix or use:

# use
ActiveSupport::TimeWithZone.new(nil, Time.zone, DateTime.parse("2012-03-25 03:29"))
# => Sun, 25 Mar 2012 03:29:00 PDT -07:00

# or if possible pass the time zone in the string
Time.zone.parse("2012-03-25 03:29 PDT")
# => Sun, 25 Mar 2012 03:29:00 PDT -07:00

# instead of
Time.zone.parse("2012-03-25 03:29")
# => Sun, 25 Mar 2012 04:29:00 PDT -07:00

It should however be mentioned that it's pretty rare that this bug surfaces and when it does it can only lose you one hour. If you can live with that you probably do best by just waiting for the patch to be accepted.

Cheat Sheet

DOs

2.hours.ago # => Fri, 02 Mar 2012 20:04:47 JST +09:00
1.day.from_now # => Fri, 03 Mar 2012 22:04:47 JST +09:00
Date.today.to_time_in_current_zone # => Fri, 02 Mar 2012 22:04:47 JST +09:00
Date.current # => Fri, 02 Mar
Time.zone.parse("2012-03-02 16:05:37") # => Fri, 02 Mar 2012 16:05:37 JST +09:00
Time.zone.now # => Fri, 02 Mar 2012 22:04:47 JST +09:00
Time.current # Same thing but shorter. (Thank you Lukas Sarnacki pointing this out.)
Time.zone.today # If you really can't have a Time or DateTime for some reason
Time.zone.now.utc.iso8601 # When supliyng an API (you can actually skip .zone here, but I find it better to always use it, than miss it when it's needed)
Time.strptime(time_string, '%Y-%m-%dT%H:%M:%S%z').in_time_zone(Time.zone) # If you can't use Time#parse

DON'Ts

Time.now # => Returns system time and ignores your configured time zone.
Time.parse("2012-03-02 16:05:37") # => Will assume time string given is in the system's time zone.
Time.strptime(time_string, '%Y-%m-%dT%H:%M:%S%z') # Same problem as with Time#parse.
Date.today # This could be yesterday or tomorrow depending on the machine's time zone.
Date.today.to_time # => # Still not the configured time zone.

Epilogue

I hope you've learned something from this post. I sure did while writing it! If you have any feedback on how it can be improved, or if you spot any errors, please let me know by posting a comment below!

Feb

Capybara and testing APIs


Before you read this blog post, go read the Capybara README. Back? Good. Did you see any mention of methods like get, post or response? No? That's because those don't exist in Capybara. Let's be very clear about this, Capybara does not have those methods. So when you're writing a test like this:

visit '/'
response.body.should have_content("Hello")

Or possibly:

get '/'
page.should have_content("Hello")

Now you know why this couldn't possibly ever work. Because while visit and have_content are indeed part of the Capybara API, get and response are not. If these methods are defined inside your test/spec context, some other framework is defining them, most likely RackTest, and that framework and Capybara do not interact. Capybara knows nothing about what happens when you call get and likewise, RackTest knows nothing about what happens when you call visit.

For most people who have used Capybara for a while, this is old news. They tried variations like the above and they didn't work, they moved on and learned from those mistakes. The interesting question is why these methods do not exist in Capybara. Omitting these methods was a very conscious decision, a decision based on the following premise: Capybara is not a library suited to testing APIs.

There you have it. Do not test APIs with Capybara. It wasn't designed for it.

Capybara simulates user behaviour. Its API was designed to run against as large a variety of backends as possible. To achieve this, we have placed severe restrictions on what you can and cannot do with Capybara. We have limited the API to the most common subset of actions that a user would want to perform in a web application. This immediately precludes exposing the details of HTTP. A user knows exactly how to submit a form, but they have no idea that that form is sent as an HTTP POST request with certain HTTP headers.

I've seen so many people hack into the internals of the RackTest driver in Capybara. I want to make it clear that doing something like the following is using a private API, which is liable to change at any point (yes, even point releases):

page.driver.browser.post('/foo', data: "here")

Don't ever do this! Why would you? You could just use RackTest directly. In fact, as we saw before, it's probably already included in your spec context. So why make it harder than it needs to be:

post '/foo', data: "here"
last_response.body.should contain("Hello World")

Why would you write page.driver.browser.post when you could just write post? See that doesn't make any sense.

By extension, Capybara assumes the response to be HTML, if you're parsing page.body or page.source as JSON or XML, why not just use RackTest instead? You'll have a much nicer, cleaner API which won't break in the future.

tl;dr:

Use RackTest for testing APIs, use Capybara for testing user behaviour.

Nov

Solving Cucumber's Problems


Cucumber took the Rails community by storm a couple of years ago. For the first time, we had an easy way of excercising the full stack of our applications. Many people didn't even realize that behind the scenes there was another library, Webrat, doing all the hard work. Cucumber became the de-facto way of writing end-to-end tests in Ruby.

When I wrote Capybara, it was mostly to improve the experience of writing Cucumber features. Over time however, with the arrival on the scene of Steak and similar approaches, people realized that the Capybara API was quite efficient at driving acceptance tests on its own and began using it with just plain RSpec or another testing framework.

And all was well, case closed, right?

Over the last year, I kind of abandoned cucumber, to write acceptance tests in plain Ruby, Capybara and RSpec. Throughout this experience, I have tried to keep an open mind. My verdict is: I don't buy it. In the beginning, it was fantastic, the overhead of Cucumber was gone, we were insanely productive. But over time, cracks appeared. As the projects grew larger, the tests became more and more difficult to maintain. I have since tried to figure out why this is.

Imagine the scenario of creating a task with a particular title, in Capybara, this might look something like this:

visit(new_tasks_path)
fill_in('Title', :with => 'Buy milk')
click_button('Create')

Simple enough. Imagine that we do this a lot, now we want to abstract this. Also quite convenient:

create_task(:title => 'Buy milk')

That looks good. Now imagine that this task is attached to a milestone:

milestone = create_milestone('name' => '1.0')
create_task(:title => 'Buy milk', :milestone => '1.0')
create_task(:title => 'Drink milk', :milestone => '1.0')

That's quite okay too, but what if this is a common pattern, a milestone with multiple tasks?

create_milestone('name' => '1.0', :tasks => ['Buy milk', 'Drink milk'])

That looks fantastic!

Here's the problem though: no one ever builds this abstraction. There is so much overhead involved in implementing the create_milestone method, that in practice, it's simply not done. It's certainly not done for the first acceptance test that could have used it. And herein lies the whole crux of the problem: the default behaviour for acceptance tests in Ruby is to be unnecessarily verbose, and you have to constantly fight this behaviour in order to write maintainable tests.

It is in abstracting these kinds of common patterns that Cucumber shines. In fact, this abstraction is probably still at too high a level for Cukes. If cukes are written like this:

Given there is a milestone called "1.0"
And there is a task called "Buy milk" for the milestone "1.0"
And there is a task called "Drink milk" for the milestone "1.0"
When I visit the homepage
And I click "Milestones"
And I click "1.0"
Then I should see "Buy milk"
And I should see "Drink milk"

Then you are not gaining any benefit from cucumber at all. You really want something like this:

Given I am looking at a milestone with the tasks "Buy milk" and "Drink milk"
Then I should see "Buy milk" and "Drink milk"

In my experience, it's very difficult to write tests at this level of abstraction with Ruby and a lot easier to write them with Gherkin, the language that cucumber features are written in.

Problems

Still, going back to Cucumber after being in Ruby land for a while, I encountered a number of problems. These are the same problems that are mentioned by many abandoning cucumber for plain Ruby.

  1. Having a separate test framework is annoying
  2. Mapping steps to regexps is hard
  3. Cucumber has a huge, messy codebase
  4. Steps are always global

I have written a new library, which I believe solves these problems.

Turnip

Turnip parses Gherkin feature files and runs them in RSpec. You run your feature files the exact same way you would run a normal spec file, and they are automatically run when you run your RSpec suite. So to run a feature file with Turnip, you would do something like:

$ rspec spec/acceptance/view_milestone.feature

Steps are implemented with strings instead of regexps, like this:

step "there is a task called :name" do |title|
  Task.create(:title => title)
end

It still allows for some variation in natural language by allowing a pseudo syntax for optional letters or alternative words:

step "there is/are :count monster(s)" do |count|
end

Just like Markdown, we're aiming for something which follows the natural conventions of writing text, instead of using the more arcane regexp syntax. The idea is to cover the 90% use case very well, instead of allowing every possible variation.

Turnip was written just to solve this particular, rather simple problem. There is no support for other programming languages, no wire protocol, it doesn't have its own runner or formatters or anything. Its only dependencies are rspec and gherkin.

In Turnip, steps can be local by scoping them to tags:

steps_for :interface do
  step "I do it" do
    click_link('Do it')
  end
end

steps_for :database do
  step "I do it" do
    Do.it!
  end
end

Now just tag the scenarios with the @interface and @database tags and you have different behaviour for the same step in different scenarios.

@interface
Scenario: do it through the interface
  When I do it

@database
Scenario: do it through the database
  When I do it

Conclusion

I don't know if Turnip solves the problems that Cucumber has. I don't know if Cucumber is the right solution for you. I do believe that Cucumber has a lot of benefits which the hivemind of this community has too easily dismissed this past year or so. I have tried to separate the ideas of Cucumber from its implementation. Try it out and see if you like the result!