Fix Flaky Rails System Tests caused by slow scrolling or animations

Reading time: 6 minutes

Chrome and Chromium Smooth scrolling flag

Flakiness on browser-based system tests (integration tests) in Ruby on Rails may have a lot of different reasons. One is scrolling, which may be slowed down by browser setup (smooth scrolling), JavaScript callbacks which result in some kind of events like for animations, etc.

Such problems are not Ruby on Rails related and even not a programming language. They are mostly not related to a single browser like Google Chrome, Chromium, and Firefox only. They are also not browser driver specific like Selenium or Cuprite. Let’s dive in and check, how to analyze them, and figure out and try fix such problems.

Randomly or constantly failing tests

This is an example of a failing test. If you encounter random or constant failing tests with a similar message or maybe other, keep on reading:

E

Error:
BikesTest#test_update_a_bike:
Capybara::Cuprite::MouseEventFailed: Firing a click at coordinates
[180.65625, 2529.96875] failed. Cuprite detected another element with CSS
selector "none" at this position. It may be overlapping the element you are
trying to interact with. If you don't care about overlapping elements, try using
node.trigger("click").
test/system/bikes_test.rb:79:in `block in <class:BikesTest>'

According to this error, there is an element at the browser pixel position [180.65625, 2529.96875] that has some kind of overlapping element. Let’s try to figure out, what is wrong first.

Analyzing the problem

First of all, you need to figure out, what is going wrong. In browser testing there are more things, you have to consider. Let’s have a look at a common browser test called system test in Ruby on Rails:

require "application_system_test_case"

class BikesTest < ApplicationSystemTestCase
  test "update a bike" do
    bike = bikes(:one)
    visit edit_bike_url(bike)

    fill_in "model", with: "Super X5v"

    click_on "Save" # <-- here the test fails

    assert_selector ".flash_message", "Bike successfully updated."
  end
end

According to the test, we open a web form to update a bike, change one value in input_field named model, and click “Save” to submit the form and store the change in a database. This is a pretty simple test. Why may such a test fail?

To figure out, what is wrong, let us open the browser and check the page. This is done through save_and_open_page in Capybara before saving the change. This may look something like this:

require "application_system_test_case"

class BikesTest < ApplicationSystemTestCase
  test "as update a bike" do
    bike = bikes(:one)
    visit edit_bike_url(bike)

    fill_in "model", with: "Super X5v"

+   save_and_open_page # <-- debugger: save html page and open it in browser
    click_on "Save"

    assert_selector ".flash_message", "Bike successfully updated."
  end
end

For me, the page to test looked like expected in Browser. Even the “Save” button was there and was visible without extra CSS or JavaScript.

Next, I checked the screenshot, which is saved for every failed test. You can make one yourself if it is not done automatically by changing your test:

require "application_system_test_case"

class BikesTest < ApplicationSystemTestCase
  test "as update a bike" do
    bike = bikes(:one)
    visit edit_bike_url(bike)

    fill_in "model", with: "Super X5v"

+   save_and_open_screenshot # save screenshot as image and open it
    click_on "Save"

    assert_selector ".flash_message", "Bike successfully updated."
  end
end

At first, I couldn’t see any problems. One thing was, that I couldn’t see the save button as it was outside of the browser window. To have it on the screenshot, you have to scroll down first. Dependent on your driver you can use one of these commands to scroll down to the relevant position:

As I am using Cuprite, the code looks like this:

require "application_system_test_case"

class BikesTest < ApplicationSystemTestCase
  test "as update a bike" do
    bike = bikes(:one)
    visit edit_bike_url(bike)

    fill_in "model", with: "Super X5v"

+    page.driver.scroll_to(0, 2000) # command for scrolling 2000 pixels down
+    sleep 1 # <-- VERY IMPORTANT: Wait as scrolling is asynchronously

+   save_and_open_screenshot # save screenshot as image and open it
    click_on "Save"

    assert_selector ".flash_message", "Bike successfully updated."
  end
end

IMPORTANT: We are working with asynchronous processes running in parallel on one or mostly many may be fast or slow CPUs with more or less RAM. If you tell the browser to do something, dependent on the setup and the page you are testing it may take more or less time to finish the task.

As scroll_to and save_and_open_screenshot are asynchronous commands, you need to wait between them a little bit to see the result of the scroll_to command before creating a screenshot.

Suddenly the test started to work and stopped failing. There was something related to scrolling or elements being outside of the Browser Viewport.

Ask search engines

It is an easy task to ask a search engine. I did some searching and found this page:

It looked very promising and seems to be the problem my test suite has.

ATTENTION: Read it first, most things won’t work for you and some are just wrong due to browser updates.

I’ve updated my Capybara setup to have this option set:

# test/application_system_test_case.rb

   # https://github.com/rubycdp/cuprite#install
   # no-sandbox must be disabled for CI / Github Actions
+  # disable smooth scrolling as it breaks long pages
   Capybara.register_driver :cuprite do |app|
     browser_options = {}.tap do |opts|
       opts['no-sandbox'] = nil if ENV['CI']
+      opts['disable-smooth-scrolling'] = true # <-- This is for old
+                                              # Chrome/Chromium versions
     end
   end

It was still not working like expected on a MacOS System using an ARM based CPU from Apple. The tests failed like without this flag.

ATTENTION: I’ve used browser control and browser automation in many different projects outside of software testing and know, that browser flags and options may change in behavior and name and get removed at any time.

As I was using Chromium in tests, I checked the existence of this flag in Chromium:

Chrome and Chromium Smooth scrolling flag

As you can see, this flag was renamed from disable-smooth-scrolling to smooth-scrolling. I have to align it:

# test/application_system_test_case.rb

   # https://github.com/rubycdp/cuprite#install
   # no-sandbox must be disabled for CI / Github Actions
   Capybara.register_driver :cuprite do |app|
     browser_options = {}.tap do |opts|
       opts['no-sandbox'] = nil if ENV['CI']
+      # disable smooth scroling as it breaks on long pages
+      opts['smooth-scrolling'] = false
     end
   end

But there was another thing that is also Operating System (OS) specific. This flag is unavailable on MacOS. Such flags may be unavailable even on a specific CPU architecutre for the same OS. Check things in software, setup, documentation, and if possible in code to really know it.

As this option was fine for my CI systems using Linux as OS, this was a useful flag to have in the Capybara setup.

Animations

As some parts of the page to test were using animations, I’ve searched for more options and stumbled on this stackoverflow question:

I’ve tried the most upvoted answer and the tests started working again. The change was simple:

# test/application_system_test_case.rb

  # Use cuprite
   Capybara.default_driver = Capybara.javascript_driver = :cuprite

+  # Disable animations to speed up tests and avoid flakiness through animations
+  Capybara.disable_animation = true
+
   driven_by Capybara.javascript_driver

This feature was introduced in Capybara v3.2.0. You can check the documentation here: disable_animation=.

Summary

By disabling animation through Capybara in Browser based tests should become more stable - especially for heavily animated applications. It is also a big help with randomly failing tests. Bootstrap is using animations to display modals.

This was again a lesson for me to take more time and look at Changelogs/History for new features or relevant changes, which may improve developer experience or encounter new problems, which did not exist before.

Newsletter

See Also

Tags