How To Run Reliable E2E Tests (And What You Should Avoid)

How To Run Reliable E2E Tests (And What You Should Avoid)

By Laurens Van Wiele Icon comments Tags: Development

Buggy software sucks. That's why we use End-to-End testing at Talon.One to validate our systems and check our integrations.

In this post, we are going to share how to avoid one very common pitfall of E2E testing: unstable tests.

When writing E2E tests, there is an unavoidable tradeoff between how thorough the tests are and how reliably tests will run in your CI environment.

Does this sound familiar?

We often hit technological roadblocks on the challenging quest to simulate real user behaviour. Selenium inexplicably starts throwing Net::ReadTimeOut errors. Your webapp goes unresponsive for no apparent reason, or all of a sudden some element becomes unclickable… even though none of these issues never pop up on your machine! As a consequence, the test gets labeled as “unstable”. This one test failing becomes “normal” and it can be “ignored”.

Eventually a few more of these unreliable tests pop up and before you know it, nobody cares about tests failing at all.

Your dream of simulating real user behaviour suddenly disintegrates into nothing more than a checkmark your startup can claim they use (E2E tests? Yeah we got those!).

We want to help you, so below you’ll read exactly how we avoid this trap.

1. The app you are testing should use a fresh build

Companies often choose to run E2E tests against a pseudo-stable dev instance of their app with a dirty database. You know, the type of server that gets re-deployed 5 times a day with untested builds.

This is a recipe for disaster and can lead to constant failures that have nothing to do with your tests at all. Maybe your tests run just as the app is restarting, a page gets redesigned without your knowledge, or somebody pushes a change that corrupts the database. All these events will lead to red lines in your reports, and none of it is within your control.

Remember: Docker is an absolute godsend to facilitate a fresh and clean build of your webapp when you run your tests.

2. Tests shouldn't depend on each other

Each test should run completely independent and its result should not depend on the outcome of a previous test. This sounds trivial, but when writing E2E tests, it often happens unintentionally. Imagine this scenario:

Test 1: user visits page A  
Test 2: user is on page A, clicks link to visit page B  
Test 3: user is on page B, clicks link to visit page C  

It might be tempting to run all of these tests right after each other, in the same browser session. Should save some time, right? However, if test 2 fails and you are still on page A at the end of the test, now test 3 will fail as well because you are not even on page B.

In this case, test 3 can only pass if test 2 passes. You've basically created a waterfall of failing tests.

How to avoid this? Test 3 should start with an empty browser, and explicitly visiting page B.

Remember: every test should start in a fresh browser window with all cookies cleared.

3. Don’t do prep work in the UI

Unstable tests are often long tests that spread multiple UI interactions over many different pages.

Remember: always keep tests short and focused.

This means they will have a lower chance of failing (when Selenium gets lost again).

For example, if you want to run the test Place new order as an existing customer, don’t run your entire sign-up process over the UI, only place the new order through the browser.

Before(test){  
  user = MyApi.create_user;
  MyApi.login_with(user);
}

Describe(test){  
  user.visit_page;
  assert(page_is_loaded)
  user.place_order
  assert(order_is_placed)
}

Any data required to run the core of the test can be created via API or by inserting data in the database. This will make the test run a lot faster and reduce the risk of unreliable UI interactions messing up your results.

4. Re-run tests before actually labelling them as failed

Almost all testing frameworks offer support for flaky tests, either natively or through a plugin. Some examples:

If a test fails unpredictably from time to time and you have run out of ideas for improving reliability, try running it again. And again, and again. We suggest 3 times, just to be safe.

5. If all else fails… just delete the test

Graphic of error on computer screen

At the very least, remove the most unstable part of it. Sometimes the technical barriers to testing a scenario are higher than the value of the test itself.

Remember, an unstable test is worse than no test at all and you should never avoid testing or writing untestable code.

It might sound blasphemous to make E2E tests only focus on one specific feature. But the most important thing is that the test are reliable.

Final words of wisdom

Whenever a test fails, everyone involved should be concerned and assume that something is actually wrong. Testing should be taken seriously and we always recommend running E2E tests to validate your systems.

Testing is a good investment for your company and has a lot of benefits for your application. E2E testing helps reliably test all aspects of your application before a release. Although testing can sometimes be painful, hopefully you have learned a few ways to make your tests more efficient, reliable, and save you from a big headache.

If you have questions about running your own E2E test or want to share your experience, leave a comment or get in touch!

Read more posts by this author.

Laurens Van Wiele

Subscribe to our newsletter

Get insights and inspiration for mastering promotion marketing.

Unsubscribe whenever you like.

There was an error subscribing to the newsletter. Please try again.

Thanks for subscribing to the Talon.One newsletter!

Related articles

Comments

Check out our Community Guidelines.