Protractor is dead, long live Cypress! - Part 1

On 24th April, Angular announced the deprecation of their E2E testing tool protractor. It was unclear if there will be a successor or if Angular delegates this to its users. At the time of this writing, WebDriver.IO, TestCafé and Cypress came up with schematics for the Angular CLI.

In this article, I will provide a short overview over the differences between the various E2E frameworks, argue why you should use Cypress and - in the second part - help you with the first steps.

If you prefer watching over reading, then this recording of my talk is for you:

Landscape of E2E testing frameworks

WebDriver-Based

We can roughly split the available tools into two groups. The ones that are based on WebDriver and the ones that operate on the Chrome DevTools Protocol (CDP).

Selenium was released in 2004. It merged with a similar tool called WebDriver. Selenium is still a major player and Protractor is more or less a wrapper around it.

With Selenium/WebDriver, developers have a common API to automate a browser. Every supported browser must provide a driver which WebDriver uses for the communication. This makes these frameworks cross-browser capable. This means that you can run the same test on Safari, Chrome, mobile browsers, etc.

Over time, WebDriver, the core technology in Selenium, became a W3C standard. That opened doors for other vendors to create their own frameworks based on WebDriver.

Unfortunately, WebDriver-based tests have the reputation to be flaky. This means that if they run under the same circumstances multiple times, they can fail or succeed. That is extremely bad for a testing framework. You spend a considerable amount of time building something that should guarantee the stability of the application and then you can't even trust that.

Chrome DevTools Protocol

Two quite popular alternatives are Puppeteer and Playwright. They don’t rely on WebDriver but talk directly to the browser via the Chrome DevTools Protocol (CDP). This gives them much better control which leads to more stable tests.
CDP binds to the Chromium-based browsers (Chrome, Edge). Cross-browser tests, as we find them with WebDriver, are not possible. Firefox meanwhile supports CDP as well and therefore Puppeteer, etc.

Some of Puppeteer’s original developers are behind Playwright. So one could see Playwright as a rewrite of Puppeteer. It also uses CDP but also supports Safari. That is possible because it ships with patched browsers.

WebdriverIO is a hybrid framework. It provides both access to CDP and WebDriver.

In which category does Cypress fall? Cypress relies on CDP as well. In contrast to Puppeteer, Cypress uses CDP only to manage the tests. The tests themselves don't use CDP at all but run directly in the browser. There is no layer between the tests and the browser anymore. Test and application code are one. Your test code has direct access to the DOM and not an indirect one via CDP or WebDriver.

That's the same approach that Selenium took before it merged with WebDriver. Looks like history repeats itself.

Flakiness vs. Cross-Browser

From a very high level view, the more abstraction we have between our tests and the browser, the flakier tests tend to become. On the other side we can do cross-browser testing. The closer we get the browser the more stable the tests become. We have to accept though that we can only run the test in one particular browser technology, i.e. Chromium.

In the future, all browsers might support CDP or the upcoming WebDriver BiDi will bring stability into the WebDriver world. Things are still evolving.

Why you should use Cypress

First, Cypress does not use WebDriver but runs its tests directly in the browser. These make the tests stable and reliable. Exactly what we wanted in the first place.

The second reason is the development experience (DX). Cypress' authors invested a lot of effort in making it as easy to use as possible. This does not mean that its API is easy to use. The tooling, the community work, and especially the documentation is just fantastic.

OK, so Cypress is better than Protractor. Why should you pick Cypress over any other E2E library?

As Angular developers, we always appreciated that we got an "opinionated framework" which pre-selected various tools for us. Although, it might look like Angular leaves the spot for E2E empty, there is another big player: Nx has been providing the best tools for Angular applications for quite some years.

Nx is a wrapper around the CLI. It started out as a candidate for MonoRepositories but over the years became more and more of an Angular CLI++. In fact, I personally would always pick nx, if I start a new Angular application. There is no lock-in, no overhead, I just get better tooling and can stay with the Angular CLI.

And guess what? Nx replaced Protractor with Cypress all the time along. So nx takes care that Angular and Cypress work well together.

The last argument is the usage of Cypress among Angular developers. Before the deprecation was made public, Angular did a survey. They asked their users what E2E they are using. That survey showed that 64% for Cypress. You call that a landslide victory in politics. And please remember, that was at a time when Protractor was the official tool! Protractor ended up with only 19%.

Screen Shot 2021-03-24 at 6 31 33 PM — Source: https://github.com/angular/protractor/issues/5502

Why you should not use Cypress

Depending on your use case, Cypress might not fit your needs:

No cross-browser testing: If you need to run your e2e tests in Safari, other browsers or even on mobiles devices, you need to use another tool. Cypress is here definitely the wrong choice.
Switching domains within one test: If your applications spans over multiple domains, you will also not be happy with Cypress.
Switching tabs: Also not possible.

This is just a logical consequence if you think of how Cypress works. Tests run inside the browser and vanish if you change the URL or the tab.

Summary

Cypress gives you both stability and great developer experience. The majority of Angular developers use it already.

If you are running E2E in Protractor, now is the time to migrate! If you haven't written E2E until now, take a look at Cypress. You will be surprised how easy it is.

Switching to Cypress means rewriting your E2E tests. You can do that in an incremental way. Cypress and Protractor don't exclude each other. They can be run in the same Angular project.

Be aware that Cypress might not fit your needs. If cross-browser support is an absolute must, you should check out alternatives like WebDriverIO, etc.

Outlook

In the second part, we will do our first steps with Cypress.

Protractor is dead, long live Cypress! – Part 1