Automatic testing of web applications is bread and butter for any Quality Assurance specialist. But among so many testing suites available and various opinions about each of them, one can lose his bearings. We've come our way of trying a few end-to-end testing tools and we ultimately chose Cypress. But his article won’t be a paean to Cypress.io (although it might seem like one), neither will I be drawing any comparisons between it and any other testing suites. You can find lots of such content in the Web.
But over the course of this brief article, I'd like to show you why we even started seeking a new automatic testing suite, how Cypress eventually worked out, and why we find it the best fit to our QA team at Monterail.
Intrigued whether we've found the Holy Grail of the end-to-end testing tool? Let's dive deeper into our real life case and see yourself.
When the app’s development was advanced enough that we began thinking the next release, our client (working in the network security industry) started asking questions about the stability and security of the application with increasing frequency.
It became clear that a 30% test coverage was not a satisfactory outcome for any involved party.
The problem was that we’ve hit a wall with Nightwatch-based testing. To give an example, the Selenium Web Driver was very issue-prone, we’ve had problems with test stability, and we were rather dissatisfied with the pace of writing new tests. It quickly became apparent to us that if we were to boost our test coverage to include most of the features, we would have to find an alternative testing suite. And a good one at that.
After reviewing relevant posts across a number of blogs and forums, and poring over roadmaps for a couple of different testing suites, we ultimately decided to give Cypress a shot. Initially, we had our app tested by two separate tools. Only after we were certain that Cypress would become our main suite did we rewrite the remaining tests for the chosen framework.
After spending a couple of months with Cypress, I have a handful of pros and cons that I’d like to share. The list of advantages is long so brace yourself.
Despite the fact that the technology is still very popular, and the community and documentation have matured, there is no denying that testing modern Web apps requires software capable of keeping up with all the recent trends in tech. Selenium can handle a spectrum of platforms, browsers, and languages (a huge advantage of that particular technology), but if you take a look at the image below and appreciate that automatic testing really can be that simple, you’ll realize that Cypress at the very least deserves a chance.
How Cypress works, Source: Cypress.io
If you’re using Node.js, installing Cypress in your project will require just a single line in the console:
npm install cypress
and presto! :) Obviously, proper configuration of the suite and the tests themselves to fit your your project is not that simple. But the truth is, that Cypress will allow you to run some basic tests in minutes (seconds even?) after completing the installation. New version available? Type in
npm install --save-dev email@example.com and you’re ready to go.
At least we’ve never had Cypress produce any issues with nodes, their versions, and related dependencies. Cypress simply works.
Well, it looks exactly as it sounds. Just look at the screenshot below. It gives you a precise overview of what’s being tested, how long will it take, how many tests have already been completed, and how many failed. Pausing, restarting, timers, previews, logs—everything’s there, in a neat and digestible package sitting in your browser window.
Running tests in Cypress.io, Source: Cypress.io
The list includes integration with dev tools, screenshots, snaps etc. The clear and informative dashboard is cool, but if you add in the ability to monitor every step and assertion of your tests (including snaps and application states in any given moment of the test), as well as additional information available to you through the dev tools you’ll quickly come to realize that Cypress has really elevated the debugging of your tests (and your application, too) to a higher level.
At present, Cypress can be integrated with around 15 CI providers. At Monterail, Google Cloud Platform (with Docker) is the CI tool of choice for this project. Thanks to the headless feature, we were able to integrate our testing with the Google Cloud Platform. At this point, tests take around 10 minutes. Recently, Cypress also introduced the ability to parallelize test runs. It should help us cut down the time it usually takes to run a test to around 5 minutes, at least that’s the results that the Cypress staff have been touting. Additionally, we were able to launch the feature that saved an app screenshot whenever the test managed to produce an error. A very useful feature, as sometimes logs are not enough to deduce what went wrong, so it’s nice to have another way of looking at what was happening with the app in a given moment.
By this I mean detection, debugging, verification, waiting for requests. Naturally, these features are nothing new in the automatic testing world. But working with network requests in Cypress is more than friendly. The Cypress dashboard, along with the dev tools console, allows us to check out each request at any moment in the test. Quickly and cleanly. Cypress also allows us to halt the tests until a given request is fulfilled by the server.
Network traffic control in Cypress.io, Source: Docs.Cypress.io
DevTools Console output of XHR Request sent by Cypress Runner
Cypress features a built-in mechanism that handles waiting for DOM elements (all we have to do is set the maximum waiting time, which is set at 10,000ms by default). This brings two advantages:
waitare more or less eliminated from the tests. The result?
Although Cypress is still a fairly new endeavor, we had no trouble finding help online throughout the nearly twelve months we already spent writing E2E tests in Cypress. The official API and the documentation look really good. The knowledge base and user activity on the official support forum and relevant corners of Stack Overflow are also very promising.
Saving the test file automatically launches the test in the dashboard. A small thing, but highly enjoyable, as it allows you to concentrate on writing code rather than running tests.
No one with any experience in writing automatic tests and basic knowledge of JS should have any trouble with handling Cypress syntax (I know I haven’t had any).
Cypress gives us the ability to work with “jQuery-like” selectors (although it’s not the only option). If we were to add to that Cypress’ built-in commands, such as
first(), we’d quickly realize that we’re capable of capturing 99% of the elements in our app swiftly (and clearly), without having to add new classes or attributes.
Alright, it’s never all roses. Cypress tends to lack in these areas:
If you need to test apps in Firefox, you won’t be able to run these tests with Cypress (at least not as of this writing). It’s a considerable drawback, but we didn’t need cross-browser testing capabilities so much that it would force us to stay with our previous testing suite. At this point, Cypress only supports Chrome and Electron.
Such as upload, download file, tab key, etc. This is one feature that we genuinely miss, and its lack gets more noticeable as the app grows. Native events don’t actually handle our killer features, but we’d feel more secure if we could at least test them in our end-to-end tests.
As of this writing, both cons are being actively handled by the open-source community, as per the roadmap. It seems that soon enough we’ll be able to both tests our apps in Firefox and Edge, as well as use the browsers’ native events in testing.
One of the reasons Cypress remains so difficult to ignore is the fact that it’s a project with a very strong open-source community backing.
Reading the forums, the project documentation, and seeing the roadmap, you can’t help but get the impression that it’s being developed by people perfectly aware of what a fast, intuitive, and easy Web app testing engine should look like. Before committing time and energy to learning an emerging technology, it’s good practice to establish whether the project itself is moving in the right direction.
Cypress is still a relatively new piece of technology. At this point, it’s hard to compare it against the capabilities offered by more mature testing suites. However, in case of our project, it worked splendidly. Writing new tests is no longer a challenge, and we’re able to extend the test coverage to cover new and upcoming features basically in real time. And it is our expectation that soon enough we will be able to put the suite to the test in other projects, too, to ascertain its effectiveness in other cases.