Choosing the right tool for performance testing is crucial for several reasons:
First of all, it must be able to carry out performance tests on the installed software base you wish to test, and therefore, offer adequate protocol coverage. Not all tools can test Citrix, SAP or Oracle Forms, for example. It also has to match the customer’s organization and way of working.
Secondly, price or subscription is also an important criterion.
Finally, ease of use, quality of support and a strategy for upgrading versions to keep pace with market developments are all important criteria.
In this article, we offer a synthetic comparison of 3 tools.
JMeter is an open-source load-testing tool now widely used by all types of customer: public sector, major accounts, various companies.
Its graphical interface is simple and efficient. It can be used to test all Web applications, Webservices (REST, SOAP), Web 2.0, messaging services, etc…
JMeter comes with native functions (in the macro sense) for dynamic data injection and/or on-the-fly data manipulation during load testing.
Developed in JAVA, JMeter offers a large number of plug-ins to various monitoring technologies.
It can be used for testing early in the development cycle – Shift Left – i.e. API, Web services and Microservices testing – as well as for end-to-end testing. Its recording interface is efficient.
JMeter’s receiver elements enable results to be visualized in the form of graphs or basic statistics, and results to be interpreted.
It’s a complete tool which, together with a Grafana-type reporting tool and a Prometheus-type metrics capture tool, enables load tests to be carried out in a wide range of contexts.
Neoload is a load-testing tool from Tricentis (since the acquisition of Neotys).
Neoload supports protocols not offered by Jmeter, such as Citrix, Oracle Forms, SAP, etc.
It also offers integrated functions for script variabilization and dynamic variable management.
In recent releases, NeoLoad has taken an important step towards the Shift Left approach, enabling teams to test earlier in the software development cycle, at API level, thanks to new features such as in-code performance testing and Swagger file import.
In addition to the scenario capture tool for scripting and the load injection control module, it also offers a powerful integrated reporting module, as well as a monitoring module for a comprehensive tool.
What’s more, it is interfaced with APM tools, notably Dynatrace, which make it easier to correlate response times with technical metrics, and thus bring great added value to recommendations for improving and tuning technical components.
The tool is not free, and its licensing model has recently changed to make it significantly more expensive.
Gatling is a basic open source performance testing tool that is being used more and more frequently, particularly in a context where the user company decides to implement performance testing very early in the release delivery cycle (mainly in agile mode) and where developers are more involved in the performance commitments of their code. Gatling makes it easy to perform API and Webservice tests.
Gatling is a “developer’s tool” in the sense that its user interface and organization make it easy for developers to find their way around, whereas traditional testers (using JMeter or Neoload, for example) have trouble finding their way around (I’ve tested …). You could say it’s a performance testing tool in code form. Note that Gatling is developed in Scala and requires a basic knowledge of this development language.
For the rest, the functionalities are there, close to those of JMeter, and the 2 tools are comparable in their advantages and limitations to an editor-type tool like Neoload.
It should be noted that Gatling offers a paying version which, for example, gives access to support and is necessary for distributed testing (controller + multiple load generators).