Load Testing with Apache JMeter

As new clients are onboarded, how do we at Qualtrics ensure that our applications can withstand the growing user base? We hammer our systems in a controlled environment via load testing. This allows us the peace of mind that new features and product launches are able to withstand the expected client usage. While there are many different load testing frameworks to choose from, various engineering teams have decided to use Apache JMeter after researching and reading resources like this blog post. JMeter not only gives us a user-friendly desktop GUI, but also allows multiple customizable plugins that can be attached to the core application.

Monitoring

One of the first steps in determining the load a service can take is to add proper hardware metrics. As mentioned in a prior blog post, we use Grafana to monitor the health of our server boxes. The most important graphs we monitor during the load test are the following 4 metrics:

• CPU
• Disk I/O
• Memory Used
• Network Throughput

Another alternative for external monitoring is to watch the error logs in Sumologic and mark when their number begins to increase. This indicates that your customers are experiencing reduced performance using your application.

Browser Configuration

Next, we need to download the appropriate software in order to conduct the load test. In this walk-through, we will use version 3.0 of JMeter and the latest install of Firefox to perform the load test. We use Firefox instead of other popular web browsers because we have found only Firefox allows for manual port proxy configuration, which will be needed for this test.

In order for Firefox to talk to JMeter, we need to setup a manual proxy configuration so that the service calls can be simulated through the tool. To do so, navigate to the Network settings under Preferences > Advanced > Network. From there mimic the configuration shown in the screenshot below.

We are using port 8080 for our tests, but if that port is already being utilized in your own system, feel free to configure to another port number.

JMeter Setup

After unzipping the download and running the bin/jmeter executable, we start by making sure we always save the WorkBench. This ensures that closing the window won’t result in any lost work.

From there, we will add a Recording Controller and a HTTP(S) Test Script Recorder to the WorkBench to listen and record web activity done on the Firefox web browser through the port proxy. To configure the Test Script Recorder to listen on the right port, change the values in the Global Settings to match the port you configured in the Firefox proxy and domain that you plan to test.

To record, hit the Start button at the bottom of the configuration screen and click through the scenario you wish to record on the web browser. Hit the Stop button once you finish the action you would like to repeat. We have seen periodic instances of root certificate warnings during testing. These can be ignored unless the recording does not work. If that is the case, we recommend checking out the JMeter documentation to resolve your issue.

Test Plan

Under the Test Plan section of the JMeter GUI, add a Thread Group element. This controls the number of users you plan on simulating: each thread acts like an individual user hitting your service. If your application is already in production, start with the number of users you have and work upward from there. If the application is still in R&D, feel free to estimate a relatively large number based on your expected user base.

App performance is always bottlenecked by the heaviest network operations, so to replicate high load for the service you will want to use a Loop Controller to repeat those network calls. You should inspect your code to determine the heaviest network operations to repeat. Endpoints with long computational steps or that call external services are usually the first culprits, but trial-and-error experimentation can be done as well to find the hardest hitters. Once you’ve found the offending operation(s) you’d like to test, drag them from under the Recording Controller into the Loop Controller.

You can then run the Thread Group to see how your app performs under this simulated high load. During this time, pay close attention to the software you chose to use for monitoring the health of your service. You should be able to see a significant increase in usage for your service once you begin to approach the maximum load it can handle.

Discovering Bottlenecks

There you have it! At a most basic level, you will be able to see the spike in load on your services via the external metric graphs on the hardware. To dig more into each individual request, you can utilize the online library of JMeter plugins to learn more about what happens at each request. The specific plugins chosen depend on what you expect to measure during the load test, but one that we found useful for recording our own services is listed below.

Active Threads Over Time

Applying to Multiple Projects

This walk-through has covered how to load test a single application, but most teams at Qualtrics own multiple applications. To help make the load testing framework reusable, we dockerized the JMeter tool and exposed command line arguments for configurable parameters of the test. Now to load test multiple applications, all teams need to do is pass in the custom configuration file and the Docker container will simulate a high load scenario on each application.

Final Thoughts

Load testing our applications has given us confidence in the number of clients we expect our services to handle. As our customer base grows, we constantly strive to figure out where bottlenecks lie in the code to make our customer experience better. We can also use this knowledge to determine how much hardware we need when onboarding new clients. Understanding the intricacies of load testing will give you the confidence in your application to support your growing user base.