JMeter – Distributed Load Testing using Docker in AWS

In the Previous post, We had learnt how to use docker in creating multiple containers running jmeter-server for distributed load testing. But we had created all the containers in the same host. Even if we can not use the single-host containers for performance testing with huge load, before pushing your performance test script related changes to AWS/any other cloud service providers, that setup will be useful to test your scripts in local.

In this post, we will see how to use docker in AWS for JMeter distributed load testing.

Creating AWS Instances:

  • I created 3 t2-micro instances in AWS.
    • Image Id: ami-d732f0b7
  • Added a security group as shows here.

aws-fw-01

  • Installed latest version of docker. Check here for installation steps.

Creating docker-containers:

As our AWS instances are up and running, Lets create a docker container on each host by issuing below commands.

  • JMeter-Master: On one of the instances, we will run below command.
    • sudo docker run -dit --name master vinsdocker/jmmaster /bin/bash
  • JMeter-Server/Slave: On remaining instances, we will run below command to create jmeter-server container.
    • sudo docker run -dit vinsdocker/jmserver /bin/bash

Now we have docker containers for jmeter-master and jmeter-server are up and running fine with all the dependencies. If we try to find the IP address of these containers, they all might look same – [172.17.0.1] for all the containers. So, our jmeter test will NOT work in this set up as master can not identify the slaves in the network.  If you remember, we ran all our docker-containers on same host. The containers on the same host will be able to talk among themselves with container’s specific IP address as docker-engine creates a default network for these containers on the same host.

jm-master-slave-host-docker

But in this AWS, the set up is almost as shown below. The master-container inside the host can NOT talk to the slave-containers on the other hosts – because containers on each host will be in their own separate network. So they can not communicate.aws-docker-container-01

The communication among the docker-containers on different hosts will be routed via their hosts. So, it can be easily fixed by using  port mapping & using host IP instead of containers.

First lets run below command to stop and delete all the containers.

sudo docker stop $(sudo docker ps -a -q)
sudo docker rm $(sudo docker ps -a -q)

Port MappingWhile creating a container, we will be mapping the exposed ports of containers to a host port. So, by talking to host on the mapped port, you will be talking to the actual container.

java.rmi.server.hostname Property: As the containers have their own ip addresses, we need to make Jmeter to communicate via host ip by updating java.rmi.server.hostname. For more info on java rmi properties, check here.

Dockerfile for JMeter Client / Master:

It does not require any change.

Dockerfile for JMeter Server / Slave:

It needs to be modified slightly as shown here.  I have added -Djava.rmi.server.hostname=$LOCALIP while starting jmeter-server.sh. LOCALIP will be a variable and the value will be passed at run time while creating the container.

 

Lets create jmeter-server containers on each host [except master] using below command. [Note: I have used different docker image – vinsdocker/jmawsserver]

sudo docker run -dit -e LOCALIP='52.10.0.2' -p 1099:1099 -p 50000:50000 vinsdocker/jmawsserver /bin/bash
sudo docker run -dit -e LOCALIP='52.10.0.3' -p 1099:1099 -p 50000:50000 vinsdocker/jmawsserver /bin/bash
sudo docker run -dit -e LOCALIP='52.10.0.4' -p 1099:1099 -p 50000:50000 vinsdocker/jmawsserver /bin/bash
  • LOCALIP should be the public IP address of the host.
  • -p 1099:1099 – is used to map the 1099 port of the container with the host port 1099
  • -p 50000:50000 – is used to map the 1099 port of the container with the host port 50000

Run the below command on master host to create a jmeter master container.

sudo docker run -dit --name master -p 60000:60000 vinsdocker/jmmaster /bin/bash
  • Container port 60000 is mapped to the host port 60000.

After creating all the containers, the setup is almost as shown below.

 

aws-docker-container-02

We can run the test now in the master container by issuing below command.

./jmeter -n -t docker-test.jmx -Djava.rmi.server.hostname=52.10.0.1 -Dclient.rmi.localport=60000 -R52.10.0.2,52.10.0.3
  • -Djava.rmi.server.hostname=52.10.0.1 -> exposes the jmeter-master IP to the slave containers
  • -Dclient.rmi.localport=60000 -> listening port of host
  • -R52.10.0.2,52.10.0.3 -> Slaves host IP address

 

aws-docker-container-03

Summary:

By using docker, we do not need to worry if we have same version of jmeter and java are installed on each host. docker take cares of all these.  Using docker-containers on a single host was very simple and straight forward. But when you use it in AWS, in order to make the containers talk among themselves, we need to use java.rmi.server.hostname property and port mapping. Once we do these, then everything works as expected.

Instead of using LOCALIP variable while creating a docker jmeter-server container, we can also use below techniques to communicate with slaves.

  • ssh [port fowarding technique] .
  • docker-multihost-network/docker swarm

We will see how to use above techniques in the next post.

 

Happy Testing 🙂

 

Share This:

26 thoughts on “JMeter – Distributed Load Testing using Docker in AWS

  1. This is really helpful, i was struggling to setup docker jmeter containers for distributed testing. I have a query: 1. If i have a testplan.jmx for 1000 users, would it be enough to use one slave for jmeter. 2. How many slave AWS instances need to be created if wants to run 1000 users with 8GB capacity of an instance. Thanks in advance.

    1. Hi Lakshmi, There is no easy formula to calculate that. It completely depends on your test plan. If you have more elements, assertions, listeners etc, your test plan itself consumes more memory.
      However, you can get an idea by monitoring CPU / Memory usage. I normally ensure that my load generators do not consume more than 80% of CPU/Memory.

      Below articles will help you with that.

      http://www.testautomationguru.com/jmeter-server-performance-monitoring-with-collectd-influxdb-grafana/
      http://www.testautomationguru.com/jmx-monitoring-using-collectd-influxdb-grafana/

  2. I am getting below error when tried to run the script.jmx file from master continer:
    Exception creating connection to: 172.x.x.x; nested exception is:
    java.net.NoRouteToHostException: No route to host
    Failed to configure 172.x.x.x
    Stopping remote engines
    Remote engines have been stopped
    Error in NonGUIDriver java.lang.RuntimeException: Following remote engines could not be configured:[172.x.x.x]

    Note: 172.x.x.x is the slave IP;

    1. This article explains that – you can not use docker’s private IP. You need to use the public ip of the host machine. Before running jmeter in distributed mode, check if you are able to ‘telnet’ that remote machine successfully.

  3. Thanks for the suggestion. if i telnet with it works. Port is mandatory here. Please suggest if i have to use port number in the .jmx file command as shown below.
    ./jmeter -n -t test.jmx -Djava.rmi.server.hostname=10.x.x.x:22 -Dclient.rmi.localport=60000 -R10.x.x.1:22

  4. Big thanks. I am able to run the script successfully after adding ports in security group.. :). Now how can i get results from jmeter master container to my local. Please help.

  5. Able to accomplish the task by running below command to get execution results, worked fine.
    ./jmeter -n -t test.jmx -Djava.rmi.server.hostname=10.x.x.x:22 -Dclient.rmi.localport=60000 -R10.x.x.1:22 -l results.csv
    And able to get this file to EC2 instane also using docker cp…
    Can you please help if i can use the jmeter master container also as slave along with the other remote host as another slave. 2 EC2 instances one as master (as well as slave) and other one as only slave. Will it be possible executing this way.

  6. Is there anyway to push/load docker Jmeter master results to S3 bukcer/AWS bucket by including some script in master dockerfile. We have a task to upload result to S3 bucket once execution is done. Please help.

  7. Thanks for all your helpful articles, changed the plan to push/pull from git than S3 bucket. Need your guidance to solve below issue: I have created my own docker files (copy pasted from your files.. :)), getting below error when run the run command. Please suggest.

    1. You are welcome to modify the dockerfile as you wish. Did the test work fine with the existing dockerfile? do you try to add any jmeter-plugins which could cause this issue?

  8. with existing dockerfile, it works fine (vinsdocker account). able to run successfully with one master 2 slaves. Since i had to use plugins added plugins in my docker file and published to docker hub. When tried to use my docker files facing the above issue (in jmbase: added plugins, jmserver: used one more port mapping 1099 3000 to use the master machine as slave also and installed this jmserver in master machine along with master docker file, jmawsservre: used this in two slaves)

  9. I am facing one issue, I have 1 master and two slaves setup (AWS EC2 instances using jmeter docker images). I have placed my csv files under the slave machines jmeter bin folder and given the same path in my .jmx file however when run non gui command from master machine, csv files are not read. It shows EOF in view results tree. My jmeter slave machine container pwd: /jmeter/apache-jmeter-2.13/bin/usernames.csv. Please suggest.

  10. Thanks for the info. Able to solve the issue by using the full path of jmeter docker bin folder of slave machine in csv data set config with /. Is there any way to get the slave machine log info when we run master command in jmeter docker jmmaster.

  11. if it has two or more jmeter-slaver containers in one host,how to map the slaver port in the container with the host port

    for exemple: if the 1099 host port is maped to the 1099 slaver container port already, how to map the 1099 port of another container in this host?

    in the others words: can a host port map to multiple containers ports?

    1. why would you need more than 1 slave on a single host? Always run 1 container per host. running more than 1 container on a single host for load test will not help at all. in fact it would affect the performance. If you still need to do, you can map any available port of the host (say 1100) to the 2nd container port 1099 using -p 1100:1099 during docker run command

  12. Thanks for sharing logs info, it helped a lot. Is it always required to expose 50000 in slave and map the master with 60000 ports (or any other different ports). is it possible to use the default port 1099 on both master and slaves?

    1. These ports can be anything. However you would need 2 ports for slave and 1 port for master. All these are for different purposes. 1099 alone will not help.

  13. We are trying to use EBS (elasticbeanstack) with Bamboo CI tool for Jmeter cluster. Using terraform files to create machines. For this purpose, can’t use already built docker image. Need to build the docker and pass IPs during run time from Bamboo. Please let me know if i can update the docker file of server like below (other stuff for this docker file is same as given in the article)
    ENTRYPOINT $JMETER_HOME/bin/jmeter-server

    CMD sudo docker run -dit –name slave1 -e LOCALIP=${slave_1} -p 1099:1099 -p 50000:50000

    Please pardon if doing/using wrong commands as i am new to docker.

  14. I am able to connect to the slave machines. The test gets triggered but the test is hung at this line..
    Waiting for possible Shutdown/StopTestNow/Heapdump message on port 4445
    Following is the command that i am using to run at my ec2 machine that has the code
    ./jmeter -n -t /home/ubuntu/jmeter/scripts/ready-load.jmx -Djava.rmi.server.hostname=52.XX.XX.XXX -Dclient.rmi.localport=60000 -R 52.XX.XX.XXx,54.XX.XX.XXx -l log.jtl

    how to get the script to complete. I have the apache-jmeter.3.3.

    Log of the jmeter.log does not have any info . Can you help me to proceed ?

      1. Are you using docker?
      2. Are your sure that ports are open in all the instances?

      Try this in master for each and every slave-ip:

      telnet slave-ip-address 50000
      telnet slave-ip-address 1099

      Try this from a slave:

      telnet master-ip-address 60000

    1. If the ‘telnet’ works as expected, are you sure that test really does not run!? It might not be logging anything in the console. Try passing these properties as well in your command.

      -Gsummariser.ignore_transaction_controller_sample_result=false -Gsummariser.name=summary

  15. Hi Sir, we are trying with different AWS regions (3 ec2 machines having public ips). From Jmeter Master exposed Port 60000, and from Slave ‘EXPOSE 1099 50000’ – using Jmeter docker images similar to your article. Added security groups with ports added to ec2 instances.
    The problem facing now is, able to telnet every slave from master but can’t telnet master from anywhere or slave. Please suggest.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.