Testing MongoDB on Kubernetes with locust — First impressions — Part 1

It was a regular Friday at office and I just had converted a standalone MongoDB to replica set for HA. The things application per se, looks fine but wanted to try out little load testing. I was searching for load testing tools to try out and found Apache Jmeter, Gatling and locust framework from results. I am from python background and sheer installation steps for java tools discouraged me from trying out Jmeter and Gatling. so i ended up using locust.

NOTE: This article may or may not suggest a standard approach but rather just gives my thoughts on trying things out for the first time in my way.

Before i write any locust tests, I need to first test connection with Mongo from my laptop.

Mongo was installed as statefulset in replicaset (https://docs.mongodb.com/manual/replication/) architecture with 1 arbiter and 4 nodes. I have very restricted access to cluster but full access to my namespace. I wanted to run locust on my laptop so i had to expose the statefulset — i had node port service and port-forwading options with me and chose port-forwading way for testing.

$ kubectl port-forward -n my-ns svc/mongodb 27017 27017

This would expose my headless mongodb service on local port 27017. I could connect to Mongo on my laptop as below:

conn = MongoClient('mongodb', 27017, replicaSet='rs0')

PS: You need to add mongodb as alias to localhost in your /etc/hosts file for this work, or just use localhost instead on special names.

This works fine on cluster but when i ran this on laptop, there were many errors like mongo-0.mongodb.my-ns.svc.cluster.local: name or service unknown This error repeated in any config i tried with connection string and the errors seemed to be DNS resolution of replicaset node members by the looks of it. From https://api.mongodb.com/python/current/api/pymongo/mongo_replica_set_client.html

Changed in version 3.0: MongoClient is now the one and only client class for a standalone server, mongos, or replica set. It includes the functionality that had been split into MongoReplicaSetClient: it can connect to a replica set, discover all its members, and monitor the set for stepdowns, elections, and reconfigs.

So i’m guessing the mongo client tried to discover all members of replicaset from service and was unable to DNS resolve them, but same worked in cluster for obvious reasons. So i had to expose all the members of mongo replicaset to my script?? I thought that would be the solution and used a service to select individual pods in the statefulset as below:

The selector statefulset.kubernetes.io/pod-name is automatically added to pods in statefulset by Kubernetes and helps us target each pod easily. Since this is statefulset, the names always have consistency. I did the same for all members of replicaset from mongo-{0–4} and mongo-arbiter-0.

Now i updated my port-forward command to expose all these services. I wrote a make target but you could use a shell command with proper trap to exit the port forward as well.

Note that i had to expose pods on different ports to make this work and ensure there is no & at the end of last command — IMPORTANT.

conn = MongoClient('mongodb://mongodb-0:27017,mongodb-1:27011,mongodb-2:27012,mongodb-3:27013,mongodb-arbiter-0:27014', replicaSet='rs0')

PS: You need to add all the names — mongodb-{0–4} and mongodb-arbiter-0 to your hosts file to make above command work.

Using this i was able to connect to replicaset properly. The next step is to write locust file to test Mongodb.