On Non-functional Testing

Here's a quick thought: non-functional testing of a system is developing for QA.


Yep. Oh, sure, not in every way—but it offers the same traps as developing for coders. Being aware of this helps testers avoid this trap, and be a bit more empathic towards the dev guys.

Let's look an example. Can you load-test the OpenWeatherMap API? It's a simple enough HTTP API, let me show you:

GET https://api.openweathermap.org/data/2.5/weather?q=London,uk

Back comes a bit of JSON:

  "coord": {"lon":145.77,"lat":-16.92},
  "weather":[{"id":803,"main":"Clouds","description":"broken clouds","icon":"04n"}],
   "// bla  bla"

So, load test: Figure out, how much it can take. Surely, you can do that, right? Go!

No worries, no worries, that's easy! Ab is installed in a heartbeat, you look up the parameters, and boom! First test done. You fire up a spreadsheet, and start filling in the data, plot a graph and there you go, not a day has gone by and the answer is: 400 concurrent requests!

400, sure? Not 401?

Come on, I don't know that exactly. I tested 50, 100, 200, 400 and 600, and 600 was too slow.

Too slow, what does that mean?

Well, 300 milliseconds or so—clearly that's too slow for a scalable API?

What's that anyway? Median response time, maximum, average?

Uhm, median?

Let's get back to that—what requests did you test with anyway?

No worries, I didn't go for the easy stuff, I alternated City ID, City name, geo coordinates and ZIP code as input parameters. And all a different city.

So, only four cities? And you did see the other API, with multiple cities?

Yeah, but, come on, we have to start somewhere?

True. So you tested from your laptop, right?

Yep, that was fastest.

Via a public WiFi network? Did you look at the ping?

Well, no. But isn't it more the concurrency we're testing?

And so on. Assumptions everywhere, none of them checked, half a day's worth of effort for nothing.

Reminds you of anything? Exactly! That sounds like nascent software development! You hear the problem and your fingers already crave to type, you can see the design in front of you, there will be three classes and the module will be very simple, no need for fancyness, and a couple of unittests, too—but you've still no idea what the problem is that you're solving.

That's why I think non-functional testing—any of performance, load or stress testing—is developing for QA. It's challenging, it's fun, and makes you oh so eager to get started.

So, if you're a QE and you feel that tingling in your fingers, do take a breath, hold it for two seconds and then tell yourself what you've probably just told that new Junior dev: I'm not going to test anything you code until you can explain me what problem you're solving and why you're solving it like this. Your load tests will be awesome—and they'll actually answer the question that someone is interested in.

When you're done load testing, spare some pity for that Junior dev next time. We've all been there, and we all love developing.