Benchmarking, performance testing, load testing, stress testing, etc.

This article/section is a stub — probably a pile of half-sorted notes and is probably a first version, is not well-checked, somay have incorrect bits. (Feel free to ignore, or tell me)

The many more names for tests. More than most of us can remember or define on the spot.

And some are fuzzily defined, or treated as interchangeable in some contexts.

Code testing

Unit testing

"is this piece of code behaving sanely on its own, according to the tests I've thought up?"

Typically for small pieces of code, often functions, behaviour within classes, etc.

Upsides:

• unit tests are often often a form of self-documentation

say, example cases are often unit tests as well

and say, if I want to dig into details, then seeing

assert uri_component('http://example.com:8080/foo#bar') == 'http%3A%2F%2Fexample.com%3A8080%2Ffoo%23bar'

gives me more information than a comment/docstring saying "percent-escapes for URI use" (which is imprecise in practice)

can be particularly helpful in helper/library functions

• it makes you be more precise about the behaviour of a piece of code

particularly useful to hammer down lower level utility functions, for you and others

• the more dynamic the behaviour of the code, the more it forces you to think about edge cases

• the more dynamic the language (as in dynamic typing), the more it forces you to think about edge cases

• sometimes you discover edge cases you didn't think of, and didn't implement correctly

this is easily overstated, but I've definitely done this before

• it helps others see your approach to code quality

practically not via the presence of unit tests, but what you're actually testing

• unit tests are part of regression testing

basically, for 'this part is fragile, and we expect future tweaks may break this again'

• when you're also looking at code coverage, you are more likely to (eventually) write tests that check error paths

Arguables:

• you will only write tests for the things you thought of (and probably coded correctly), not the things you forgot (and probably didn't code)

this a decent indication of completeness, but not a guarantee against bugs.

Downsides:

• the less dynamic the behaviour, the more that unit testing converges on testing if 1 is still equal to 1

which wastes time

which can give a false sense of security

• the more dynamic the behaviour (in the 'execution depends on the actual input') sense, the less that a few tests actually prove correctness at all.

In fact, tests rarely prove correctness to begin with (this is an extremely hard thing to do), even in the most overzealous forms of TDD

most of the time, they only prove you didn't make the most obvious mistakes that you thought of

• while on paper the "try your hardest to think of everything that would break it" idea is great

...if you look around, a buttload of unit tests are of the "think of things you know probably work anyway" sort

...because a lot of people write unit tests only because someone told them sternly (often by someone who barely understands when they are useful and when not)

• a lot of real-world bugs sit in interactions of different code, and unit tests do not test that at all

• The more OO-ey code is, the less unit tests do anything (TODO: elaborate)

• if it involves locking, IPC, network communication, or concurrency, or interact with other parts of the program that has state (think OO), or other programs that have state, the less you really test - or can even say what you have tested or not.

such things are hard to test even with much fancier techniques

• there is no good measure of completeness of your unit tests

if you think code coverage is that thing, you are probably a manager, not a programmer.

Regression testing

"Is this still working as it always was / doing what it always did / not doing a bad thing it had previously done?"

Refers to any kind of test that should stay true over time.

In theory any test that you do not throw away will act as a regression test, but this is about tests you don't just keep it around because you wrote it before, you keep it around because it's a "a test we wrote when we fixed a nasty bug, to ensure we won't regress to that bug later" - hence the name.

Regression tests are often as simple as they need to be, and frequently a smallish set of unit tests is enough.

Reason you may keep a test around include:

• a fixed bug was subtle and easy to miss, so more likely to be reintroduced. A guard against that is great

• this code is often touched and often altered

• this code is used by a lot of the codebase, so bugs (or breaking changes, or false assumptions) are far reaching

Upsides:

• should avoid such bug regression well

• May also help avoid emergence of similar bugs.

Arguables / downsides:

• very specific, may cover less than you think

• which can mean a false sense of security

Integration testing

"Does this code interact sanely with the other code / parts of the program?"

Where many code tests (particular unit tests) test components in isolation, integration tests takes components (which presumably already have unit tests) and aims to test whether they interact properly.

It's not a test of your product as a whole, because that tends to be later in the process. Or, these continuous-delivery days, sometimes never ('user tests means deploying to users and see if they complain, right?').

Integration tests are often the medium-sized tests you can do during general development.

Acceptance testing

"Are we living up to the specific list of requirements in that document over there?"

Said document often says 'functional design' at the top.

Which can involve any type of test, though in many cases is a fairly minimal set of tests its overall function, and basic user interaction, and is largely unrelated to bug testing, security testing, or such.

These draw in some criticism, for various reasons.

A design document tends to have an overly narrow view of what really needs to be tested, you're not necessarily even testing whether the whole actually functions, or acts as people expect.

The more formally it's treated, the less valued it is when people do their own useful tests.

Also relevant

Self-testing code

This article/section is a stub — probably a pile of half-sorted notes and is probably a first version, is not well-checked, somay have incorrect bits. (Feel free to ignore, or tell me)

Self-testing code is code that includes some checks inside its own code.

This often amounts to mean

assert() statements within a function, e.g.

testing important invariants

doing your own regression checks

intentionally borking out earlier rather than later when a bug could have wide-reaching implications (e.g. around concurrency)

https://en.wikipedia.org/wiki/Self-testing_code

Testing larger chunks

End-to-End testing

This article/section is a stub — probably a pile of half-sorted notes and is probably a first version, is not well-checked, somay have incorrect bits. (Feel free to ignore, or tell me)

Basically, testing whether the flow of an application works, basically with a 'simulated user.

The goal is still to test the application at mostly functional level - whether information is passed between distinct components, whether database, network, hardware, and other dependencies act as expected.

End-to-end testing is often still quite mechanical, and you might spend time specify a bunch of test cases and expected to cover likely uses and likely bugs.

This is in some ways an extension of integration testing, at a whole-application and real-world interaction level, finding bugs

While you are creating a setup as real users might see it,

It's not e.g. letting users loose and see what they break.

Fuzz testing

Fuzz testing, a.k.a. fuzzing, feeds in what is often largely random data, or random variation of existing data.

If software does anything other than complain about bad input, it may reveal bordercases you're not considering, and e.g. the presence of exploitable buffer overflows, injection vectors, ability to DoS, bottlenecks, etc.

Perhaps used more in security reviews, but also in some tests for robustness.

See also:

• https://en.wikipedia.org/wiki/Fuzzing

Load, performance, stress testing

This article/section is a stub — probably a pile of half-sorted notes and is probably a first version, is not well-checked, somay have incorrect bits. (Feel free to ignore, or tell me)

Load testing is a name that groups various others, broadly meaning 'put load on a system and see what happens', yet often your goals is more specific, so the name is too.

Consider:

• stress testing: find if it stays well behaved under heavy load

e.g. disconvering issues related to resource contention, memory leaks, timeouts, overflows, races, non-elegant failure, etc.

• longevity/endurance testing: find if it stays well-behaved over time, probably with a constant load of jobs

(doesn't necessarily test stress conditions)

should e.g. reveal memory leaks, things like counter overflow bugs

• soak testing: somewhere between longevity testing and

see if a realistic production load reveals bugs that targeted tests do not - like resource leaks

• performance testing: push it until it won't go faster, to see how fast that is, often related to:

finding the minimum expectable time to handle any unit of work (e.g. with near-empty tasks)

finding whether there are unusual variations in that time, and if so where they come from

finding the maximum rate the system handles for real-world load

check whether you get reasonable response under expectable average load (black box style - not yet caring why or why not)

finding the maximum rate the system handles under unusually complex load

finding which parts are the bottleneck

also to eliminate or relieve them where possible (not so black-box; may involve a lot of detailed system diagnosis)

finding a baseline speed to compare against in future tests, to see changes in speed

• see what effect code/feature changes have on any of this

Notes:

• One useful distinction:

performance testing is seeing how fast it can go,

stress testing is seeing whether it does something unexpected while doing that.

• The "how fast can it go" part is, compared to most other tests, more of an artform to do meaningfully, because specific benchmarks represent specific uses, hidden assumptions, are often unrealistic workloads, don't test degradation under load, integrated parts, etc.

(In some cases done primarily to please your PR department - "Lies, damned lies, and benchmarks")

Longevity/endurance testing, soak testing

This article/section is a stub — probably a pile of half-sorted notes and is probably a first version, is not well-checked, somay have incorrect bits. (Feel free to ignore, or tell me)

Longevity testing means running the system for a longer time, to evaluate system stability under production use.

It often means giving it a basic load to deal with during that time, not to stress it but enough to e.g. reveal memory leaks within reasonable time.

Notes:

• Endurance testing, capacity testing, and some other names are basically the same thing.

• in modern release-often cycles, this is often replaced with a some extrapolation and assumptions

or, if the only purpose is finding memory leaks, a test that tries to evoke them specifically

Soak testing is very similar in that it also asks you to run it for a long time if you can, but specifically asks you to try to make load simulates real production load, not necessarily stress testing but enough load to reveal issues like memory leaks, hidden problems in transaction processing.

The term seems borrowed from electronics, where it's testing it above e.g. temperature ratings for a while to see when and how it fails.

You might go further and think about the contents of tasks you submit, e.g. some repetition and some fuzz testing to this, to reveal hotspots and such.

Stress testing, recovery testing

This article/section is a stub — probably a pile of half-sorted notes and is probably a first version, is not well-checked, somay have incorrect bits. (Feel free to ignore, or tell me)

Stress testing is basically seeing if you can break the system by attempting to overwhelm its resources.

Often done to see whether a system will merely slow, fail on specific tasks, or fall over in general.

Recovery testing often amounts to stress testing plus seeing what happens what happens when resources are taken away, simulating partial failure. Consider not only CPU but also IO, services, connectivity, memory.

Useful in general but perhaps more so in cloudy arrangements.

Often done to see

• whether it will recover from specific resource failures at all - seeing if it becomes slower (preferable) or just falls over.
• whether stress during recovery will lead to repeated failure

Common pitfalls in benchmarking

This article/section is a stub — probably a pile of half-sorted notes and is probably a first version, is not well-checked, somay have incorrect bits. (Feel free to ignore, or tell me)

Measuring latency / client rate when you wanted to measure system rate

Network clients that do things sequentially are implicitly limited by network latency, because that is the amount they will sit around just waiting.

So a client's latency is oftne limiting factor to a single client's interaction rate, but in a (properly written) server this says nothing about what the server can do.

It may be that any single client can get no more than 50 req/s (because it was elsewhere on the internet), but the system you're requesting things from could reach 5000 req/s if only you pointed enough distinct clients at it.

Measuring rate when you wanted to measure latency

Say you do 20 small writes to a hard drive.

End-to-end, the act of going to platter disk might take 10ms for each operation.

Yet if your set of writes was optimizable (e.g. very sequential in nature so they could get merged) it's possible that a set of operations take only a little more than 10ms overall.

If you then divide that 10ms by 20 writes and say say each operation took 0.5ms, you will be very disappointed when doing each individually takes 10ms.

That 0.5ms figure may numerically the average, but is almost meaningless in modelling operations and their performance.

Measuring the overhead

tl;dr:

• Overhead is very important only when it's on the same order of magnitude as the real work.
• Overhead is probably insignificant if it's an order of magnitude lower

Let's say you are testing how fast your web server requests can go.

One framework points out it can do 15000 req/sec, which is clearly multiples faster than the other one that goes at a mere 5000 req/sec. For shame!

...except via some back-of-napkin calculations, I would actually expect that to be less than a percent faster in any real-world application.

Why?

Because the difference between those two rates is on the order of 0.1 milliseconds.

Maybe those microseconds are spent doing some useful utility work, that your hello world test doesn't need, but almost every real app typically does. Maybe the faster one is a bare-bones you have to do absolutely everything yourself framework, and when you do, there is no longer a difference. In fact, it might even be faster at that extra work, and the only criticism is that it apparently puts it in there by default. You don't know.

But even if we assume is 100 microseconds of pure waste - consider that many real web requests take on the order of 10ms, because that's how long the calculation or IO for something remotely useful tends to take.

So now we're talking about the difference in responding in 10ms and 10.1ms. That difference is barely measurable, it falls away in the latency jitter.

If I cared about speed, it would be my code I should be examining, because I can almost certainly improve it by lot more than 100 microseconds.

Measuring the startup, hardware tuning, etc.

This article/section is a stub — probably a pile of half-sorted notes and is probably a first version, is not well-checked, somay have incorrect bits. (Feel free to ignore, or tell me)

The more complex the thing being executed, the more likely it is that something is being loaded dynamically.

Say, if there's something as complex as tensorflow under the covers, then maybe the first loop spends 20 seconds loading a hundreds-of-MByte model in to the GPU, maybe doing some up-front autotuning, and all successive loops might take 10ms to run.

Even if you pre-loaded the model, the first loop may still be 100ms..1s because of the autotuning or just because some further things weren't loaded.

Or even during the first iterations - some libraries might optimize for your hardware, at a one-time cost.

Or even during later ones - JIT compilers will often do some analysis and tweaking beyond the first iterations.

The low-brainer workaround is often just to run a bunch of loops on real-ish data, before you start actually timing things.

Timing inaccuracies

This article/section is a stub — probably a pile of half-sorted notes and is probably a first version, is not well-checked, somay have incorrect bits. (Feel free to ignore, or tell me)

It's easy to do:

time_started = now()
do_stuff()
time_taken = now() - time_started

The shorter that do_stuff takes, the more it matters how fine-grained this now() function is, and how much overhead there is in calling that now(), and the less that time_taken means anything at all.

For example

• python's time.time() was meant to give the seconds-since-epoch timestamp, it just happens to do it as precise as it can

...but assume that it's no better than ~10ms on windows (it can be, but don't count on it), and ~1ms on most linux and OSX

time_ns (since py3.7) was introduced to deal with precision loss due to time() returning a float.

but only gives nanosecond [[|resolution versus precision|resolution, not precision]] - the precision might be down to the 100ns..250ns range on linux(verify). It's still 1ms on windows. (also note that at this scale, python execution overhead matters a lot)

Measuring your cache more than your code

This article/section is a stub — probably a pile of half-sorted notes and is probably a first version, is not well-checked, somay have incorrect bits. (Feel free to ignore, or tell me)

Okay, you've read the timing inaccuracies, and wrote "repeat this small code fragment a million times and/or at least a few seconds, and divide the time taken by the amount of runs"

That is better, in that it makes the inaccuracies in the timing around that code negligible.

And it is also worse, because you now run into another potential issue:

The smaller that code fragment is, the more likely it is that running it as a tight loop means it gets the bonus speed from various caches (L1, L2, RAM, page cache, memcache).

When optimizing inner-loop number crunching, this may be sensible, but if this code is never run that way in practice, you are now really testing your cache, and not even realistically at that (because of access patterns).

When your aim was to to get an idea of speed of running it once in everyday use, you just tested basically nothing.

This isn't even a fault in the execution of a test, it's a fault in the setup of that test, and in thinking about what that test represents.

See also

• http://agiletesting.blogspot.com/2005/02/performance-vs-load-vs-stress-testing.html
• http://www.soft.com/News/QTN-Online/qtnsep02.html

http://en.wikipedia.org/wiki/System_testing