Difference between revisions of "Benchmarking, performance testing, load testing, stress testing, etc."

From Helpful
Jump to: navigation, search
m (Unit testing)
m (Common pitfalls in benchmarking)
 
(3 intermediate revisions by the same user not shown)
Line 20: Line 20:
  
  
They are also useful in [[#regression testing|regression testing]]
+
They are also useful in [[#Regression testing|regression testing]]
  
  
Line 76: Line 76:
  
  
Tests that should always be true (because things build on it),
+
Basically it's any tests (unit tests or more complex) that should ''stay'' true,
but you suspect might be altered, e.g. because it is often-touched code.  
+
because other things build on it.
  
Useful to avoid regressing to older bugs, as well as emergence of similar bugs.
+
...but you suspect the code might be altered, e.g. because it is often-touched code.
  
Makes sense for parts that may be rewritten, may be precarious in their interactions.
+
 
Often introduced after a specific bug was found, to ensure it won't be introduced again later.
+
Often introduced after a specific bug was found, to ensure we won't regress to that bug later.
 +
This can also help avoid emergence of similar bugs.
  
 
===Acceptance testing===
 
===Acceptance testing===
Line 97: Line 98:
  
 
If software does something other than complain about bad input,
 
If software does something other than complain about bad input,
it may reveal cases it's not considering, and e.g. the presence
+
it may reveal bordercases it's not considering, and e.g. the presence
 
of exploitable buffer overflows, injection vectors, ability to DoS, etc.
 
of exploitable buffer overflows, injection vectors, ability to DoS, etc.
  
Line 218: Line 219:
 
<!--
 
<!--
 
tl;dr:
 
tl;dr:
* Overhead is very important when it's on the same order of magnitude as the real work.
+
* Overhead is very important only when it's on the same order of magnitude as the real work.
 
* Overhead is insignificant if it's an order of magnitude or more away.
 
* Overhead is insignificant if it's an order of magnitude or more away.
  
Line 224: Line 225:
 
Let's say you are testing how fast your web server requests can go.
 
Let's say you are testing how fast your web server requests can go.
  
If one framework can do hello world at 5000 req/sec and ananother at 15000 req/sec, you would go for the latter, right?
+
If one framework can do hello world at 5000 req/sec and ananother at 15000 req/sec
 
+
You would go for the latter, right?  
 
+
But the thing above doesn't mean three times as fast, ''at all''.
+
It means the overhead of handling differs approximately 70 ''micro''seconds.
+
  
 +
No? That's a do-nothing test, which basically means there is a 70 ''micro''second difference in overhead. If that's a constant, you would probably not care at all.
  
If you are writing a memcache-like thing, you might care.
 
  
 
Yet most public-facing serving is faster than a few dozen milliseconds,
 
Yet most public-facing serving is faster than a few dozen milliseconds,
Line 244: Line 242:
 
<!--
 
<!--
  
'Repeat this small code fragment a million times and divide the time taken by a million' often effectively measures the speed of your various caches (L1, L2, RAM, page cache, memcache), not the speed of the code.
+
'Repeat this small code fragment a million times and divide the time taken by a million' may well measures the speed of your various caches (L1, L2, RAM, page cache, memcache) as much as the speed of the code.
  
  

Latest revision as of 00:52, 27 June 2020

This article/section is a stub — probably a pile of half-sorted notes, is not well-checked so may have incorrect bits. (Feel free to ignore, fix, or tell me)


The many more names for tests. More than most of us can remember or define on the spot.

And some are fuzzily defined, or treated as interchangeable in some contexts.


Code testing

Unit testing

Often essentially "is this part behaving sanely on its own, according to the tests I've thought up?"

Typically for small pieces of code - often functions, behaviour within classes, etc.


They are also useful in regression testing


There is some discussion about unit tests.

Which you can probably summarize as "They can be a useful tool, but if you claim unit tests as proof of anything, one has to worry." and just the fact that you rarely test interactions.

Integration testing

Often essentially "does this interact sanely with all the existing parts"

Tests the combination and interaction of units of code.

Regression testing

Often "is this still working as it always was?"


Basically it's any tests (unit tests or more complex) that should stay true, because other things build on it.

...but you suspect the code might be altered, e.g. because it is often-touched code.


Often introduced after a specific bug was found, to ensure we won't regress to that bug later. This can also help avoid emergence of similar bugs.

Acceptance testing

Often essentially "are we living up to the formal requirements in that document over there?", which is often its functional design.


Which can involve any type of test and scale of testing, though in many cases is relatively basic.

Fuzz testing

Fuzz testing, a.k.a. fuzzing, feed in what is often largely random data, hence the name.

If software does something other than complain about bad input, it may reveal bordercases it's not considering, and e.g. the presence of exploitable buffer overflows, injection vectors, ability to DoS, etc.


Used in security reviews, but also in some extended tests for robustness.


See also:

On code coverage

Load, performance, stress testing

Stress testing, recovery testing

Common pitfalls in benchmarking

Measuring the overhead

Measuring your cache instead

Timing inaccuracies

Microbenchmarking

Caring about users

Usability testing

Accessibility testing

See also

http://en.wikipedia.org/wiki/System_testing