A/B testing: Difference between revisions

@@ Line 2: / Line 2: @@
 A/B testing test two variants (A and B),
-usually showing just one to members of a population
-(often with a 50-50 split).
-It usually amounts to statistical hypothesis testing - vary one thing in what you show to people, and who complains less, clicks away less, etc.
+Say,
-The setup and analysis can get complex - try to improve engagement metrics ''while'' trying to not affect use experience metrics,
-vary many different things, etc.
-...but each test is usually fairly simple, e.g.:
 * trying different wordings in your <strike>spam</strike>marketing,
 * different ways to display an ad.
 * what price are people more likely to buy the thing at?
 : probably more than two - you'd want to find a curve
+Often splitting a population so each person sees just one,
+often trying for a 50-50 split (it makes the statistics simpler).
+It frequently amounts to statistical hypothesis testing - vary exactly one thing in what you show to people, and see who complains less, clicks away less, etc.
+A lot of A/B testing fails to be that rigorous (it is hard to vary just one thing), but it still useful.
+Even when it is possible, the ''goal'' may still mess it up. Say, "try to improve engagement metrics ''while'' trying to not affect use experience metrics" is useful, but not longer an isolated test, vary many different things, etc.
 Sometimes it's even ''just'' about improving an app's user experience, by letting your.

Revision as of 16:29, 27 January 2024