Use the correct statistics

There will be times when you have statistics that seem to be telling you exactly what you want to know.  They seem to be so clearly tied to your goals that there wouldn’t be any reason to think they might be giving you the wrong idea.  But sometimes, even these stats can be leading you astray.

Let’s say you have the following two options:

• Option A - Earn 1 point every turn.
• Option B – Roll a fair die every turn.  Earn 5 points if you roll a 6, otherwise earn nothing.

Your goal is to analyze the options.  Which is better?  Well, A gives you 1 point every turn guaranteed, and, using very basic statistics knowledge, B gives you 5/6 points every turn on the average.  So…  A is better.  Right?


Well, sure, I guess.  There’s nothing wrong here really, except for making the conclusion that A is better from stats alone.  The question was unclear, because you need to define the word “better.”  And to do that, you need to have a clear goal.  Before you say anything, yes, yes, you’re assuming that points are good and the goal is to get more points in fewer turns, and that’s fine.  That’s not what I mean.

Let’s say you have the same two options:

• Option A - Earn 1 point every turn.
• Option B – Roll a fair die every turn.  Earn 5 points if you roll a 6, otherwise earn nothing.

Only this time, there is actually a game with a goal too.  Two people simultaneously choose from option A or option B every turn.  The winner is the first person to score 5 or more points, and the game ends in a tie if both players reach 5 or more points on the same turn.

Now you have a well-defined goal.  So now you can say which option is better.  So, which option is better?  Well the same statistics are still there.  A gives you 1 point every turn, and B, on the average, gives you 5/6.  So A is still better.  “But wait, I see the title and the intro,” you say, and you think about it differently.  The goal isn’t how many points you can get, it’s how many turns it will take you to get 5 points.  So, with that in mind, if you only choose A, you’ll take 5 turns every time.  If you only choose B, well, statistics knowledge will tell you that you’ll take 6 turns on the average.  So A will win in fewer turns on the average.  So… A is still better.  You can probably guess what I’m going to say though.

A isn’t better.  B is.  Let’s look at two more statistics.  If one person only chooses A and the other only chooses B, what is the probability B will win, and what is the probability B will tie?  First, B will lose if the die fails to come up 6 for 5 consecutive turns, and the chance of this is (5/6)^5 = 40.19%.  So B will tie or better 59.81% of the time.  Now the chance of B not rolling 6 in the first 4 turns (and therefore not winning) is (5/6)^4 = 48.23%.  Which means the chance B will roll a 6 in the first 4 turns and win outright is 51.77%.  Let’s look at the stats summary:

A vs. B
Misleading but accurate stats:
Avg. pts. per turn:              A – 1, B – 5/6
Avg. turns to score 5 pts:    A – 5, B – 6
-------------------------------------------------
Stats that matter here:
Chance of B winning over A: 51.77%
Chance of B tying with A: 8.04%
Chance of B losing to A: 40.19%

B will tie or better almost 60% of the time, and B is expected to win outright more than half the time on the average!  If both players always choose A, the result will always be a tie.  On the other hand, if one person chooses B when the other chooses A, then the person choosing B will do better 51.77% of the time, and do worse 40.19%.  So if your goal is to succeed in this game (with winning being as much better than tying as losing is worse), then B is the better choice.  No doubt about it here.

On the top, if you know the other player will choose A, you'll win more than you lose by choosing B.
On the bottom, if you know the other player will choose B, you'll win more and lose less by choosing B.
In other words, you should always choose B.

“So… why are the first stats wrong?” you ask?  They aren’t wrong.  They just don’t match what your true goals in this game are.  The number of turns for B to score 5 points can vary greatly.  There is a very reasonable chance that you take 10 turns or more, or even 20 turns or more.  So when you take the average number of turns, a large amount of the contribution comes from very high turn count but relatively low probability cases.  But for the sake of this game, it doesn’t matter if you take 10 turns or 20 turns or 1000 turns.  The game’s already over if you take more than 5 (and the other person is choosing A).  So there’s no reason to put more weight on the 1000 turns case than the 10 turns case.  All that matters is: will you take more than 5 turns, and will you take less than 5 turns.  And although it sounds like a very good statistic to use in this case, using the average number of turns is putting more weight on these high turn cases, to a significant extent.  So much so to make A look like the clearly better choice when B is actually better.

As you could probably imagine, this concept applies to more than just a probability game.  In general, when the quality of a choice is measured using some metric or statistic, you’ll want to stop and ask, “Well, is that metric the thing we care most about?  Is maximizing (or minimizing) that metric the same as maximizing our chances to reach our goal?”  Sometimes it will sound so obvious that it should be the case, and sometimes even then, it won’t be.

The more direct application is to using the statistics from (a heavily skewed distribution, more generally, but in this case) a geometric distribution (for non-stats people, this would be the cases where people say, say, “This happens 1/100 times on the average” and you watch it over and over until you see it happen once).  It doesn’t exactly work like you might imagine.  If there’s some machine that gives a winning prize 1/100 times on the average, and a bunch of different people play the machine, how many times would most people have to play before winning once?  100 times?  No.  It’s expected that more than half the people playing would win in 69 tries.  To add to that, 63% of people will win in less than 100 tries.  Think about that.  The average you are told, correctly, is one win in every 100 tries, and yet, more often than not you’ll win in your first 69 tries.  The chance of you winning before your 100th try is 63% compared to the 37% chance of you having to go through 100+ tries.  Now, the reason the average is much higher is because there’s a reasonable chance you just don’t win.  You take 200 tries and you don’t win.  You take 400 tries and you don’t win, even though the expectation is 100 tries.  It’s not very likely, but it’s not unreasonable either.  This is why you must take the scenario, and your goals, into account.  In some cases, 400 tries is no different than 200 tries, because the game’s already over or whatever situation you’re in has already ended; you can’t actually take that many tries.  In these cases, you don’t want to use stats that overly weight the very high end.  On the other hand, in some cases 400 tries is quite a big difference than 200 tries, and you do want stats that take this into account.

The conclusion?  Don’t make a conclusion from statistics without considering your goals properly.  Make sure the statistics you look at truly reflect the goals you hope to achieve.  And it's not always obvious that your stats don't properly suit your goals.  You could have a game where the goal is to win in the least number of turns, and you use the average number of turns to win in determining the best strategy.  One more time, the goal is to win in the least number of turns, and you’re considering the average number of turns it takes to win.  And you might still end up with the wrong implications if you use that to compare strategies.  So, it’s worth it to keep in mind that it’s one thing to know the statistics, and it’s another thing to determine the correct statistics to know in order to take meaningful information from them.