The Schools League Tables Fallacy


Look at the following table showing the scores achieved by a set of schools  (that we have made anonymous by using numbers rather than names). School 38 achieved a significantly higher score than the next best school, and its score is over 52% higher than the lowest ranked school, number 41.

If I told you that these were the available state schools in your borough and that, instead of your child being awarded a place in school 38, he/she was being sent to school 41. You would be pretty upset, wouldn't you? Now scroll down to below the table.


School Number

Score

38

175

43

164

44

163

25

158

31

158

47

158

11

155

23

155

48

155

40

153

7

151

30

151

6

150

9

149

33

149

19

148

10

147

12

147

32

147

2

146

27

146

42

146

28

145

35

145

49

145

45

144

46

143

1

142

18

142

22

141

26

141

4

140

14

140

29

140

39

139

8

138

5

136

17

136

34

136

3

134

24

133

36

131

37

131

15

130

21

130

16

128

13

120

20

116

41

115



In fact the numbers do not represent schools at all. They are simply the numbers used in the UK National Lottery (1 to 49).  And each ‘score’ is the actual number of times that particular numbered ball had been drawn in the 1172 draws of the UK National lottery that had taken place up to 17 March 2007. So the real question is: Do you believe that 38 is a 'better' number than 41? Or, making the analogy with the schools league table even more accurate, do you believe the number 38 is more likely to be drawn next time than the number 41? (after all your interpretation of the schools league table is that if your child attends the school at the top he/she will get better grades than if he/she attends the school at the bottom).

The fact is that the scores are genuinely random. Although the ‘expected’ number of times any one ball should have been drawn is about 144 you can see that there is a wide variation above and below this number (even though that is still the average score). What many people fail to realise is that this kind of variation is inevitable. To see why look at this explanation. If you rolled a die 60 times you would almost certainly not get each of the 6 numbers coming up 10 times. You might get 16 threes and only 6 fours. That does that not make the number three better than the number four. The more times you roll the die, the closer in relative terms will be the frequencies of each number (specifically, if you roll the die n times the frequency of each number will get closer and closer to n divided by 6 as n gets bigger); but in absolute terms the frequencies will not be exactly the same. There will inevitably be some numbers with a higher count than others. And one number will be at the 'top' of the table while another will be 'bottom'.

I am not suggesting that all schools league tables are purely random like this. But, imagine that you had a set of genuinely equal schools and you ranked them according to a suitable criteria like average A-level results. Then in any given year you would expect to see a wide variation like the table above. And you would be completely wrong to assume that the school at the top was any better than the school at the bottom.

In reality, there are inherent quality factors that help determine where a school will come in a league table. But this does not disguise the fact that much of the variation in the results will be down to nothing more than pure and inevitable chance.



Norman Fenton


Return to Main Page Making Sense of Probability: Fallacies, Myths and Puzzles