Lady Luck is just an illusion. So why are we so much in thrall to her?

When humble heating engineer Steve Whiteley won well over £1m on a two-quid punt at Exeter races last week, he made for a popular winner. He rarely gambled, he got to the races using a free bus pass, and, after his windfall, he decided to continue doing his job out of a sense of responsibility to his customers.

It's easy to be happy for him, because he's clearly a nice chap. However, the satisfaction goes deeper than that. His win helped underpin a number of very reassuring narratives: that the good are rewarded, that we're all in with a chance to transform our lives through luck, and that an ordinary Joe can beat the odds.

How different the story of "lotto lout" Michael Carroll made us feel, back in 2002, when the bin man hooligan turned up to collect his £9.7m win wearing an offender's electronic tag. Rarely have the tabloids poured so much invective in the direction of someone who wasn't a paedophile.

Carroll's win seemed to throw sand in the face of one of our most cherished beliefs: a primitive attachment to a just world, that the bad must ultimately come to no good in the grand scheme of things. In the case of Carroll he did, satisfyingly, squander it all and went back to being a bin man. The only slight irritation is that he displayed no regrets whatsoever.

Luck is never neutral. That is intrinsic to the meaning of the word, as opposed to "chance" or "probability". Luck implies that it "belongs" to someone, that you have it or you don't. It is personal, which is why we feel annoyed when Michael Carroll profits for no good reason and countervailing delight when Steve Whiteley does well. We believe in lucky streaks, lucky people, lucky numbers, and lucky stars.

Luck is probability taken personally. Luck differs from chance in that it is always "good" or "bad". You might chance to see an unusual breed of pigeon in the park, but (unless you are a passionate birdspotter with a special interest in pigeons) you won't think of it as lucky, because it means nothing to you. Unlike mere chance, we ascribe values to luck.

The power of the belief in luck in the world at large is truly phenomenal. Witness the new casino complex in Macau, where the booming Chinese come to spend their new-found wealth. It is predicted that, by the end of year, the Chinese will gamble £1trn in Macau. Think of all the money taken out of all Britain's cash machines in a year, multiply it by five and you get an idea of the size of the operation. Chinese punters flock there in their droves.

This, in my view, makes most of them mugs, since you can't buck the odds in the long term, although you can in the short term, as both Steve Whiteley and Michael Carroll have proved. Because this is where luck bumps heads with probability. And where luck is a human belief, probability is a statistical fact.

It is, however, a statistical fact that most people don't fully understand, which is why bookies are so successful. And we have no reason to cock a snook at the Chinese, when we have the biggest lottery in Europe and a betting shop on practically every high street in the land. The bookies, who you might think might be upset at losing more than £1m to Steve Whiteley, will in fact be cheering from the back of their limos because it appears to be proof that it "can happen to everyone".

Well, yes – what happened to Whiteley can happen to everyone, even though the odds are 879,137-1. So you would probably have to try almost a million times for it to happen to you as well. Which is fine, but if you are willing to genuinely believe in that sort of luck, then you should also spend a realistic amount of time worrying that you will be struck by lightning or die as a result of a shark attack. Yet while we don't worry about these latter events – who insures against a shark attack? – we are happy to spend money to indulge the possibility of the former.

Therein lies the strength of the bookie. It's not in the fact that the world is random, which flies in the face of the lucky punter's belief that they have been individually singled out for their fortune, but in the fact that it is not random at all. It is governed, very strictly, by mathematical probability. The bookies know and understand this, and that is why you never see a poor bookie: something akin to a law of physics is on their side.

The phenomenon of statistical probability is extraordinary; it's like the workings of an invisible hand. Take the most basic measuring instrument for chance, the coin toss. Even this simple example is misunderstood by most punters. For instance, if a rookie gambler throws a coin and it comes up heads seven times in a row, they will imagine that the chances of it coming up tails next time are going to be higher than 50 per cent. But they are always 50 per cent. And it is being armed with this sort of counterintuitive information that makes sure the bookie always wins in the long run.

"In the long run" is the key phrase. Toss a coin 10 times, and it is quite likely to come up heads twice, say, and tails eight times. But toss it 100 times, and it will, with extraordinary reliability, come up divided between 48 and 52 times heads and tails. Which is almost magical. It is the force of probability in action, and it is very powerful.

But most people do not want to know about probability. They want to believe in luck, because it personalises the world and their place in it. However, there are only a few groups of – largely unconnected – people who tend not to believe in luck, outside of scientists and statisticians. These are very successful people and New Age enthusiasts.

Very successful people are the ones most inclined to say "there's no such thing as luck". This translates as, "I got where I am by skill and effort alone, and if you are poor and unsuccessful, you ended up there for the same reason".

Nassim Nicholas Taleb exploded this myth in Fooled by Randomness, in which he studied a series of successful City traders and discovered that they consistently undervalued the role of accident when ascribing reasons for their success. In other words, it was chance and not skill that makes them their fortunes, an interpretation that they naturally resisted.

Even more mystifying in their denial of chance are the New Agers. They insist that everything that happens is a result of karma, or your own choices – most memorably, Glen Hoddle, the England football manager who lost his job for suggesting that disability was as a result of karma.

It is not hard to find otherwise perfectly intelligent people subscribing to this belief. I remember having a conversation with an otherwise highly intelligent friend of mine who insisted that everything that happened to you was brought about by your own design. I asked whether, if a plane crashed on our heads right at that moment, it would be our fault? She said that it would be. Presumably she is worried right now about the karmic burden of those who are suffering and dying in the wake of the Japanese earthquake.

Why all these crazy ideas about luck and karma? Simple ignorance, mainly, along with wishful thinking. Probably the most useful thing that could be taught in Britain's schools would be statistical probability – the relative likelihood of everyday events. Then we might, for instance, work out that letting your child walk to school alone is a great deal safer than driving them. But we don't want to "leave these things to chance", even if we don't understand the nature of chance in the first place.

This is all, perhaps, a long way from the amiable Mr Whiteley and his popular lottery win. Anyway, he doesn't seem yet to have made any claims that the win was a result of his star sign being in the ascendant or because his wife bought him some lucky heather the week before. And either way, I am genuinely glad for him.

But the reality is that the old lottery motto – "It could be YOU" – is incomplete. Yes, it could be you. But it almost certainly won't be, and it isn't worth flushing money away in the hope that it will be. On the plus side, it almost certainly won't be you who gets killed by a terrorist bomb or bitten by a snake.

The Bell Curve of chance runs through our lives like a constant corrective to outlandish fortune, one way or the other, and like the coin tossed 100 times, tends to even out in the end. Steve Whiteley is just a statistical fluke, as was his less attractive counterpoint, Michael Carroll. Life is governed not by randomness or luck: it is governed by probability. And probability always wins the day in the long run.

Which, of course, won't stop me having a punt at Cheltenham.