Tim Lott: Cheer up! It may never happen. (But a pessimist half hopes that it will...)

I'm fairly sure that you will find this article pretty uninteresting. In fact chances are, the editor will not commission me again, and then I won't be able to make my mortgage payments, and then the bailiffs will move in, and then....

You get the drift. I am a dyed-in- the-wool pessimist, which according to a recent article in Time magazine, puts me in a minority of 20 per cent over the optimists' majority of 80 per cent. However, unpopular though it is, pessimism does have its uses. The reason this article will probably have turned out OK is because I was convinced that it wouldn't. Pessimism is the little voice in the ear that tells you that you're no good, and you better watch yourself at all times to make sure that nobody notices.

Pessimists get an unduly bad press. Most people would be likely to agree with George Bernard Shaw's view that a pessimist is "a person who thinks everybody is as nasty as himself and hates them for it", rather than his much less well-known contemporary Edward Hubbard, who believed, more realistically to my mind, that a pessimist is a man who has been compelled to live with an optimist.

However, pessimist that I am, I can't escape the conviction that most optimists are idiots. David Cameron's conference speech last week, despite the accolades it received from Ian Duncan Smith and Kenneth Clarke for its optimism, struck me as particularly ill-judged, not to mention slightly nauseating. (He has form on this – remember the "Let the sunshine in" speech of 2006?)

Wasn't it optimism that got us into this monstrous pickle in the first place? The irrational exuberance of the markets, the belief that boom would continue without bust, that low taxes could co-exist with high public spending, that the world economic system had entered a period of permanent growth. That, in short, money grew on trees?

However – politically at least – Cameron is no fool. After all, politicians are duty bound to behave as if they are optimists. Anything else is seen as political suicide, or "talking down the country". Usefully, he has the grumpy yin of Osborne, giving us the "if it isn't hurting it isn't working" line to balance his wildly improbably yang.

All the same, Cameron's bottom- of-the-barrel imprecations to tap into our "indomitable bulldog spirit", his conviction that this was a "time of opportunity" and that just round the corner"the sense that our country is on the way forward once again" and that there will soon be "more money in your pocket" were pushing the credibility of even this tired convention. Yet the speech will undoubtedly pick him up a few votes because even in grumpy old Britain, people remain at heart inclined to look on the bright side. This isn't because of their bulldog spirit, however, indomitable or otherwise. It's because they are hard-wired to do so.

Neuroscientists have discovered that the inclination to look on the bright side is rooted in biology and very resistant to inconvenient facts. Tari Shalot conducted a fascinating experiment where in which optimists were asked what they thought they chance of getting cancer within their lifetime. They guessed, on average around 13 per cent; the likelihood is actually around 30 per cent.

Furnished with this information, they upgraded their estimates only slightly, putting their chances at still well below 20 per cent. Optimism, it seems, takes little account of reality. Psychologists are also well acquainted with Optimism Bias. Collectively we can grow pessimistic – about the direction of our country or the ability of our leaders to improve education and reduce crime. But private optimism, about our personal future, remains incredibly resilient. A survey conducted in 2007 found that while 70 per cent thought families in general were less successful than in their parents' day, 76 per cent of respondents were optimistic about the future of their own family.

People hugely underestimate their chances of getting divorced or losing their job, expect their children to be extraordinarily gifted, envisage themselves achieving more than their peers, and overestimate their life expectancy, sometimes by 20 years or more. These psychological facts doubtless contributed to Cameron's judgment in peppering his speech with such cues (along with the fact that he had very little else to offer in terms of policy details). But in some senses, this optimism sits ill with the Conservative tradition – one of the reasons perhaps that his words rang so hollow.

After all, Conservatism was founded on pessimism. The predication that human beings were fallen from grace, and that unless restrained and guided by the firm hand of an authoritative (albeit limited) state, they would fall prey to their natural instincts towards greed and destructiveness. This is a view that stretches back to Hobbes and Burke, but in more recent times the same kind of pessimism displayed itself in Margaret Thatcher's "There is no such thing as society" views – that people only cared for themselves and their families.

As the Conservative philosopher Roger Scruton points out, "The belief that humanity makes moral progress depends upon a wilful ignorance of history. It also depends upon a wilful ignorance of oneself – a refusal to recognise the extent to which selfishness and calculation reside in the heart even of our most generous emotions, awaiting their chance."

The socialist and liberal tradition, conversely, looks back at least to Rousseau and the idea that human beings are intrinsically good, or at least blank slates, and that this good nature could be liberated given the proper environment. This optimism, one could argue, led to Jacobinism, the excesses of Marxism and the social engineering that led to modern brutalist housing estates and the worst excesses of progressive education. (You might point out that some of the grumpiest, most pessimistic commentators and politicians come from the left wing, but that is because there is nothing so cynical as a disappointed idealist.)

So which is the most realistic – optimism or the pessimism? This is a no-brainer. Pessimists are rarely so misguided as optimists since it is inevitable that things will go wrong, but it is not inevitable that things will go right. Things going wrong is in the nature of things and systems and people. Things going wrong is why we have to work so hard as human beings to get them to go right.

This does not mean, however, that pessimism is necessarily a good philosophy for a nation or an individual to live by. Optimism is what you might call a "mobilising myth". It gives us energy, dynamism and hope in times of hardship. Much as I disagree with it philosophically, I recognise that we cannot do without it.

All I would suggest is that there is a distinction between authentic and inauthentic optimism. There is no such thing as inauthentic pessimism incidentally: as a pessimist, one is entering a simple deal to trade away the possibility of disappointment on the one hand, and the grim recognition of intractable reality on the other.

Cameron's speech last week was very much in the category of the inauthentic – calculated soundbites to fit in with a slice of market researched advantage in the face of the lack of anything material to focus on. But there is a more respectable form of optimism, an openness to what might be, without expecting it to automatically come about through the machinations of divine fate or "our bulldog spirit". An openness to possibility recognises that while things consistently go wrong, sometimes they will, against the odds, go right. It recognises that the human spirit is unquenchable, and that human energy and imagination are formidable. It suggests that people are more good than bad, more sensible that stupid, and more kind than cruel. This is the kind of optimism that a pessimist can subscribe to and still feel intellectually honest. The world is going badly wrong at the moment, but nothing is for ever and the worst, while yet to come, is also transitory.

Conservatism, like liberalism and socialism, doesn't really know what to think or say any more. We are all lost in the dark, the only difference being that the optimists are still tunelessly whistling.

We do not have the bulldog spirit, any more than any other country, but we do share the human spirit and that is a remarkable thing. So dare to hope, yes, but be realistic. After all, it's a difficult trick to keep your eyes open while your head is in the sand. Although David Cameron last week, with his unflagging instinct for political opportunism, came depressingly close to pulling it off.