Do economists know any more than us?

I've been conducting an experiment. I've supplemented woeful headlines with a rich infusion of economics texts. Macro, micro, behavioural and neo-classical, I've tried them all. I've been to Adam Smith and Milton Friedman, but I've also scoured the work of lesser-known DWEES (Dead White European Economists). And I've tried financial gurus, too, like George Soros. Of course I reread Keynes.

But it hasn't been easy. The closed-off, deterministic ambience of economics still gets to me. So does the conspicuous absence of humans, except as producers or consumers. And the language (anyone for "quantitative easing"?) remains, with rare exceptions, dreadful. For all its apparent methodological sophistication, a lot of economics is best experienced as a pseudo-science, valuable often for what it doesn't purport to tell us – the odd flashes of illumination rather than the extrapolation of often misleading statistics.

Economists like to pad out their credentials with references to the area in which they claim expertise. So-called behavioural economics is voguish now, but I am not sure whether this means more than a rueful acknowledgment that "the dismal science" is cracking apart under the pressures we place on its practitioners. The message I've brought back from the economic front is that we must listen to economists, but they must change the way they speak to us. And it would be best if the change occurred immediately, before the world is further damaged.

There are 30,000 economists with PhDs, and this smallish tribe appears to be overwhelmingly male, with an inherited over-rational view of life. Many economists really do believe, mountains of evidence from Darwin or literature to the contrary, that we are a rational species. A survey conducted among the economics departments of MIT, Harvard and Stanford reveals that most young economists place overwhelming trust in mathematics. (A "broad knowledge of the economy" seemed important to only 3 per cent of respondents, whereas to 68 per cent it seemed unimportant.)

Is the practice of economics really that different from what we hacks do? Giles Keating, head of research at the Credit Suisse Private Bank, tells me that the models used by economists should be regarded as "partial representations" of reality. "One should guard against certainty," he says. "And one should also realise that no model ever applies 100 per cent."

Economists can seem dogmatic while ceaselessly changing their minds. No wonder that when they are depicted in popular culture, it is as uber-nerds, stuck in their own backrooms and requiring rescuers. A clue to how economists might change comes from Richard Layard, who has tried to reconfigure economics by marrying statistics to the utilitarian, pleasure-based philosophy of Jeremy Bentham. Layard thinks that social sciences can be made to deliver happiness. Criticising the number of economists who "don't do useful things," he tells me that each one should also be "a bit of something else."

Economics was, by and large, a 19th century invention rooted in a confident sense that information would aid progress, and early practitioners assembled data with the zeal of demented Casaubons. By the mid-20th century economists acquired the status of secular seers. They were supposed to supervise developed economies, raising up less developed ones. Although it was never clear whether what they did should be considered as a science or not, it was accepted that economists had a patent on the future. Ensconced in foundations, finance ministries, planning departments of corporations, ubiquitous on cable TV shows, Homo Economicus ruled. Who else would tell us how our complex societies were likely to develop? Who else was capable of predicting the future?

By the 1980s, it became fashionable for financial firms to acquire economists. Economists lent their skills to anticipating the performance of markets, at the price of becoming salesmen. The economist Robert Shiller, who did predict the slump, believes that many pre-slump practitioners in the vicinity suffered from "Groupthink" – a polite way of saying that they were so far in on the game and failed to notice anything was amiss. Dissenters from the prevailing optimism were abused as spoilsports bent on wrecking a good party. As Keating explains, people cease to listen when you write the same memo predicting catastrophe for the fifth time.

Although economists are not as unpopular as their banker patrons, they, too, have suffered from the backlash since the Crash. Among others, the Queen has wondered aloud why economists hadn't anticipated disaster. For Robert Skidelsky, author of the biography of John Maynard Keynes, economics consists for the most part of rehashed fads. "A few geniuses aside, economists frame their assumptions to suit existing states of affairs," he says. "They are intellectual butlers, serving the interests of those in power, not vigilant observers of shifting reality. Their systems trap them in orthodoxy."

I'm not sure that Skidelsky isn't exaggerating. Some of the most important writing in our time comes from the vicinity of economics – not, to be sure, from the now discredited school of so-called "classical" economists, but from outsiders on the periphery of the profession. There have been shrewd best-sellers like Freakonomics, or the recent Nudge (recommended by David Cameron) using economics methodology to render the kinks of human nature.

This year's cult book in New York is Lords of Finance, a gruesomely topical account of the blindness of central bankers during the 1930s slump by Liaquat Ahamed, formerly an economist at the World Bank.

Economists now come from diverse academic backgrounds, and some of them nowadays are women. Gillian Tett, shortly to publish a book on the Crash, says she learnt about economics amid the wreckage of the failed Soviet system, working in a Tajikistan village on a PhD in anthropology. She tells me that we should redefine Home Economics. Everyone should be sufficiently numerate to understand how credit card debt is accumulated, and how economies, national or international, do or don't function. The best economists are now touchingly ready to admit that they do not really know what is going to happen. When I approached Martin Wolf four months ago, I asked him how long it would be before we knew if the world financial system could recover fully, he paused. "I don't really know the answer to that question," he said. "It could be months, but it could be years, too." When I asked how we would know, he almost shrugged. "I'm not sure," he said. It would seem from recent columns that Wolf is uncertain about our prospects. "I am becoming ever more worried," he wrote recently.

My first encounter with economics took place in the 1970s when, as a young journalist, I was required to construct for the BBC a device illustrating the effects of inflation. The machine, which looked like something built by Wallace and Gromit, had glass tubes and valves and coloured liquids. When operated by an economist, it began to leak; and it was consigned to the props department without further ado. But I absorbed the work of Keynes on the Central Line between power cuts during the Three Days' Week. A Keynes revival is under way now, but the great man's memory has been wrongly evoked before.

Keynes was a speculator, patron of the arts, entrepreneur, promiscuous homosexual, unsuccessful philosopher and purveyor of the best cerebral one-liners. He ripped up the rule book of classical economists. Keynes saw that slumps didn't disappear of their own accord. If we wanted to, we should cause them to end. Keynes gadded about from one conference to another during the 1930s in a not entirely successful effort to persuade governments to take action, borrowing money to create jobs and pumping money into the economy.

This is the Keynes daily celebrated in the editorial pages of posh newspapers. But Keynes changed his mind a lot. In the 1930s, he was dealing with governments pathologically averse to debt but it's not clear that he would approve of today's gigantic debt levels in the US and Britain.

As his Russian ballerina wife said, Keynes was "more than economist". He believed that he was set on earth for the purpose of saving liberal civilisation from extinction. Keynes was the ultimate British sceptic, rudely, gloriously provocative. He began his career as a philosopher, later pondering the difference between risk and uncertainty. At best, economists could prepare us for uncertainty. "We do not know what the future will bring," he once told a group of bankers, "except that it will be different from any future we could predict."

Keynes would not have been surprised to see his ideas fall from fashion, as they did in the late 1970s. I once had the pleasure of eating a chicken salad sandwich for lunch with the free-market guru Milton Friedman, and he seemed to be one of the most happy people I have ever met. Friedman did truly believe that left to their devices, markets worked, and that governments should be encouraged to do as little as possible, and he chided me gently when I expressed my admiration for Keynes. ("Brits always love him", he said tolerantly. "I'm not sure why.")

The Friedman sunshine is to be found distilled in Alan Greenspan's memoir The Age of Turbulence. Greenspan advised four Presidents as Chairman of the Federal Reserve Bank, and he should be seen as the most influential figure in Planet Money of the past two decades. The Greenspan boom was based on cheap money and minimal regulation.

Last autumn Greenspan appeared before Congress, delivering an astonishing mea culpa. "The whole intellectual edifice collapsed in the summer of last year," Greenspan admitted. He meant that the neoclassical, mainstream economics practiced by him had failed, because of a "fundamental flaw". Markets weren't, as he had believed, self-correcting. They didn't always display rationality – because individuals didn't behave rationally. Greenspan was saying that at certain times, irrationality would rule. More important, he was suggesting that the guiding principle of rational selfishness he had espoused, and promoted for so long, was an illusion.

It's still hard to comprehend the levels of greed and stupidity that characterised the era which ended with last October's crash. The excess is best understood in relation to the ridiculous art bubble that accompanied Wall Street's Glory Years. One instrument invented in the past years of madness was the Collateralized Debt Obligation, in its latter stages of evolution made not from real debts, but from so-called synthetic materials. Rather than acquiring securities or bonds or other debt instruments in order to sell them, salesmen invented them – and then sold them.

The idea of fake debt or securities legitimately traded has allure, it's equivalents today "zombie bank" and "vulture fund", have rather less.

The worst thing about this slump is that one cannot imaginatively share the experience of financial misery. Unlike other events – wars, royal weddings, rock concerts – crashes induce nothing but fear experienced strictly in isolation. Rather than bringing people together, slumps keep them separate. It would appear that Maoists have reappeared in China, calling for the abolition of capitalism, but we are nonetheless lumbered with the system that failed us – a crucial difference from the 1930s, when millions, not just in the Soviet Union, believed that markets could be superseded.

Much has been shattered in the past months, most of all the illusion that a borderless world, having broken out of the cycles of boom and bust, would run on, held together by little more than the free exchange of goods. Not just New Labour, but the politics of the West are held together by what has proved to be a fiction. For the moment, too, the slump has also diminished the appeal of the notion that, left to itself, abetted by the astute manipulation of interest rates capitalism spreads wealth efficiently, if unevenly, pulling the poor out of their misery. Economists did their best to promote such ideas, but we are guilty of collusion. Did anyone really, outside the think tanks of the market fundamentalist right, buy into the neo-liberal world vision?

A vision of our future was wholly absent from many of the texts I read. But I did find a distinctive hell relevant to our present. Within the world of economists a cult surrounds the darkly pessimistic views of the Austrian-born economist Joseph Alois Schumpeter, who died in 1950. He is remembered for his definition of capitalism as "creative destruction".

Schumpeter bequeathed two important if controversial insights. The first was that capitalism couldn't function even half effectively without a succession of brutally successful entrepreneurs ready to overthrow the existing order. The second was that capitalism wasn't in the least moral, whatever Adam Smith had intimated. It was so unappealing that it was always likely to cause revolts on the part of those whom it had destroyed, ultimately ensuring its own destruction.

Schumpeter would have been fully at home in the present, pooh-poohing worries over millions of jobs lost each month. "I often wonder," he wrote in his diary, "whether there is anything that ever happened that could not be turned into a business opportunity."

Now that the momentary euphoria of the G20 summit is past, economists have returned to the dismal daily figures. Is there anyone who doesn't think that a prolonged slump will cause widespread political upheavals?

Fear and greed are the primary capitalist emotions, but for the slump befuddlement seems more appropriate. As I read about economics, I learnt to identify an expression on people's faces. One could see it not just in the job centres of Middle Britain or Ohio, but among those locked out of Guangzhou toy factories. It was present among the nouveaux pauvres of Palm Beach or Park Avenue defrauded by Bernie Madoff. Japanese Finance Ministers succumbed, so on occasions did Gordon Brown, and economists themselves.

And yet knowing a bit more does help, as I discovered. At least you begin to comprehend what cannot be known to any reliable degree. This is what Liaquat Ahamed told me. "Economics doesn't always tell you about what really happens. There are too many theorists, and practitioners concoct them to order. They are a bit like lawyers." For real understanding, Ahamed thinks we have to turn to history. The fascination of his account of the last slump lies in his insistence that so much suffering could have been avoided in the 1930s – if the powerful had been less blind, cleverer, more compassionate.

If not a magician, we'd like another Keynes, perhaps. However, in a lifetime of polemical activity, Keynes never uttered a word of false reassurance. Economics, he once said, could provide "the circumstances of civilisation". He meant that it was up to the rest of us to decide what we wanted. Economists could only help us achieve that goal. So we do need to tell economists how we want the world to be. And if economists are to fix our world, they need to know more about it.

Nick Fraser is editor of the BBC documentary series 'Storyville'

Ten books you must read to understand this crisis

NUDGE

Richard Thaler and Cass Sunstein

David Cameron's favorite guide to public policy making

THE TRILLION DOLLAR MELTDOWN

Charles R Morris

Shows how financiers failed

JOHN MAYNARD KEYNES

Robert Skidelsky

Britain's key economist

LORDS OF FINANCE

Liaquat Ahamed

How the last crash happened

HAPPINESS

Richard Layard

The relationship between economics and success

THE AGE OF TURBULENCE

Alan Greenspan

Fails to predict the crisis

PROPHET OF INNOVATION

Thomas K McCraw

Joseph Schumpeter's life

THE STORM,

Vince Cable

How we got into this mess

THE CREDIT CRISIS OF 2008

George Soros

THE ASCENT OF MONEY

Niall Ferguson

How money shaped society