Rory Stewart: It's hard to believe we can build a credible state after so many years of failure

Three years ago, I started working on a documentary about foreign invasions of Afghanistan. This week in Chicago, Nato and the Afghan government were agreeing to withdraw foreign troops, having failed in the objectives they set themselves four years ago of defeating the Taliban and building a "credible, effective Afghan state".

It is easy to describe the disasters of the last 160 years. We filmed in the narrow, five-mile long gorge where 5,000 British soldiers and camp-followers were picked off, one by one, in the 1842 retreat towards Jalalabad. We filmed the ridge above Kabul, from which Highland soldiers were driven in 1880.

We sat in the Kombat bar in Moscow, with Russian veterans, toasting the memories of the 20,000 Russians who were killed in Afghanistan in the 1980s; and with mujahedin commanders, talking of the million Afghans who were killed. But I am still struggling to understand how we got trapped in these situations – what exactly we were thinking, and who exactly was responsible.

Four years ago, Sir Sherard Cowper-Coles, our ambassador in Kabul, one of our most senior diplomats, seemed to disagree with the Afghan policy. He suggested that he could do nothing because he was "tied to the wheel of an American chariot", whose charioteer was Richard Holbrooke. But Holbrooke – the charismatic legend who had run the Balkan peace talks and had been made the Presidential superenvoy for Afghanistan – also seemed to disagree with the policy. He told me that the decisions were being made by the military and in particular General Petraeus.

So I went to see Petraeus – the author of the counterinsurgency doctrine in Iraq, a man tipped to be the next President, and the commander of all the operations, spending almost $150bn a year, with almost 150,000 foreign troops under his command. And he felt that he was being sidelined and undermined by politicians.

But politicians didn't seem in control either. When the General demanded a surge in troops, President Obama asked for nine weeks to decide whether to send them. But it was clear to everyone except himself what he was going to do. In theory the US President made the ultimate decision, and he may have even have had the illusion of free will, but he had no choice: if his commanding general on the ground requested the troops, he would have to send them.

Yesterday, some of the senior officials and generals who have run Afghanistan over the last decade explained brilliantly what had gone wrong: and all suggested it was someone else's fault. We can show that the Emperor has no clothes. The problem is that we are the Emperor.

The documentary is built out of 100 interviews in four countries, shot in a rock concert in Moscow, on a 9,000ft mountain in Afghanistan, and riding in a canoe in Vermont with the former director of operations in the CIA. In the Russian countryside was a young Soviet development worker who stole a mobile-cinema truck from Kabul in 1983 and drove it 350 miles to the community where he was working in north-west Afghanistan. (He took great pride in the work he'd done educating Afghan women and developing political parties.)

Back in Afghanistan, an Afghan said that he'd been so cold and hungry and lonely when hiding from the Russians that he'd almost chosen to walk into a minefield. He showed the graceful poplar tree by the clear Panjshir river which had been festooned with Russian body parts after an ambush. In the States, a charming, well-read American official who had helped to finance the mujahedin said that the civil war and terrorism that followed was not his fault.

But no amount of interviews, or looking in the British Library at the blood-stained journals and letters of British men and women who'd been killed or kidnapped in 1842, or reading previous debates in Parliament, ever quite explained who made these decisions to invade, why they invaded, and why they kept investing in a failing project.

It was easier to find people who disagreed with the decisions than people who supported them. Andropov in the Russian Politburo was one of many who argued powerfully against invading Afghanistan, and predicted that the Soviet Union would become trapped in a futile occupation. And yet he signed off on the invasion.

In 1838, almost everyone who knew anything about British India seemed opposed to invading Afghanistan. Mounstuart Elphinstone, Britain's leading authority on Afghanistan, said: "If you send 27,000 men and can feed them I have no doubt you will take Kabul; but for maintaining [the new king] in a poor, cold, strong and remote country among a turbulent people, like Afghans ... it seems hopeless. The Afghans were neutral... they will now be disaffected and glad to join any invader to drive you out."' And yet Palmerston pushed ahead.

Why? The answer in every case is that Britain or the Soviet Union convinced themselves that Afghanistan was vital to national security; that somehow this place, which Richard Wellesley, Governor General of India, and his brother the Duke of Wellington warned "was nothing but a land of rock, ice, sand and snow", was a deadly threat to our great empires; that it needed to be invaded and occupied and settled, and that failure was "not an option".

How subtly brilliant men elaborated these fears and convinced themselves; and how empty this paranoia proved to be. Three times, superpowers argued that departure would lead to catastrophe. Three times they left, and the people who suffered most were the Afghans, not the superpowers.

But superpowers seemed to find it ever harder to acknowledge failure and get out. The British invaded, and were humiliated, twice in the 19th century, but left both times within three years. A century later, the Soviet Union knew it was making no progress, but couldn't get out for eight years. More people were killed after the decision was made to withdraw than in the proceeding period. This time, Nato troops have been there for a decade, and the Nato conference this week suggests the final departure won't be for a couple more years.

Since the documentary is filmed amongst Afghan rocks, in mud forts, and with mujahedin around camp fires, it might be easy to think that there's something magical about Afghans and the Afghan landscape that makes it unconquerable. But the story is less about Afghanistan and more about ourselves. Consider our envoy, Sir William Macnaghten, trying to talk and bribe his way to a deal with Afghan insurgents in 1841; or Lord Salisbury, over-ruling the caution of his officials in 1878 and insisting that there was no alternative to invading Afghanistan; or the fluent Dari-speaking Russian diplomat, showing pictures of the model farm he built near Jalalabad: they show something peculiar, not in Afghanistan, but in ourselves.

It's a story of how, with all our good intentions, with all our wealth and guns and erudition, we find it so difficult to resist exaggerating our fears, and inflating our pride: we find it so difficult to acknowledge the limits of our knowledge and power. How difficult we find it, however often we are humiliated, to achieve humility.

The writer spent 32 days walking across Afghanistan in 2002. His documentary, 'Afghanistan: The Great Game', will be broadcast on BBC2 on Monday 28 May and Wednesday 30 May at 9pm