Big brother is watching – but can he still see a future for himself as party leader?

Last May, as hundreds of MPs and peers gathered in Westminster Hall to hear President Obama address a joint session of both Houses of Parliament, one man had more reason than most to rue his position six rows back on the tightly squeezed chairs in the great hall.

In front of him, alongside the President at the top table, were David Cameron and Tony Blair, Gordon Brown and William Hague, and an even more familiar face: his brother Ed, the recently elected leader of the Labour Party. For David Miliband, the former Foreign Secretary, who has made maintaining neutral facial expressions something of an art form, it was an especially difficult occasion.

According to a friend who was sitting close to him, at that moment David was unable to hide his feelings. "I've never seen him look more dejected," he said. "I remember looking over to him at the time and being struck by how despondent he was."

Eight months on, some in Westminster are suggesting that as Ed's star has waned, so his brother's might rise again. A poll this week suggested that of all the senior Labour figures, only David could threaten the Tories. With Ed's popularity falling below even that of Nick Clegg, the ComRes survey found that if David was in charge, Labour would now have a three point lead.

But after the trauma of defeat in the leadership election and his self-imposed exile on the back benches, does David still want the Labour crown, and would he get it even if he did?

Talking to friends and former colleagues who have spoken to him regularly over the last 18 months, it is clear that the leadership election not only hurt him more than was acknowledged at the time – but that he has far from recovered.

The brothers now meet only at family occasions, and generally avoid talking about politics. David has not attended Prime Minister's Questions since Ed became leader and does not go to the same political functions.

At the last Labour conference in September, David put in a token appearance at a fringe meeting before leaving for a conference about China in the US and missing his brother's keynote speech. The public explanation is that David believes his presence would be a distraction. The private truth is that he has no desire to be there.

"I think the best way of describing it is to say that he has good days and bad days – and as time goes on he has more good days than bad days," said one person who knows him well. "But he still finds it incredibly difficult. It is hard to make the transition from Government to opposition, especially if you have been Foreign Secretary.

"Imagine you lose to your brother in a leadership election and have to be publicly supportive of him. On a purely psychological level that's tough."

One of the ways he copes with it is by travelling. Since losing the election he has visited San Francisco, Islamabad, Zurich, Washington, Paris, Istanbul, Utah, Abu Dhabi, Berlin, Boston and Krakow. Some of the visits were for speeches, others conferences and he also did a teaching stint at Stanford University. He is still fascinated by foreign affairs, the international lecture circuit pays well, and on trips abroad he is less likely to be asked about his brother.

He is also starting to use the freedom of the backbenches to take on more long-term paid work. Earlier this month it emerged he had been hired by a Pakistan-based City firm as a consultant. The part-time job with Indus Basin Holdings, which funnels investment into Pakistani agriculture projects, is worth about £50,000 a year.

It comes on top of the £92,000 a year he receives from the Californian clean energy firm VantagePoint, and the £75,000 a year he gets as vice-chairman of Premier League football club Sunderland. For this reason, among others, his friends say it is unlikely that David will take on a front bench Labour role, at least before the next election.

That is not for want of asking. Close political allies of Ed have made approaches to David on several occasions, asking him whether he would be prepared to return.

"David has made clear that he doesn't want to come back," said one shadow cabinet ally. "I think that's for several reasons. He can earn more than if he had a full time opposition brief, and after being Foreign Secretary, being an opposition spokesman would be a bit of a come down.

"But the main reason I think – although he has never said this to me – is that he still finds it very difficult to be in a room with Ed and he can't avoid that if he's in the shadow Cabinet."

Several people who know David do not believe that he has entirely given up hope of one day leading Labour should Ed lose the next election. Intriguingly, one of the few areas of domestic politics he remains involved with might help him achieve that.

David is a trustee of a group called Movement for Change, which helps local people bring about change in their communities. He describes it as "two-thirds Labour history, one-third American community organising." In concept it is very similar to the kind of community activism where President Obama cut his teeth. Should he decide to run again for leader, the Movement would give him a ready-made platform and campaign organisation.

But the odds of him succeeding his brother are still long. After he lost the leadership, he was consoled by the number of senior Labour figures who said the party had made the wrong choice and that he'd be back in a few years. Those voices are now diminishing.

And given Ed's faltering leadership so far, it is hard to see the party turning from one Miliband to another should he leave before 2015. More likely is that Yvette Cooper or Alistair Darling would claim the crown.