David Cameron's reputation is in tatters - and the people rejoicing the most are within his own party

What a difference six months makes. Last October, David Cameron was at the height of his power as he celebrated his general election victory in his speech to the Conservative Party conference.

He set out his ambitious legacy agenda to tackle Britain’s “deep social problems” before standing down by 2020, promising an all-out assault on “the scourge of poverty”; to end racial and sexual discrimination to ensure “real equality”; and to transform our failing prison system. He has since fleshed out his bucket list in a series of speeches.

This weekend, Cameron will address the Tories’ second largest gathering of the year, their spring conference. The grassroots members will doubtless be loyal. But the atmosphere will be very different.

That will not be caused solely by the damaging allegations of hypocrisy on tax avoidance, after Cameron’s belated admission that he did after all benefit from his father’s offshore investment fund.

When Parliament returns from its Easter break on Monday, the unanswered questions will keep coming. The barrage will not be halted by his decision to publish his tax returns - a promise he made in 2012 - raising the question whether he had hoped it would remain in the long grass.

Politicians put themselves on a pedestal and must meet higher standards than the rest of us. On the tax issue, the bar is higher for Cameron because of his privileged background. The fact that he was subject to the UK tax regime when he sold his shares in his father’s fund will not limit the damage. It may be unfair, but that’s politics.

The tax avoidance row is corrosive enough for the Tory brand and Cameron’s attempt to end its reputation as the “party of the rich”. But it is not his only headache: before the Panama Papers exploded onto the scene, his legacy agenda was already in deep trouble.

Cameron had no alternative but to allow Cabinet ministers the freedom to speak out on Europe in the run-up to June’s referendum. He hoped that he and the ministers campaigning for Brexit would agree to disagree on Europe but would not attack each other on other issues. This has proved a very serious miscalculation.

The Tory Outers, who claim angrily that Cameron has tilted the referendum playing field against them, are quite happy to trash the Tory brand if it they think it will help them win. When Iain Duncan Smith quit as Work and Pensions Secretary, he attacked the Tories’ record on welfare since 2010, saying their “all in it together” mantra was at risk and suggesting they cared only about Tory voters.

Cameron’s record on the NHS, the biggest symbol of his Compassionate Conservatism, is called into question by the Outers. These Brexiteers accuse his Government of starving the NHS of funds, claim the EU’s single market is “threatening patient safety”, and exploit the pressures on the NHS to make a “dog whistle” attack on EU migration. They turn a good news story about the new national living wage into a warning that it will attract more EU migrants. Hardline backbenchers threaten Commons defeats on non-EU issues.

Perhaps the most ludicrous claim was Vote Leave’s assertion that Downing Street rushed out the Government’s leaflet on the referendum being sent to every home to “distract the media’s attention from the issue of whether the Prime Minister’s family money is kept in offshore trusts.” It showed the damage the Outers are prepared to inflict on him.

Cameron is fighting for his very survival. If the Out camp wins, he will be out too.

His understandable focus on the referendum has led to Downing Street taking its eye off other balls. That is the only explanation for ministers' flat-footed response to Tata Steel’s announcement about selling its British business.

Where was the Government's early warning system? Does anyone in Number 10 watch telly? I saw Tata’s crucial board meeting in Mumbai being discussed a week before it took place.

Again, Cameron’s legacy goal for the Tories to be the “party of working people” is undermined by the Outers, who wrongly blame the steel crisis on the EU. Any stick will do. Whacking Cameron on anything helps them, they calculate.

This will make it very hard to reunite the Tory family after the referendum – whatever the result. Even if Cameron wins, which looks increasingly in doubt to me, he will struggle to repair the damage to him and his project inflicted by his own side.

Cameron’s allies hope the tax affair will not inflict long-term harm, since voters already know he is “posh” and many believe all politicians are “in it for themselves.”

However, the tax row, the bloody Tory civil war and the threat to steel jobs could all undermine trust in Cameron at the very moment when he is asking voters to trust him on the EU.

This toxic combination could repel some swing voters who gave him the benefit of the doubt at last year’s general election, partly because the alternative was so weak. Voters may see the referendum as a free hit because it won’t change the Government. But it could now change our prime minister.