Andrew Grice: It was the right speech for David Cameron. It came from Nick Clegg

After two weeks of chaotic government, the politician said: “It's time to get back to governing; providing the leadership and focus the people of Britain deserve in these difficult times… Our priorities must be people's priorities: boosting business, creating jobs, helping with the cost of living.”

The speech should have been made by David Cameron. Revealingly, it was made by Nick Clegg. On the same day, the Conservative Prime Minister delivered a watered down version of his Liberal Democrat Deputy's "we must get back on track" message. After backbench Tory revolts over Europe and gay marriage, Mr Cameron told his MPs he would not bring forward more socially liberal measures like same-sex marriage.

True, it was easier for Mr Clegg than Mr Cameron to rebuke the fractious Conservative Party. Yet the Prime Minister's more conciliatory tone suggested weakness rather than strength. The Liberal Democrat leader also spoke out of deep frustration. After this month's Queen's Speech, he wanted attention to focus on two important reforms being introduced by Liberal Democrat ministers – on social care and a flat-rate pension.

Conservative splits and claims that a close Cameron ally attacked Tory activists as "swivel-eyed loons" created a perfect Tory storm. Although the Liberal Democrats are currently more united, disciplined and grown-up than their senior partners, the Coalition as a whole suffered some collateral damage. Insiders report that Mr Cameron was left rattled and bad-tempered by the incessant pressure from his own party. He managed to disguise it well in his calm and assured response to the horrific act of terrorism on the streets of Woolwich. The Prime Minister is good in a crisis, something his Tory critics should remember.

They won't. They don't give him much credit. Events of the past fortnight have reignited plotting against him by Tory MPs determined to force a vote of confidence in his leadership. It melted away when Margaret Thatcher's death reminded Tories they are all in the same family. I doubt there will be a confidence vote but the usual suspects are talking about one again. To trigger such a vote, they require the backing of 46 of the 304 Conservative MPs. "We would need 90 to have any chance of getting him out," one rebel said, explaining that Mr Cameron could squash a revolt by 46 of his MPs and carry on but could be fatally wounded if about a third of them voted against him.

There is relief in Labour circles that the Tory trauma has turned the media spotlight away from its own problems. Yet Labour cannot afford to relax, as the spotlight will soon fall on it again. There are growing rumblings about Ed Miliband's leadership amongst Labour MPs. There is no plot to oust him, but he is under growing pressure to spell out more clearly what his One Nation Labour vision would mean in practice – and how his party would cut the deficit. Labour MPs worry that an economy that is recovering in 2015, however slowly, would leave their party looking backwards. They fear Labour is too stuck on talking about what it would do now rather than what it would do in 2015.

Mr Clegg, pictured, is not without his Liberal Democrat critics but, bizarrely, most speculation about his leadership prospects seems to be fuelled by Conservatives. Michael Gove, the Education Secretary, claimed two weeks ago that Mr Clegg was blocking Tory childcare reforms to fend off a leadership challenge by Vince Cable, the Business Secretary. It was widely seen as a diversion from Tory divisions.

Privately, Tory ministers continue to talk up the prospect of Mr Clegg being toppled by his own party before the general election. I detect an element of wishful thinking here. It would suit the Tories for the Liberal Democrats to have a centre-left leader like Mr Cable in 2015, as he would have a better chance of wooing back the centre-left 2010 Liberal Democrat voters who defected to Labour when the Coalition was formed. That would boost the Tories' chances in the Labour-Conservative marginals that will probably decide the election.

Mr Clegg has no intention of veering left to chase these voters. Similarly, talk of the Coalition ending before the 2015 election seems to emanate from Tories rather than Liberal Democrats, although Mr Cameron joined Mr Clegg this week in insisting it will last until polling day.

The Liberal Democrat leader believes detaching his party from the "big bad Tories" six months before the election would not fool the voters and wants to fight it on the Coalition's record. "We can't disown what happened in the previous four-and-a-half years," said one Clegg ally. "You can't dive off the board and then change your mind just before you hit the water."

Who is the only party leader not facing internal grumbling? Step forward Nigel Farage. The "swivel-eyed loons" row was another gift for the UK Independence Party leader, who continues to spook the other parties. Mind you, when you are a one-man band, it is difficult to lead a rebellion against yourself.