No 10 in a spin over briefing against Prince

David Cameron was forced to try to head off a rift with Buckingham Palace last night after his new spin-doctor appeared to question the Duke of York's future as a British trade envoy.

Downing Street sources – understood to be Craig Oliver, the Prime Minister's new director of communications – briefed the BBC that Prince Andrew's position might become "untenable" if further revelations about his links to the controversial American billionaire Jeffrey Epstein came to light.

Vince Cable, the Business Secretary, then threw fuel on the flames by saying that it was up to the Duke to judge whether he should stay in the role – and that "conversations" would be taking place with him on that subject.

Just a few hours later, however, Downing Street was in full retreat. The Prime Minister's official spokesman insisted that Mr Cameron had full confidence in the Duke and was "fully supportive" of him remaining in the post.

He also dismissed suggestions that Prince Andrew's role could be downgraded following the furore over his other business connections.

The prince today faces fresh questions over his links with Kazakhstan, whose President's son-in-law bought the Duke's former marital home for £3m over the asking price. It is reported that senior figures in the former Soviet republic hoped that the Royal envoy would help attract City investment to the country.

The Duke has been the UK's special representative for international trade and investment since 2001, travelling around the world and at home promoting Britain's business interests. There have been long-running complaints about the lavish nature of his official foreign trips.

Prince Andrew has faced criticism for entertaining the son-in-law of the ousted Tunisian president Zine El Abidine Ben Ali at Buckingham Palace. He has also maintained links with Mr Epstein, who was sentenced to 18 months in prison in 2008 for soliciting a minor for prostitution.

The controversy started on Sunday night when a political member of the Downing Street press office briefed reporters covering the Conservative spring conference that there would not be "many tears shed" if Prince Andrew was forced out of his role. After the story was given prominence in several newspapers, the BBC contacted Mr Oliver – who left the Corporation only a fortnight ago – for confirmation of Mr Cameron's position. According to sources in the BBC, Mr Oliver then attempted to dampen down the story. But, apparently unaware of the Downing Street convention not to comment on matters concerning the Royal Family, he suggested that the Duke might yet be forced out of his role.

The resulting story led the early-morning bulletins. Last night Buckingham Palace refused to say whether it had called Downing Street to complain.

Mr Cameron's official spokesman said: "The Prime Minister thinks he is doing an important job and is making a major contribution and he is supportive of him in that role. We are not reviewing that role in any way. He has made a very important contribution to UK trade through the role and continues to do so."

Speaking during a Cabinet away-day in Derby, the Chancellor, George Osborne, also gave the Duke his backing, telling reporters: "We have confidence in him doing the job and we think he's done a good job in recent years.

"He's promoted British exports. What we want is everyone promoting British exports at the minute and Prince Andrew has done that."

No 10's backing for Andrew was welcomed by Buckingham Palace, which for a number of days has being trying to stress that the Duke would continue working at a job he relishes. A Palace spokesman said: "The Duke of York remains committed to the role of special representative and we are pleased the Government recognises this."

Prince Andrew himself appeared to draw a line under the criticisms as he busied himself with public engagements during the day.

But a source said the Duke recognised that being pictured with Mr Epstein last December was "unwise" and that "you shouldn't expect to see a photo of the Duke of York with Epstein again".

However, Labour former Foreign Office minister Chris Bryant said it was time for him to go. "I think we should be dispensing with his services. I think the charge list now against him is so long that he is a bit of an embarrassment," Mr Bryant told BBC Radio 4's Today programme. "I just can't imagine the next time it is proposed that he goes to a foreign country what the ambassador would think."

Time waits for no man (not even the press chief)

David Cameron's new spin king Craig Oliver was so busy berating his old colleagues at the BBC over their reporting of the PM's views on Prince Andrew that he was late for his first away-day.

Mr Oliver had been due at No 10 in time to leave with Mr Cameron for a trip to Derby. But by the time he tried to get into Downing Street, it was in "lockdown" – so the PM drove straight past his crestfallen Director of Communications.

Mr Oliver went in the side door of No 10 before emerging from the front a few minutes later, a phone clamped to each ear, before hurrying onto Whitehall to hail a taxi.