Exclusive extract: How Cameron tried to evade Murdoch's embrace

For the first 15 months after Cameron won the leadership election, his media team, led by George Eustice and with the support of Steve Hilton and Oliver Letwin, had adopted a strategy of arms' length engagement with the press. Hilton was among those who believed that no longer was the printed word a major leader of public opinion. People were now less deferential to thunderous editorials from on high. They made up their own minds, thank you.

Hilton and Eustice had sold the idea of a new arrangement with the press barons, whereby a greater, healthier distance was kept and the democratic process would be all the better for it. Television was to be the new battleground. There was to be "no more sucking up to Murdoch". It was a defiant and refreshing departure from the Tony Blair textbook.

There were to be other, tougher aspects of this new approach, and they had largely been vindicated, not least when a well-prepared Cameron outfoxed Jeremy Paxman during the leadership campaign. Cameron and his team had taken a muscularly non-committal approach to mounting calls to "come clean" on whether the leader had ever taken Class A drugs. Editors in supportive papers who offered to "do the drugs story sympathetically" were politely told that Cameron was allowed to have a past, before he became a politician, that was free of scrutiny.

In other words, Cameron's people expressed a novel "no thanks, we'll play this game our way" attitude. And it seemed to be working. In August 2006, the Tories had established a good lead in the polls. Such insolence was not appreciated by the Murdoch-owned Sun, which asserted in its leader column that Cameron did not deserve his lead. Murdoch's critics took this to mean "we have not endorsed him yet".

But the plan was to turn out to be no more than an experiment. By early 2007, the sunny hopes for a new politics were already fading. Cameron was struggling to convince a great many in his party that he was a Conservative at all, and with the Labour government anticipating a rise in their poll ratings with the expected accession of the comparatively untainted figure of Gordon Brown, the arms' length relationship with the Murdochs looked needlessly bold.

The previous summer he had made the notorious proclamation that became known as his "hug a hoodie" speech. The press reception had been hostile, but for Eustice and Hilton this was no more than was to be expected. "Yes, yes, we'll carry on," said Cameron, evidently less convinced than his aides. At their encouragement, "to show we wouldn't be bullied", as one confidant put it, he followed up with a speech of comparable tone and surprise value. Cameron was the object of yet more consternation from his natural supporters and their mouthpieces in the press. When his closest media advisers proposed a third such speech, Cameron, in a typically Cameronian way, said: "I'm sure you're right... but how much more of this do we have to do?"

To emphasise his declining faith in the new approach, he invited comparison with the never-say-die "invincible knight" from Monty Python and the Holy Grail, whose bravado led him to having all his limbs cut off but who insisted on shouting fatuously at his triumphant adversary: "Tis but a flesh wound!"

This was Cameron's lighthearted way of saying enough was enough. In fact, he felt things were getting very serious. With the austere but principled Gordon Brown waiting to sweep away the perception of Labour as polished but shallow, Cameron needed to be at the top of his game. Indeed, what would happen if Brown were to call an early election?

Cameron knew Brown had good relations with both Rupert Murdoch and Paul Dacre, editor-in-chief of the Daily Mail. Why should either newspaper group go against the personal preference of its most senior executive and endorse Cameron? Charlie Brooks, who at that point was at the start of his relationship with Rebekah Wade, told a friend that Rupert Murdoch – whose pedigree in picking winners was second to none – believed there would never be another Etonian in No 10 again. The thought occurred to Cameron that he could be out of his job by the end of the year.

"David got very tetchy at around that time," says Cameron's friend. "His apparent breezy confidence often hides a lack of exactly that, and David had a really major wobble that spring."

"We tested the new strategy to destruction," remembers George Eustice. "But the knowledge that Gordon Brown, the ultimate licker-of-the-boots of the Murdoch regime, was coming in meant that Cameron was likely to be portrayed as loftily out of touch rather than right, so we had to abandon the experiment. David stuck with my strategy as long as he could," remembers Eustice, "but having reluctantly abandoned the 'keep your distance' approach, he then embraced the 'OK, let's do what everyone else does', and let's do it properly stance."

Someone else close to Cameron's thinking added: "George Osborne in particular, and to a lesser extent Michael Gove, thought it was part of the New Labour playbook to get Murdoch on side. Keen on the traditional idea of wooing the media. We might not like it, it's just the way of the world. However unpleasant these people may be, however much you may not like them, you have to play their game and go to their parties."

So, notwithstanding Cameron's previous coolness to Murdoch, a legacy of Carlton TV's defeats at the Australian's hands, it was all systems go to get Murdoch on side. The brave talk ("We don't need bloody Murdoch!", as one of Cameron's strategists had put it privately) was set to one side, and every opportunity to impress News International was to be grasped with alacrity.

As luck (if that is the word) would have it, there seemed an admirable way of killing two birds with one stone close at hand, the appointment of Andy Coulson, who had recently resigned from the editorship of the News of the World.

For all his outward lack of concern, by 2008 David Cameron was becoming worried about the phone-hacking story. With the next election still a couple of years away, according to a friend of Cameron, the leader asked Coulson outright if he had known anything about the phone hacking, to which he replied: "Categorically not."

In a private context, he also asked people with extensive dealings in newspaper regulation, who he had good reason to believe would be familiar with the workings of the grubbier end of Fleet Street. In good faith, it is fair to assume, they too reassured him that he had little to worry about.

When, in August 2008, Cameron and his family were invited to visit Rupert Murdoch's yacht Rosehearty, the flights to Santorini were paid for by Matthew Freud (which concerned some of Cameron's advisers more than it did him). For Freud, this was the ultimate piece of political matchmaking. This was jet-setting of the old school.

But in a private one-on-one, Cameron asked Murdoch directly about the continued rumblings about his press chief. Murdoch replied that, as far as he was concerned, the police knew everything. There was no new evidence of wrongdoing. He blamed his political enemies in the British media for stirring up trouble.

The trouble was that the "stirring" didn't stop. The House of Commons Select Committee for Culture, Media and Sport spent a considerable amount of time investigating media standards, eventually concluding that it was "inconceivable" that none of the News of the World employees past and present had known about the phone hacking and accused senior executives of suffering from "collective amnesia".

Cameron was to seek assurances from Murdoch again before the 2010 election, and Murdoch's answer was the same. In that case, said Cameron, he would stand by Andy and make sure the "witch hunt" was unsuccessful. The Murdoch reassurance chimed with Cameron, whose cast of mind inclined him to believe that if the police had found nothing, there must be nothing there.

But a slight switch to how Cameron responded to questions about Coulson was introduced. As a close colleague says: "When there's a bit of doubt, the position shifts, the language changes." And so it did on this occasion. Previously, on the rare occasions he had been asked, he would offer a vague suggestion that he was led to believe Coulson knew nothing about the phone hacking. But now, on 10 July 2009, the words "I believe in giving people a second chance" received their first airing.

By 2009, the Tories began to show a new antipathy to the BBC, floating the freezing of the licence fee and urging the corporation to "do more with less". The Tories and the Murdochs shared a striking identity of interests. With the Tories looking good in the polls, what more could the media empire ask for?

Just a few weeks later, at the most damaging of moments for Labour (ie, during its conference), The Sun announced it was to support the Conservatives in the next election. But still the phone-hacking story wouldn't go away.

Representations were believed to have been made by senior courtiers at Buckingham Palace, which had always been unhappy at Coulson working by Cameron's side in opposition, but hitherto had been pacified by Coulson's intention to find another job after the election, according to an unimpeachable royal source. The Palace insists the Queen herself did not initiate an attempt to influence Downing Street staffing, but acknowledges the possibility that Cameron's circle was sent a message informally about the feelings of the Royal Household about the prospect of the former News of the World editor in Downing Street.

Expressions of concern were also made to Cameron's office, at least, by Nick Clegg, Lord Ashdown, Zac Goldsmith, Sir Max Hastings, Daily Mail editor-in-chief Paul Dacre and others. And a more surprising voice spoke to him to question whether Coulson should follow his master into Downing Street: that of Coulson himself, who worried that he was becoming an embarrassment to Cameron.

Cameron waved his concerns away – he needed him, he was coming to Downing Street. It was a terrible mistake for both of them. Had Coulson left at the election, the interest in hacking might have largely died away. His presence at the heart of Britain's government, however, redoubled determination to expose the truth.

On 1 September 2010, The New York Times published a lengthy account of the extent of phone hacking at News International. Cameron was lobbied yet again by those who felt he was being overgenerous to Coulson, and told a friend at the time: "I can't go round sacking people on hearsay. There's no evidence against Andy." But he did also say to a close friend at the time: "If it does turn out he's been lying to me, he'll be out of here tomorrow."

But while Cameron toughed it out, there had been a change of attitude inside News International. Emails emerged that, it was reported, showed much wider knowledge of unlawful activity than had been previously admitted.

At around this time, with Murdoch furious at not having been told the full extent of what had been going on, it was made known to Cameron that he should no longer feel obliged to protect Coulson. In fact, Coulson had been telling an unheeding Cameron for some months that he felt he should stand down, and on 21 January he resigned as director of communications.

Extracted from 'Cameron: Practically a Conservative' by Francis Elliott and James Hanning, published by Fourth Estate, £10.99. Copyright Francis Elliott and James Hanning, 2012