Andy McSmith: The mystery of how a hands-on editor could know so little

The story told by Andy Coulson after The News of the World phone-hacking scandal broke was always a strange one. The man who is now constantly at David Cameron's side, directing his media strategy, was then an ambitious, hard-driving interventionist editor who expected his staff to produce results. Yet, he apparently had no curiosity at all about how they did it.

In April 2006, the newspaper's royal correspondent, Clive Goodman, was able to quote verbatim a voicemail from Prince William to his younger brother, Prince Harry, teasing him about a visit to a strip club.

This was one of a sequence of chillingly accurate tales about the princes turned in by Goodman – so accurate that they prompted the Palace to call in Scotland Yard. But Coulson apparently never wondered how his correspondent managed to be so well informed.

The scales fell from his eyes after the newspaper offices were subjected to a police raid and Goodman was charged with a criminal offence, for which he later went to prison. Then Coulson accepted responsibility for an employee's wrongdoing, about which he claimed to have known nothing, and resigned.

What began as a scandal about newspaper ethics now threatens to become a serious embarrassment for David Cameron, after the latest carefully researched allegations by The New York Times, which suggest that Coulson was much better informed than he made out.

One former News of the World journalist, Sean Hoare, told the US reporters that he too hacked telephone calls and that Coulson knew and "actively encouraged me to do it". Another, quoted anonymously, said he had been at "dozens if not hundreds" of meetings with Coulson, and heard people admitting to pulling phone records or listening to messages. (Coulson denies this.)

Coulson is not some minor figure on the political scene. He is, in effect, Mr Cameron's Alastair Campbell. He has the same job title – Downing Street's director of communications – that Campbell once had. He was personally appointed by Mr Cameron, and if he is drawn into the phone-hacking scandal, it will reflect badly on the Prime Minister's reputation. Mr Cameron will be asked whether he really believed the story that Goodman was the only News of the World journalist indulging in phone hacking, and that the newspaper's editor really knew nothing about it.

Mr Cameron was aware of this risk when he brought Coulson into Downing Street from the Leader of the Opposition's office in May. He was prepared to take it, because Coulson is so valuable to him.

He is Mr Cameron's link to the Murdoch media empire, which the Conservative leader has cultivated with the same care that Tony Blair did 15 years ago. Coulson is also the experienced insider who can predict the behaviour of the tabloid press.

Most of his advice has proved sound, though the handling of the case of William Hague and his now ex-adviser Christopher Myers has Tories wringing their hands in despair at its amateurishness. Though Coulson is a highly influential adviser, he is not so powerful that he could tell a politician of Hague's stature to issue a statement giving details of his wife's miscarriages – or stop him from doing it, when that is what he had decided to do. Coulson would be able to advise on timing, however. He may have calculated that if the highly revealing, over-long statement was to go out at all, it was best done on the day that Blair's memoirs were published in the hope that the publicity hype surrounding the book would bury the bad news. The tactic is well known, but this time it manifestly did not work.

Assuming that bad news always comes in threes, Coulson must be asking himself after the Hague fiasco and the New York Times revelations, what next?

The former tabloid editor is not universally loved in the middle ranks of government. Ministers resent his constant access to the Prime Minister, and complain that he is too exclusively interested in the Conservative tabloids, particularly the Daily Mail and The Sun. This is particularly exasperating for Liberal Democrat ministers, who need to communicate with their voters, who mostly do not read those two newspapers.

But he has earned Cameron's gratitude by taking a hefty pay cut to enter No 10 on a special adviser's salary of £140,000 a year, having helped with the negotiations that created the Coalition Government.

One of the conditions he laid down was that there should be an integrated media operation. It meant working alongside spin doctors who had spent months or years attacking the Conservative Party, but Coulson has succeeded in pulling them together as a team. His deputy, Lena Pietsch, was Nick Clegg's press secretary before the election.

The first big test of this combined operation came only three weeks after it was set up, when David Laws, Chief Secretary to the Treasury, faced allegations about his expenses.

As a Lib Dem MP, Mr Laws would normally have turned to his own party machine for help during the crisis, but the Lib Dems soon found that Coulson was the man giving advice. He was the one urging a quick resolution to stop the scandal from building up. Mr Laws quietly stepped down on a Saturday, denying the Sunday newspapers their quarry.

That was a skilful damage limitation operation which illustrated why Mr Cameron needs Coulson. The next big challenge for Downing Street's chief spin doctor could be to protect his own reputation.