Yasmin Alibhai-Brown: Freedom of speech is fine until the invective is against you

The ship flying the flag for free speech is often unsteady, sometimes leaky, as it sails capricious, tempestuous seas. Sometimes even the captains jump off and struggle to keep faith with its mission. Like the supremely erudite Stephen Fry who has always, to my knowledge, been an uncompromising champion of free expression, keeping watch on deck whatever the provocations.

Yet this Friday came the moment when Mr Fry couldn't abide by his own credo and ferociously assailed the Daily Mail columnist Jan Moir for her freely expressed views on the young pop star Stephen Gately. His gay lifestyle, she suggested, was "more than a little sleazy" and his death was unlikely to have been from natural causes. Now Fry commands a virtual army on the web. He can make or break someone with under 140 characters.

He went for Moir on Twitter, later expanding to full-sail wrath on his blog: "a repulsive nobody writing in a paper no one of any decency would be seen dead with has written something loathsome and inhumane". Other big-name liberals and gays have joined in. Advertisers are, apparently, worried and may abandon the best-selling newspaper.

I can understand their rage. The column was ugly, insensitive and homophobic. However, though I passionately believe in free speech, I am not an absolutist nor a hypocrite. The only real argument is where the line is drawn. Perhaps fundamentalists like Fry will now be more honest and accept that there are limits. Even for them.

Milton, one of the fathers of freedom, brazenly excluded some from this fundamental right: "When I speak of toleration and free expression, I don't mean Catholics. Them we extirpate." Professor Stanley Fish, the American culture critic, is incisive in his analysis. Everyone, he says, in the free-speech zone understands what is permitted: "Everyone has a trigger point, which is either acknowledged at the beginning or emerges at a point of crisis." Opinions are not abstracts, they enter society and have to deal with its needs, too.

Seven events this month reveal the increasing tension between freedom and responsibility. Each case is testing and spawns its own, particular dilemmas. Only libertarian fools and fanatics would give set-piece answers. Test yourself.

First came the national furore over "Pakigate" and Strictly Come Dancing. Then a picture of Brooke Shields, aged 10, nude, was withdrawn from view by the Tate Modern. The photographer had paid her mum $450 for the image. Shields herself has tried to have this object of exploitation removed from the public eye. So a good call, I think, by the Tate.

The BNP's bulldoggish Nick Griffin, a white supremacist, admired by the Ku Klux Klan, opponent of Jews, Muslims and mixed-race families, is invited on to the nation's most prestigious TV programme. He, who would deny millions of us the vote, is an emblem of democracy and his violent thugs who try to silence so many of us black and Asian Britons become beneficiaries of free-speech doctrine.

Hitler won the votes of the majority. Would the BBC have done him the honour, too? I say the BNP should be interrogated on news programmes but an appearance on Question Time is a privilege which the BBC now bestows on racists. It sickens those of us who expect better of the corporation.

Then comes the ghastly Dutch MP Geert Wilder who overturned the order banning him from entering Britain imposed by the former Home Secretary Jacqui Smith. He curses the Koran, damns and insults European Muslims, is a fearless xenophobe and seems to enjoy the hurt he churns up.

Invited by a UKIP MP, they both celebrated their victory for freethinking. So why then didn't Wilder accept any of the invitations from Muslim intellectuals to debate his ideas in public? Because he, like many others of his ilk, appears only to want to incite Muslims into behaving like "savages". How disappointing it must have been for him not to have a fatwa to take back home.

I agree that he should be allowed into Britain and I was proud Muslims responded with good sense. But to see him fêted as a hero in parliament was an affront. This must mean free passage for proscribed hate-makers – rabid imams, anti-Semites, homophobic black rappers. If not, it only confirms outrageous double standards.

The most serious threat to free speech has come from David Miliband, now a skilled double-dealer. He talks the talk on good British values and yet rejects the judgement of two senior judges who demand disclosure of information that could prove our intelligence services colluded with the US and others to torture captured Muslims in the "War on Terror", in particular Binyam Mohamed who was held in Guantanamo Bay for many years.

Next the drama over a scientific study on toxic dumping in West Africa by the company Trafigura, whose lawyers obtained an injunction to keep the information secret, including debates on the scandal in Parliament. The gaggers were duly defeated but commercial confidentiality remains an effective weapon used by big business to keep us in the dark.

Lastly, the scientist Simon Singh (a good friend) is being sued by the British Chiropractic Association which objects to his attacks on the profession. He, who is supported by Fry and others, got leave to appeal against an earlier ruling that went against him. Many of us are silenced by the might of libel law. Money, as Orwell wrote, "controls opinion". Singh wants more "freedom to criticise fairly and strongly" on the blogs and scientific writing. I agree but too many bloggers are mad or malicious. So what to do about them? Not easy.

Libertarian ideologues such as journalist Brendan O'Neill have no such moral conundrums: "Offensiveness is part of life; the politics of inoffensiveness is a threat to free speech and open debate." Yes, until people's deep feelings are roused as were Fry's by Moir.

Any woman who has been sexually abused would not have seen the art in the image of Brooke Shields. And Muslims, Asians and Black people are human, too – experiencing the pain of gratuitous invective piled on us, day after day, by toffs like Martin Amis and Wilder and racists like the BNP. Words do violence to humans, more sometimes than sticks and stones. They can disable you to the point of insanity.

Don't get me wrong. More and more freedom is what we must strive for, but a complete lack of restraint leads to anarchy and dehumanisation. Those who react against one set of expressed prejudices should imagine themselves into the pain of others who feel incensed and violated. People like me want to come here to live and breathe freer than we can in our old homelands.

I vehemently object to the way all legitimate questioning of Israel's illegal policies is stamped out and the way minorities try to silence all those who expose community oppression. But freedom is precious and needs to be protected from dictators and censors, and sometimes from itself. That is something even freedom ideologues seem suddenly to understand. Perhaps now the twittering classes will band together to object to Miliband's dirty secrets or stand by racial groups who are needlessly demonised. Just as Gately was.

y.alibhai-brown@independent.co.uk