What if I don't want to wear a white shirt on Sunday?

A long time ago I was having lunch with Dr (now Sir) Rhodes Boyson, then a rising Tory politician. Norman St John-Stevas did not approve of him and used to call him "The Colossus" after the Colossus of Rhodes. In the course of the meal Dr Boyson announced that he believed in democracy and in freedom. What did he believe in, I asked, when there was a conflict between the two?

A long time ago I was having lunch with Dr (now Sir) Rhodes Boyson, then a rising Tory politician. Norman St John-Stevas did not approve of him and used to call him "The Colossus" after the Colossus of Rhodes. In the course of the meal Dr Boyson announced that he believed in democracy and in freedom. What did he believe in, I asked, when there was a conflict between the two?

Anticipating Mr William Hague's speech of 10 days ago, Dr Boyson replied: how could there be any conflict? Quite easily, I said. I had recourse to that useful philosophical concept, the desert island. Suppose there were 10 people marooned on an island. Luckily they had managed to take all their clothes with them. By a majority of six to four they decided that it would be compulsory to wear white shirts on Sundays.

Which group did Dr Boyson support? The insisters on white shirts? Or those who wanted to wear whatever they liked (and who might by this stage have numbered fewer than four, for some of them might have decided to accept the majority decision)?

Dr Boyson contemplated for some time, as if this was the first occasion on which the problem had been presented to him. He concluded that he was with the majority. White shirts would have to be worn. For a former headmaster such as Dr Boyson, this conclusion was not surprising. A former headmistress might well have decided that white shirts would have to be worn irrespective of any vote. Indeed, she might have felt that holding a vote at all was subversive of desert island discipline.

For my part, I said to the doctor, I was of the freedom party. Nor did vague considerations of solidarity or increased morale among the 10 affect my view. It would be different if they were being menaced by an invading force, all wearing shirts of different colours. In that case white shirts might be a military convenience, to enable the defenders to recognise one another, as with a football team. But unless solid reasons for compulsion such as these could be produced, people should be allowed to do as they liked.

I am not perhaps entirely consistent about this. Few of us are, though on the whole we are more so than the politicians. In the 1980s the young men and women who worked in the City frequented wine bars in large numbers. Many of these bars were, indeed, set up specially for their benefit. One of their characteristics - then as now - was that if a group of, say, eight entered licensed premises they would either stand at the bar or, if they sat down, refuse to split into two groups of four each. Instead they would form themselves into a flat oval.

A few years ago, not long after the Fleet Street diaspora and the consequential influx into the district of persons connected with finance, such a group entered El Vino's wine bar (or, as Lord Beaverbrook used to call it disapprovingly, "El Vino's public house"). They immediately began to rearrange the furniture, placing tables together and shifting chairs till the requisite space was secured. I was sitting in a corner and summoned the manager.

"Tell them to stop all that, would you, Christopher?" I said.

"But they're not doing you any harm, Mr Watkins."

"It's not a question of whether they're doing me harm or not. It's a question of whether they're behaving properly."

If I were trying to make out a case for consistency, I should say that they might well have being doing me and others harm if the room had been more crowded than it was at the time - that accordingly my attempted prohibition, futile though it turned out to be, fell squarely within John Stuart Mill's principle that people should enjoy liberty provided they did not either harm others or restrict their liberty. The other answer is that there is a crucial difference between disapproving of certain conduct and wanting to prohibit it by law.

These are plausible, serviceable answers. But they are not entirely satisfactory. In a secular (even though credulous) age, most people nevertheless believe that a funeral cortÿge should be treated with respect: that, if hats are no longer doffed, onlookers should at least not fool about or shout: "'Ere we go, 'ere we go, 'ere we go." We believe this not simply because we do not want to hurt the feelings of the deceased's nearest and dearest or of other mourners (though that certainly comes into it) but because it is, we think, the right way to behave in the circumstances.

The fallacy of modern politics is not only to believe, as Mr Hague does, or purports to do, that there can be no conflict between freedom and democracy, but to proclaim, as Mr Jack Straw tends to do, that any restriction on freedom must be justified by the increase in the supposed freedom of somebody else. Thus the insistence on driving on one side of the road, and the prohibition against driving on the other, is justified by the freedom not to be killed, seriously injured or merely inconvenienced. But it is simpler and more honest to admit that the freedom of motorists is clearly being restricted in the worthy cause of safety, convenience or - in radical circles long a suspect word - order.

Wholly unnecessary confusion has been caused not only in political theory but in governmental action through the importation of the positive notion of freedom into all discussions of the subject. It confuses the freedom to do something with the ability, the capacity or the power to do it. For example, someone with no money is perfectly free to go to Paris for the weekend. If he or she is deprived of a passport by executive action, that freedom disappears.

This old-fashioned definition of freedom as the absence of external restraint is now out of favour with all three political parties, as it has been for ages. When Labour people object to it on the basis that it is out of touch with the "real world", I cite the example of Aneurin Bevan as a young man at the Central Labour College in the 1920s. Bevan was fond of good food and good drink early in life. When he was at the college he found he could not afford them as he would have liked. So he subsisted on tea and buns for six and a half days of the week, saved the money he would otherwise have spent on more substantial but still unattractive fare, and enjoyed a blow-out at a good restaurant once a week.

In the Spectator a fortnight ago, Mr Bruce Anderson predicted the present mess in the Tory party about drugs. It is an even bigger mess in the Government. During the course of his observations he wrote that, drugs apart, all the social changes of the past 20 or so years had been in a direction that was liberal, even libertarian. We are in our present mess about drugs owing to panicky legislation in the early 1970s. We subsequently became allies, as we always do, of the United States, this time in Mr George Bush senior's doomed "war on drugs".

But in other areas I have been unable to detect any countervailing movement. Mr Straw has done something about the licensing laws: that is all, more or less. Last year at Bournemouth Mr Tony Blair was scornful of "libertarian nonsense". Indeed, this is the most pettily bossy government since 1945. Last week Ms Yvette Cooper, the Minister for Public Health, was urging the virtues of virginity. I was reminded of nothing so much as the American actor who said: "I knew Doris Day before she was a virgin."