The golden jubilee: Ben Pimlott tells why we are obsessed by royalty

The British monarchy is possibly the most famous the world has ever known, and certainly the most celebrated in the 21st century. When we talk about monarchy we talk about a political and often religious form that is very primitive, very general and very durable. Indeed no other political form goes back so far, or has lasted so long. In discussing any monarchy, you are not just talking about an institution. You are also, inevitably, talking about particular personalities.

Monarchy and royalty have not been areas that have engaged me much until recently. I had boyhood memories of the Coronation. But I could not have told you the names or ages of minor royals – probably still can't – and, while never a republican, shared the typical Independent on Sunday reader's disdain for a subject that seemed peripheral to mainstream politics and thought.

The year in which my perspective changed was 1992 – the annus horribilis, in the Queen's phrase. That was the year that Windsor Castle was gutted by fire, three royal marriages hit the rocks, Andrew Morton's kiss-and-tell ghosted autobiography of Diana, Princess of Wales, hit the streets and deference hit rock bottom. There was a frisson: a new, almost furtively exciting twist to the permissive revolution.

For the first time since the Victorian if not the Regency period, conventions of reticence were replaced in the media by the compulsion to reveal. At the same time, not just the tabloids, but the most sober broadsheets were taken over by a single theme. Never before had the media been so critical of royalty. Yet never before had there been a topic on which so much had been written and discussed.

It was a curious cultural moment and it intrigued me, partly because it seemed to be combined with a collective lack of self-awareness. The nation – indeed, the world – was transfixed by the fortunes and misfortunes of Britain's titular head of state. Yet nobody paused to consider why that should be so.

What struck me was that an institution and a family dismissed by the sophisticated as trivial and irrelevant was nevertheless the subject of fascinated analysis, humour and the comment. Much of it, indeed, was along the lines that here was a topic so trivial and irrelevant that it wasn't worth talking about.

It was this paradox that engaged me, and it is that I decided to write about in my biography of the Queen – at some cost, I suspect, to my reputation among the seriously serious-minded. "I'm not terribly interested in the Royal Family," people say to you at parties as a put-down, before asking you if you've ever met the Queen. Yet I was, and remain, defiant. Public interest in monarchy is a social phenomenon whose scale alone should arouse serious attention.

Seeing people en masse brings the point home. On the blustery Saturday before the Queen Mother's funeral, I walked the length of the queue waiting to spend a few seconds filing past the catafalque in Westminster Hall – from the House of Lords, along Millbank, across Lambeth Bridge, snaking along the South Bank as far as the London Eye. The walk took nearly an hour – people in the queue were waiting up to seven hours, people of all ages, ethnicities, styles of dress, some chatting or reading a book, others just standing patiently. What was going on? It is possible to deride, but not surely to ignore. Such a curious way of spending time, not among a loopy minority, but among a sizeable proportion of the population of south-east England, cannot, it seems to me, be dismissed as trivial, without dismissing people in general as trivial.

By any straightforward reading of the constitution, the political role of the monarchy in Britain has dwindled to something close to nothingness. We should hesitate to call the monarch a cypher. The role could still imaginably be invoked in some future crisis – a hung parliament with the parties deadlocked, or a war perhaps. There are also regular meetings with the Prime Minister. Lip service is paid to the monarch's right to be consulted by the premier, to encourage and to warn him or her.

But in the living, day–to–day world of decision-making, a political role barely exists – which is one reason why talk of reforming the monarchy offers so little to grasp on to.

Thus the political role of the monarchy cannot account for the scale of the emotions that followed the death of George VI's widow. At the same time, the claim that the public grief for the Queen Mother can simply be taken as a tribute to the unique qualities of Her late Majesty, can only be taken as a partial explanation of its extent. Such a claim may be countered by the obvious point, that the public reaction was equalled or exceeded by the response, five years ago, to the death of Diana.

However "special" these two women may have been, it is surely the case that the public reaction to their loss was – in some sense – out of proportion. Indeed, since nobody else in the past century – apart from kings, and possibly Churchill – has caused such an upsurge of posthumous feeling, it would be making a remarkable claim for them, to say that their "specialness" was merely about them as people. In 1997, Earl Spencer did make such a claim about Diana, when he spoke of her as "classless", needing "no royal title to generate her particular brand of royal magic".

However, Diana was unknown before her marriage, and might well have remained so without it. What Earl Spencer meant was that she no longer needed such a title – it was surely one of the key ironies of the Diana tragedy, that her legend depended on her status as an ex-royal, whose children were royal, and who was herself therefore also for ever royal.

It could be the case, of course, that media hype plays a large part in all this. However, in both the Diana and the Queen Mum cases the press was caught on the hop – it took several days before it came to terms with a reaction whose spontaneity took everybody by surprise.

If it is the case that people turn out for royals in numbers so large, and with emotions so bewildering, that they cannot simply be explained by their personalities or media over-heating – then we need to look much more closely at what "royal" is or might be in the 21st century world. Roll the word around in your head a little – royal, literally "to do with the king". We British are so familiar with it, that even the most secular and republican of us feel it in our bones. Americans, by contrast, are no less fascinated by it – but because they don't feel or understand it, wonder what it is.

So what is royal? At the end of several years of looking at the subject, I am still perplexed. My hope is to convince you that the question exists – a question that hovers tantalisingly over our understanding of the post-Enlightenment, Post-Modern psyche.

There is the question of defining royalty. And there is the question of a cultural understanding of royalty, and the psychology of the public relationship to it.

In Britain, as every schoolchild knows, the definition of royalty is ultimately, and on occasion directly, in the hands of Parliament. This was made plain in 1688, when Parliament declared that the fleeing James II had, in effect, abdicated. If there was any remaining doubt, it was banished to the profound humiliation of the Windsor dynasty in 1936, when a Prime Minister with the backing of Parliament required Edward VIII to choose between a marriage deemed inappropriate and the throne. Again, there were reminders in the 1950s, when it became clear that the monarch did not even have control over her dynastic name. After the death of George VI, Lord Mountbatten had boasted "that the House of Mountbatten now reigned". It had previously been the case that children of the male consort – in this case Philip Mountbatten – took their father's name.

Churchill was outraged, and took the matter to Cabinet. "The Cabinet's attention was drawn to reports that some change might be made in the family name of the Queen's children and their descendants," declare the minutes for 18 February 1952. "The Cabinet was strongly of the opinion that the family name of Windsor should be retained; and they invited the Prime Minister to take a suitable opportunity of making their views known to Her Majesty." And so it was.

As every schoolboy or girl also knows, the Stuart dynasty of the 17th century defended a doctrine of the divine right of kings, which was, however, rejected by Parliament. By the time of the Hanoverians, whose succession had little to do with lineage and everything to do with Parliament, the doctrine had effectively been discarded. The sense of the monarchy as no more than a canny constitutional convenience, indeed, lies behind the chapter on the subject in The English Constitution, by Walter Bagehot, published in 1867, a monarchist, indeed anti-democratic, yet highly secular and refreshingly irreverent work which has had an extraordinary influence on the subsequent interpretation of Britain's unwritten rule-book. It remains the starting point for any discussion of the place of the monarchy and the Crown in the British order of things.

Bagehot was a traditionalist, but he was also a believer in Parliament. He was contemptuous in his dismissal of a monarchical "right" that comes from anywhere except Parliament. He notes that when the Hanover family came to the English throne, "it was a sort of treason to maintain the unalienable right of lineal sovereignty".

However, there is a significant twist in Bagehot's account. In the process of providing his justification for monarchy, Bagehot acknowledges that superstitions that were, by the 1860s, long dead among the thinking classes not only continued to thrive among the ignorant ones – but that, indeed, such credulity provided a vital underpinning to the whole system. Divine right might not be the official justification, but it remained – however contrarily – the popular one. "If you ask the immense majority of the Queen's subjects by what right she rules," Bagehot said, "they would never tell you that she rules by parliamentary right... They will say she rules by 'God's grace': they believe they have a mystic obligation to obey her."

It was essential to his argument in favour of the monarchy that, although neither he nor his sophisticated audience believed for one moment in unalienable lineal right, a degree of tradition and sentiment had built up around the actually reigning family to the point at which most ordinary people took for granted a version of it. What Bagehot seemed to identify was a kind of free-floating superstition in relation to monarchs which was likely to accrue to anybody who the system threw up as sovereign.

Since Bagehot's day the landscape has changed substantially. In the 21st century, the monarchy has become a different institution in a different system. We need to consider, however, whether we can nonetheless recognise traces of "mystic obligation" and a latter-day belief in "the unalienable right to lineal sovereignty" in the actually existing relationship between royalty and public.

What is it to be "royal" in an ostensibly equal society, which affords rich and poor, aristocrat and proletarian, notionally equal rights?

There is certainly something peculiar, and at odds with almost every other social value, about royalty's survival. One answer lies in what might be termed the phantom relationship that exists between the reigning Royal Family and the non–royal general public. At the root of the phantom relationship is the idea of reciprocity – the feeling that the iconic figure has feelings about us, in return for the feelings we have about them.

Let us now return to the reaction to Diana's death – a near-hysterical episode, which has largely faded from memory, perhaps because it is so disturbing and even embarrassing to our rational selves.

One particularly disturbing feature was that not just the scale, but the particular form, was apparently so new. Nothing similar had occurred in living memory. Anybody who visited Kensington Gardens during that strange episode was likely to be moved by the sincerity of the emotion – but also by the weirdness of it all: the people hugging each other, the shrines, the candles, the dolls and teddy-bears, the scribbled prayers to Diana and Dodi, the shoulder-high plateau of flowers covering several acres of land, cellophane coverings glinting eerily in the autumn sunshine.

There was a shifting of the boundaries of the normal. Following the death of Diana, an estimated 1.3 million bouquets were laid at symbolically appropriate sites nationwide, at an estimated cost of some £25m. "It was all so unBritish," as one elderly royal retainer put it to me.

Or was it? In her broadcast on the eve of Diana's funeral, the Queen spoke of the "extraordinary and moving reaction" to the Princess's death, from which there were, she said, "lessons to be learnt". It was certainly extraordinary. However, while there is no recent precedent, there have been other occasions when the British public has gone relatively bananas over a royal death, if never quite as bananas as in 1997. When George VI died in 1952 at the age of 56, for instance, the nation was taken aback by how shocked it found itself to be. "The King's death really has swamped politics," wrote Richard Crossman, recording that the office of the New Statesman was convulsed by a debate about whether the magazine's front page should carry a black border or not. More dramatic still was the reaction to the premature demise of the justly admired Prince Consort in 1861. People spoke of that event as a catastrophe, and wondered at a loss that had made "thoughtful men and maidens, with unrestrained tears exhibit such emotion".

However, by far the most widespread and intense response to an early death in the 19th century – and the one which is significantly comparable to the Diana reaction, in a number of ways – followed the tragic loss in childbirth of Princess Charlotte Augusta, 21-year-old daughter and only child of the Prince Regent in 1817. The public had been preparing to celebrate the birth of a second in line. Thirty years later, Harriet Martineau wrote that "never was a whole nation plunged in such deep and universal grief. From the highest to the lowest, this death was felt as a calamity that demanded the intense sorrow of domestic misfortune". Church bells tolled, public functions were cancelled, shops and theatres were shut and mourning was worn by the whole nation, of whatever rank. There was also a literary outpouring. Byron wrote that "The fair–hair'd Daughter of the Isles is laid, The love of millions!" Interestingly, Charlotte Augusta had been something of a people's princess, on the side of social outcasts, and like Diana, renowned for her empathy.

In both the Charlotte Augusta and the Diana cases, grief was combined with grievance: the death of a cherished female, identified as champion of the have–nots, sparking an unexpected popular protest disproportionate to the nature of the loss. But there was also, of course, another common factor. Both were royal.

As royal princesses, they were pegs to hang a whole encyclopedia of myths on. In the case of Diana, the narrative of ordinariness leading to extraordinariness became a feast for theorists. One interpretation, drawing on both feminism and Freud, presents the Princess (much as Charlotte had been presented) as an archetype of female suffering, violated both by her husband and by the world.

There are two central ideas – first, the tension between the fairy story surrounding a royal marriage and the pressures on a private union; second, and more important, the physicality surrounding the relationship – the phantom relationship – that existed between the general public and Diana the sufferer and healer who reached out to the poor and, in a variety of ways, touched them.

The history is familiar, with a developing theme. In 1987, the Princess of Wales accepted an invitation to open the first dedicated Aids ward. Her willingness to be seen in physical contact with sufferers from one much-feared disease was soon extended to another, leprosy. Finally, she turned to the maimed and disfigured victims of landmines in Africa, visited them, and made herself their champion. In each of these missions, Diana appeared to relate the pain of others to herself.

If the cult of Diana had as much to do with the Princess's physical being – unmarked limbs caressing maimed ones, and so forth – as with her soul, what invested the whole with special importance was the ever-mysterious attribute, shared with the equally instinctive Charlotte Augusta, of being "royal".

And so we come back to this strange concept, which has a habit of not going away, however much we want to sugar-coat it, sentimentalise it, kitschify it, in order to place it on the historical margins, along with Easter eggs and Christmas trees. Diana was not just a pretty face, or a do-gooder. She was a royal face, and a royal do-gooder.

The trouble with embarrassing subjects like royalty is that serious people avoid them to the point at which a serious discussion becomes impossible, because nothing serious is ever written or said on the topic and there is no canon of literature to provide a reference point. In the case of royalty and royalness, historians and other intellectuals shy away, placing modern royalness at the less interesting end of the study of ritual. Thus in the study of contemporary history – unlike in the study of medieval and Reformation history, or of European 20th century history – there has been a preference for the rational over the ideological and romantic. Perhaps it is because we like to think either that our constitution is rationally based, or – alternatively – that we are a rational people who, by abolishing or toning down the anomalous royal bits, can learn to behave even more rationally. And yet the actual behaviour of the British public in 1997, even the comparatively sober behaviour of the public in 2002, points to instincts and emotions that are much more primitive.

At the time of the Coronation, one astute Canadian psychologist identified in the old dominions what he called "a regal feeling", acquired like a mother tongue, he suggested, through birth or residence. In Britain, some version of that "regal feeling" persists in Golden Jubilee year. Yet to identify it as a mere survival is simply to restate the question. What has it survived from, and should we expect it to diminish?

Some kind of "regal" or royal feeling – as distinct from awe or deference afforded to the merely famous and powerful – has certainly existed since prehistoric times. While some of its characteristics and manifestations have varied widely, others have remained remarkably persistent.

In almost every culture, there is a concept of "royal" that has distinctive features, but also a striking degree of universality. Almost everywhere, "royal" has components that are familial, dynastic, traditional, sacred and religious. Above all, "royal" is personal: the concept of "royal" links the well-being of the nation or tribe to the flesh and blood of an individual. In some societies, indeed, the body of the king and the body politic have been seen as the same.

Many royal-linked traditions are modern adhesions, often Victorian in origin. New ones are being added all the time. But there are other elements, and talismanic rituals, of genuine antiquity.

In particular, there is a core to the Coronation Service as it was conducted in the 20th century that it is very ancient indeed. Anointing with holy oil, in particular, is a particularly ancient ritual and has had a continuing centrality. Much was made of this part of the ritual in 1953. When I was writing my book about the Queen, the late Lord Charteris, the Queen's most intimate of former private secretaries, frequently impressed on me the powerful impact the physical act of anointing had had on her, and on her commitment to her purpose, as an impressionable young woman of 27.

You can see the Coronation service as a high mass, a unique uniting of the temporal and the spiritual. Or you can see it as an imposition upon, even a violation of a hapless individual who has no choice in the matter by church and state. Those old enough to remember the televised service 49 years ago will recall the ominous force with which Archbishop Fisher brought the heavy crown down on to the Queen's head. They will also recall the moment when she donned what looked like a white nightie and disappeared into a kind of tent to be privately anointed by the elderly primate. More alarming still are Fisher's sermons delivered over preceding weeks. According to the Archbishop – and today it is almost painful to read – the Coronation was about sacrifice. The service defined her special relationship with the Deity – a relationship based on self-denial.

Anointing and crown-wearing were two physical manifestations – the one magical, the other symbolic – of the link that supposedly existed between monarch and people, and between monarch and God. There was also a third, even more direct and personalised and possibly the most interesting of all – royal touching. Here we may possibly get a step closer, if not to understanding, then to appreciating the psychic power of a concept that continues to elude us. Royal anointing and crowning gave the king power. Royal touching showed that he had it. Today nobody believes or would confess to believing that the touch of a monarch can perform a miraculous cure. The belief that it could do so, however, was once so universally held that for many centuries the efficacy of the king's touch was virtually a test of monarchy itself – inseparable from the concept of a lineal institution.

The history of the royal touch is interesting partly because of its persistence but also because of its significant separation, in the Christian tradition, from the history of God–given miracles in general.

From early on, the king's touch was supposed to be specifically beneficial to sufferers from the "king's evil" – scrofula, or tuberculous adinitis – a condition involving suppurations which made the face putrid with sores. One suggested explanation for this particular illness, rather than any other, is that it replaced the "biblical" leprosy, when leprosy in Europe became less common. At any rate it was significant that – as with leprosy – scrofula made the sufferers repugnant, thereby enhancing the charity of the touch.

There were spin-offs from the royal touching power – so-called "cramp rings" for example, which involved rings fashioned out of coin touched or blessed by a monarch and used as a cure for epilepsy. But the best results came from hands–on contact. Performance of the touch was frequently a mass event. There is plenty of witness testimony as to its supposed efficacy.

Touching survived the Reformation, and was even enhanced by it. The English Civil War of the 1640s seemed to enhance it still further – indeed, a belief in the King's miraculous gift became one of the dogmas of the Crown.

After the Restoration of Charles II in 1660, the King underlined his right to the throne by holding huge ceremonies dedicated to the miraculous cure. Between May and September 1660, no fewer than 23,000 people were touched by him – one particular session in the Banqueting Hall is recorded by Pepys.

The most famous recipient of Queen Anne's touch was the infant Dr Johnson, who had contracted scrofula from the milk of his wet-nurse. Johnson who was touched at St James's Palace in 1712. Though only two-and-a-half at the time, Johnson always remembered the "lady in diamonds and a long black hood" who cured him of his ailment. He wore the touchpiece – a symbolic coin handed out on such occasions – on a ribbon round his neck for the rest of his life.

Such was the long and remarkable history of the king's touch – which has no parallel in, for example, priestly powers, which required sanctity in the healer for miracles to occur. Touching was performed routinely by generations of English and French kings, regardless of their moral standing, and it was specifically a royal capacity.

The rise, flourishing, and decline of the "king's touch" legend in Western Europe closely shadows a belief in witchcraft, suggesting that the two reflect a similar pattern of understanding. However, while witches ceased to be identified by the beginning of the 18th century, monarchs – repackaged and modified – continued to reign, carrying with them half-remembered associations linked to, but possibly going beyond, consciously retained rituals of anointing, crowning and so forth.

Officially, modern constitutional monarchy has long eschewed any notion of divine right. Arguably, however, divinity is so close to the universal concept of kingship that it is impossible for theologians and constitutionalists to separate the two, however much they may try. Similarly, the notion of a royal race – or Royal Family – which is not a necessary aspect of constitutional, parliament-driven, monarchy, is an aspect so much taken for granted that it remains hard to conceive of the institution, even in its stripped-down form, without it.

Royalty today is an institution which modern, civilised citizens like to think of as merely symbolic – no more than a splash of colour, in Churchill's words, providing a comforting element of tradition. It is this institution that stares at us from chocolate boxes and which tourists turn out to see. On the other hand, we have great upsurges of public feeling, way beyond the bounds of normal behaviour in other circumstances. We have the rational, and the irrational. A law-encased dynasty – and a bubbling witches' cauldron of folk memories and unidentifiable magic.

Diana the healer, Diana the physical... "Her touch was almost a spiritual thing," the Prince of Wales was heard to say in private, after her death. "People felt that if they touched her, they would be cured of what they had – and sometimes they were. It may have been auto-suggestion." Many others remembered Diana's touchings – and, indeed, drew sharp contrasts between what the press called the touchy-feely Princess, and other non-tactile members of the Royal Family, who were considered to be failing in their duty by touching people so little.

So, is the idea of royalty dead – or reduced to the plastic, McDonaldised property of celebrity magazines? In Golden Jubilee year, 115 years after the celebration of the last Golden Jubilee, it is possible to say that it is neither. The perception of monarchy has changed, is changing and will presumably continue to change. But in a number of respects it is remarkably the same, containing echoes of the past, not just in the rituals themselves, but in the feelings aroused by them, that it may be too disconcerting to disinter. Nor may it be entirely fanciful to see – attaching to the political construct of royalty as to the fairy-tale mythology of magic princes and princesses – a wistful hope for the miraculous kiss and the salving touch.

Royalty is about a fantasy relationship with people we do not know, and their imagined ability to come to our aid. It is about an experience of that fantasy, and of the soap opera associated with it, that neighbours share with each other, and is part of our identity. This year marks the celebration of half a century of extraordinary change in Britain, and also a half century of abnormal stability. It could be that one of the things that has changed less than we often believe is our communal instinct, of which the "royalty relationship" – sometimes against our better, conscious judgement – is a uniting part.

Ben Pimlott is author of 'The Queen: Elizabeth II and the Monarchy – Golden Jubilee edition', published by HarperCollins at £9.99. This article is based on a lecture recently delivered at St Paul's Cathedral.