When Britten's world was all at sea: How Billy Budd reflects the composer's own turbulent times

In his centenary year, Benjamin Britten is proving that he can still stir up controversy. With a full programme of his works at the Proms about to culminate in Tuesday's semi-staged performance of Michael Grandage's acclaimed Glyndebourne production of Billy Budd, it would be difficult to ignore the composer's work this summer. But he continues to make news even beyond the musical arena.

Earlier this year film-maker Tony Palmer claimed that the Ministry of Defence refused to release footage for his documentary on Britten (due to air this November) because of the composer's status as a conscientious objector and perceived "flight" from Britain, with his partner, Peter Pears, on the eve of the Second World War. Meanwhile, the letters page of the East Anglian Daily Times reverberates with descriptions of Britten "cowering under his piano on the other side of the Atlantic".

Such furore merely points up Britten's talent to disturb – as much as by his personal life and politics as by his music. It is a wonderful, and very British, paradox that our greatest composer of the 20th century should represent such a subversive outsider, yet be claimed by the very establishment which once railed against him. And nowhere is that contrast more apparent than in the creative genesis of Billy Budd, a work which premiered in 1951, but whose tense genius can be traced back to its source material: Herman Melville's elegiac and profoundly disturbing novella, Billy Budd, Sailor.

Billy Budd was the last piece of prose produced by the man responsible for the greatest American novel, Moby-Dick. Yet it had lain fallow and unpublished after Melville's death, in 1891. Only its chance discovery, in a tin bread box, lead to its rebirth as one of most celebrated British musical works of the 20th century. Published in 1924, Billy Budd had a delayed, time-bomb effect, as if it had to wait for a new age for its power to be revealed.

When Melville wrote Billy Budd, it was as a "failed" writer. Moby-Dick had sunk without trace in 1851, and he had published nothing successful since. He'd been left high and dry, reduced to the post of customs inspector on the wharves of New York. Billy Budd was his final farewell. It looked back to his glory days, to the pull of the sea and the men with whom he had shared his youthful adventures. But it was also seeped in unexpressed desire and rage against abusive power. It is those heady themes which, 60 years later, made it a such an apt vehicle for Britten's own confrontational nature.

In following Auden and Christopher Isherwood to America in 1939, Britten and Pears were seen as cowards by the British establishment. That sense of opposition sensitised the composer to the story of Billy Budd. The Handsome Sailor is described as "such a fine specimen of the genus homo, who in the nude might have posed for a statue of young Adam before the Fall". He is an innocent, like the young boys to whom Britten became attached, yet from whom he kept his distance. Billy, known as Baby Budd, is the unattainable, unspoiled boy-man.

Press-ganged during the Napoleonic War, Billy is a victim of authority. As Melville writes, "there are Englishmen forced to fight for the King against their will. Against their conscience, for aught we know." It was a condition that resonated with Britten, who also stood accused. "Cads and cowards," as Pears said. "Deserters, even, if you will."

But Billy's fate is sealed by the sinister figure of Claggart, the ship's "policeman", who fixates upon him, and who, in frustration, seeks to frame Billy, accusing him of fomenting mutiny on board the ship. Confronted by Claggart in front of Captain Vere, Billy lashes out against his accuser and, without meaning to, kills him.

"Struck dead by an angel of God! Yet the angel must hang!" – Vere's dilemma is at the heart of the story. Billy is morally innocent, yet the State demands that he must die.

There are other, hitherto unexamined aspects of Billy Budd which seem to have fed Britten's work. Melville wrote his story in the 1880s, at a time when fin-de-siècle literature in Europe was at its height. There are descriptive passages in the novella which seem to speak of Wilde and Baudelaire as much as they evoke the sea literature of Joseph Conrad. Lines such as, "The bonfire in his heart made luminous the rose-tan in his cheek," could have come straight from the pages of The Picture of Dorian Gray or Salomé.

And while Melville's setting may be 1797, his underlying tone seems to prefigure Jean Genet's 1947 novel, Querelle of Brest, set in the French naval port and following the adventures of a sailor lured into a queer underworld of overweening masculinity, corrupt policemen, and sadistic power. Filmed by Fassbinder in 1982, Querelle was in turn reflected in Claire Denis's sensuous, brutal film, Beau Travail (1999), based on Billy Budd but relocated to modern French East Africa.

It is a fascinating, subcultural lineage: from Melville's "pre-homosexual" age to Britten's measured "coming out" in the mid-20th century. In his new, magisterial biography of Britten, Paul Kildea vividly describes what Cyril Connolly called "the authentic rallying cries of homo-communism", which shaped the composer's creative and personal life. Auden, Isherwood and Stephen Spender were far more world-weary than Britten, the innocent young boy from Lowestoft. Their tales of decadent Berlin prompted Britten to write cabaret-style songs such as "Funeral Blues" to Auden's dark lyrics (as revived by Mark Ravenhill at this year's Edinburgh Festival), while Isherwood introduced Britten to a British subculture, via the "notorious" Turkish baths in Jermyn Street in 1937.

And it is that awakening - Gildea speculates that Auden may have been in love with the young composer – contrasted with the tensions of ‘normal’ society, which informs Britten’s works, Billy Budd and Peter Grimes. In both, Britten tapped into the sense of persecution and otherness; the same society that judges Billy Budd is merely a seaborne version of the small-town mentality which pursues Peter Grimes.

There is another interesting counterpoint here, too. If Auden had enlightened Britten, then Melville's mentor was the decidedly heterosexual, darkly handsome Nathaniel Hawthorne with whom he became obsessed; in an extraordinarily suggestive image, Melville wrote that Hawthorne "shoots his strong New England roots in the hot soil of my Southern soul". As Auden's modernism inspired Britten, so Hawthorne's gothic mindset infected Melville. Both were vital creative relationships but both would fracture and fall apart, as if unable to withstand their own intensity.

Hawthorne owed his sense of foreboding to the fact that one of his ancestors had been a judge at the Salem witch trials in the 17th century – the beginning of the American fear of the other, one which would resurface in post-war McCarthyism, as acerbically dramatised in Arthur Miller's The Crucible.

Both Britten and Auden, who had spent formative time in mid-century America, were all too aware of this punitive era. Indeed, in the anti-gay purges of the 1950s, Britten himself would be interviewed by Scotland Yard during its investigation into high-profile homosexuals, while Melville's biographer, Newton Arvin – known as the "Scarlet Professor", and who was lover to the young Truman Capote – was accused and acquitted of being a communist, only to be hounded from office in 1960 when he was found in possession of "pornographic" semi-nude images of men.

That poisonous atmosphere is implicit in Britten's scenarios. The vengeful crowd in Peter Grimes might as well be the townsfolk in Hawthorne's The Scarlet Letter – in which a young woman is tyrannised for her adultery – or the prosecutors of the Salem witch trials. Is it any coincidence that it was in Britten's Suffolk that Matthew Hopkins, Cromwell's Witchfinder General, held sway, responsible for the identification of seven suspected witches in Aldeburgh alone, for which he was paid £2 by the borough?

Pursuit and persecution run through these narratives – from Salem and Portsmouth to Aldeburgh and into the present day. It is one reason why Billy Budd, with its sense of forbidden desire and affronted morality, retains its potency in a fraught 21st century. Sadly, Melville's relationship with Hawthorne fizzled out in recriminations; Auden and Britten parted in the 1940s, and barely spoke again (for all that Alan Bennett reunites them in The Habit of Art). In this light, Billy Budd becomes a melancholy coda to the two relationships. Thomas Mann called it "the most beautiful story in the world". It went on to inspire DH Lawrence, Albert Camus and CLR James, who found its utter darkness almost unbearable; Peter Ustinov filmed it in 1962, with Terence Stamp making his movie debut as a bleached-blond Billy; and Morrissey recycled the title for one of his plangent songs.

But perhaps the last words should be left to EM Forster, the opera's librettist, who introduced Britten to the story in the first place. "Melville…reaches straight back into the universal," he wrote, "to a blackness and sadness so transcending our own that they are undistinguishable from glory."

Billy Budd' is at the Proms on 27 August. There's a free pre-performance discussion by Philip Hoare at the Royal College of Music at 5.15pm broadcast on Radio 3 that evening. His book, 'The Sea Inside', is published by Fourth Estate