Greatest stories never told: Ten famous writers reveal their works that never made it into print

"Heard melodies are sweet," observed Keats in his "Ode on a Grecian Urn", "but those unheard are sweeter" – a dubious proposition for the music lover, but a potent image for the creative artist. The works that exist in artists' heads are beautiful, perfect things – no wonder their creators despair when the works refuse to emerge, beautiful and perfect, and they contemplate the compromised, spavined, half-cocked, half-baked monstrosity before them on the easel, the stave or the computer screen.

Writers of books have often lamented their inability to make concrete what seemed in the abstract so original, clever, funny and moving. "Ever tried. Ever failed. No Matter. Try again. Fail again. Fail better," concluded Samuel Beckett in Worstward Ho. TS Eliot in "Four Quartets" called his performance in writing the poem "not very satisfactory," and his writing "a raid on the inarticulate with shabby equipment".

For every book that reaches publication day and gives its author his two minutes of (if he's lucky) fame, glory, reviews and the admiration of the opposite sex, there's a nasty pile of debris, of aborted riffs, stillborn metaphors and banished chapters. There's a multitude of titles abandoned halfway through, or consigned to the bottom drawer, to be rescued or reheated half a lifetime later. Herman Melville, finding he was making no headway with a novel called Agatha, passed the manuscript to Nathaniel Hawthorne, who tried his best with it before sending it back – and the two authors passed it back and forth, issuelessly, for years.

Some books never get written at all, but stay, wraith-like, in their owners' heads, often understandably: the three-decker Victorian novel in verse, the definitive biography of Shakespeare, the translation from the Sanskrit, the cowboy/SF crossover romance, the novel written entirely without the letter O. There are books that are overtaken by history (John Fowles apparently spent years "blocked" because his Cold War novel was eclipsed by the events of 1989), and some overtaken by (as it were) the future: in Granta 100, Martin Amis explains how he abandoned The Unknown Known – the story of a Muslim terrorist taxed with rounding up every subversive rapist in the US and turning them loose on Colorado – because "in the end I felt that the piece was premature, and therefore a hostage to fortune; certain future events might make it impossible to defend."

In a major contribution to the non-genre, George Steiner now publishes My Unwritten Books, describing in tantalising detail the seven books he never got round to writing. His reasons for inaction are varied – he couldn't write the one about the state of Israel because he lacked the "clarity of vision. And the Hebrew"; he abandoned the biography of a brilliant Sinophile scholar because he found him, in person, mendacious; and he couldn't write the book about his polyglot sexual partners for fear of hurting certain unspecified ladies. But his conclusion is melancholy: "A book unwritten is more than a void. It accompanies the work one has done like an active shadow, both ironic and sorrowful. It is one of the lives we could have lived, one of the journeys we did not take."

John Walsh

Kate Mosse

A confession – I've been obsessed with Robert Falcon Scott and his doomed journey to the South Pole in 1912 ever since watching a repeat on BBC television of the classic, and lavish, Ealing Studios 1948 film Scott of the Antarctic, starring John Mills as Scott with music by, for crying out loud, Ralph Vaughan Williams himself (these were the days when I didn't realise that great composers could also be alive composers.)

It is the great heroic British story: man against nature, pride and hubris versus – well, listening to advice, British spirit in the face of catastrophe at the end. And the moment when they see, from a distance, the Norwegian flag fluttering in the chill, howling wind and understand that Amundsen and his dogs have beaten them.

I started to scribble a few notes (following my first attempts at a novel, Nanouk the Eskimo, with illustrations, at the age of six) with the thoughts of some imagined extra member of the expedition... Of course, lacking any personal experience of a) polar expedition, and b) being in sub-zero conditions (having grown up in Sussex, which is, even at the coldest of times, really nothing to remark upon); and crucially c) having no actual story in my teenage mind, I realised PDQ that there was no way I could bring the landscape, let alone the men, to life. Besides, it had been done.

But I remain wistfully drawn to Scott and his last expedition and admire enormously those writers – say, Jack London, Stef Penney, Sarah Wheeler, Ted Tally – who bring to silent life the snowscapes and ice and endless unchanging skies of the frozen Terra Nova.

Will Self

Looking back over my files of unrealised projects, I found this one from 2004 about a group of average man – and woman – superheroes, entitled The Fantastic Four: "Phillip Glass read a lot of newspapers and magazines. A great unruly thatch of them covered his kitchen table. Often his wife, Thelma, would take armfuls out into the garden and burn them. Phillip Glass discovered, through his assiduous – and yet casual – newsprint trawling that as his life winnowed down to seeming-nothing and suburban entropy, he had become, statistically, the most common individual in Britain – if not the wider world.

Phillip Glass was, he realised, a member of a silent minority that lay in the densely-shaded intersection of the most complex imaginable Venn diagram. His marital status was typical; his sexual orientation predictable; his income, median; his consumer profile, lacking in salience; his occupation, office bound. But far from Glass relapsing into the mild depression we might expect (we recall at this juncture, Beckett's 'Soon we will be old, then we will be dead'), one ordinary morning, when a senior colleague from Corporate HQ was late for a meeting, Glass discovered that he had Amazing Powers. His very anonymity and intuitive ubiquity made it possible for Glass to assume the role of the missing colleague, while yet attending the meeting himself. This was not a straightforward impersonation – but something more numinous and complex. Glass was the ultimate faceless bureaucrat, and as such anyone present could see anyone not present's face projected on to his fleshy screen..."

It goes on; Statman – as Glass styles himself – discovers Bigman, Crazy Cat Woman and Neurogirl, all of them bog-standard types with incredible powers. Together they begin to right the world's wrongs. Of course, then along came the TV series Heroes and rendered my novel idea otiose. Am I pissed about this? Am I hell – nothing is ever lost in the creative economy – unlike the real one.

Deborah Moggach

Most of the unwritten works are stage plays, actually, rather than books. This is because I long to hang around with actors during rehearsals, smoking and drinking coffee from styrofoam cups while they call me "darling"; then of course my name in lights in Shaftesbury Avenue. In fact, I once dreamed an entire play called Gravity, which was a very sub-Stoppardian piece set in a Cambridge don's apple orchard. I even dreamed its accompanying reviews, which, annoyingly enough, were rather poor – honestly, if one can't get rave reviews in one's dreams, what hope is there?

But an idea I worked on much longer still sends a frisson through me. I've always been drawn to what-might-have-been – paths not taken and so on. This one opens with a young honeymooning couple, back in the Forties (they're probably Fabians or young communists), taking a walking holiday in the Swiss Alps. The bridegroom stumbles and falls down a crevasse, never to be seen again. The years pass. His bereaved young wife eventually gets married, has a life, a job, has children... grandchildren... 60 years pass... Until, due to global warming, the glacier melts. Her bridegroom reappears. And he is, of course, perfectly preserved as the beautiful young man she married all those years ago, still wearing those same clothes, still with that haircut. Meanwhile, she's become an old woman, she has had a whole life. He has had none. But all those possibilities are within him, the possibilities of the life they would have had together, like petals packed into a bud.

You can see the problem with this: where do we go from here? And, even more worrying, I discovered that Shared Experience had staged a piece about something horribly similar. Perhaps, oh God, it had seeped into my unconscious? So I abandoned it.

But lost plays are somehow more poignant than lost novels, because one has also lost the actors who would have given them life – they have also been shut away in that drawer, their own journeys never taken. And they'll never even know about it. Which, in this case, is probably a good thing.

Amanda Craig

I have a few "sock drawer novels" knocking around – a dreadful romantic thriller set on Capri, a historical tragedy inspired by the life of the poet Catullus and a mock-Gothic mystery involving the Brothers Grimm. All were half-written in my teens and early twenties, when I was under the delusion that fiction was about fame, money and the love of beautiful men.

However, nothing fills me with as much relief as the idea that my space opera, The Abyss of Time, will never see the light of day. Like all budding writers, I was trying to copy what had excited me most when growing up. In those days, Faber still published its best SF short story collections, and there were some genuinely cutting-edge writers such as Robert Heinlein, John Christopher, Ursula Le Guin, Philip K Dick and Cordwainer Smith who excited me enormously, especially when I was feeling cramped by reading Eng Lit at Cambridge and being told that Modernism was the last gasp of civilisation. I wanted monsters, magic, marvels and bombs, not bloody Leopold Bloom trudging round Dublin. When I conceived of a towering space opera based on The Tempest, with Prospero as a genetic scientist living in exile with his "daughter" Miranda on an asteroid, and Ariel and Caliban as warring aliens, I was thrilled.

A mixture of magniloquent philosophy and stilted pornography, its climax involved a lot of intergalactic explosions and a hermaphroditic elopement. Really, I just needed to live longer, calm down and get out more.

Joseph Connolly

Twenty books I've published now – 10 fiction, 10 not – but the number I haven't written but wanted to, or did write but never saw the light of day, amount to a damn fair few. If one added to those the existing books one wishes one had written (in my case, Oliver Twist, Alice, Brideshead Revisited, Billy Liar, Lucky Jim... I could go on) the score becomes faintly ridiculous.

The first book I didn't write was to be my inaugural novel – the work that would unveil me to a panting planet. We're going back more than 30 years, when all I'd really read were Buckeridge's Jennings books and James Bond, so I thought I'd set it in a boarding school (my sole experience of life to date) with snobby, sexy and sadistic overtones. The role I had in mind for Matron was unspeakably filthy, though not, it occurred even to me, entirely plausible. No need to explain why that never left the launch pad. Soon, I'd jilted Jennings and 007 in favour of Thomas Hardy, so I thought I might write a very long, um – hommage. After a few awful stabs at weather and lambing and reddlemen and strong, independent women in long rustly dresses, I had to admit that my grasp of 19th-century rural life and romance was less than total.

The third novel I never wrote nearly came to be when I was an antiquarian bookseller. In an article in Punch, Paul Jennings had said we were a miserable lot, and one would never see a pub called The Laughing Bookseller. Maybe not – but a novel? I actually wrote bits of this and had the cheek to show them to Kingsley Amis, who lived down the road. He said he very much liked the recurring leitmotif of the cheese sandwich, but as to the rest... he passed it on to Martin, who at the time had published one fairly coolly received novel, and no one much expected a second. He very much liked the leitmotif of the cheese sandwich, but as to the rest... Paul Jennings was right: the bookseller wasn't laughing. And as to the cheese sandwich, I didn't even remember putting it in.

Andrew O'Hagan

When writers speak of their juvenilia, they pretend to be disclosing their embarrassments, but actually they are disclosing their vanity. It would be much more meaningful if people stopped calling it juvenilia and started calling it rubbish. But writers, as much as anybody, are addicted to the notion that first steps, even very faltering ones, can provide a wonderful indication of the giant leaps that are to follow. That is the vanity encoded in all discussions of early things we failed to publish: wasn't I cute, wasn't I muddled, wasn't I clumsy, but look, my darlings – don't you see the very seedlings that threaten at any moment to burst forth into the glorious light of day!

One summer, when I was a teenager, I spent every hour writing a novel called On a Clear Day. No, it wasn't a dream of a future without pimples, but a perfectly earnest ream of trash about somebody being caught dodging their fare on the train from Glasgow to Ayr. I wrote it in several stolen school exercise-books – talk about unity of form and content – and was always looking for impressionable girls to read it to. Even at the time, this deathless manuscript lacked charm: it was boring. All over Scotland there are girls called Alison and Heather and Christine whose taste for the glories of world literature were spoiled for ever by my relentlessly passionate recitation of said book. I think I got the title from a Barbra Streisand song, which tells you everything you need to know.

But it's really people's adultilia we ought to be afraid of, the rubbish we write when the muse is off enriching somebody else. I have two short stories from a year or more ago, and, I tell you, I can't find a drawer deep enough to bury them in. They were both started in a chardonnay rush – one of them is about Muriel Belcher, the famous former hostess of the Colony Room in Dean Street, Soho, written almost entirely in Polari; the other is a not-entirely-lovely piece about a respected contemporary artist who goes around burning down old buildings. They are not the work of youth, but of a grown, professional man, who should be taken out and shot some time this afternoon. But maybe I'm being hard on myself. They obviously just lack the poignant Streisandesque title. Hello Muriel? What's Up, Muriel?? The Muriel We Were?

Elaine Feinstein

In an old cardboard box, high in an attic I can no longer reach, sit a few chapters from two books abandoned but not entirely forgotten. The first was intended as an ironic take on Brideshead Revisited, set in a Cambridge just after the Second World War; the story was to be told by a poetry-loving Newnham College scholar from the working-class Midlands, drawn into a set of smart young people she can't handle. I enjoyed writing about the wicked assurance of landed beauties and the whiff of darkness and snobbery in King's College parties but, to give the novel some edge, I needed to invent an English working-class home with genuine Richard Hoggart credentials. Using my own Russian Jewish background distorted the whole structure. So I put the book aside to write Lady Chatterley's Confession, and have never gone back to it.

I feel more regret for the second, a biography of Benjamin Disraeli, one of those luftmenschen – men of air – who live on little more than quick wits and impudence. Always distrusted for his dazzling clothes, mercurial conceit and the scandals that pursued him, he nevertheless bewitched rows of stolid gentlemen from the Shires into choosing him as leader of the Tory party. Once Prime Minister, he even captured Queen Victoria's affections, rather to her royal surprise. I put the book aside because there are several good biographies of Disraeli already and I am not a parliamentary historian; but even now he tempts me as a figure in a novel.

DJ Taylor

Hugh Walpole used to spend each year end noting down the endless stream of volumes he intended to write. Driven and indefatigable, Walpole managed to conjure a surprising number of these projects into existence.

I don't have his tenacity. Various biographical subjects are off limits, the territory already staked out by better-qualified parties: Anthony Powell because the inestimable Hilary Spurling is already hard at work; George Gissing because Professor Pierre Coustillas, the world's greatest Gissing authority, has been unwinding the melancholy threads of his career these 30 years and more; Ronald Firbank because – extraordinarily – two books about him are due out in the next few months.

Then there are those PhD theses that got away: fantastical monsters with (notional) titles such as Twentieth Century English Literary Culture and The City and the Book: Urban Themes in Victorian Literature. Or the study of the Singing Postman, Allan Smethurst, composer of the immortal "Ha Yew Got a Loight, Boy?", and his symbolic importance in the decline of popular culture. And the update of Cyril Connolly's Enemies of Promise, demonstrating why it is impossible to write great literature in our debased age. Finally, there is a shelf-full of phantom fiction, including a religio-political epic entitled God: The Novel, for which I fear the world is not, and perhaps never will be, ready.

Jonathan Coe

It's too risky to talk about the novels I've never got around to writing, because a lot of those early, sunken ideas have a habit of rising again, years later. Often, the reason you set a story aside is that you don't feel strong enough to write it – not yet.

But there is a non-fiction idea I toyed with, which I'm pretty certain I'll never write – a big, wide-ranging history of post-war Britain, reflected entirely through its comedy. It would have started with the consensus era of comedians like Tommy Handley, who drew the country together in a community of laughter. Then it would have moved on to The Goons and Beyond the Fringe, the comedians who sounded the first explicitly anti-establishment voices, dividing their audience: baffling and outraging some, while providing for others an outlet for long-pent-up angers. With Monty Python, the 1970s would see even more anarchic pressure being applied to that consensus, finally causing it to fracture – but not until it had reached a high-water mark with the Morecambe and Wise Christmas show of 1977, watched by 27 million people in a moment of shared enjoyment that would never be repeated nationally again. Then the bombshell that was Mrs Thatcher, the polarisation of comedy into "mainstream" and "alternative" camps, reflecting the battle lines drawn up all over the country. After which, moving into the 1990s... well, you get the idea. I still think it would have been an interesting book.

Lynne Truss

For at least a dozen years, I have been collecting material for a book about Captain Cook and the South Pacific. I don't know why. It feels like a grand lifetime's project, justifying the purchase of voyage journals edited by the wonderfully named New Zealand scholar JC Beaglehole, but I certainly haven't found my story yet – and my vague, slow-burning ideas keep getting overtaken by events, which is discouraging.

For example, for a while I thought about concentrating on the visit to England of Omai, the native of Huahine brought back on the second voyage; but then the sale of Reynolds's heroic Omai portrait became front-page news, and my weedy idea started to look a bit pitiful. Going back further, in the mid-1990s I got excited about telling the story of Cook's voyages from the point of view of a cat – in fact, Dr Johnson's famous cat Hodge, who had (I liked to imagine) stowed away on the Endeavour after Johnson made that wounding remark about having had better cats than him. Hodge might be good at etymology (having overseen the composition of the Dictionary), and might therefore take an active interest in the naming of Botany Bay and so on. Anyway, I had no sooner mentioned this rather terrible idea to my closest friends than a very good book came out about Shackleton and the Endurance – told from the point of view of a cat. Damn.

I haven't given up, though. I feel sure that my path through this material will be revealed. Meanwhile, a small model of Cook sits always on my desk, I listen to the soundtrack from Master and Commander with a wistful faraway expression, and anything written by anyone called Beaglehole will always be welcome in my house.