Does Philip Hoare's Samuel Johnson Prize victory confirm that formal biography is now a dying art?

In spring 2001, Philip Hoare published Spike Island, an eerie and evocative history of a military hospital near his Southampton home and the sad secrets that it harboured. After its release, he got an admiring letter from the very writer whose inspiration lay behind the work. To receive the compliments of WG Sebald, Hoare told me on Tuesday after he had won the BBC Samuel Johnson Prize for non-fiction with his latest book, Leviathan, or The Whale, "was like getting a fan letter from EM Forster". Too soon, Sebald died in a road accident in Norfolk. Since then, the posthumous influence of the unclassifiable, genre-crossing German author who lived and taught in Norwich for more than 30 years has never ceased to grow.

Leviathan is Hoare's extraordinary, multi-dimensional quest for the whale through memoir, biography, zoology, history, travel and – of course – the fictional monster of Herman Melville's Moby-Dick. It richly embodies the new directions in narrative that Sebald took himself, and opened up for others. Hoare had written biographies of Noel Coward and the aesthete Stephen Tennant before The Rings of Saturn, Sebald's mesmerising tour through the myth and history of coastal East Anglia, struck him like a thunderbolt.

Of course, other writers had already shown that non-fiction narrative could evolve into a skittish mongrel beast. This hybrid style of storytelling mingled its literary bloodlines, dramatised the author and – crucially – threw off the tight leash of cradle-to-grave biography, cause-and-effect history or neutral travel reportage. Pioneers such as Iain Sinclair and Bruce Chatwin had also broken acres of fresh ground.

Yet it was the drily witty German expatriate who helped Hoare escape from the chafing fetters of conventional biography. "It has been his spirit and encouragement that freed me from the constraints that I'd felt myself labouring under," he says. Researching Noel Coward, Hoare had felt caught up in a grinding machine of anecdote, rivalry and recollection. "I met people like Katharine Hepburn and Gore Vidal, who were still rehearsing old feuds. You realise that you just become part of this ongoing fabrication."

He became "very frustrated by the supposed objectivity of writing history or biography". The thinking and feeling self that gets to swim – ecstatically - with the whales in Leviathan clamoured to be let back into Hoare's work. As his investigation spanned forms – from childhood memories and cetacean biology to literary criticism and biography – as well as continents and oceans, he knew that the book had to move around as freely and fearlessly as the superlative beasts it evokes. "I had to write it in a very personal, passionate, eclectic and madly digressive way. There's no structure to this book. It happened as it happened."

In non-fiction, such free-style fluidity now seems to net the spirit of a restless age. In contrast, orthodox biography – for so long a bankable jewel in the crown of British publishing - has entered stormy waters. No "straight" biography of a major figure has won the Samuel Johnson Prize since TJ Binyon's life of Pushkin in 2003. Victors since then have included Jonathan Coe's experimental, semi-fictionalised life of the avant-garde novelist BS Johnson, James Shapiro's microscopic study of Shakespeare through one landmark year, 1599, and Kate Summerscale's Victorian true-crime mystery, The Suspicions of Mr Whicher. All, like Leviathan itself, shunned the chronological slog to exploit the new territories that non-fiction has occupied since big-name biography began to lose its bloom.

This year, some of the season's best-received lives – Jackie Wullschlager on Marc Chagall, Philip Norman on John Lennon, Gerald Martin on Gabriel García Márquez, Julian Evans on Norman Lewis – did not even reach the Samuel Johnson longlist. In contrast, three titles on a science-heavy shortlist – Richard Holmes's The Age of Wonder, Manjit Kumar's Quantum and Liaquat Ahamed's Lords of Finance – adhered to the now-favoured format of the group biography that weaves the linked stories of friends, rivals or colleagues.

Michael Holroyd's 1967-68 life of Lytton Strachey is often seen as the inaugural act of a golden age in biography. It set the tone for later works that allied the full and frank disclosure of private lives with storytelling skills to win commercial and critical success. Last year, though, even Holroyd joined the drift away from the single-focus blockbuster. Instead, he wrote about the conjoined careers on and offstage of actors Henry Irving and Ellen Terry – and their tangled families – in A Strange and Eventful History.

Perhaps the balance of achievement has not shifted. The balance of fashion definitely has. In terms of advances offered, sales expected and excitement generated, large-scale biography in the full-dress, life-and-times, warts-and-all mode seems to have passed its peak.

Yet the post-Holroyd heyday – with towering talents such as Graham Robb, Claire Tomalin, Jenny Uglow and Richard Holmes among its luminaries - can still cast a lingering glow. Look at the autumn schedules for 2009 and any transition in taste will not be obvious. Biography will still put on a brave face: John Carey on William Golding; Selina Hastings on Somerset Maugham; Michael Slater on Dickens; Ion Trewin on Alan Clark; William Shawcross's authorised life of the Queen Mother.

However, for DJ Taylor – the biographer of William Thackeray and George Orwell - each of these high-profile lives counts as a special case. "It's against the trend. There are very good reasons for all of these books". Golding, for instance, was "the one major figure of the postwar era" lacking a big "Life". The coming cluster "hasn't anything to do with a renaissance of the art".

Taylor believes that, for the 20th century at least, almost all the ranking figures now have a worthy monument. The novelist Anthony Powell, a prominent exception, has the award-harvesting biographer Hilary Spurling on his case. In a post-heroic period when it feels as if "everyone who matters has been done", authors search for attractive second-division figures. Attendant lords may illuminate their age. "You've got to find someone who's on the margins," Taylor says, "but can be sold to the public for quirkiness and charm."

Elsewhere, new-wave life-writing has pushed speculation beyond the limits of the documents until it almost slips into outright fiction. It has crept up on famous names from the side entrances to their lives rather than knocking on the front door. It has shifted its spotlight from the stars centre-stage to the extras chatting in the wings. It has given the biographer a leading role in the pursuit of a quarry. Above all, it has sought to tell a breathing story rather than dump a dead weight of data in the reader's lap. Whether tracing the model for Robinson Crusoe into the sea-mists of rumour, or deducing Shakespeare's daily round from faint echoes in the archives, Diana Souhami and Charles Nicholl – quoted below – exemplify this hunt for a truth about the elusive past that burrows beyond the "facts".

Not all biographers can detect a decline in the genre. Next week Hermione Lee, who has written lauded lives of Virginia Woolf and Edith Wharton, publishes a study of biography in Oxford's "Very Short Introductions" series. For Lee, the "popularity and ubiquity" of biography - in British culture anyway - may provoke a sporadic backlash against intrusion and exposure.

Yet "predictions of doom from the publishing industry about 'the death of biography'" should carry no more clout than prophecies of "the death of the novel". And if the art mutates – fewer heavyweight portraits, more mixed-media sketches or avant-garde miniatures – then, for Lee, that means a reversion to older and looser ways of writing lives, in Classical moralists such as Plutarch or post-Renaissance tale-tellers such as gossipy John Aubrey.

Four decades before what Lee calls James Boswell's "garrulous hold-all" of a Life in 1791 made Dr Johnson the archetypal subject of biography, Johnson himself had written a vivid unofficial account of Richard Savage – the penniless wild poet and close friend of his youth. Lee notes that Richard Holmes, in his book Dr Johnson & Mr Savage, argues that Johnson created a "hybrid, 'romantic', non-fictional form, combining drama, romance, folk ballad, journalism and morality".

That doesn't sound so alien to the experiments of a Souhami, a Nicholl or a Sinclair. In its late maturity, the literature of real lives may be shedding girth and grandeur and returning to its more informal roots. As Philip Hoare says about Sebald's mixed-up modes of non-fiction storytelling, "It's like the way we experience culture, art and life. We don't experience life in genres."

New lives for old: biographers who shun the 'womb-to-tomb' story

Quality: that's what matters – imagination, wit, style, narrative pace, conceptual underpinning and feeling drawn to the authorial voice as well as the subject matter. I hope the huge one-damn-thing-after-another biography, where the biggest crime is to omit a reference, has had its day. I'd like the same expectation of literary quality to be applied to biography as to fiction: the same desire for innovation, range and surprise. In 'Coconut Chaos', I crossed genres and moved between fiction, biography, autobiography, history and adventure. In the shops, the book ended up in Travel, where only the devoted could find it. I don't see why one biographical form should live at the expense of another. Nor is any individual definitively "done" - except perhaps Hitler and Diana. I've tried to do something different in each of my books. Not to develop or diverge as a writer – that's boring. Nor is it a good idea to try to conform to what we imagine will sell.

Diana Souhami (author of 'Selkirk's Island', 'Wild Girls' and 'Coconut Chaos').

The traditional "womb-to-tomb" biography has perhaps come to seem rather high-handed and old-fashioned. I think there have been too many 800-pagers, particularly about 19th- and 20th-century figures, where the wealth of surviving documentation makes them unwieldy with superfluous knowledge. These big books also presuppose a readership with a lot of time on its hands. The interest now is in something at once more oblique and more close-focused, something that offers a brief frisson of historical intimacy rather than a grand funeral oration. As Dr Johnson himself noted, the best biographies are "levelled with the general surface of life".

Charles Nicholl (author of 'The Reckoning', 'Leonardo da Vinci: the flights of the mind' and 'The Lodger: Shakespeare on Silver Street').