Tall tales from history: Are historians best placed to write historical fiction?

Historians turning their hands to fiction are all the rage. Since Alison Weir led the way in 2006, an ever-growing number of established non-fiction writers – Giles Milton, Simon Sebag Montefiore, Harry Sidebottom, Patrick Bishop, Ian Mortimer and myself included – have written historical novels.

So successful has been the experiment, with many of the books making the bestseller lists, that earlier this year Penguin bought two novels from Kate Williams, one of our finest young historians, for the staggering sum of £1m.

If not nearly as lucrative, my own contract with Hodder in 2007 was still quite a leap of faith. On the strength of a four-page proposal and no sample chapters (let alone a finished manuscript), I agreed to write a trilogy of books about an Anglo-African soldier called George Hart who fights in the wars of the late Victorian period. My only previous stab at fiction was in the late 1990s when my then agent, having read the first two chapters, told me not to give up the day job.

So what was different now and why was my editor at Hodder prepared to take the risk? A key factor was the runaway success of Alison Weir's debut historical novel, Innocent Traitor, about the brief 16th Century reign of Lady Jane Grey.

Suddenly all the major publishers in London were looking to replicate Weir's sales by finding their own popular historian who, as one editor put it, "could write" and was willing to give fiction a go. I was in the right place at the right time.

But if getting a contract was relatively straightforward, writing fiction was far harder than I could have imagined, and there were moments during the long and torturous edit process when it seemed that Zulu Hart, the first of the trilogy, would never be fit for public consumption. Not that all historians struggle to make the transition. Sebag-Montefiore found writing Sashenka, his debut novel about a headstrong young woman in pre- and post-revolutionary Russia, "a joyous experience, no footnotes, and the freedom of emotions and imagination". It helped that he realised early on that "historical fiction is simply fiction set in the past, and should be judged as such".

I, on the other hand, wanted to give history an undue prominence in my fiction until I was advised by my editor to "let go of the past". Not an easy task for someone who'd spent the last fifteen years writing history, you might think? And you'd be right. Historical facts are the vital framework around which non-fiction writers construct their narratives; they are, quite simply, indispensable. Yet now I was being told that if I wanted to write decent historical fiction I had to avoid being constrained by events as they actually happened.

Eventually I saw the sense of this. I wasn't being asked to sacrifice historical accuracy per se. Just to accept that a historical novel, or any novel for that matter, stands or falls on plot and characterisation; period detail is important, but only in so far as it gives a sense of authenticity. It must remain in the background and never be allowed to dominate the story.

Historical fiction, as a result, often takes liberties with the 'truth': it compresses time, invents conversations and motives that real people never had, and generally tampers with the historical record for the purposes of plot.

The trick is to minimize those liberties, and to make sure that when you're writing about historical figures you "stay true to the spirit of that person". This was the advice given to me by the late and great George Macdonald Fraser, author of the Flashman novels, whom I interviewed shortly before his death in 2008.

He claimed to have broken this rule only twice – with Otto von Bismarck, the architect of German unification, and Nicholas Ignatieff, a Russian adventurer and spy – and felt justified in doing so because the former "was a swine" and the latter "a pretty hard man" if not an arch-villain. A made-up protagonist, of course, gives the author the greatest licence, but even he or she must not stretch credulity too far.

Thus I created George Hart, the son of an English VIP and brought up a gentleman, but of mixed Irish-African descent on his mother's side and therefore a man with a foot in both camps, capable of seeing the British Empire from the perspective of both ruler and ruled. His unnamed father, on the other hand, is a real historical figure who had a penchant for actresses, secretly (and illegally) marrying one and having three illegitimate sons by her.

Alison Weir prefers her central characters to be people who actually existed because fiction enables her to fill in the gaps in the historical record. "While I was researching my biography of Eleanor of Aquitaine," she says, "it occurred to me that I wanted to write a novel about Eleanor, in which I could develop ideas and themes that had no place in a history book, but which – based on sound research and educated guesses – could help to illuminate her life and explain her motives and actions." By getting inside her subject's head she is, she believes, able to gain "insights that would not be permissible to a historian, and yet can have a legitimate value of their own".

Yet Weir offers an important word of caution: "It is liberating to be able to use one's imagination, but you can't simply indulge in flights of fancy. That sells short both those who know nothing of the subject, and those who know a great deal. I know – because my readers regularly, and forcefully, tell me so – that people care that the historical fiction they read is close to the truth."

This, perhaps, is the main reason why books by historians-turned-novelists are selling so well. Many readers of historical fiction like to be entertained and educated, and the only authors they can entirely trust to do both are historians. Sebag-Montefiore's central character Sashenka is, he says, "totally invented but the history, the details, the background, the real historical characters that all appear as minor characters are all accurate historically".

It helps, therefore, to know the period. Yet much of the research required for a historical novel is, I discovered, very different from that done for a history book. For Zulu Hart I already knew a lot of solid factual information because I'd previously written a history of the Zulu War. What I didn't have was the sights and smells. What, for example, did the inside of the War Office look like in 1879? Or what was the experience of steam travel from England to South Africa at that time?

I had to find out, and for this sort of detail the internet was invaluable; not only for pinpointing sources like diaries and books, but also for providing online maps, plans and even first-hand accounts.

Kate Williams' debut novel for Penguin, due out next year, is about a young woman obsessed by a serial killer in 1840. Though the research she had done for her biography of the young Queen Victoria was "helpful", she "still needed to do more".

In particular she needed to find out about "death, domestic life and the streets of the period". For his first fiction, Warrior of Rome, Oxford don, Harry Sidebottom, found himself researching things like "food, clothes and aspects of social history" that he "had never had to think about before", and that it took just as much time as his first history book.

We've all faced the charge that our novels are history lite, and to some extent that's true. Yet for some, historical fiction is a way into reading history proper. Weir was 14 when she devoured Katherine by Anna Seton in two days and then "had to rush off to the history books in my school library to find out what really happened".

I was about the same age when I read my first Flashman novel and it's no coincidence that most of my history books are about the same Victorian wars. Sebag-Montefiore has many letters from fans of his fiction who have since read his histories.

Occasionally historical fiction can have a powerful modern resonance. My new novel, Hart of Empire, is set during the second Anglo-Afghan War of 1879 when Britain's motives for invading were remarkably similar to NATO's in 2001: to replace an allegedly anti-Western regime with something more amenable. But it didn't work out then and it doesn't look like it's going to now. My own dismay with the current war is mirrored by my protagonist's fury with our last doomed attempt to impose a ruler on the Afghans.

It was, thank Heavens, a far easier book to write than its predecessor; and it helped that I wrote it while the tricks of the fiction trade were still fresh in my mind. But, having made the transition, how difficult would it be to pen history again? I had no worries, and quickly rattled off the first three chapters of my much-delayed history of the British soldier.

When I showed them to my non-fiction editor, however, she was appalled: too much detail; not enough narrative drive. It was almost as if, free from the tyranny of plot, I was overdosing on incidental detail, a complete no-no in fiction. Suitably chastened, I cut furiously and got the story moving. The feedback this time was generally positive.

Williams is convinced that being a historian has helped her fiction because she wasn't tempted to put in too much of her research. But does it work the other way round? Will her writing of history benefit or suffer from her foray into fiction? It's a question we all want an answer to and, for now, the jury is still out.

Saul David's 'Hart of Empire' (&12.99) is published by Hodder & Stoughton