Can the magic last forever?

Who among us can honestly say that our response to a book is detached from fashionable opinion of it? We cannot help but be mindful that it is not quite the done thing to dislike, say, Mark Haddon's The Curious Incident of the Dog in the Night-time, or Philip Pullman's His Dark Materials trilogy. Publishers, who have to be aware of the market, are certainly fashion-conscious. That is why, eight years ago, so many of them turned down a struggling writer's children's novel about a young wizard at a boarding school. Children's novels weren't fashionable. Nor were wizards. Nor boarding schools.

Who among us can honestly say that our response to a book is detached from fashionable opinion of it? We cannot help but be mindful that it is not quite the done thing to dislike, say, Mark Haddon's The Curious Incident of the Dog in the Night-time, or Philip Pullman's His Dark Materials trilogy. Publishers, who have to be aware of the market, are certainly fashion-conscious. That is why, eight years ago, so many of them turned down a struggling writer's children's novel about a young wizard at a boarding school. Children's novels weren't fashionable. Nor were wizards. Nor boarding schools.

Worldwide sales of countless millions of Harry Potter novels by that one-time struggling writer, JK Rowling, have persuaded publishers to revise their opinions. Now they are throwing the kinds of advances once associated with high-concept thrillers or chick-lit at children's novels, particularly ones with elements of fantasy. Film companies are responding to the trend, and the media are suddenly taking an interest in authors who, a few years ago, would have been regarded by most journalists as entirely uninteresting.

The latest to get this attention is Michelle Paver, who was all over the press earlier this month when Orion published Wolf Brother, the first novel in a six-book sequence set in the Stone Age and called the Chronicles of Ancient Darkness. Paver has secured advances worth about £2m from publishers worldwide. She is not a first-time author: she has published five adult novels, none of which has received a similar level of attention.

Other authors are making this lucrative transition to children's books. Scholastic (JK Rowling's US publisher) is about to bring out The Akhenaten Adventure, the first volume in a trilogy called The Children of the Lamp, by one PB Kerr. As Philip Kerr, the author has an enviable track record in attracting big advances and film deals, and he has maintained that form in his new field: Scholastic UK and US paid about $1.8m for the trilogy, while DreamWorks bought the film rights.

As Kerr and his agent, Caradoc King of AP Watt, set out to do these deals, they had many encouraging examples to inspire them. DreamWorks also has the rights to Lion Boy by Zizou Corder, the nom de plume of Louisa Young - also the author previously of adult novels - and her daughter Isabel Adomakoh Young; Puffin paid a six-figure sum for the publishing rights. Eoin Colfer's Artemis Fowl novels have been sold to publishers in 40 countries, and to the film studio Miramax. Fox has film rights to Inheritance, a fantasy trilogy by Christopher Paolini, who had a US hit with the first volume, Eragon, when he was still a teenager.

Lian Hearn's Tales of Otori trilogy ( Across the Nightingale Floor, Grass for His Pillow, Brilliance of the Moon), set in a mythical version of ancient Japan, was a six-figure purchase for Macmillan; Universal bought film rights. Jonathan Stroud's Bartimaeus trilogy (volume two, The Golem's Eye, is out next month), following the adventures of a djinni and wizard, won a six-figure publishing deal in the UK from Random House, and a $3m book and film deal in the US from Miramax.

You may have noticed a predominant theme in these deals. "I get a lot of letters beginning, 'I'm writing a fantasy trilogy,' says Sophie Hicks, a literary agent whose clients include Eoin Colfer. The model here is Philip Pullman and the His Dark Materials trilogy ( Northern Lights, The Subtle Knife, and The Amber Spyglass). Pullman's books show how the best fantasy writing transcends terms such as "children's book": think of Alice's Adventures in Wonderland, The Lord of the Rings, The Chronicles of Narnia, Watership Down, Harry Potter.

Bloomsbury, JK Rowling's publisher, responded to adults' enthusiasm for Harry Potter by publishing editions of the novels packaged to appeal to an adult audience. Scholastic followed suit with His Dark Materials. Macmillan is bringing out adults' and children's editions of Lian Hearn's Otori novels. This year's runaway publishing hit, Mark Haddon's The Curious Incident of the Dog in the Night-time, is available in a children's edition from Red Fox and an adult edition from Vintage.

These are "crossover" novels. One suspects that, since the successes of Rowling and Pullman, publishers have been desperate to find books that will have a similarly broad appeal. But they deny that they have become obsessive on this subject. "Absolutely not," says Philippa Dickinson, who runs Random House Children's Books (a stable that includes Jacqueline Wilson, Terry Pratchett, Jonathan Stroud, Christopher Paolini, Mark Haddon and Philip Pullman), when I ask her whether her editors are constantly on the lookout for crossover fiction. "Everyone talks about 'crossover' fiction, but really you can only make it work if you have a very special book. Retailers don't like crossover novels very much otherwise, because they can't be sure whether to put them in the adults' or children's section - and they don't often want to put them in both."

Kate Wilson, now running Scholastic and previously in charge at Lian Hearn's publisher, Macmillan, points out that many of the authors now winning large advances and enjoying unprecedented sales are writing squarely for a juvenile readership. One of Macmillan's best-publicised recent purchases has been Georgia Byng's novels about Molly Moon, an orphan with hypnotic powers; the company also publishes Meg Cabot's hugely popular The Princess Diaries novels. You don't see many grown-ups reading them on the bus. After JK Rowling, the most successful children's writer in the country is Jacqueline Wilson, now the most popular author in the nation's libraries. There has been no talk of issuing adult editions of Wilson's Girls in Tears or The Sleepover Club.

However, children's fiction has won acceptance and even kudos among adult readers - save one or two disenchanted observers such as Howard Jacobson, who has written in these pages of his desire to knock copies of the Harry Potter novels out of adult readers' hands. Since the mid-1980s, the Whitbread Book of the Year award has gone to the winners of subsidiary Whitbread awards in one of five publishing categories: novel, first novel, poetry, biography, and children's novel. No children's novel came within a sniff of winning until 1999, when Harry Potter and the Prisoner of Azkaban just lost out to Seamus Heaney's Beowulf. A year later, Philip Pullman won the overall award, with The Amber Spyglass. He has won numerous author of the year awards since, as has Jacqueline Wilson. Mark Haddon won the Whitbread this year, and at the British Book Awards he took the awards both for children's book of the year and for literary novel of the year.

Every publishing phenomenon is the result of one outstanding success, which finds an enthusiasm that publishers hadn't previously tapped. A novel by Judith Krantz called Scruples created the sex-and-shopping boom of the 1980s. The novels of Joanna Trollope - though she shudders at the charge - stimulated an appetite for "Aga Sagas" about the provincial middle classes. Bridget Jones's Diary opened up the market for chick-lit.

There are sociological reasons for these booms too: sex and shopping reflected the materialism of the 1980s, while the Aga Saga signalled a disillusionment with those values. What, then, is the enthusiasm for children's books about?

It seems to reveal an appetite for strong, old-fashioned if you like, storytelling. Populists will argue that publishers and authors, particularly of so-called "literary fiction", have neglected this aspect of fiction; and it is certainly true that from the early 20th century there developed a belief, expressed in EM Forster's "Yes - oh dear, yes - the novel tells a story", that telling a story was not the novelist's highest aim. Whereas children's writers know, as Puffin managing director Francesca Dow says, "that unless they tell a good story, they haven't written much of a children's book". The BBC's Big Read poll of last year did not set out to find the best book, but the book people most enjoyed reading. The Richard and Judy book club is based on the same premise. Some book industry insiders wondered how daytime television chatshow hosts could bring any credibility to book recommendations; but it was because Richard and Judy claimed only for their selections that they were books people would enjoy that they proved so influential. One of the hits from their summer selection has been Jennifer Donnelly's A Gathering Light, first published by Bloomsbury as a children's book.

You can see why adult authors such as Philip Kerr might want to try their hands at this genre. Of course, there have always been authors, such as Penelope Lively and Nina Bawden, who have worked in both fields. But there have been a noticeably large number making their debuts in children's books recently: in addition to Kerr, Louisa Young and Michelle Paver, there have been the US crime writers Elmore Leonard and Carl Hiaasen; the ex-SAS thriller writers Andy McNab and Chris Ryan; Julie Burchill, whose teenage novel Sugar Rush is out this month; Jeanette Winterson, who wrote a picture book for Bloomsbury called The King of Capri (and who will be followed in the same series by Margaret Atwood). Personalities want to get in on the act too: Madonna with her picture books for Puffin, Ricky Gervais with his forthcoming Flanimals (Faber), and, also for Faber, Sir Paul McCartney, who is working on a collaboration with author Philip Ardagh and animator Geoff Dunbar.

Not all these books are going to fulfil their authors' and publishers' expectations. It is the nature of every publishing boom that it results in overproduction, which causes disillusionment in the market and then decline. "Publishers have a terrible record for jumping on bandwagons," admits Richard Scrivener, publisher at Scholastic. "There's always the potential to become the publishing equivalent of Leeds United, lashing out loads of money in the hope of creating a dream team." (Amid financial turmoil, Leeds were relegated from the Premiership last season.)

However, Scrivener and the other publishers I spoke to all said that they were mindful of the dangers, and approached every new alleged hot property with caution. "We want to buy 'superlead' titles," says Francesca Dow, "but that doesn't mean we feel we have to spend money just for that reason." At Random House, Philippa Dickinson says, "We recently lost out in three auctions, one after the other. I regret that we didn't get the books, but I don't [regret] passing at the price that was being asked."

No doubt there are observers of these events who think, "Why don't I write a children's book? It can't be that difficult." Perhaps there are some ephemeral children's books that do not require genius to create. But the deals described here are all for books that, rightly or wrongly, publishers believe will last. Let a top agent, Sophie Hicks, offer a corrective. "You're trying to write a book that is timeless and ageless. That must be the hardest thing of all."