Why this is a golden age for children's literature: 'Children's books are one of the most important forms of writing we have'

The first book I read to myself was Dr Seuss's The Cat in the Hat – something I still recommend to learners – and the Cat's wild anarchic humour has always struck me as the perfect analogy for what happens to a child who discovers books. We think it's about being good – remember how the kids in the story are made to "sit-sit-sit-sit/ And we did not like it/ Not one little bit"? Reading involves sitting, and getting a child to stay still is one of its most unattractive aspects. But once you master this, everything changes. People who love reading are often called bookworms – but that's the wrong way around. It's not you that worms into a book, it's books that worm into you.

And worm into us they have. Children's books, new and old, are actually what is keeping publishers in business. They are what parents and grandparents, aunts and uncles, teachers and godparents actually buy, and go on buying. Many children's authors – not just JK Rowling, but David Walliams, Julia Donaldson, Francesca Simon and Anthony Horowitz – have sales of the kind that would make any author of fiction or non-fiction weep with joy. You are doing well as a literary novelist if you sell 10,000 copies. A chick-lit author can hope for 100,000. A top historian such as Antony Beevor might shift that many too. However, just one book by Walliams, Awful Auntie, earned his publisher, HarperCollins, more than £6m last year. John Green's The Fault in Our Stars earned Penguin UK more than £4m. According to The Bookseller, in 2014 the Children's category earned £336.5m, a year-on-year rise of 9.1 per cent, in a books market that declined 1.3 per cent.

That is not to say that all the books earning these sums are any good. And most children's writers barely earn the minimum wage if they are lucky. The kind of books we think of when talking about children's literature are hardly ever those given large advances, and often they show little sign of being the best-sellers they become. Yet children's literature is hugely important. Indeed it is so important that you could hardly overstate how crucial it is to the kind of world we want to live in.

Children's books are, paradoxically, one of the most important forms of writing we have, and the most overlooked. It is children's authors who are what Shelley called "the unacknowledged legislators of the world". From them, as much as from parents, a child receives an idea of how the world could or should be. They are the first real visual and literary culture that an unformed person receives, and this is one reason why we tend to remember children's books as our favourites. But they also give a child a lever with which to prise open the world. They tell us that life is much bigger and more complex than we might have imagined, and that it contains people who are both like and unlike ourselves. This may seem daunting, but a great children's book portrays an environment in which the young are not powerless. Such books confront our deepest fears of being lost, hungry or in mortal danger, and they reinforce a child's inner ability to cope with this fearfulness and to discover where true strength lies. Almost any great story which people enjoy conforms to this model, whether it is about David and Goliath, Alice in Wonderland or Varjak Paw. Yet for most of the time, children's books are treated as infantile, escapist or easy to write. Believe me, they are not.

For the past 15 years I've reviewed children's books. I was lucky in that I happened to start just when JK Rowling's first Harry Potter book was published, and I was one of the very few people to review it. Her success, and that of Philip Pullman, excited more and more interest in children's books which, up till then, had been seen as the Cinderella of publishing.

No longer. I'd argue that we are now living in a Golden Age of children's literature. But it's not the first time that children's books as an art-form have blossomed. The First Golden Age, which took off in the 1850s, thanks to the increasing number and status of children, was particularly dominated by five authors. JM Barrie, Robert Louis Stevenson, E Nesbit, Frances Hodgson Burnett and Anna Sewell all wrote in distinctly different genres that have been passed on, like DNA, to present-day authors.

Barrie's Peter Pan, which few children find readable today, was the first novel in which ordinary children enter a magic world and have an adventure there – something that readers of Alice in Wonderland, The Chronicles of Narnia, Harry Potter and His Dark Materials will recognise. These are what many now think of as quintessential children's books. They allow a child to escape from what is real and, as importantly, return to reality in a better frame of mind.

Robert Louis Stevenson's Treasure Island has also fallen out of favour with present-day readers, but any number of adventure stories, from Rudyard Kipling's The Jungle Book to The Action Hero's Handbook derive from it. Stevenson's young hero, Jim Hawkins, foreshadows the plucky resourcefulness of Anthony Horowitz's reluctant teenage spy, and Eoin Colfer's criminal mastermind Artemis Fowl.

E Nesbit has perhaps given us the strongest DNA of all. Her stories, whether magical or not, are firmly rooted in the real world and have been hugely influential on books of the Second Golden Age – Diana Wynne Jones's Chrestomanci books, Lucy M Boston's Green Knowe series, Joan Aiken's Wolves of Willoughby Chase sequence and Eva Ibbotson's hilarious witches and ghosts. Her quarrelsome, highly believable brothers and sisters were echoed by Enid Blyton, Arthur Ransome, Noel Streatfeild, Roald Dahl and, more recently, a constellation of contemporary authors like Hilary McKay, Anne Fine, Cathy Cassidy, Frank Cottrell Boyce, and Francesca Simon. Above all, she is acknowledged as the single greatest influence on JK Rowling, presumably because her conception of mixing the magical with the mundane is sharply satirical. The most recent winner of the Costa Prize for Children's fiction, Kate Saunders, updated one of Nesbit's most famous books with Five Children on the Western Front – having cleverly worked out that, in just a few years, her famous Edwardian family would have been embroiled in the First World War.

Frances Hodgson Burnett is also one of the most sympathetic to modern children, despite being a raging snob. Her stories of impoverishment, bullying and exile have a consistent following and a marked influence on authors such as Jacqueline Wilson. She is the patron saint of the sensitive, romantic child, and her influence is to be seen in some of the best writers for teenagers, such as Sophia Bennett, Anthony McGowan, Tim Bowler and Frank Cottrell Boyce. In a society where, once again, the gap between rich and poor children is widening, it is crucial for young people to imagine what it is like not to be themselves.

Finally, there is Anna Sewell, whose equine autobiography Black Beauty left a lasting legacy to those who write animal stories. From AA Milne's rodents in tweed to Michelle Paver's Wolf Brother, this is a kind of story which delights in animals as creatures very like children. Even if I personally never wish to be sent another book about a little lost kitten, Black Beauty laid the foundation for the Royal Society for the Prevention of Cruelty to Animals.

What makes this First Golden Age so particularly distinct is, in fact, its didactic and moral purpose. The little chimney sweep in The Water-Babies, the death of Black Beauty's great love Ginger and even Peter Rabbit's father's fate in a pie leave powerful impressions on the minds of those who encounter them. Yet this First Golden Age offers reassurance that everything will turn out all right in the end.

If you think of some of the books which you may have loved in your own childhood – novels such as Little Women, say, or Charlie and the Chocolate Factory – and try to re-read them to your children, you quickly come up against concepts such as duty, self-sacrifice and not complaining which tend to be quite alien. Above all, the children in this First Golden Age do, rather than think. They are expected to be resourceful and resilient, proud of their class and country. When Nesbit's Five Children time travel in The Story of the Amulet, they have no qualms whatsoever about telling Caesar how jolly marvellous Britain is – thus inciting him to invade us.

The Second Golden Age, which fed the imagination of the baby boomers, ran roughly from the 1950s to the 1970s, and is quite different in that it reverberates with a new, global moral consciousness. It portrayed the battle between good and evil – most famously in Tolkien – as an absolute struggle, and it did so in the wake of the Second World War. To grow into adulthood aware that someone, somewhere, can destroy the world with a nuclear bomb, does tend to have a profound effect on the imagination. To also discover that atrocities like genocide were carried out because a populace did not question authority is also to understand why subversive authors such as Roald Dahl, Judith Kerr and Maurice Sendak became so popular.

It's not surprising to find that the genre of choice for the Second Age was fantasy – beautiful and magical but fraught with danger. However, Lewis also provides spiritual balm and instruction in the figure of Aslan, the talking lion.

There are many examples of this guiding, protective, mysterious figure in the Second Golden Age. Will in Susan Cooper's The Dark is Rising has a wise, magical old teacher in Merriman Lyon – or Merlin, as he turns out to be. Alan Garner's Colin and Susan have the wizard Cadellin, and Frodo Baggins's Companions have Gandalf. All of these draw on myth, but above all they draw on the European concept of God, and it's no surprise to find the same figure popping up in Harry Potter's Professor Dumbledore. And no wonder we needed him. In the 1960s, it wasn't enough for a child to find her father or restore the family fortune. This time, we were told, we needed to save the world. But by the time you get to Philip Pullman's His Dark Materials, it's not just the world which needs saving, but the multiverse – and God, if he exists, has dementia.

Tolkien and CS Lewis made such a profound impression on my own generation that many of us have talked and written about how desperately we wanted to find Narnia and Middle Earth. Today's children and young adults tend to have this feeling about the Harry Potter books. I have lost count of the number of people now in their twenties who confess their disappointment at turning 11 and not getting a letter admitting them to Hogwarts. Yet Rowling's enchanted world, even when it darkens, is an anomaly. For in our Third Golden Age, we ask children to confront not only the traditional enemies such as school bullies or Dark Lords, but Death itself. Far from reassuring readers that some authority is benign, it is portrayed as unequivocally malignant, corrupted by ecological disaster, politics, greed and war. The child's heroism lies not in following instructions but in defiance, truancy and rebellion. With the advent of YA fiction for secondary school pupils, not only death but sex has become part of the picture – in fact, with novels such as Twilight, it's central to it.

This Third Golden Age is ripe with an extraordinary amount of talent. Where there might have been, say, five or six leading authors in my own childhood, today there are probably over a hundred. It is an enormous field, and is publishing vastly more books of all kinds, including genres such as horror which would once never have been considered appropriate. What might once have been called comics are now graphic novels, treated seriously. Ipads have enabled books to become animated, with sound, music, special pop-up features that young children especially love to explore. The creative potential for books in a visual sense is growing all the time. Yet you still need kids to learn how to read – and to sit, sit, sit, sit – before they can love doing it.

All kinds of attitudes have changed, mostly for the better. Bullies were hated in Tom Brown's Schooldays but now, as in Louis Sachar's Holes, they are both villains and victims. Racism has become completely unacceptable, and there are authors from Mary Hoffman to Malorie Blackman who tell wonderful stories of children for whom skin colour is no obstacle. The prim Alice has morphed into Cressida Cowell's kick-ass Viking girl Camikaze in How to Train Your Dragon.

Dystopian novels become a genre of their own, in which adults, politicians and leaders are consistently portrayed as deceitful, greedy, vainglorious and wicked. Occasionally, as in Sally Gardner's Maggot Moon, the dystopia portrays an alternative history – a Fascist 1950s Britain along the lines of 1984. More usually, they are set in the future, against the cataclysms produced by current trends.

Dystopia isn't new: in my own childhood there were superb writers such as John Christopher, whose Prince in Waiting trilogy should be much better-known. But these futures were the product of natural catastrophe or alien invasion. Now, the darkness and violence of contemporary dystopias is highly politicised. The most famous is Suzanne Collins's The Hunger Games, in which teenagers are required to fight to death for the ultimate TV reality show. Or, you might say, the ultimate high-school show-down. Plenty of other terrific novels such as Scott Westerfeld's Uglies, Malorie Blackman's Noughts and Crosses, Meg Rosoff's How I Live Now and Moira Young's Blood Red Road depict our future as ravaged by science, racism, war, genetic mutation or most credibly, exams.

The emphasis in the First Golden Age was very much on being healthy in mind and body – if a child became sick, he or she usually got well as part of their story. Today's reader has no such encouragement. There is an alarming trend in what has been termed "sick-lit" which seems to wallow in the idea of a child self-harming, being ill, dying, or even committing suicide. I trace this back to a novel which was not written for children at all, Alice Sebold's The Lovely Bones, but it has spread into best-selling books like Ways to Live Forever, Before I Say Goodbye and many more. John Green's The Fault in Our Stars which I mentioned at the start, is about two teenagers with terminal cancer who fall in love. It is a huge international best-seller. I find the portrayal of such tragedy to the vulnerable teenager mawkish, reprehensible and even dangerous.

Does this make the Third Golden Age less remarkable? It may make a good deal of it less long-lasting: we tend to treasure what comforts us. Deep down, we all still want the story to end happily. This is an astounding time in which to be a parent, or a child. We have never had so much access to so much information, and we have never had so many ways of enjoying it through story. Above all, this is an astounding time in which to be a reader.

This is an edited version of a speech made in aid of Belsize Park Library