Is Tom Stoppard's Arcadia the greatest play of our age?

What will endure from the plays of the late 20th century? Already, the theatre that caused the greatest fuss at the time – the in-yer-face shockers by Mark Ravenhill, Martin McDonagh and friends – look flashy and shallow and strangely dated; only Sarah Kane's psychological slashing seems to have survived from this flashing pack of playwrights. Yet one genre seems to have solidified as the decades pass into bona fide masterpieces, and will perhaps define that period: the play of ideas.

It looks now like the theatre from the 1980s and 1990s that tried to dramatise the great intellectual mudslides and forest fires of its time has thrived better than any other – from Michael Frayn's Copenhagen to Caryl Churchill's Top Girls to Terry Johnson's Insignificance. Using the old theatrical forms of the comedy or the thriller, they ask the most profound questions – what is human life for, and how it should it be lived? Standing above them all, making the case for the entire genre, is perhaps the greatest play of its time: Arcadia by Tom Stoppard.

As it is about to have its first major revival in the West End since its premiere in 1993 – starring Stoppard's own son, Ed – the vindication of Arcadia seems close at hand. Yet Stoppard compresses so many ideas and guffaws and griefs into less than three hours that any attempt at a summary of the play will sound paradoxical. It is an English country-house farce about the death of the universe. It is a laugh-filled tragedy about what happens if you take the intoxicants of poetry and science seriously. It is a play where Stoppard turns himself into a clown whose juggling balls are Romanticism, Classicism, and the meaning of life.

The play is set in Sidley Park, an English stately home, in two different centuries. It opens in 1809, in the style of an Oscar Wilde drawing-room farce. A handsome young science graduate, Septimus Hodge, is living there, tutoring the precocious 13-year old girl of the house, Thomasina Coverley. Reading through her Latin homework, she wants him to explain what "carnal embrace" means. When he tells her, she is appalled. "Now whenever I do it, I shall think of you!" she rasps. "Is it like love?" He replies: "Oh no my lady, it is much nicer than that."

And he has been demonstrating this conviction: Septimus has just been spotting having "a perpendicular poke" in the gazebo with Mrs Chater, the wife of a visiting poet. The lesson is interrupted by a note from Mr Chater, demanding he receive "satisfaction" for his wounded honour in the form of a duel. Septimus sighs: "Oh, Mrs Chater demanded satisfaction and now you demand satisfaction. I cannot spend my day and night satisfying the demands of the Chater family." When Mr Chater arrives in a fury, Septimus insists he won't engage in a pistol-fight to defend the honour of "a woman whose reputation could not be adequately defended with a platoon of musketry deployed by rota."

The play then shifts suddenly to the 1990s, and a more realist style. In the same house on the same set, a historian called Hannah Jarvis – a role written for Felicity Kendal – is delving into the history of Sidley Park with the permission of the Croom family. She is a cool woman who has stripped herself of emotion and stocked her heart with icy frigid air, as she buries herself in piecing together stories from the past.

Her work suddenly is interrupted by a braying, patronising English don called Bernard Nightingale who – we soon realise – has discovered the note that Chater wrote to Septimus in an old book. Only he is convinced it means something more – something much more. He believes the note was written by Lord Byron, the great Romantic poet, who happened to be visiting that weekend – and that he fought the duel and killed Chater. This would explain his until-now mysterious fleeing to France in 1810. It will be "the literary discovery of the century", he neighs, turning him into a "media don – book early to avoid disappointment."

And so the structure of the play is set. We watch the action unfold from 1809 to 1812, while the characters in the late 20th century try to figure out what happened using the surviving scraps of their lives. The stories alternate until, in the final scene, they appear on stage together, stumbling past each other, unseen, unseeable, yet locked in a waltz.

Hannah – and Stoppard – are obsessed with the way the garden at Sidley Park was redesigned while Thomasina was swotting and Septimus was shagging, because it represents the intellectual shift that was sweeping Europe at the time. Until 1809 the garden was in the classical style, modelled on Virgil and ancient Greece – ordered and clean and geometrical. But then the garden was demolished and remade to conform to the vision of the new romantic craze sweeping Europe – wild and irregular and disordered. Lady Croom exclaims: "Where there is the familiar pastoral refinement of an Englishman's garden, [there will soon be] an eruption of gloomy forest and towering crag, of ruins where there was never a house, of water dashing against rocks where there was never a spring. My hyacinth dell is to become a haunt for hobgoblins." Hannah calls it "the Gothic novel expressed in landscape. Everything but vampires."

The idea of what Arcadia – paradise – looks like flipped in one generation, from order to disorder, from classical calm to romantic chaos. Hannah believes she has uncovered – in the crags of the garden's history – a perfect symbol of this degeneration. When they were carefully constructing their fake wilderness, the gardeners built a fake hermitage – and Lady Croom demanded that the gardeners provide a hermit to live in it. "If I am promised a fountain I expect it to come with water," she says. The bemused gardeners suggest advertising for a hermit in the newspaper, causing her to retort: "But surely a hermit who takes a newspaper is not a hermit in whom one can have complete confidence." But a hermit was found – and he is the subject of Hannah's new book.

He spent decades scribbling away in his fake hermit's hut, unremarked on by the Croom family. When he died at the age of 47, he was discovered to have been writing tens of thousands of pages of incomprehensible equations and Cabbalistic proofs that the world was coming to an end. "He's my peg for the breakdown of the Romantic imagination... the whole Romantic sham!" Hannah explains. "It's what happened to the Enlightenment, isn't it? A century of intellectual rigour turned in on itself. A mind in chaos suspected of genius. In a setting of cheap thrills and fake beauty... The decline from thinking to feeling, you see."

And so the tension that runs through the play is set up, in the very set itself. It's the old division that obsessed the 18th century. The classical order – which mutated into the Enlightenment – believed the world was ordered and was governed by rules that could be slowly uncovered. The Romantics believed this was a suffocating cage in which humanity was being imprisoned, and sought to overthrow all rules in the name of individual creativity. You make up your own rules as you go along: every man is an artist. There is no order other than the one you invent.

Against the backdrop of this transformed garden and the transformed ideas it embodies, a strange story begins to unfold. The young Thomasina is, it soon becomes clear, a genius. Even as she girlishly prances around failing to spot the series of sexual farces unfolding in her family, she can grasp the implications of the newest scientific discoveries way ahead of any of the adults around her. Septimus teaches her about Newton's laws of physics. They are clean, clear, promising an underlying, predictable order to the universe. Thomasina frets about what becomes of free will in a world where we are all merely atoms moving in line with laws of motion – then, suddenly, she spots a series of dark flaws in Newton.

She explains that in Newton's universe, equations can run in either direction – forward or back. But there is one equation that runs only one way: heat turns to cold. The same thing is happening everywhere, all the time: it's called the Second Law of Thermodynamics. The implications – only just being grasped by the generations after Newton – were plain, and bleak. "It'll take a while, but we're all going to end up at room temperature," says one character. Septimus – sobered by Thomasina's explanation – adds softly: "So the Improved Newtonian Universe must cease and grow cold."

These are characters who take the implications of their ideas seriously. Septimus and Thomasina are stricken by the realisation that instead of setting up a perfectly ticking and well-oiled machine, Newtonian physics exposed us as living in an irrevocably doomed world.

In the present day, Bernard the aspirant media don scorns the implications of science. In his struggle to prove Byron was a killer, he thinks gut instinct and aesthetics trump boring old scientific facts. He announces to the scientist who lives at Sidley Park, Valentine: "We were quite happy with Aristotle's cosmos. Personally, I preferred it. Fifty-five crystal spheres geared to God's crankshaft is my idea of a satisfying universe. I can't think of anything more trivial than the speed of light. Quarks, quasars, big bangs, black holes – who gives a shit?... If knowledge isn't self-knowledge it isn't doing much mate. Is the universe expanding? Is it contracting? Is it standing on one leg and singing 'When Father Painted the Parlour'? Leave me out, I can expand my universe without you."

Bernard's romantic passion – laced with a little charlatanry – is in opposition to Hannah's classical reserve. She is afraid of emotion and passion; Bernard is afraid of sobriety and the nagging sensation that the facts might not justify his flights of fancy. He "just knows" Byron murdered Chater.

Meanwhile, nearly two centuries before, Septimus is clearly falling in love with Thomasina. He is thrilled by her discoveries, not only of the "heat death" implicit in Newton, but of another, deeper flaw in Newtonian physics. Why, Thomasina asks, can Newton's laws and equations only predictably describe the physics of linear, manufactured objects like squares and cones and pyramids? "Armed thus, God could only create a cabinet... [But] if there is an equation for a curve like a bell, there must be an equation for one like a bluebell, and if a bluebell, why not a rose?" There surely must be a mathematical pattern underlying the things of real life too. She determines to draw these equations.

But the audience slowly realise this is impossible – not because she is wrong, but because she is so far ahead of her time. When Hannah finds her old notebooks, she gets Valentine to explain them to her. He is a mathematician living in the house, pining for Hannah, and trying as part of his PhD to unlock the numerical patterns underlying the changing population of grouse at Sidley Park, as recorded in the old game books. He explains: "When Thomasina was doing maths it had been the same maths for a couple of thousand years. Classical. And then for a century after Thomasina. Then maths left the real world behind, just like modern art, really. Nature was classical, maths was suddenly Picassos. But now nature is having the last laugh. The freaky stuff is turning out to be the maths of the real world."

It turns out that so much of the world around us – rainfall averages or measles epidemics, say – follow bizarre equations. Valentine explains: "People were talking about the end of physics. Relativity and quantum looked as if they were going to clean out the whole pattern between them. A theory of everything. But they only explained the very big and the very small. The universe, the elementary particles. The ordinary-sized stuff which is our lives, the things people write poetry about – clouds – daffodils – waterfalls – and what happens in a cup of coffee when the cream goes in – these things are full of mystery... because the problem turns out to be different."

How do we understand them? With a new kind of maths, known as chaos theory. Traditionally, scientists expected dynamic systems to be stable, predictable. But tiny variations in inputs can cause huge changes. Simple equations can produce complex patterns. The way to decode them is a process known as an iterated algorithm. This is a piece of algebra where you take the solution to an equation, and plug it back into the start of the same equation, and keep repeating the process, again and again. Out of a simple equation, you get complex patterns. But the maths is so complex and so time-consuming, it can only be done with computers. It was inaccessible to Thomasina.

So Thomasina, the audience realises, glimpsed a truth, centuries earlier than anyone else. "She didn't have the maths, not remotely. She saw what things meant, way ahead, like seeing a picture," Valentine says. And she knew that if she was right, she could help us escape from the trap laid by Newton – of a predictable, determined universe shorn of free will, and doomed to freeze. With the day-to-day unpredictability of chaos theory, "determinism leaves the road at every turn," she says. "The unpredictable and the predetermined unfold together to make everything the way it is. It's how nature creates itself, on every scale, the snowflake and the snowstorm."

And maybe it offered a form of hope beyond the freeze. When it is explained to her, Hannah asks Valentine: "Do you mean the world is saved after all?" He replies: "No, it's still doomed. But if this is how it started, perhaps it's how the next one will come."

But what became of Thomasina's insight? Hannah reveals its fate casually, in the sixth scene. (Skip this paragraph if you want to avoid a plot spoiler.) Thomasina died in a fire on the eve of her 17th birthday. Her insights came to nothing. Then we see her alive again, skipping onto the stage, trying to persuade Septimus to kiss her. It is, we realise, the night of her death. And suddenly, it hits the audience. The hermit in the garden is Septimus, trying to prove Thomasina's equations, alone and half-mad in the romantics' garden after her death. His mind and pencil didn't have the capacity to do what a computer can manage in a few minutes – but he tried, scribbling endlessly, for decades, trying to prove there is hope after all, and it can only be discovered "through good English algebra."

The stale cliché about Stoppard – and about this genre – is that he is a brilliant manipulator of ideas, but with no heart. Yet here – at the core of his best play – is the greatest love story on the British stage for decades. Yes, the characters bond over ideas – but some of the most interesting people in life do just that.

That would be enough to make Arcadia a masterpiece – but it is even more than that. The play stirs the most basic and profound questions humans can ask. How should we live with the knowledge that extinction is certain – not just of ourselves, but of our species?

The play suggests that we are forever re-enacting the patterns of the past with mild variations – or, in other words, that the human heart beats to an iterated algorithm. Thomasina's distant relatives echo her lines through time, with a word misplaced. When Thomasina weeps for the destruction of the library of Alexandria and all the lost plays of the Athenians, Septimus says: "You should no more grieve for [them] than for a buckle from your first shoe, or for your lesson book which shall be lost when you are old. We shed as we pick up, like travellers who must carry everything in our arms, and what we let fall will be picked up by those behind. The procession is very long and life is very short. But there is nothing outside the march so nothing can be lost.

The play is both a vindication of this speech, and a repudiation of it. Thomasina's notebooks are picked up again by Hannah – but what about when the march ends? In our time, science suggests a threat to our ability to survive far more imminent that the frozen universe implied in the Second Law of Thermodynamics: our "heat death" could come under a blanket of our own warming gases. Arcadia asks, in part, how do you live with the certain knowledge of extinction – not just you, but your species'?

In the most important speech in the play, Hannah suggests the answer lies in the process of trying to understand, while you can. You find meaning by questing on, even in the face of failure and extinction. She tells Valentine: "It's all trivial – your grouse, my hermit, Bernard's Byron. Comparing what we're looking for misses the point. It's wanting to know that makes us matter. That's why you can't believe in the afterlife, Valentine. Believe in the after, by all means, but not the life. Believe in God, the soul, the spirit, believe in angels if you like, but not in the great celestial get-together for an exchange of views. If the answers are in the back of the book I can wait, but what a drag. Better to struggle on knowing that failure is final."

And so in the end, Stoppard suggests the division that obsessed the 18th century – between romantics and classicists – exists in all of us. Hannah prides herself on her classical reserve, but by the final scene, it is faltering. She finally agrees to dance with Gus, the mysterious, mute young son of the house who seems to have an inexplicable knowledge of the distant past. He is a symbol of all the things that lie beyond her rational explanations – and she embraces him. Septimus is a stern scientist who venerates geometry, but he ends as the most romantic figure of all – hermit in a Gothic garden trying vainly to vindicate the theories of his lost love.

Stoppard seems to believe that without both halves of the 18th century self – an impulse to understand the rules that govern the world, and an impulse to overthrow them and create ourselves anew – we are not fully human.

In the last scene, the characters from the 18th century and the 20th century are on stage together, occupying the same space. They cannot see each other, yet they seem to be speaking to each other all the same, as the implications of Thomasina's discoveries tumble out. As the music rises, Thomasina and Septimus waltz together for the last time – a dance that is another iterated algorithm, always the same, always slightly different – and Hannah takes Gus's hand for a dance of their own. The sound of the coming fire slowly rises. The waltzing couples dance in circles past each other, oblivious to each other, and intensely aware of each other, all at once.

It's a moment that shows the power of the play of ideas to fuse together concepts and characters into a theatrical grenade. This final scene is the waltz that takes place inside all of us – of our ancestors dancing with our present, of reason dancing with irrationality, and of hope dancing with despair, as the roaring, crackling sound of the heat-death draws ever closer.