NW, By Zadie Smith

We make out of the quarrel with others, rhetoric," wrote WB Yeats, "but of the quarrel with ourselves, poetry." Ever since White Teeth in 2000 (the debut novel of which she wrote a scathing anonymous review) Zadie Smith has been pursuing a rowdy and rewarding quarrel with herself. Out of her divisions and contradictions – not for nothing did she call her 2009 essay collection Changing My Mind – has come fiction that, for all its vitality and virtuosity, never feels remotely satisfied with itself. Meanwhile, her sparkling critical pieces have in their heartfelt engagement with icons and mentors, from Franz Kafka and David Foster Wallace to TV sitcoms, opened a door into the writer's studio.

Briefly, NW – her fourth novel – tells us that the quarrel goes on, as knotty and raucous as ever. Truce or settlement looks as far away as ever. The book revisits her home turf in clamorous sectors of north-west London – Willesden, Kilburn, then upmarket into Queen's Park. Here the spectacular collisions and disjunctions of a divided city enact its author's doubts about what kind of novelist she is – and how the novel might make sense of these jagged splits and rifts.

In essence, NW follows the paths of two young women from the same council estate – half-Irish Leah, Caribbean Keisha (later, self-rebaptised as Natalie) – through a time and place "crazy busy with self-invention". Two men shadow their trajectories, snakes to their ladders: Nathan, who sinks back into the street-life hustling of drugs and crime, and Felix, whose intermittent toehold on a "creative" career is dislodged by the demons of addiction.

For all its close focus and refusal of a hackneyed state-of-the nation narrative, NW reads like a post-meltdown novel, in comparison with the breezier ebullience of White Teeth: "Not everyone can be invited to the party. Not this century." From the outset, when a crack-addled neighbour scams cash out of trusting Leah, we grasp that the ladder of mobility has been hauled up and stashed away. Yet Leah and Keisha, born (like their creator) in the mid-1970s, have both climbed it – though to very different heights.

Leah, a philosophy graduate whose perception of "time as a relative experience" lends NW a discreet foundation of big ideas, has settled for a lowly admin job dispensing funds to charities. She builds her life around passionate marriage to the kind and beautiful French-African hairdresser, Michel. The driven Keisha-Natalie has taken a high-speed escalator. Self-possessed, ruthlessly disciplined, she qualifies as a barrister, quits the legal-aid ghetto and flies high in commercial chambers: early-QC material. Life becomes "a problem that could be solved by means of professionalization". Marriage to the debonair and loaded Italian-Trinidadian Franco follows, with two kids and a Queen's Park pile to polish the illusion of bliss.

"What's going on up there, Keisha?" asks her old friend, the sexily life-enhancing Tonya. We never quite know, and perhaps neither does she. A secret misery pushes her into high-risk adventures. Behind the puritanical self-creation, "the desire for chaos" lurks. Meanwhile, ticking clocks and shrinking opportunities – for advancement, for love, for motherhood – punctuate a novel keen to remind us that "women come bearing time".

Such an outline tells you next to nothing about the exhilarating crackle and sparkle of Smith's prose. Her dialogue sings and soars; terse, packed and sassy. In the novel's first half we absorb the tangled racial, class and family histories of her characters through backchat and badinage rather than plodding exposition. Page by effervescent page, NW delivers all the jostling, aromatic savour and spice of a stroll up Kilburn High Road: the centre of its urban world. As a sculptor of London conversations, each stray phrase chiselled into throbbing life as a signature of character and context, Smith is simply wonderful: Dickens's legitimate daughter. Felix's valedictory row in Soho with his druggie-aristocrat lover Annie leaps and roars along with jaw-dropping panache and finesse.

And then Felix's story stops. Dead. Which brings us back to that nagging quarrel. In 2008, Smith published an essay in which she juxtaposed the lyrical humanism of Joseph O'Neill's novel Netherland with the avant-garde dismantling of narrative conventions in Tom McCarthy's experimental Remainder. Her willed preference for the latter struck me then as a case of Smith not so much changing her mind as twisting her own arm. Somehow, the supremely confident straddler of two 20th-century traditions – the empathetic realism of EM Forster, showcased in On Beauty, and the immersive stream-of-consciousness flow and jump-cut montage of Virginia Woolf, more prominent here – felt herself cowed by a bleak theoretical diktat. Smith surely wants to intensify our experience of urban shocks and shifts, not (in the manner of the hard-core avant-garde) to deny that the novel can or should attain such a full-spectrum truth.

So has Smith knuckled under to a crew of up-themselves posh theory-boys? Felix, the occasion for some magnificent writing, exits in a spasm of melodrama. Then she withdraws from the mind of Keisha-Natalie to recount her upward mobility in an alienating, deadpan sequence of numbered sections, speckled with bossy injunctions such as "Reader: keep up!" Yes, it breaks the illusion, undermines the realist narrative, and so on. But it can feel too as if the most gifted British novelist of her generation has opted for a bout of gruesome self-harm. Well, critics can do exhortation too. Stop it, now!

In the last act, tragic, prophetic Nathan returns. The social chronicle morphs into an existential pilgrimage: "She had no name, no biography, no characteristics. They had all fled into paradox." Faithful to its riven and polarised city, NW cussedly refuses to hang together. Read it as a bold embrace of an age of "sudden and total rupture", when every clock keeps its own time, or else as a self-sabotaging flight from coherence in the service of abstract dogma. Either way, Zadie Smith's quarrel with herself still gives rise to a fierce and fractured poetry.