Amy Jenkins: Why confessional writing captures the female mind

Eat, Pray, Love, the movie of Elizabeth Gilbert's bestselling confessional memoir, opened in the US this week.

Sony has thrown a huge amount of money at it – $80m – and cast Julia Roberts in the lead role in an effort to ape the success of Sex and the City. But early reviews are lukewarm – which only goes to show that the current craze for the female confessional doesn't translate well to the big screen.

Bad films, though, are fodder for misogynistic attacks. The thirtysomething confessional is commonly accused of being narcissistic, navel gazing – or the biggest put-down of all – just "too much information". Eat, Pray, Love, which detailed Gilbert's real-life journey back to happiness following a painful divorce, and involved her eating in Italy, meditating in India and falling in love in Bali, sold seven million copies and is much loved – but critics accuse it of privileged self-obsession.

If the book had been a novel about a woman searching for some kind of self-realisation – a respite from internal demons, even if they are the demons of privilege – no one would have thought twice. As it is, real-life women who admit in print to thinking about themselves get shamed. It seems that soul-searching is seen as a luxury afforded only by the relatively well off and is therefore an unforgivable indulgence – especially if not done at an artistically tasteful arm's length.

There's an idea that the simple things in life should make us happy. Enough food, a roof over our heads – and the more we have the happier we should be. But that's only the illusion of capitalism. The new science of happiness has shown us that happiness has very little to do with income. Still, some elements in society will always be uncomfortable with the idea of middle-class angst. They are like the Yorkshire men in the famous Monty Python sketch who don't think the "young people of today" should be allowed to complain about anything: "Cardboard box? You were lucky! We lived for three months in a paper bag!"

Happily, the confessional genre is defying its critics and flying off bookshop shelves. In the US, Meghan Daum's Life Would Be Perfect If I Lived in That House and Emily Gould's And the Heart Says Whatever are making waves. Over here, my own personal favourites have been Stephanie Merritt's The Devil Within, Susie Boyt's My Judy Garland Life and Hephzibah Anderson's Chastened.

I devour these books. I don't think it's easy to write as honestly as these women write, but I do think these books are often more successful than contemporary novels. The fact that it's for real carries a great deal of fascination. The books are an opportunity to get inside another woman's head and hear the truth – stuff they'd never tell you at a dinner party. What could be more compelling?

Plus, as I'm reading, I'm scanning the whole time for evidence that I am, in fact, sane. Everyone's head is a maelstrom of neurosis – it's not just me – and for that comforting information I will forgive the author a multitude of first-person participles. Their neurosis doesn't even have to be the same as mine – as long as they have one. Meghan Daum's new book is about her obsession with moving house. I've only ever lived in two houses, the one I grew up in and the one I live in now. But I still found the book a great read for the simple reason that the nature of the presenting problem (whether men or real estate) doesn't really matter; what is important is the underlying struggle to make sense of life and what it takes to be happy.

When Nick Hornby wrote Fever Pitch, nobody accused him of excessive self-regard – of having an unhealthy compulsion to expose himself. Women, it seems, are encouraged to bare their bodies, but never their souls.