Neil Diamond: Has he finally become hip?

Neil Diamond, 67, has displaced Bob Dylan as the oldest chart-topper in US album history, grabbing his first UK No 1 for good measure. The album, Home Before Dark, like its predecessor 12 Songs, was produced by Rick Rubin, previously responsible for the stark last act of Johnny Cash's career. Next month, Diamond will make his Glastonbury debut.

But Diamond will never really be hip. He is not a Mount Rushmore rock monument as Cash was. Nor does he have a reservoir of quicksilver genius waiting to be rediscovered, like Dylan. He is, instead, a throwback, among the last of the Tin Pan Alley songsmiths. His kink is that he aspires to be Beethoven, yet sinks into self-doubting despair when he writes. When Cash sang Diamond's "Solitary Man", it had the weight of life and death, sung by an ageing, caged lion. Diamond's existential dread came from writing it.

His US No 1 combines two factors: the numerical weight of a different branch of the baby-boomers who less than two years ago put Dylan's Modern Times there, and an appearance on American Idol, where his pop craft really does look like genius. The comparison with Dylan is instructive. For most rock fans, Diamond remains notorious for his performance of "Dry Your Eyes" in The Last Waltz (1978), Martin Scorsese's film of The Band's final show. "Follow that," he reputedly bragged to Dylan, who whipped back: "What do I have to do, go on stage and fall asleep?"

His appearance was part of a doomed attempt at credibility. Talking in 1976 to Rolling Stone , he confessed: "I was 26 or 27 [when the hippie movement began], had already been married, had two children. I knew that I was out of it... [But] I was trying to be George Gershwin. 'Hip' was something frivolous people had time to be... hip is bullshit." The cultural conflagrations of the Sixties that made so little sense to him destroyed his old home, Tin Pan Alley: "It exploded with the rest of it." Diamond's crowd, his management and the media both noted, were older, "average, upward-bound people".

"The time is probably just about right for a spot of critical reappraisal," Melody Maker's Chris Salewicz grudgingly conceded, reviewing a Diamond greatest hits in 1974, in the wake of Robert Wyatt's joyful, frail cover of Diamond's "I'm a Believer", which The Monkees had made a transatlantic 1966 No 1. "But who cares anyway," he sniffily concluded, "you're always going to hear this on every Wimpey estate in the country."

That would doubtless make Diamond more proud than a good review. As he says: "I'm a meat and potatoes kind of guy." But an ageing catalogue of self-written standards ("Cherry, Cherry", "Cracklin' Rosie", "Forever in Blue Jeans", "Love On the Rocks") doesn't touch the heart of the man who has climbed back to the top with Home Before Dark.

Because Diamond's background has always been unconventional. His father, a successful Jewish dry goods seller in Coney Island, sang old Russian songs about the house, while hearing Pete Seeger strum folk songs at a Catskills summer camp determined Diamond to start performing; experiences common to many classic New York songwriters. But his father's travels across America for business made Diamond an itinerant, transplanted, lonely child. His first hit was called "Solitary Man" for good reason. Outsized ambitions silently brewed in his head. At the Brill Building, he failed in writing for others because he poured out "too many words", instead of crafting classical pop structures, like Burt Bacharach or Carole King. It took seven years of such failure to finally make it. The garish white outfits that made him a joke for hipsters were for him the result of a slow, nervous attempt at positivity. Before that, like Cash, he had been a sombre man in black. Bizarrely, auditioning to play the revolutionary, scabrous comedian Lenny Bruce in 1972 was what peeled open this inner torment. "Bruce's language and thoughts were so violent," he told Rolling Stone. "He was just saying all those things I had been holding in, 'fuck' and 'shit' and 'death' and 'kill'... It was all the anger that was pent up in me. I went into therapy almost immediately after that."

"I Am...I Said", his 1971 hit, is an early example of such navel-gazing. It agonises over the worth of his, or anyone's, songwriting. But Diamond has also been giddily vain enough to state: "I don't dream of being George Gershwin, I dream of being Beethoven and Tchaikovsky and Robert Frost. That's how much I think I can do musically." Blowsy numbers such as the 1978 Barbra Streisand duet "You Don't Bring Me Flowers" have been far more typical. Diamond is odd, but no lost genius.

Rubin looked to the singer's older, better days. "I've always been a fan of his early records," he said. "Growing up, I always listened to FM radio. And he was always on." Said Diamond: "He very politely asked me why my songs changed. Without realising it, the records became bigger and bigger, and the songs became smaller and smaller."

"Loneliness" was the mood Rubin found in Diamond for 12 Songs, part of a rejuvenation the singer had already been seeking. Home Before Dark sees Diamond take charge and cut closer to the bone. Mature, wordy reflections and the ghosts of Brill Building tunes are a fitting way for this strange 67-year-old finally to reach No 1.

'Home Before Dark' is out now on Columbia