Ben Folds: New 'So there' album features eight rock songs performed by a classical sextet

"Don't make a science of the mojo, it's going to happen or it's not," stresses Ben Folds, animated about how he "crafts" his songs as he sips his scotch. "I think whether you're Mozart or Bob Dylan, you're crafting, you're a craftsman and you've decided to be a professional. I'm hard-nosed about these things."

I meet the hard-nosed but affable singer-songwriter just before his barnstorming performance at the Royal Opera House, at which he showcases his particularly bold new album, So There, which features eight "rock" songs performed by the New York classical sextet yMusic (consisting of cello, viola, violin, clarinet, flute and trumpet), with Folds on vocals and piano, plus a 21-minute Concerto for Piano and Orchestra.

"Most of my initial listening as a child was Sixties R 'nB stuff, I didn't even know that white people made music," claims the 48-year-old, "but when I was about 10 or so I started playing in youth orchestras. My new record is a more orchestral record and that's the kind of music I cut my teeth on." Folds grew up in Winston-Salem, North Carolina, a place he was keen to escape from ("I was born in the wrong region") because "Southern thinking is so tiny".

"My upbringing was working-class Southern and if you sang you were [called] a faggot, if you danced you were a faggot, if you drew you were a faggot, you couldn't really do anything but play football or drive souped-up cars and get into fights," he claims. "I thought, 'so, I'm a faggot, that's awesome', but then I'd think, 'no I'm not, but if I was then that's fine'.

Even though he says he isn't gay he has spoken out about gay rights and he gave an interview with the Huffington Post condemning the gay marriage ban in North Carolina. Although he doesn't tend to speak out about politics, he has also said in interviews that he supports President Obama's stance on gay marriage saying, "We don't have to talk about everyone having the right to eat or breath, so why this? It feels like a no-brainer. What a weird time."

He has been married four times and is currently single. "I thought I had to be married and I kept on trying to be married," he says. "I was trying to be what I was supposed to be from the age of 19 and being married was infused into my upbringing by osmosis."

Ultimately, Folds, who flits between residences in LA and Nashville, rejected his "judgmental" background "because it's restrictive to art" – and there's no doubting the droll pianist's commitment to his art. Over a long career, which took off 20 years ago with his pop trio Ben Folds Five and their self-titled debut album, Folds has proved to be a serial collaborator with a diverse and rich legion of artists, including a supergroup called 8in8, comprising the writer Neil Gaiman, his wife the singer and pianist Amanda Palmer, and Damian Kulash of OK Go.

They formed in 2011 with the goal of making eight songs in eight hours, and played just one concert. He has also collaborated on an album with the novelist Nick Hornby ("I think his popularity has overshadowed the literary weight that's in his books") and also, perhaps most memorably, with William Shatner, producing Captain Kirk's 2004 album Has Been, where Folds created the arrangements for Shatner's prose-poems.

"Shatner eats directors for lunch, you are really hit hard by him," maintains Folds. "I saw some footage of us in the studio and he was yelling at me, 'Hey, you are pecking at me like a chicken, stop it'. I was hitting him hard at the time because I thought he could do better. I had his back though because I cared about him and I cared about the record. And I learned a lot from this fearless performer."

Folds is pretty fearless himself, with his searingly honest ("Brick", his first big hit from 1997's exquisite Whatever and Ever Amen, concerned his high-school girlfriend having an abortion) and frequently acerbic ("She stabbed my basketball/And the speakers to my stereo," he claims on "Bitch Went Nuts") lyrics and openness to a challenge, such as his recent foray into classical music.

His vocals and song themes aren't unlike arch experimentalists Rufus Wainwright and Stephin Merritt (The Magnetic Fields) but his "microscopic" and adroit songs smack more of British artists such as Neil Hannon (a friend of Folds), Chris Difford and Elvis Costello. Britain has always embraced Folds ("I feel I'm given a certain respect here") and his sensibilities and humour appear very British ("I have a dry sense of humour and make jokes back home that no one laughs at"). However, for all the George Gershwin-like composing on the new record, he's lost none of his trademark nimble lyricism as evidenced on "Not a Fan", where he pleads, "I grew up on sugar cereal and TV/I wonder what you see in me?".

So how autobiographical is So There? "My songwriter friends say So There is my personal record but I don't see it that way," he says. "This record is very much an anything is possible kind of record. The other times that I remember feeling that way is on the first Ben Folds album and on the William Shatner record, those are records where I experienced that sort of clarity." Neil Hannon and the likes of Billy Joel and Elton John seem to agree, with the latter two ("both sweet guys") describing So There as a "brilliant piece", which touches Folds.

"Elton's been so kind to me, he buys all my music and he's just really encouraging of the Piano Concerto," he says of his hero. "I've done presidential gigs but a couple of people freak me out and one of them is Elton. As a consequence of that I feel a lot more comfortable communicating with him over email than over the phone but I hate that too because I can see how alienating that can be, but you'd have to go to a shrink to get over that." So There is a mature, rousing work that proves Folds is at the top of his creative game, and it's clear over our two-hour chat that he's thought long and hard about his "craft".

"You have to relax all the things you don't need – if you don't, you'll die," he maintains. "I can play the piano louder than I did 20 years ago, but can I play it harder? Hell no. I can only put in half the physical energy that I used to but I'm just trying to peel away the shit that I don't need, the facial or vocal expressions I don't need or the notes that I don't need. And what's left, well I just kill it."

Ben Folds's new album, 'So There', is out on 11 September