Why is director Todd Solondz returning to the film that nearly destroyed his career?

Todd Solondz knows he has an unenviable public image. "There will always be people who will tell me I'm a horrible human being," he says, in a whiny New Jersey accent that borders on self-pity. "So there's not much I can do about that." Those who loathe Solondz have called him "cruel, misanthropic and mean-spirited", as he puts it, accusing him and his films of exploitation. His work is certainly provocative, personal and prickly, as uncomfortable as sackcloth on skin. And such has been the criticism that even Solondz seems detached from his output: "It's not even really what I would call a career," he says.

Wearing grey trousers and a lime-green short-sleeved shirt, Solondz cuts a nebbishy figure – "the quintessential nerd with attitude" as one critic joshed. Now 50, and with tufts of grey hair, the last few years have been tricky for him. His last film, 2004's Palindromes, a disconcerting exercise that cast eight actors as the same 12-year-old girl, was largely self-funded and grossed a measly $700,000 globally. Long gone, it seems, are the days of 1995's Welcome to the Dollhouse, which won the Grand Jury Prize at Sundance; rather, Solondz finds himself on the margins of a US independent film-making scene that itself has been suffering as the studios tighten their belts.

Little wonder he's shocked by the fact he has a new film to promote, Life During Wartime. "It's such an anomaly, particularly in these economic times. Things are so grim for the non-studio film-maker. I'm just lucky. It's just weird that this movie got financed, finished and so forth. I can't account for it." Admittedly, the financing failed several times before the film finally got off the ground. But given Solondz's reputation and the "difficult" material contained within, that's hardly surprising. "If I had wanted to do something that would be more marketable, I think I wouldn't have had the same struggle."

In Life During Wartime, Solondz revisits his greatest triumph, 1998's Happiness, a lacerating depiction of suburban malaise. Happiness won the critics' prize in Cannes and earnt a Golden Globe nomination for Best Screenplay – remarkable when you consider its main talking-point was Solondz's sympathetic portrait of a paedophile (Dylan Baker). And if you ever wondered what happened to him and the film's roster of screwed-up characters, Life During Wartime catches up with them 10 years on. But don't expect to see Baker, or Jane Adams, Lara Flynn Boyle and Cynthia Stevenson as the dysfunctional three sisters at the heart of the story.Solondz has recast the entire film, often daringly.

The very first image is a close-up on the distinctively scarred face of the black actor Michael Kenneth Williams (aka Omar, from The Wire), who has taken over from the white, sandy-haired Philip Seymour Hoffman in the role of the sad-sack sex-pest Allen. "I didn't want someone who was going to evoke Philip Seymour Hoffman for me," says Solondz. "I had to go some place else." It's why he calls the film "a variation as much as it is a sequel". He admits it would have been "a nightmare logistically" to reassemble the old cast, but it was primarily an artistic decision. "For me, it just felt much fresher. Audiences are smart. You want them to think they know where we're going and then come at them from another angle."

There's a suspicion that he can no longer afford an actor like Hoffman anyway, but Solondz has nevertheless assembled a first-rate cast, Shirley Henderson, Paul Reubens and Charlotte Rampling among them. Most impressive is Ciarán Hinds, an admirable replacement for Baker as Dr Bill Maplewood. As before, it's this character's story that's the most intriguing as, recently out of prison, he tries to reconnect with his two sons. The film may originally have been titled Forgiveness, but Solondz provides no easy path to redemption. "These are characters with troubles and struggles and that's what makes it compelling for me," he argues. "You have to have a struggle."

If the film is less overtly controversial than Happiness, Solondz reckons this is because the past decade has seen a considerable shift in attitudes to the material – in particular the issues surrounding paedophilia. "I suppose when I made Happiness, 10 years ago, the subject matter was more shocking. I don't think there's anything shocking about it today. It's shocking when you don't read about this stuff. It's in all the media and so forth. I went to the park recently back home. There's a little sign and it says: 'No adults may enter unless accompanied by a minor.' That kind of thing never existed when I was growing up, and I don't even know if it existed 10 years ago."

With its cast overhaul, Life During Wartime could easily be read as a study of how we all tend to reinvent our own identities over time – is it a coincidence that there's a poster in Billy's student dorm-room for Todd Haynes' Bob Dylan movie I'm Not There? Rather like Palindromes, Haynes' film cast several actors as his subject; Solondz, though, denies that the use of the poster is a nod to his own interest in this device, and instead puts its presence down to expediency. "It was so much trouble to get clearance on anything. And we could get clearance on that poster because Ed [Lachman, his cinematographer] shot the movie, and is friendly with Todd."

Solondz raises an interesting point. So dangerous and incendiary is his work considered, that even companies usually desperate to promote their goods in films have backed away. He cites the fact that he wanted Maplewood to drink Diet Pepsi in one scene. "Pepsi would never give us clearance. Nor would Coke," he says. "When it comes to product-placement, I'm on everybody's blacklist. Nobody wants to give us anything." Is it because of the paedophile angle? "Well, I don't want to say. Just every angle. All I can say is, nobody wants to have their product or their brand contaminated by this movie."

It's no surprise, given Solondz's past experiences. When he made Happiness, outraged executives at parent company Universal Studios forced their specialist arm October Films, who owned Solondz's film, to drop it from release (it was released by another company). He fared little better at New Line when he made 2001's portmanteau Storytelling. After the furore around Happiness, he was contractually obliged to deliver a more palatable R-rated film. Cutting one explicit gay sex scene (featuring Dawson's Creek's James Van Der Beek), he was also forced to place a red box over the genital areas in a graphic multi-racial heterosexual sex scene.

Still, there's something about Solondz and his work that suggests that, like his characters, he will always suffer. Even his debut feature film was calamitous – so much so, he doesn't like to include it on his official CV. Made in 1989, Fear, Anxiety & Depression – in which he starred – still evokes the title's emotions in him. The film was poorly received, and Solondz quit film-making for a time, taking a job teaching English to Russian immigrants (an experience repeated by the Jane Adams character in Happiness).

It was a shock to the system. Raised in a Jewish household and sent to a series of private schools before he went to Yale to study English, Solondz was an academic high achiever. Even his entry into the film industry was covered in glory.

He made a series of shorts at New York University film school, one of which caught the attention of the industry and won Solondz two separate studio deals. His career looked set – until he made his inauspicious debut. It was only when a family friend found him some funding that he decided to take a second crack at film-making and produced Welcome to the Dollhouse. Contrary to popular belief, its unpopular teenage heroine, Dawn Wiener, was not based on his youthful experiences, he says. Even so, I'm curious how his teenage years were. "Fine," he says, visibly stiffening. "I survived."

Solondz has no qualms about admitting he's happy to have left his adolescence behind. "To have to relive my youth – oh my God!" he cries. "I'm so happy to be older." While he may not be in any better position in the film industry, at least ageing seems to suit him. Calling it "the cure for everything", he's finally grown into his skin – a fact that may account for the slightly less brutal tone of Life During Wartime. "We'll see how long the body holds out," he says. "But in every way up here" – he points to his head – "it's so much better."

'Life During Wartime' opens on 23 April

Fear and loathing: The Solondz CV

Fear, Anxiety & Depression (1989)

In his little-seen debut, Solondz stars as a Beckett-worshipping playwright trying to penetrate the New York art scene.

Welcome to the Dollhouse (1995)

His first hit, this study of unpopular 7th-grade schoolgirl Dawn Wiener was full of Solondz's trademark caustic humour and excruciating observations.

Happiness (1998)

A career high point, this controversial story of a screwed-up extended family and those around them was bold, uncompromising and bleakly funny.

Storytelling (2001)

Shocking, and uncomfortable to watch, but not as effective as its predecessors; this two-part tale about race, sex and the Holocaust was a minor misstep.

Palindromes (2004)

Perhaps Solondz's most daring film, it's also his most gimicky, as eight different actors, of differing race, weight and sex, all take turns playing the same 12-year-old girl. JM