The problem with Ben Affleck

At passport control at the Eurostar terminal, I'm asked why I'm going to Paris. I tell the controller that I'm on my way to interview Ben Affleck, and his response is one that many people have when the Boston-born actor crops up in conversation: "Can you please tell him that he's a terrible actor!"

I defend Affleck by telling the man he's in for a pleasant surprise if he sees Affleck's directorial debut, Gone Baby Gone. The thriller is based on the novel by Mystic River author Dennis Lehane. It starts with the abduction of a young girl whose redneck single mother clearly struggles with the responsibility of bringing up a child.

Affleck's younger brother Casey plays a private investigator, Patrick Kenzie, called in to investigate by the girl's aunt, who fears the police are not doing enough. Similarities to the Madeleine McCann case ensured that this film sat on the shelf for nearly a year awaiting a UK release.

On the way to meet the director, I mull over why Affleck attracts such venom. Once, it seemed the world was at his feet. He and his best friend Matt Damon wrote and starred in Good Will Hunting. Their reward was a Best Screenplay Oscar and a passport to an A-list acting career in Hollywood. Damon's reputation took off, but Affleck had a knack of picking dud flicks; Pearl Harbor, Armageddon and Daredevil are on his CV. But the real opprobrium began when the actor started dating Jennifer Lopez, and people chose to forget that Affleck had shown his acting qualities in several films by Kevin Smith, Shakespeare in Love and Richard Linklater's teen classic Dazed and Confused.

The adventures of "Bennifer" were a staple in the media. The couple even insisted on acting together – the resulting film, Gigli, was panned and quickly gained a reputation as one of the worst films in history. When their relationship hit the rocks, Smith even thought it prudent to cut scenes of the couple getting married from his ill-received comedy Jersey Girl. It seemed that everything Affleck touched turned into box-office poison.

Then, at the Venice Film Festival in 2006, Allen Coulter's film about the mysterious death of the actor George Reeves, Hollywoodland, was screened and Affleck picked up the prestigious Best Actor prize. His troubles seemed over: he hooked up with the actress Jennifer Garner after they met on the set of Daredevil, and they have a daughter. And he decided it was time to sit in the director's chair.

Affleck says: "What attracted me to the book [Gone Baby Gone] was that it was a complicated story; it had a moral ambiguity and a complexity to the characters; it was about a rejection of the idea that there is a black and white, that you can banner people as good or bad and that you can diagnose people as such. I've really felt a strong negative reaction to that type of pigeonholing in my life, and when I came across this I really responded to how this book refused to do that to any of its characters."

He speaks slowly, as if every word has to be considered. He reveals that he could have appeared in Gone Baby Gone rather than his brother: "When I originally bought the rights, I was going to act in it and try to find a director. Then I started to scare up the money to direct it, and once I did that I thought I couldn't be in it. I guess it was too difficult and therefore I thought I wouldn't be able to do either one well and I'd do a bad job and I just wanted to do a great job.

"Just in terms of my soul, I didn't want to spend all this time in my life on this and it not be that good. The first movie is five times as important as your second or the third in terms of the impression that it makes."

The 35-year-old is well aware of the negativity circling him. It's unavoidable when the lampooning is so open – an episode of South Park depicted Affleck as a wooden hand and commented that his "face is as flat as a pancake". I mention this, and his reaction is possibly the eeriest I've ever seen in an interview: "I don't know. I don't ever watch South Park. But I think, er..." He pauses. "I've met those guys and they're perfectly nice guys."

Then he stares at the wall for what seems like ages, not even blinking. I don't know if he's hoping I'll ask another question, but I take a guilty pleasure in waiting. Then he snaps out of it and continues being diplomatic: "I don't think that if you can't handle being criticised and you can't take a joke and you have skin that is too thin that you should be in the entertainment business. You have to be able to deal with it. You have to be grown up."

He admits his career choices have been erratic. "I've had different approaches to my work at different points in my life. At certain points, I've felt I've cared more than at other times: sometimes it's just been about making money; sometimes it's just been about trying to get a job; sometimes it has been like being a deer in the headlights and being on certain types of sets I've never been on before; sometimes it's been about the thrill of working with certain people, or the pressure of trying to be Jack Ryan [Affleck played the Tom Clancy character in The Sum of All Fears]. For the last three years, stepping into doing Hollywoodland and then doing this movie and going forward, it's been about nothing else except wanting to do something and making it really good."

The biggest change to his life has been the birth of his child. "Fatherhood has made me more sensitive and probably more caring, in a way. I always thought I was a caring guy, but it has made me feel things more acutely."

But it wasn't fatherhood that led him to Gone Baby Gone. "It wasn't the missing-child storyline that grabbed my attention so much, although it's a powerful aspect of the story. This story says that we're not taking care of children as well as we could in society, and I found that pretty moving. Also, it's about the way the media cover certain cases of kidnapping and that there is a certain hypocrisy in shouting so loudly on the television about one thing and totally neglecting to talk about thousands and thousands of others."

Why cast his brother Casey, who, since his turn in The Assassination of Jesse James by the Coward Robert Ford, is getting the plaudits once seemed destined for him? "He was in Good Will Hunting, so I've had professional experiences with him before," Affleck says. "I have respect for him as an artist and that sounds strange to say, but it wasn't like we're doing a movie and all of a sudden your little brother turns up on set. The only thing that did concern me was that my decision could be viewed as nepotism. But I knew he was good for the movie and had every confidence he'd be positively received when the film came out. This film was too important to me to take any risks or to do anybody any favours.

"Casey has the benefit of not being very well known so he can surprise the audience. That is a tremendous asset. Now, when Casey does his next movie, people know he's good and he'll make less of an impact."

'Gone Baby Gone' opens on 6 June