Jacob Rees-Mogg is posing for The Independent's photographer by the rather beautiful village green in Hinton Blewett where he grew up as a child and which now lies within the boundaries of his Somerset constituency.

Nearby is a traditional red phone box – looking a little worse for wear.

"You know the police once found a stash of cannabis in there," he tells us in between patiently posing for endless pictures. "My father used to say that it was easier to get cannabis in Hinton Blewett than a copy of The Spectator."

Rees-Mogg's father William – former editor of The Times – is, understandably, much on his mind. He died shortly after Christmas and everywhere we go people come up to him to share their memories and offer condolence.

It must be hard being a public figure with a public father and constantly have to address your own private sadness. But he doesn't let it show and responds to their solicitations with unfailing courtesy and a kind of old-fashioned formal politeness which is his hallmark.

They are mannerisms which were a cause for ridicule when he first became an MP in 2010. Newspaper sketch writers couldn't wait to get their teeth into "The Mogg" – whose unashamedly upper-class style was at odds with the image of the Conservatives David Cameron was trying to portray.

Quentin Letts called him the "Hon Member for the Early 20th Century" who was "41 going on 90", while Ann Treneman noted that if there was "ever anyone who personifies the words 'out of touch', it may be The Mogg".

But three years on – and in Westminster at least – perceptions of Rees-Mogg, 43, are changing.

He was recently described by a Tory colleague as a "mini Boris" – charismatic despite being posh and an able politician who could explain complex arguments simply and intelligently – without recourse to vacuous sound bites.

He has become a frequent guest on programmes such as Newsnight where he has rather effectively defended the Government on tricky subjects such as benefit reform and rejecting statutory regulation of the press.

So are his appearances now sanctioned? "I think they're no longer de-sanctioned," he smiles. His agent, Margaret, sitting next to him, adds pointedly: "That's not as funny as you might think."

Rees-Mogg tries to explain with deliberate understatement: "I think in the early days they'd probably have been happier if I didn't do it – but I don't get the feeling now."

When you spend time with Rees-Mogg – watching him talk to constituents about mundane subjects like child benefit, parking restrictions and planning disputes – you wonder whether his fogeyish media image is a bit affected.

Yes, he has old-fashioned mannerisms – but nothing like the pantomime portrayal of him. He is also undeniably clever. Before going into politics he set up and ran his own highly successful investment company – not the mark of an upper-class bumbler. So is it a bit put on?

He laughs. "I've got a nasty feeling it's the truth. Reading my father's obituaries he had exactly the same image when he was my age. Everyone thought he was a young fogey. It's probably something to do with the Somerset air."

He then says something interesting and also rather revealing. "I think Boris has the answer to it. Boris makes no pretence about being anything other than he is. I've always been very suspicious of politicians who try to be 'men of the people' because actually it's a rather condescending view of the world. Everyone has their own individual or different experiences and you cannot, as a politician, have lived the life that anybody else has lived. "You can't say, I understand how millions of people live their lives because of how I've grown up."

So does that mean Cameron (who like Rees-Mogg and Boris also went to Eton) is putting it on? He is too smart to fall for the question. "I think Cameron got it absolutely right in his party conference speech when he said, I've been very lucky – I've had a great education. I want that for everyone else," he adds a little too quickly.

Rees-Mogg says one of the defining experiences for him politically was campaigning in the safe Labour seat of Central Fife in the 1997 general election. The campaign has gone down in Westminster folklore because Rees-Mogg was supposed to have canvassed for votes in his Bentley with his nanny (the nanny bit was true, the Bentley bit not). But he worked hard in the seat and got a very respectable nine per cent of the vote (higher than his successor managed in the 2001 election).

"It was probably the most useful learning experience I've had politically," he says. "I met thousands of people who told me a little bit about their lives and the problems of living in an area where almost everybody was dependent on the state. It was important to see how clever ideas in Whitehall didn't work in people's daily lives."

Rees-Mogg is on the right of the Tory party and has a record as one of the Coalition's most rebellious MPs. He slightly disputes this but says on certain issues he is prepared to defy the whip.

"Parliament and politics is interesting if you think about what the issues are, how strongly you feel about them and whether it is important to express your views," he explains. "As a general rule I think the Government ought to expect to get its way on the routine business.

"[However] When they are doing things that are constitutional then that is different and I think you have greater responsibility not just to go along with the latest idea of the day." David Cameron is due to make a defining speech on his European position later this month and Rees-Mogg is clear that it will have to be ground-breaking to prevent Ukip denying the Tories victory at the next election.

"It's not going to be easy for him. But I think to neutralise the Ukip threat he needs to be talking about a pretty thorough renegotiation. But he also needs to be saying, if I can't get that renegotiation then we've got to leave."

And does he want to leave?

"I am not frightened of leaving but it is not what I think is necessary."

But later he adds: "My neighbouring two seats both have Lib Dem majorities that are smaller than the vote which went to Ukip at the last election so that's a real problem for us."

And what does the future hold? He says he is not interested in ministerial office because as a backbencher you've got more independence and "I'm not going to be made a minister anyway".

It is ironic that without the accent and the mannerisms he would certainly be tipped for the Cabinet.

Maybe we don't live in such a meritocratic society as we think.

Old values, fresh thoughts

Age: 43.

Education: Eton and Oxford.

Status: Married with four children.

Nannies or nanny state? "Oh nannies. I'm all in favour of nannies. Nannies are wonderful."

Elected or appointed Lords? "Am I not allowed to say hereditary?"

Thatcher or Churchill? "God that is difficult. Presuming we're not going into a world war, Thatcher."

Modernisation or tradition? "Tradition."

Republican or Democrat? "Oh Republican without a question. I'd even have voted for Mitt Romney."