Pedigree chum: Is Alexander Armstrong the poshest man in comedy?

More so than Miranda Hart, Stephen Fry and David Mitchell, Alexander Armstrong seems to be the acceptable face of posh comedy. With his comically large ears (a gift from his father), crinkly smile, affable demeanour and (crucially, perhaps) lack of smarty pants, he's the cuddly side of the upper classes in an age when, rather oppressively, toffs seem to be taking over again. Even Armstrong's overgrown Hooray Henry, 'Harry', in those adverts for Pimm's – alcopops for the privileged – is cherishable. Not that he drinks the stuff in public, he says, for fear of wags shouting, "It's Pimm's o'clock" – one of the great advertising campaigns, by the way, that helps explain some of Armstrong's wider appeal. The more you parody the posh, as the creatives at the advertising agency Mother realised, the more accessible they become to other groups.

Not that Alexander Henry Fenwick Armstrong – 'Xander' to his friends and his PRs (sometimes the two are the same thing) – is himself a parody of a toff; he's the real deal. He has true blue running though his arteries, as explored in a 2010 episode of the BBC1 genealogy show Who Do You Think You Are?. Armstrong's lineage stretches back through landed gentry in Northumberland and Ireland (he is actually listed in Burke's Landed Irish Gentry) to William the Conqueror himself – and that's seriously well-connected, unless you happen to be one of those people who thinks that you are a social upstart unless descended from Ethelred the Unready.

So what better place to meet this hard-working actor/comedian/quiz show presenter/panel show host than in the 'library' of the Ivy Club in London's West End – a slightly poky annexe in which I didn't notice any books; perhaps members come here to re-charge their Kindles. I'd have worn my smoking jacket if I owned one. "A secret staircase," quips Armstrong, a piece of wood panelling opening, as in a stage farce, to reveal the man himself, eyes crinkling. "Sorry, the traffic was terrible. How are you?"

I have seen Armstrong likened in print to "a big friendly labrador", but I think I discern a core of shrewdness in those twinkly, crinkly, button-brown eyes. How could it be otherwise, when he is one of Britain's most accomplished comedians? He isn't Bertie Wooster, after all, but the sort of man who observes, writes and performs Bertie Woosters – creations like his street-slang spouting Second World War RAF pilots in BBC1's The Armstrong & Miller Show. But more of his fruitful – and sometimes fraught – partnership with Ben Miller later, because my burning question is why he insists on channelling so much of his time and energy into an afternoon quiz show, BBC1's brilliantly named Pointless?

Armstrong answers this indirectly when discussing his decision not to become Des Lynam's successor on Channel 4's words and numbers game show Countdown, a job that was offered to him in 2007. "Countdown came about at a time when we'd just had our first baby," he says, "and, if I'm entirely honest, it looked like an income – a salary – something I'd never had before." Don't underestimate the lure of a regular pay cheque, in other words. Even at the glamorous heights of light entertainment, it seems, a freelance is a freelance and a monthly salary holds as much attraction as a Bafta nomination.

"This happens a lot in our business," he elucidates, "that just when you think things ought to be going well and you're looking for things to start happening, nothing's happening... you lie awake at night thinking, 'How many more weeks can we afford our nanny on half-time?'."

Don't hate him for that last remark, because, to put it into context, we had just been discussing his childcare arrangements. Anyway, Armstrong films three editions of Pointless a day, four days a week, for about three months of the year, which, while not exactly coal-mining or cleaning lavatories, sounds one hell of a slog, even if it is my mother's favourite daytime quiz show. "It's no accident that it's ended up being a teatime trivia fave," he says when I tell him this. "They threw lots of very clever people at it. I genuinely love doing it... it's very relaxed... and it's one of those nice shows where the more we go off-piste the better. Richard Osman who I co-host it with is a very old friend of mine – he was at Trinity with me."

Trinity College, Cambridge, is also where Armstrong first met his future comedy partner Ben Miller. Armstrong arrived wide-eyed and socially backward after an idyllic but isolated rural childhood as the youngest child of a land-owning Northumberland GP, Henry, and his magistrate wife, Virginia – the parent with the ancestors dating back directly to the Norman Conquest. He describes growing up near (albeit not that near) Rothbury as a mixture of Tolkien's Middle-earth and Jane Austen.

"It was an incredibly sheltered upbringing," he says. "It was so far away from the modern world. We lived so far from anywhere that our newspaper wouldn't be delivered up to our house, it would be left at the post office which is about a mile and a bit away down at the bottom of the valley. So one of the treats would be to walk with the dogs to go and pick up the paper."

It's the sort of upbringing he wouldn't mind giving his three young sons – Rex, Patrick and Edward – although his London-bred wife Hannah, a former events organiser for (obviously) Harvey Nichols – is less sure. "I don't want them to be too sophisticated too soon," says Armstrong. "I'd really quite like them to be wide-eyed... I mean there's a lot to be said for boredom. If you grow up somewhere where the pace of life is very slow you enjoy the gaps between the pulses. I read a lot and, boy, did I practise the piano."

And music, singing in particular, took him to cathedral school in Durham – a public school which he once likened to "a country club for the children of rich Northern industrialists... I was the only one who didn't speak with a Geordie accent" and where he claims now to have been "teased, but not in an unpleasant way" – and thence on a choral scholarship to Cambridge. "I'm staggered when I look back at how, when I got to Cambridge, I didn't know anything of the world."

His more worldly Cambridge contemporaries included Sacha Baron Cohen, Sue Perkins and Mel Giedroyc, as well as, more fatefully, Ben Miller. "Ben was a bit of a star there... mainly because he was in a rock band, but also because he was in Footlights. And he was going out with Rachel Weisz, who was the great beauty of our Cambridge generation. She still is."

After graduation, Armstrong shared a flat in London with the fledgling playwright Jez Butterworth, later of Jerusalem fame, performing sketches with his first comedy partner, another Cambridge chum David Wolstencroft. "We'd set up a comedy club in Notting Hill of which Ben became a member, and I was doing stuff with Ben as well as doing stuff with David, and the stuff that Ben and I were doing was just better. As simple as that." Not, however, before an embarrassing scene on the King's Road, Chelsea. "Ben and I were buying props for a sketch and we bumped into David and he's going, 'Oh, what's this?'. And we went, 'Oh, just some stuff... for tonight... for a comedy gig'. It was a tough thing to do, but you have to be a little bit pragmatic. We had quite different tastes in comedy."

Wolstencroft went on to create Spooks, and Armstrong and Miller to be rejected by the BBC, the Corporation going through one of its periodic anti-Oxbridge phases. "Fair enough," says Armstrong, before adding that he wishes they could show some consistency. "I mean, by the time we were most recently on the BBC, so were Mitchell and Webb and Harry [Enfield] and Paul [Whitehouse]. They're not Oxbridge but they're a white middle-class double-act – although they probably wouldn't thank me for saying that."

Channel 4 eventually picked up Armstrong and Miller for four series, before the duo fell out with each other, or rather Armstrong threw a hissy fit. "I suddenly got very claustrophobic", he says. "I did walk out rather stormily sometime in 2001. It was timely and actually rather useful because we had only ever worked with each other, and tied up with that was a dread that when things dried up for the pair of us then that was it."

That perennial insecurity again. Anyway, the friendship was strengthened by the split, says Armstrong, and indeed their sketch show did return in 2007, this time on BBC1, with renewed vigour. They won't be making any more of it however. "I was just watching Watson & Oliver last night and I just found myself thinking, 'Thank God we don't do a sketch show any more'. You never build momentum in a sketch show – they're a series of stops and starts... you have to rebuild your credibility every 90 seconds."

With backing from Hat Trick the duo have recently formed a production company called (yes) Toff Media, "specialising in snugly-fitting bespoke comedy", and at the moment they are supervising scripts for a new comedy drama for the BBC in which they play Footlights graduates who also work for MI6. "I won't say too much about it because it's still very much in its infancy. We worked on making it as credible as possible that two entertainers would be spies. They tour with the British Council."

In the meantime, Armstrong continues with solo ventures – further nanny-subsidising quiz shows (ITV's Epic Win which, he says, won't be coming back), adverts (playing an irate customer in the Direct Line insurance ads), investing money from these adverts into a restaurant chain, hosting a rather good new comedy panel show that's currently showing on Dave, Alexander Armstrong's Big Ask, and appearing on Have I Got News for You. In fact, Armstrong is the most prolific and, by general consent, the most successful of the post-Angus Deayton guest-hosts, with a record 19 appearances. And he was nearly named as the new permanent host – until the fallout at the BBC over the suicide of weapons expert David Kelly in 2003 scuppered his succession.

"We'd drawn up contracts, we'd done everything and it was just a matter of waiting," he says. "And then there was a sudden battening down of the hatches at the BBC by which time the guest host thing had proved to be such a successful formula – it was of a piece with the tub of lard thing in a way. Have I Got News for You has a tremendous knack for extemporising when things went wrong."

It was HIGNFY regular, Ian Hislop, who advised Armstrong not to take the Countdown job, arguing that it would be too restricting. "That was very timely, because I had sort of forgotten what the other side of the career is, which is that it has to be something that interests you."

And one of those interests is acting, playing Caroline Quentin's husband in three series of ITV's Life Begins, a sex-addicted guest in Hotel Babylon, inventor Clive Sinclair in Micro Men, with Tim Roth in Sky1's Skellig and with Martin Clunes in Reggie Perrin, as well as this year's Doctor Who Christmas special. "I would really love to do a film," he says. "I've done a couple of Woody Allens, and I was in a film called Plunkett & Macleane that Working Title made... Erm, sorry, I'm going to rephrase that. I'd really love to be in a good film."

In the meantime he's starring in a new ITV drama, Love Life, playing a bathroom showroom boss who gets one of his employees (played by Andrea Lowe) pregnant. It also stars Rob James-Collier from Downton Abbey – although Armstrong hadn't watched Downton while they were filming Love Life, and only recently caught up with the box-set of series one while nursing his children through a bout of flu.

His marriage to Hannah, and his relationship with his three sons, is an obvious source of joy, and one of the reasons he's happy with the regular hours (and income) provided by hosting Pointless. "We met through a mutual friend," he says of Hannah. "She was actually being lined up for this friend of mine. I remember thinking, 'How could someone be so beautiful?'. It's all marvellous... marvellous."

The family has just moved into a new house in "W10... Ladbroke Grove... the Oxford Gardens end, sort of Latimer Road... Wood Lane" – oddly precise directions until you realise that perhaps he's trying not to mention the 'N' word. And if Armstrong lives now on the fringes of Notting Hill, he's also in the outer orbit of the Notting Hill Set, as the circle of acquaintances surrounding Prime Minister David Cameron and his Chancellor of the Exchequer, George Osborne, became known (neither politicians live there now).

"George Osborne's wife, Frances, is the sister of a great friend of ours," he says. "And I've met David Cameron since I played him," (twice – in the Channel 4 satires The Trial of Tony Blair and Hacks). "He was very friendly." In fact, Armstrong remains supportive of Cameron the PM. "I think he's an honest person," says Armstrong, "although once they become important they have to put up the carapace that's going to work with John Humphreys or Paxman. They all become glib somehow.

"I'm a bit of a floating voter actually – I've voted for all parties. Our family always voted Lib Dem – our MP is Alan Beith (MP for Berwick-upon-Tweed since 1973) who's brilliant, so our background is Lib Dem. I'm not greatly impressed by party politics but I am by individual people. I'm a centrist, and very suspicious of any tribalism."

Armstrong also supports the rural campaigning group, the Countryside Alliance. "I shoot very occasionally... I grew up shooting... but because I do shoot I think it's important to stand up for it," he says. "And I'd like people to be honest about what they don't like about country sports, because if it's actually the people you don't like then I'd much rather they would actually just say that."

It's perhaps understandable then that he's friends with keen countrywoman and former Two Fat Ladies chef Clarissa Dickson Wright – although he is happy to scotch the regularly reported 'fact' that Dickson Wright is a cousin. "This is hilarious," he says. "My uncle by marriage – his half-sister is the sister-in-law of Clarissa. I am more closely related to you than I am to Clarissa." It is true, however, that she did go on his stag do in 2003 to help the stags ice the wedding cake. "We thought it might be fun – and it was. But we did do other things, too. We arranged flowers and everything."

'Love Life' begins on ITV1 on Thursday