Looking after No.1: Jeffrey Archer on David Cameron, 'Fifty Shades of Grey' - and his past

The sentiments are very Jeffrey Archer. There is the precise, slightly self-congratulatory nod to Corlies Smith, who not only edited Salinger and discovered Thomas Pynchon, but unleashed Kane and Abel on the world. There's a keen awareness of an audience and what makes them tick. And as happens throughout our conversation, there is the eerie sense we are also talking allegorically about Archer's entire life: a succession of triumphs, disasters and comebacks that would try the patience of even his own fictional universe.

In the past decade, this seemingly inexhaustible cycle of booms and busts has slowed. It is now nearly 10 years since Archer was released from prison, having been found guilty of perjury and conspiracy to pervert the course of justice. "It has been very much more relaxed, not that I can ever relax," he says. Apart from writing, he has business interests in art and theatre, and extensive charity work. "I have raised £15m as an auctioneer. But it has been a very quiet stage."

Others have filled his shoes in various areas of public life. There are, he is the first to admit, bigger and better writers: Dan Brown and JK Rowling receive honourable mentions, but his current fixation is EL James. A new generation of politicians has captured the public attention as Archer once did in Margaret Thatcher and John Major's governments. Even his impressive record as a scandal magnet has been usurped.

We talk before Chris Huhne and Vicky Pryce were sentenced for perverting the course of justice. Their downfall contains echoes of Archer's own fall from grace – although his own wife, the famously "fragrant" Mary, has remained loyal throughout his troubles.

I was advised before the meeting that Archer would not speak about the case. An enquiry after Huhne and Pryce's sentencing was also met with "no comment". Yet, one of my questions – about the sort of reception Huhne could expect in prison – might have been answered some years before, when I interviewed Archer following his own release from jail: "In prison, I never met any boring middle-class people. These people were murderers, people on drugs, these people had five wives and 18 children. What prison gave me, aged 61, was 1,000 new characters I would never have met, and a world I never would have touched. I just sat there absolutely mind-boggled."

Archer may have withdrawn from the limelight in recent years. But such is the diversity of his achievements that it has required an entire rogues' gallery of bestselling writers, crooked politicians and tabloid favourites to replace him on the public stage. It's enough to make you chant, "There's only one Jeffrey Archer."

We talk at his Thameside penthouse overlooking the Houses of Parliament. It is a symbol of his success, which Archer does not take lightly. "Every day I look at that view, I actually stand up and say, 'Being a simple storyteller gets you this?' That's all I am. I'm not a writer. I am a storyteller. Every day I'm aware how lucky I am," he says, beating his chair in time.

Examples from Archer's art collection adorn every wall. Tables are packed with photographs: Archer with Princess Diana, playing cricket with Ian Botham and Viv Richards, talking to Lyndon B Johnson. Alongside many family portraits is one of Margaret Thatcher signed to Mary. Has he seen the former PM recently? "Yes. She's not well. She doesn't know who I am, or Mary. She adores Mary. They are both chemists and were taught by the same woman."

Drinking from a mug emblazoned with "Lordship" and looking a few years younger than his 72 years, Archer is almost aggressively good company. By turns charming and combative, he remembers my name and uses it often. On other occasions, he is less friendly, albeit with a gleam in his eye. He mentions his new-found superstardom in India. I ask how the reception compares with England. "They are very much more serious than idiots like you. You are more interested in the past. Finding some excuse to raise some subject that's irrelevant. They're not. They are fascinated by the process of writing."

India seems to be experiencing something like Archer-mania. On a recent tour, he appeared in front of crowds of up to 3,000. "They go bonkers," he says. "Why are 14-year-old girls screaming as if you're a pop star? When it first happened, I burst into tears. Very strange reaction. I've never been able to explain that. It doesn't happen now."

A compelling drinking game could be constructed from Archer's repeated motifs and phrases. "I can't explain that" would get you pleasantly sloshed, as would "Storyteller" or panegyrics to his wife, Mary. I ask why Archer thinks people are so fascinated by their marriage. This prompts questions of his own. How old am I? How long have I been married?

"You don't have any opinion at all. When you have been married 20 years – I have now been married for 46 – everyone goes through problems. Don't pretend it's me. Others wonder, 'How did they survive when we didn't?' We survived because I am never not fascinated by Mary, and have total regard and respect for her. We bring something to each other the whole time. We are very aware of it." Does he mind my asking about his private life? "That's your job. It's your problem if you didn't get a proper job. As your mother no doubt told you many times."

Of course, it's Archer's job that we are really here to discuss. His new novel is Best Kept Secret – another punning phrase that seems to toy with Archer's colourful past. Indeed, certain scenes echo his own alleged misdemeanours: one character dallies with a prostitute; there are hints of insider trading, albeit for the greater good. But as the panegyric to Mary implies, this is an eminently readable page-turner about heroic men and strong women who are smarter and occasionally more deadly than the males.

riting, like interviews, seems akin to bare-knuckle boxing for Archer. I begin a question about his literary peers. "Competitors," he corrects me gleefully. "You look at EL James or Dan Brown and ask, 'What am I not doing? Why are my figures less than theirs?' JK Rowling I understand, because Harry Potter is a pretty strange thing. But I don't understand EL James. I am told by people who know that there are a lot of erotica writers who haven't had an amazing breakthrough."

Archer tried Fifty Shades of Grey, but quit after 30 pages. "I am told the sex doesn't come until after page 100. I never got there." Did James's success ever tempt him to enhance the sexual content of his own work? "I don't do erotica. I don't do sex, really. I am not going to change my style. I tell stories. You ain't getting any violence, sex or bad language." I point out that there is a single "fuck" towards the end of Best Kept Secret. "That was very bad. That would have been the first thing my mother, no longer with us, would have said, Not necessary."

In certain respects, a writer's life suits Archer. It fulfils the image of the self-made man who relies on his own resources. "Mary has always kindly said I have a first-class mind. But I have always had to fight. Though Mary says – and I accept it – that it's an advantage to be disadvantaged."

Archer is the first to admit that he is not a natural-born novelist. His book-a-year output is the result of a prodigiously strict writing schedule. Working in concentrated two-hour blocks, he writes by hand at his home in Majorca. "6am to 8, 10 to 12, 2 to 4, 6 to 8. Bed at 10. Head down. Get on with it." Archer mimes a prayer. "Can I have another page, please? Can we go again today or is it over?"

If it is difficult to imagine Archer the social butterfly existing in this splendid isolation, it may be because one mistakes his creative act for self-expression. Writing for Archer is all about self-control. "The rule is that no one can speak to me until dinner. If I talk, that's fine. A bit cruel. But if something is going on in my mind, I don't want it interrupted."

What happens if someone does interrupt? "I have been interrupted twice. Once 20 years ago by Mary. I was about to kill her. She said, 'I thought you might like to know that James has got three A-pluses and will be going to Oxford.' That was all right. The second one was the death of a friend and I had to fly straight back."

Despite the discipline and financial success of his work, Archer still sounds like a gentleman author – a novelist by accident. I ask at one point whether he would have given up writing if his early books had not sold. "Yes, certainly. I'd have become a barrow boy. I wasn't going to do 15 books with 3,000 people reading. No thank you."

rcher's pastimes have brought him far greater success than any day-job. His art collection, including Picassos, Hockneys and Warhols, is worth tens of millions. Investments in the theatre have proved lucrative, if erratically so. "I have got The Bodyguard, which is making £100,000 a week. I had Flashdance last year and lost every penny. It's fun. I love the theatre. An industrialist told me that if everyone's hobby had made half a per cent every year, they'd be very happy. I don't have a plane, a yacht or a football club."

Of course, Archer's day-job was always meant to be politics. "I was a failure," he says briskly. "I never got anywhere. I never achieved. I am very proud of the election results we had with Margaret Thatcher and John Major. We stayed in power for 13 years, and I played a minor, minor, minor, minor part in that. But I haven't had a successful political career."

Nevertheless, it is noticeable how Archer's tone brightens whenever politics comes up. He speaks with particular enthusiasm about Michael Gove ("A triumph for this government. He is clearly a formidable man") and Boris Johnson: "Do I want him as prime minister? No. Do I think he's brilliant at what he does? Yes."

When conversation moves to the Eastleigh by-election, he positively fizzes over with memories of battles past. "I love by-elections. Love them. Me versus Mandelson. That was terrific. Get the knives out, kid. Let's get going." A worthy adversary? "More than worthy. He was the best I have seen at a by-election. From the moment you got out of bed he was doing something. You had to watch him. So I had to do something to make sure he watched me."

Archer declares himself surprised and grudgingly impressed by Ukip. "Farage is no fool. What I underestimated is the Romanians and the Bulgarians filling up my wife's hospital, taking our jobs. Ukip cashed in on that. It was a clever cash-in because they weren't discussing black people. It was genuinely a fair statement. If the vote had been a week later, I think Ukip would have won."

Archer argues that the Conservatives' third place is not the disaster some are making it out to be. "David Davies is clearly up to Machiavellian politics. When he said, 'If we come third it's a crisis,' he wanted us to come third. He still wants to lead the Conservative Party." At the same time, there are causes for concern. "If the Liberals had come second, we could both say, 'We are the governing party. But the Liberals won! And it's Hampshire!'"

Archer's advice to David Cameron is to split from Nick Clegg before he does a runner. "We cannot sit and wait for Clegg to decide when he will break from us. We must find the right issues, kick him into the long grass and call an election." Archer's bet is that Cameron won't take the lead, and Clegg will announce the divorce at next year's Liberal Party conference. 'It's tough being prime minister. Cameron is a nice human being and a gentleman. He will probably sit and wait to be kicked in the goolies whereas he ought to be kicking Clegg in the goolies first."

Archer's own future involves more of the same: more books to write, more money to raise and most importantly more readers to reach. "An MP stopped me in the House yesterday and asked, 'Are you still writing?' I wasn't cross. It makes you realise how many millions out there don't even know you are a writer. Let's get them as well."

Before he heads to lunch, I ask whether he has any regrets. He looks unusually sombre. "It will make you laugh. It's that I don't have any daughters. I wanted daughters more than anything. I wanted to take a generation of women though to the next era. Mary's father believed in women's rights. On his death bed he made me promise I would never stop her doing what she wanted. I made that deal with him. I would have liked two daughters, three daughters… to have seen what they could do."

'Best Kept Secret' is published by Macmillan, priced £20