'Hollywood never suited me': Elizabeth McGovern on fleeing LA and Downton Abbey’s Lady Cora

To morning coffee with Lady Cora, chatelaine of Downton Abbey. Or rather, because I'm not one of those sad sacks who confuses an actor with their TV character (happens to soap stars all the time, I'm reliably informed), it's one cappuccino (for her) and one latte (for me) with Elizabeth McGovern, the LA-raised actress who plays the American heiress wife of Hugh Bonneville's the Earl of Grantham in ITV1's recent hit Sunday night costume drama. She's looking very fetching this snow-swirling slushy morn in a grey beanie hat and carmine lipstick that accentuates the almost vampire-like pallor of her skin. In fact, McGovern looks altogether très chic in a snug, layered woollen way – although she wouldn't get past the tradesmen's entrance at Downton.

Cor, Downton Abbey, now there's a thing. Mocked by pedants for its anachronistic dialogue, unlikely master-servant informality and the odd gatecrashing TV aerial, Julian Fellowes' lords-and-footmen saga swept doubters before it and gave ITV its biggest rating success since, well, The X Factor, the Simon Cowell talent show juggernaut that directly preceded it. And I have to admit that it was Downton Abbey's proximity to The X Factor that fuelled my own misgivings. Was the same audience that embraced the likes of Cher, One Direction and Matt going to give a hoot about the fictional upstairs/downstairs business of an Edwardian stately home? Isn't the age of deference as dead as Wagner's chances of staging a comeback?

"It could have gone either way," admits McGovern, once she's shaken off the photographer and come inside into the warm. "It could have felt like a slow, completely deflated piece of work after the hype of The X Factor, but I think what it proves is that there is a hunger for this sort of story. I think it's less about class and more to do with the fascination of throwing a group of people into a very contained situation and seeing how they bounce off one another."

A bit like I'm a Celebrity... or Big Brother then? "Yes, but when there's a great mind giving you the writing rather than what you get from the hodgepodge of Big Brother, I think it's something that lifts people's spirit rather than drains it."

It has certainly lifted McGovern's profile, although not so far that people are stopping her in her local Sainsbury's. "I have such a different look on the show," she says. The actress has said she "auditioned/begged" for the role, whereas you might imagine that as an American living in England, on tap, as it were, and very much like Lady Cora, she might have had the casting director beating a path to her home in west London. "No, much to my annoyance," she says. "They're much more excited about flying somebody over, even when they've got somebody on their doorstep in Chiswick."

A second series of Downton Abbey starts filming in February, with McGovern's involvement ("as far as I know"), but in the meantime she has been elevated within the hierarchy of the peerage to play yet another ennobled American, the Duchess of Windsor, in a new Radio 4 play to be transmitted over Christmas. Rose Tremain's The Darkness of Wallis Simpson imagines the woman for whom King Edward V111 abdicated, as a 79-year-old, 14 years after Edward's death and bedridden in Paris, being attended to by her formidable legal guardian Maitre Suzanne Blum (played here by Miriam Margolyes) – an abusive relationship that involved much wrist-slapping of the duchess if Tremain's script is to be believed.

"Yes, I'm afraid so," says McGovern. "In some ways she's very much a prisoner at the end of her life." The play pivots upon a single dramatic conceit – that Wallis, now approaching death, has forgotten every single thing about Edward. Other moments in her life she can vividly recall, but the world-shaking events at the heart of it are lost to her, apparently forever. The Duchess of Windsor is of course very much in vogue, what with the recent sale of her jewellery and Madonna's upcoming biopic. As an American married to a Briton, McGovern feels some sympathy for the traditionally demonised divorcee.

"I read the same sympathetic biography of Wallis as Rose Tremain and I was really exercised when reading it how badly she had been treated by history and by the press," says McGovern, adding that the disregard for the personal feelings of the British monarchy ("although it's probably changed now") is one of the few things that still make her feel like an American."Otherwise I'm totally assimilated," she says. Any advice for Kate Middleton? "Get out while you can."

Born into an academic family in Illinois in 1961, McGovern moved to Los Angeles when she was 10 and her father took up a law professorship at UCLA. She started acting while at high school in North Hollywood, which, I say, sounds very Beverly Hills 90210, but she says, was very different. "North Hollywood isn't actually Hollywood, it's in the San Fernando Valley... it's not the most glamorous part of LA."

She studied acting at the American Conservatory School in San Francisco, and then at the prestigious Juilliard School in New York City, before being picked up by an agent. "That sort of morphed into a career," she says – a career with stellar beginnings. Having won the part of Timothy Hutton's girlfriend in Robert Redford's Oscar-winning Ordinary People in 1980, she was herself nominated for a Best Supporting Actress Oscar the following year for her portrayal of the early-20th-century actress Evelyn Nesbit in Ragtime.

"It's a blur, to be honest," she says (she says "to be honest" a lot). "Much of it was the shock anybody goes through leaving home, but exacerbated by this heightened experience of attending the Academy Awards... I was a bit shell-shocked for many years, to be honest."

And to cap it all, by the age of 22, McGovern found herself engaged to be married... to Sean Penn. They co-starred on the "rather sweet little movie" Racing with the Moon. How close did they get to the altar? "Not that close," she says. "I don't think I ever expected we'd get married, I have to be honest with you. We were so young – I was 22 and he was 23."

So who dumped who? "There was a lot of back and forth as I recall. I think basically my feeling about it was it was an experience I was going to see through to its logical end, which is kind of what happened. I don't remember what happened exactly, but it was kind of a natural sequence of events." Penn married Madonna the following year. Was it on the rebound from McGovern? "I wouldn't put it that way exactly, no."

Despite playing Robert De Niro's girlfriend in Sergio Leone's gangster epic Once Upon a Time in America, Mickey Rourke's girlfriend in Walter Hill's Johnny Handsome, and Brad Pitt's girlfriend in The Favor ("he seemed like a very sweet man, but he didn't strike me as being the overwhelming sex magnet that he's become"), McGovern spent the rest of the 1990s putting distance between herself and the life of a Hollywood A-lister. "I was bit overwhelmed by it, to be honest, I did a lot of turning down. The way it works in Hollywood is that if you're hot it doesn't matter if you're right for the part or not, you're just offered it. And I really didn't listen to anyone's advice. I just went my own way, which isn't always the right thing to do, but, for better or for worse, that's what I did.

"I did a lot of regional theatre and a lot of theatre in New York, which has become more common now, with people going back and forth, but in those days it was unusual. But for some reason – I don't really understand why – I was absolutely determined to do it. And I'm glad I did because it's given me a view of my profession as a craft rather than a sort of cult of personality that it can easily become.

"Hollywood never suited me, I didn't ever feel comfortable with it, it took me a couple of years but I found where I was always meant to be... Chiswick!" she laughs (she has a good laugh). "That's it... Chiswick High Road, that was where I was always meant to be."

The long road from Hollywood to Chiswick began when she met the English director Simon Curtis, of Cranford fame, while he was producing a BBC version of Christopher Hampton's Tales from Hollywood. At this point in the interview, Curtis – once described by McGovern, apropos her love of backpacking, as "an amusing, lovely but sedentary and podgy Jewish guy who has to be coerced into walking somewhere for lunch" – himself makes an uncannily-timed appearance in the brasserie to see if his wife has finished with me. She hasn't, and goes on to tell me of their courtship, McGovern inviting Curtis to a Bonnie Raitt concert. "It was a feeling of comfort and rightness," she says of their mutual attraction. "But it was a shock to me because I had a pregnancy quite quickly... and giving up my country and my career."

And that's how she came to be inhabiting a "comfortably eclectic" house in Chiswick, with Curtis and their two daughters, Matilda, aged 16, and 12-year-old Grace. "They get frustrated with me because I'm not particularly organised and I think they think I'm a bit... stupid," she laughs. "But they think I'm fun and I think they're fun.

"Chiswick is the place I've been looking for my whole life," she says, and she means it. Neighbours include Colin Firth, with whom McGovern did a play at the Donmar in 2008, and Firth's Italian-born wife Livia, who runs an eco-design shop on Chiswick High Street, in which I have never seen anybody. And I do a lot of shopping in Chiswick. "They've got two kids, and we're neighbours, but it's like with most people raising kids, having jobs, we don't really hang out that much because we're too busy."

A heightened glimpse of what life might be like chez McGovern-Curtis was provided by the BBC sitcom Freezing, which Curtis directed and co-produced and in which McGovern and future Downton Abbey co-star Hugh Bonneville first played at being husband and wife. Scripted by James Wood, who went on to write this year's well-regarded BBC2 sitcom, Rev, Freezing was an unusually acute (and very funny) adult comedy of manners that was none the less disliked in some quarters for its showbizzy insiderishness. The fact that these same critics don't complain about 30 Rock suggests there may have been a class bias going on as well.

"I was surprised how many viewers embraced it," says McGovern. "A lot of people told me they could relate to the aspect of Freezing that was about being middle-aged in today's world. I was joking with Simon the other day that we should do a new series of it, but this time call it Hot because of Downton Abbey, and me and Hugh being 'in'."

On the subject of middle age, it's not often that a person waits until their forties to launch a musical career. And that, rather amazingly, is what McGovern has done, becoming the singer-songwriter with the acoustic band Sadie and the Hotheads. The Hotheads include The Nelson Brothers, musicians and producers who instructed McGovern in the art of song writing, and who brought in Ron Knights on bass and borrowed Goldfrapp's drummer, Rowan Oliver, to record a CD. There are three tracks from it on MySpace if you want to listen to them. They're quite good really – Carole King without the angst – and not what you expect from someone called Sadie.

"I write about being a middle-aged, well-adjusted, middle-class woman," says McGovern. "I've just finished a rock'n'roll song about somebody at the school gates waiting for their kids to come out and I just love the juxtaposition of that subject-matter with that music... why not?"

Why not, indeed; we've had dad rock, why not mum rock as well – although I don't see Simon Cowell having a Sadie and the Hotheads night on The X Factor. In the meantime, McGovern is shooting a low-

budget film based on a novella by Lytton Strachey's niece, Julia Strachey, Cheerful Weather for the Wedding – about a girl who thinks she might have made a mistake on the morning of her wedding day. Her co-stars are Mackenzie Crook, Felicity Jones and John Standing – and it's this project she mentions when I ask her whether she now feels fully assimilated into Britain. "It's really my dream come true – I feel part of that community. I never thought it would happen... it's unbelievably gratifying for me; I'm so happy."

But with Downton Abbey starting in the States in January, and no doubt proving as popular there as it was here, would she take a second bite at Hollywood if producers suddenly remembered that actress who was big in the Eighties, and they started picking up the phone? "I wouldn't say no. Sure," she says. "But I'd be happy not to. I'm really happy here."

'The Darkness of Wallis Simpson' is on Radio 4 at 2.15pm on Tuesday 28 December