The Tonight Show - for one more night only...

It isn't so much a changing of the guard, as American broadcasting's version of a royal abdication. Tonight, after 17 years and more than 3,800 episodes, James Douglas Muir Leno, the stand-up comedian better known as "Jay", will hand over his crown as the undisputed king of late-night chat.

Shortly after 11.30pm, from behind the faux-walnut desk that has always been his throne, Leno will introduce viewers to Conan O'Brien, a 46-year-old performer who has been given the considerable task of safeguarding the grand old national institution that is The Tonight Show.

"I have something really unusual planned, something really out of left field that we're going to end on," Leno informed his seven million viewers, who have helped maintain the programme's status as a cash cow, generating most of NBC's estimated $300m (£187m) in late-night revenue.

It had better turn out to be good. The Tonight Show is a cultural as well as financial powerhouse which has for generations boasted an uncanny knack of shaping the news agenda. Its host is a household name. O'Brien will be just the fifth man to take the hot seat in the show's 55-year history.

Leno leaves behind a franchise that has amused, entertained, and informed in roughly equal measure. He swapped anecdotes with pop stars and talked "scoops" out of some of the biggest names in Hollywood. Recently, a serving President of the United States sat down on his famous leather sofa.

The frontman also leaves a legacy of extraordinary commercial success, writ large in his reputed annual salary of $27m. When he took over from Johnny Carson, who presided over The Tonight Show for three decades until 1992, Leno trailed in the ratings behind David Letterman. Today, he boasts some seven million viewers, to his rival's five.

The success, which today sees Tonight syndicated to 15 countries, is all about comic timing. Leno is a gentle interviewer in the Parkinson mode but is beloved of audiences around the world for his comic monologues. He kicks off each show with a tirade of well-timed zingers.

Throughout the 1990s, he scored a string of scoops. His famous interview with a puce-faced Hugh Grant, who had just been caught consorting with a prostitute, was one of the most famous moments in 1990s television, and marked the point when his audience share first outstripped that of Letterman.

This week, Leno proved that he still had a story-getter's instinct, when Mel Gibson appeared on his show and was persuaded to reveal that his new girlfriend, a young Russian singer called Oksana, who is reputed to be at the centre of his $800m divorce battle, is pregnant.

Leno's disappearance from the slot therefore puts a very valuable market right back in play. O'Brien has won Emmys in NBC's 12.30am "graveyard" slot but his viewing figures are unspectacular.

When the new Tonight Show launches on Monday, Letterman, together with the third big beast of the lucrative late-night market, ABC's Jimmy Kimmel, will be hungrily circling his audience, looking for disaffected viewers in search of an alternative.

O'Brien's first week of guests seems promising: it includes Tom Hanks, Will Ferrell, Pearl Jam, Green Day and Ryan Seacrest. But even he is unwilling to predict that he will keep hold of Leno's audience.

"Nobody has the answer," O'Brien told reporters this week, adding that he intended to avoid "overthinking" or tinkering too much with the show's format. "I can try to bullshit you and say it's all going to be fine, but I don't know. This move is unprecedented in TV."

The apparent lack of confidence may leave you wondering why NBC, the otherwise-ailing US television network which controls both Leno and O'Brien, didn't simply keep the two men in their previous jobs. Circumstances make that almost impossible: in 2004, when Leno, now 59, signed his most recent five-year contract extension, he publicly predicted that by 2009 he would be ready for retirement.

Anxious to ease the succession crisis, NBC rushed to sign O'Brien as his replacement. To ensure he joined, the network promised that O'Brien would take over The Tonight Show by this year – and agreed to pay him $45m if he didn't. When the deadline approached, it turned out that Leno – a workaholic, whose life revolves around his show and his classic car collection which he keeps near his Burbank studio – was reluctant to shuffle off into retirement. So to prevent him leaving for a rival channel, NBC have offered him a new show.

That programme, which launches in September, will occupy the prime-time slot from 10pm, five nights a week, and will see the comedian return to his roots as a traditional stand-up performer.

"We'll give it a chance and see what happens," Leno said, describing the opportunity to do the new show at this stage in his career as "pretty much gravy".

As for his old job, he added with characteristic understatement that he's happy simply to have avoided ruining it. "The Tonight Show was sort of like the America's Cup. You don't want to be the guy who screws it up," he said.

"So when I hand it off to Conan, it'll be like, 'Here you go. I left it in exactly the same shape.' Like when you bring the rental car back and there are no dents in it."

'The Tonight Show': Leno's greatest moments

Barack Obama

"Like the Special Olympics"

Shattering convention shortly after taking office, Obama became the first serving President to appear on a late-night chat show. Leno asks him if he's found time, in between basketball games, to check out the White House's famous ten-pin bowling alley. The President replies with a self-deprecating joke. "I've been practising... I bowled a 129... It was like the Special Olympics, or something!" The studio audience laughs politely. But America's disabled athletes are not amused. Within hours, Obama is forced to issue a grovelling apology.

Hugh Grant

"What were you thinking?"

June, 1995. Grant, the blue-eyed boy wonder of British cinema, is caught in flagrante with Divine Brown on Sunset Boulevard, having apparently paid her $50 for oral sex. A few days later, he rocks up on The Tonight Show. "What the hell were you thinking?" asks Leno. "I'm not one to go around blowing my own trumpet," mumbles Grant, to uproarious laughter, before apologising to his girlfriend Liz Hurley.

Michael Jackson

"Fondling Neverland"

Having subpoenaed Leno to give evidence at his 2005 molestation trial, Michael Jackson had his lawyers persuade the judge to forbid Leno from telling jokes about it. Leno responds by arriving at his studios in pyjamas and exploits a loophole to hire comedians such as Roseanne Barr, Drew Carey and Dennis Miller, to tell jokes for him. "Let's see what's new in the Michael Jackson trial, or as it's now being called, 'Fondling Neverland'," goes one zinger. "What does Michael Jackson call a school bus?" goes another. "Meals on wheels."

Arnold Schwarzenegger

"I'm an outsider"

He'd dropped the odd hint but even Schwarzenegger's inner circle had no idea he intended to use his appearance on Leno in 2003 to announce his candidacy for Governor of California. "It is my duty to jump in the race," he reveals, to gasps from the audience. "I'm the most unique candidate because I'm an outsider." A Leno campaign launch has a mixed chance of success. John McCain also used the show to announce his intention to run for President. We know how that ended.

OJ Simpson

"The Dancing Itos"

With the "trial of the century" transfixing America, Leno employs a troupe of dancers made to look like the flamboyant judge, Lance Ito, to perform a song and dance routine. Their act, which runs each evening throughout the court case, becomes a cult hit, satirising Ito's alleged showboating, which some blamed for OJ Simpson's acquittal on charges of murdering his wife and her lover.

The Cheers Finale

"This is known as losing control"

To mark 1993's final episode of the sitcom Cheers, Leno records an episode from the bar in Boston on which the show is based. The entire cast turns up and get uproariously drunk, shouting, giggling and slurring their way through interviews before beginning a raucous sing-song. "This is known as losing control," announces Leno.