Watch and learn: BBC Four's success is a sign that Britain is regaining a hunger for intelligent broadcasting

"Everyone Needs a Place to Think". That was the slogan under which BBC Four was born. In TV trailers ahead of the channel's launch, the message was set to intriguing footage of Lord Byron's Villa Deodati on the shore of Lake Geneva, the Antioch Baptist Church in Cleveland, Ohio, and sweeping views of the Chiltern Hills.

These locations were the inspirations for Mary Shelley's Frankenstein, for Jesse Owens' supreme athleticism at the 1936 Berlin Olympics, and for Ian McEwan's novel Atonement. And then from 2 March 2002, we had our own spot to sit and ruminate, in front of the BBC's first digital channel.

This was a time when intellectualism was in full retreat, in the face of the rise of celebrity culture. The words "dumbing down" were on everybody's lips, especially in conversations about television. Channel 4 was preparing to launch its third and most successful series of Big Brother, a show that redefined the network that was born two decades earlier with Walter, a demanding Ian McKellen film about the life of a disabled man, as the prime-time treat in its launch-day schedule. BBC Two, once the unquestioned home of high-brow entertainment from the Corporation, was being led in a pursuit of ratings by its ambitious controller Jane Root, who introduced lifestyle programming such as Top Gear and What Not to Wear.

If there was a hunger for something more serious, BBC Four was intended to be a slap-up feast. When the channel launched, The Independent's reviewer, Robert Hanks, was instantly infatuated. "I'm not sure the earth didn't move for me," he wrote of a network that opened with a triptych of films about artists (a profile of Michael Landy, Robert Hughes on Goya, a Salvador Dali drama starring Ewen Bremner and Stephen Fry). "Every day on television should be like this."

But what has happened to BBC Four in the eight years since? Sadly, it remains a minority channel, taking a 1 per cent share of the total audience and being sampled by slightly less than 14 per cent of television viewers during the course of a week. Only seven BBC Four programmes, almost all of them drama biopics, have ever pulled in ratings of more than 1 million.

This could be a problem in the current climate, where the prospect of an incoming Conservative government brings with it the likelihood that the BBC will have to rein in its ambitions. Jeremy Hunt, the shadow culture secretary, told the Royal Television Society's Cambridge Convention last September that BBC Four, and the more youthful BBC Three, needed to do more to justify their existence. "I think for some of the new channels, BBC Three and BBC Four, they have very small audiences but still cost a lot of money," he said. "The case needs to be made for this kind of thing." Two months later, the BBC director-general, Mark Thompson, who is conducting a strategic review of the corporation's operations amid widespread questioning of whether it provides licence fee-payers with value for money, indicated that he was looking at cutting some of the organisation's digital broadcasting services after the analogue signal is switched off in 2012.

Both Thompson and the Tories should think very carefully before taking the knife to the BBC's most cerebral television network. For a growing number of viewers (the audience share has gone up 500 per cent since 2002 and the number of weekly viewers has tripled to 7.2 million in that period) BBC Four is their refuge, the place on the electronic programme guide (EPG) to which they head in search of intelligent television, whether it be Paul Morley's introduction to the music of Brian Eno, the art historian Gus Casely-Hayford's Lost Kingdoms of Africa or the Shooting the War series based on black-and-white footage from German and British home movie-makers.

It is perhaps not fully appreciated that this is the channel that introduced British television viewers to Larry David's Curb Your Enthusiasm, created its own horrifyingly authentic political satire The Thick of It, and first screened the cult Kiwi music comedy Flight of the Conchords. It has given a platform to the acting talents of John Hurt, Jane Horrocks, Michael Sheen and Helena Bonham Carter, and acquired for a snip Mad Men, the ultra-glamorous (and much garlanded) dramatisation of the culture of New York advertising in the Fifties; a hit which was missed by the rest of British broadcasting.

The channel has given opportunities to presenters new and old, such as Paul Merton, who made a four-part series Silent Clowns on the genius of Buster Keaton, Charlie Chaplin, Harold Lloyd and Laurel & Hardy. "I did 35 minutes of documentary about them and the last third of the programme was showing a silent film in its entirety. Some of the films were 100 years old. That's extraordinary that BBC Four did that."

Merton, who has made a programme about pre-First World War European cinema, which will be shown on the channel next month, was also trusted to direct a film on Alfred Hitchcock, in spite of his limited experience. "It's a great channel to work for," he says. "BBC Four has become the TV equivalent of Radio 4. You expect the documentaries to be pitched at a certain level and the people making the programmes to think that they're being watched by an intelligent audience." By contrast, he is mystified by its more youthful but less distinctive sister digital channel: "What BBC Three is for, I don't know."

Roly Keating, now the head of BBC Archive, was the launch controller of BBC Four, which emerged from the prototype digital channel BBC Knowledge. "We had really high ambitions. We wanted seriousness, idealism and staying power – something that would be visibly different in the television landscape," he says. "We debated back and forth about how to position it."

In its first year, BBC Four showed its ambition by adapting for television Michael Frayn's stage play Copenhagen and delighted JG Ballard with its dramatisation of his novel Home. Controversially, it screened The Falklands Play, which had been shelved by the BBC for 16 years after Michael Grade's decision that its sympathetic stance to Margaret Thatcher could influence voters. The Ian Curteis production, starring Patricia Hodge as Baroness Thatcher, attracted a record BBC Four audience of 174,000, a welcome number for a channel then facing criticism that less than 5,000 were tuning in to some of its shows.

With most British households still having access to only the terrestrial channels, the BBC enraged some viewers by spending licence-fee money on high-quality productions that could be received by only a minority. So BBC Four's first great breakthrough commission, The Alan Clark Diaries, starring John Hurt as the rakish former Tory minister, was well received critically but generated anger because of its limited availability. "I had to defend why the BBC had dared to put something so interesting on this channel and say 'Because we commissioned it for this channel'," says Keating. "We were on the cutting edge of the controversy the BBC always gets when it moves into new media forms and gets complaints that some people don't have access."

He is not unhappy to relate that he clashed with BBC executives over the wisdom of commissioning Curb Your Enthusiasm. "Some colleagues inside the BBC were saying it was too niche and that it would never find a British audience. I said this is not a mass audience channel but it is a channel where we want to recognise brilliant creativity."

BBC Four quickly saw value in scheduling credible music programmes on Friday evenings, notably the Britannia series which has explored the origins of often ignored genres of British music including jazz and folk. "It has become part of the landscape of telly that there will be something for music lovers at the end of the week on BBC Four," says Keating.

Documentaries have been at the heart of the offering from the outset, the channel becoming home to the Storyville strand edited by Nick Fraser. The channel has screened more than 500 Storyville documentaries, with the most recent being this month's Last White Man Standing, on the trial of Tom Cholmondeley, a wealthy Kenyan landowner charged with killing a poacher on his land. Fraser worries that BBC Four doesn't do enough to promote its wares. "It's sophisticated and niche, without being obscure. It's a real national asset, a small one but not all assets have to be large," he says. "They should put more money into marketing, it's hidden under a bushel."

Storyville has changed subtly according to the interests of each BBC Four controller. Keating, who Fraser describes as "a very creative magpie person with the skills of a cultural bureaucrat", wanted eclectic and provocative films. Keating's successor Janice Hadlow, who took the top job at BBC Four in 2004, developed the channel's reputation for history coverage.

Now promoted to controller of BBC Two, she raised the profile of the digital channel by staging a succession of history-based seasons, such as The Edwardians, a collection of 18 programmes on such diverse subjects as scouting, music hall and groceries. Presenters included Waldemar Januszczak, Dan Snow, Ian Hislop and Kelvin MacKenzie.

The Medieval Season, which was wittily marketed with a Jimi Hendrix track played on a lute, launched the presenting career of television historian Professor Robert Bartlett. Hadlow is even more proud of the examination of post-war culture that was The Lost Decade, a mix of programmes which she believes was important in building audience loyalty. "We looked at the music of the period, the history, the cookery, the personal experience, mixing the voices and tones into something that I hoped would feel bigger than the sum of its parts," she says, claiming to have identified interest in an under-reported era before films such as An Education highlighted the complexities of the period.

Hadlow's breakthrough drama was 2006's Kenneth Williams biopic Fantabulosa, which won Michael Sheen the RTS best actor award ahead of his film portrayals of David Frost and Brian Clough. "We didn't have very much money, but I always felt that if you had a great script and brilliant central performances, people wouldn't mind. They know they're not watching the experience you get from a hum-dinging BBC One costume drama," she says.

Ben Stephenson, the BBC's head of drama commissioning, says BBC Four has "revolutionised" the television genre. "It has made low-budget something to be celebrated rather than be seen as second-best, a creative endeavour rather than a limitation. I think BBC Four drama is our equivalent of theatre and that's why we get extraordinary writers and actors to do it."

Since becoming controller in 2008, Richard Klein has taken BBC Four's use of the biopic to another level entirely. Klein last year oversaw the channel's most successful project, a celebration of three British artistic heroines.

Jane Horrocks whose portrayal of Gracie Fields in Gracie! brought an audience of 1.5 million, was "thrilled" by the part and emails that the channel is "a new and exciting home for quality single drama on television". Helena Bonham Carter, who played Enid Blyton, similarly enjoyed the channel's no-frills approach, says Stephenson. "She said it was one of the best jobs she had done, because it's the equivalent of a stage play. There's not enough money to do massive landscapes so it has to be all about her relationship with the other characters." The season was completed with Anne-Marie Duff's portrayal of Margot Fonteyn.

Klein, who, like the other BBC controllers, occupies a glass box of an office on the sixth floor of Television Centre in west London, is anxious to bring a more contemporary feel to his channel. Divorced and bringing up his eight-year-old daughter alone, he has commissioned a season on modern fatherhood. "I passionately believe in the re-examination of the role of fathers in the 21st century, I think that's a very contemporary piece of work."

The documentary maker Morgan Matthews is making a two-hour film exploring the 50 violent murders of teenagers that took place in Britain in 2009. "It's a brave commission," says Matthews of a film that will be shown on BBC Four at the end of this year. "What we are trying to do is present the real story, which isn't necessarily about gangs. It's lot of different types of violence and some cases that you have never heard of."

Another show which gives the channel a modern relevance is Charlie Brooker's Screenwipe, a caustic and brilliant appraisal of British television. "He brings a ferocity fire and venom as well as outrage to the subject matter which is truly entertaining and shocking sometimes," says the controller.

Klein, who has a German father and an English mother, carries a German passport and spent part of his childhood on a farm that bred New Forest ponies in northern Germany. He is anxious to combat German stereotyping, which is no bad thing in a World Cup year. So he is sending Al Murray (yes, The Pub Landlord) on an exploration of the land that is the butt of much of his humour. "He is learning to speak German and discover what's great about Germany," says Klein. "That's a very BBC Four approach because he – in terms of his alter ego – represents quite a lot of what some British people think about Germany, seeing it through the prism of a pointed helmet and Nazism. But he's a very erudite and intelligent man and he's going to go to Germany and find out about the culture, architecture, visual arts and music. It will be fun as well."

Andrew Graham-Dixon, the channel's art presenter, is also heading to Germany, having already completed a forthcoming series on Russia. Given that his Art of... series has so far taken him to Spain and Italy, viewers might have anticipated him focusing on France. "You might argue that is quite an unusual decision – people would have gone for Art of France or Art of Holland," says Klein. "I chose Germany because it is a country that has had an extraordinary long and fruitful relationship with Britain, until about 90 years ago when it stopped."

The new controller likes to be "counter-intuitive" and "provocative". He has commissioned a "humorous drama" on the American journalist Heather Brooke's attempts to uncover details of expenses claims by British MPs.

A current series which well captures BBC Four's ambition is the The Great Offices of State. Part of the channel's attempt to explain how Britain is really run, it features the great documentary maker Michael Cockerell (who has taken on the mantle of Anthony Sampson), persuading former Whitehall mandarins and Secretaries of State from the Home Office, Foreign Office and Treasury to tear each other's reputations apart on camera, revealing the inner machinations of their departments as they try to define their personal roles in history. The series is part of a season, Corridors of Power, which also includes Getting Our Way, in which former ambassador Sir Christopher Meyer, through his access to fellow diplomats, sheds light on the practices and strategies of British foreign policy.

Does all this amount to value for money? BBC Four has an annual budget of £55m. "We reach a large audience," protests Klein. "Sky Arts is a very good channel but it's playing to audiences of 2,000-3,000. BBC Four plays to audiences of hundreds of thousands. Bizet's Carmen reached 120,000, that's Wembley twice over!"

These are not figures that will appease all the critics. But Klein knows that to broaden the offering risks diluting the channel's identity. "To some extent we are quite clubby," he admits. "We are unashamedly who we are and if you are looking for something that will give you food for thought, we are the channel for you. That's not an age differential, that's not a class differential, it's a way of thinking." Time to join the club.

The New Intellectualism: From podiums to podcasts

The "studious and curious persons" for whom Sir Hans Sloane founded the British Museum two-and-a-half centuries ago have good reason to hope for a new intellectual golden age, propelled by technological advances and aided in no small part by the broadcasting skills of the museum's own director, Neil MacGregor.

In recent weeks, MacGregor's hugely ambitious project to tell the world's history through his description of 100 artefacts – from the label on the sandal of a Pharaoh to a 7,000 year-old Japanese pot – has established itself as one of the modern success stories of British talk radio. Downloads of this Radio 4 series, A History of the World in 100 Objects, are topping the podcast chart, and are more popular than the comedy of Ricky Gervais or the music selections of Radio 1's DJs.

The venture, described by Mark Damazer as the "most exciting history project" in his five years as BBC Radio 4 controller, is symbolic of how digital technology is fuelling a thirst for knowledge that runs counter to the idea that Britain is in cultural decline.

That appetite is also reflected in the growth of the organisation Intelligence Squared, which has restored the tradition of debating in Britain, drawing thousands of people to hear arguments on subjects such as the role of the Catholic church, the future of news, or US President Obama's foreign policy. Set up by John Gordon and Jeremy O'Grady (founder of The Week current affairs magazine), Intelligence Squared has outgrown its original London venue at the Royal Geographical Society and moved to the 2,500-capacity Methodist Central Hall, where hundreds are being turned away from sold-out events that charge £25 to hear clever people talk.

"Our idea was that instead of getting information from a report or opinion column in a newspaper, you could develop the excitement of learning through pitching one heartfelt view against another," says O'Grady. Speakers include Stephen Fry, Martin Amis, US moral philosopher Susan Neiman, Palestinian political leader Saeb Erekat and The Independent's Howard Jacobson. Only one debate, on the subject of hunting, has required the intervention of security staff. The business has developed with the help of a website that now includes a substantial archive of filmed debates, on which users are invited to comment.

Other debate-based organisations have sprung up, such as the younger-skewed Institute of Ideas, while the British publisher Chris Anderson will bring his American-based ideas movement TED (Technology, Entertainment, Design) to Oxford for a global conference in July.

The BBC has, with a fanfare of publicity, woken up to the public hunger for science programming. How Earth Made Us, recently screened on BBC Two, was the most successful science programme on the channel for four years. Four major science commissions will go out in March: Invisible Worlds, Seven Wonders of the Solar System, Brain Test Britain and Bang Goes the Theory.

And though it attracts small audiences, Sky Arts has become a key element in the portfolio of the satellite broadcaster, breaking new ground in the genre by staging live theatre in a purpose-built studio during the summer, and partnering with the Glyndebourne Festival to offer live opera coverage in high definition.

Technological advances offer great opportunities for museums and galleries to make their collections more attractive. More than 100 British museums have signed up to a scheme that allows them to download up to 3,500 hours of Pathé news footage from 1897-1977. "Having archive on a big screen makes your exhibition much more dynamic," says Alastair White, general manager of British Pathé.

So the Dales Countryside Museum has moving images of early farming practices, and the Lakeland Motor Museum screens original film of Thirties cars. Roy Clare, chief executive of the Museums, Libraries and Archives Council, says institutions are transforming themselves from keepers into sharers of historical treasures.

He encourages the public to blog about museum collections, and during this month museum visitors are being asked to take digital images of collections and post them on Wikipedia. "It's suddenly absolutely plain that the real joy is digitised content shared widely," he says. "We have moved on from a situation where collections were interpreted by professionals and the public was only allowed to view. Museum professionals now have the tools to enable the public to participate in the narrative."

MacGregor's 100 Objects series embraces a similar spirit. The pieces he describes in audio can be viewed with a zoom facility on a BBC website, and 13 of them are featured in a spin-off children's television series for CBBC.

"Neil is the portal to making this accessible. He's prepared to explain things without being laborious and he's genuinely enthusiastic," says Philip Sellars, the executive producer of a series that will have impact well beyond the traditional Radio 4 audience. "I cannot imagine this won't be a driver in getting people to go to museums and think about the past."

Ian Burrell