Yasmin Alibhai-Brown: The new MPs I'm glad to see

At 3am on election night – when results were still uncertain – I surrendered to sleep and turbulent dreams. In one scene, Roy Hattersley, in a kilt, was saluting a uniformed and heavily armed Margaret Thatcher. On Friday my husband brought in the coffee as always and asked: "Do you want the good or bad news?" "Both and fast," I replied, half-awake and already hyperventilating. Asthma inhalers are no damn use when it comes to election wheezes. The Lib Dem surge turned out to be a chimera. Encouraged by polls and commentators, their supporters – including me – believed that this time the party would make significant gains and shake up the political duopoly. It didn't happen. Bitter? Yes.

Democracy – that wilful and capricious mistress – once more stymied predictions and deceived pollsters who got rich but got it wrong. But, we did get the hung parliament many of us wanted. David Cameron did not get his big mandate for his "Big Society". And Labour, though no longer New but old and shabby, was brought down but was not annihilated.

Brown had a bad election (much of it his fault but also because sections of the media relentlessly ripped into him as if he was dead meat and they a pack of mad hyenas) but he galvanised many. Millions of Britons still don't want to entrust power to the right. And though down, and again contemptibly undermined by a hostile press after the first TV debate, the Lib Dems have become key players in the next parliament. Their policies on tax, Europe, immigration, PR and reformed politics will provide intelligent and modernist challenges to the regressive instincts of the other two parties.

All three leaders talk of national interest and consensus, new deals; they have come down from their perches and sound more humble and compromising – just what the country needs. Though it is unnerving, Britain's pragmatic people will survive and make it work. This isn't simply optimism of the will. The election has delivered remarkably enlightened results, which are at present unrecognised because the constitutional impasse has overshadowed everything else.

Appallingly, the election was largely a white, male fest, both in terms of who was out and about representing the parties and who was interrogating them. The results do not reflect that whitewashed nation. The number of black and Asian MPs went up from 14 to 27. Six Asian women were elected and three were Muslims. Rushanara Ali, whose parents are Bangladeshi immigrants, won in Bethnal Green and Bow, thus crushing the organised misogyny of many local Muslim men. Shabana Mahmood replaces Clare Short in Ladywood Birmingham. Both these women were local girls who got to Oxbridge. Barrister Yasmin Qureshi won in Bolton South-East, not an easy place. Here, reactionary Islam and social conservatism dominate and she must have had a hard time. Clever, ambitious, their hair flowing with defiance, these candidates have made history.

The Tories too have delivered diversity. Ugandan Asian Priti Patel won in Witham and, most impressively, there was victory for mixed-race Helen Grant, who was selected for Ann Widdecombe's Maidstone and Weald seat. How on earth did she manage to influence and move that constituency? And that she did says so much about how the country has changed. Labour has a brilliant new MP for Streatham, Chuka Umunna, the nearest we have to Barack Obama, a compelling orator and intellectual. Also in the new Parliament will be the first ever black British MP of African origin, Chi Onwurah, and an engineer.

Gay MPs like Ben Bradshaw and Chris Bryant kept their seats and the Tories elected their first ever confident and out lesbian MP, Margot James. And Caroline Lucas, the most impressive candidate I heard in this election, now represents Brighton Pavilion, the first Green MP in our Parliament. Tristram Hunt (Labour, Stoke Central), the charismatic historian, and Rory Stewart (Tory, Cumbria) the impossibly young ex-diplomat and human rights champion, bring real substance to the House. More joyous news: in spite of the respect and importance given to the BNP, particularly by the BBC, they were roundly thrashed. The population was overwhelmingly anti-immigrant but citizens knew not to vote for the party that gives shelter to "patriotic" Brutes, Nasties and Psychos. Gone, erased, even in Barking.

Voters also seemed to dismiss celeb politics. Esther Rantzen lost her deposit in Luton and George Galloway with his big cigar and bigger ego was seen off in Poplar. His Respect party, with its confrontational style, didn't appeal and may explain why loyalist Salma Yacoub didn't make it in Birmingham Hall Green.

Losing must devastate candidates and some are very dismaying indeed. Evan Harris, for example, the Lib Dem Oxford MP who was always impressive on science, freedom and civil liberties. Charles Clarke was a real character and will be missed. It is such a pity that the excellent independent candidate Richard Taylor (Wyre Forest) lost to the Tories.

But behind some defeats one can pick up signs of democratic sophistication. It is bloody marvellous that Annunziata Rees Mogg failed to make it, though her brother Jacob and other toffs did. Class doesn't always rule. Nor are people fooled by a change of image. Shaun Bailey, black, street and fanatically new Tory, lost because voters detested Cameron's social policies. The first black female minister, Dawn Butler, lost to the Lib Dems' Sarah Teather. Sad, yes, but the better woman won.

Elections in the end cannot only be about racial, religious and gender entryism. Shahid Malik, the first Muslim minister, lost Dewsbury. He voted for authoritarian laws, against any investigation into the Iraq war, for draconian asylum laws and he lost because some Muslims did not vote for one of their own. And Sadiq Khan kept his seat but with a reduced majority – a perfect result. He voted for the 42 days and needed reminding he should not betray fundamental rights. How heartening is that?

The US president Theodore Roosevelt said: "A great democracy must be progressive or it will soon cease to be a great democracy." This election, for all the trouble it has caused, shows our system is in good health, progressive and indeed great.

y.alibhaibrown@independent.co.uk

Further reading: 'Making Minority Government Work', edited by Robert Hazel and Akash Paun et al, published by the Institute for Government 2009