Duncan Smith fighting for his survival

Iain Duncan Smith's short reign as leader of the Conservative Party was under threat last night after senior Tories defied his desperate plea for the party to "unite or die".

Mr Duncan Smith astonished MPs by warning that his party faced oblivion because a "small group" of them were trying to sabotage his leadership. But his dramatic personal statement, issued at a hastily called press conference, failed to stem the growing crisis of confidence in him.

Several Tory MPs said Mr Duncan Smith's words had only deepened the crisis engulfing the party. There was growing speculation that Michael Portillo would throw his weight behind Kenneth Clarke, another defeated candidate in last year's Tory leadership election, to take over. "That is an option," said one leading Portillo supporter.

The Tory leader suffered another setback last night when the House of Lords dropped its opposition to plans to allow unmarried couples, including gays, to adopt – despite Mr Duncan Smith's campaign against it. Eight Tories, including Mr Portillo and Mr Clarke, defied a three-line whip on Monday by voting in favour.

The rebellion prompted Mr Duncan Smith to appeal to the leaders of his critics and the party's 350,000 members in the hope they would order their colleagues to rally behind him.

"We cannot go on this fashion. We have to pull together, or we will hang apart," Mr Duncan Smith warned. He claimed that for "a few", Monday's vote on gay adoption was "an attempt to challenge my mandate to lead this party".

"I cannot allow the efforts of a dedicated team in Parliament or hundreds of thousands of hard-working volunteers to be sabotaged by self-indulgence or indiscipline," he said. "It [the Conservative Party] will not look kindly on people who put personal ambitions before the interests of the party."

Far from bolstering his authority, Mr Duncan Smith received an immediate rebuff because the Commons rebels rejected his charge of disloyalty.

Mr Portillo said: "I voted against a three-line whip for the first time in my life because I believed it was wrong and inconsistent to use coercion on adoption and it was my only reason for doing so. I reject entirely the unwarranted misinterpretation of the motives of those tens of MPs who were unable to support the party last night."

Mr Clarke added: "It would be very much easier to unite if Iain Duncan Smith would refrain from imposing three-line whips on subjects which have always been left to the judgement of individual MPs."

Tory MPs also juxtaposed Mr Duncan Smith's call for unity with his role as a backbench rebel who refused to back John Major's government in 58 votes on the Maastricht Treaty. "How does he square that with his attacking a quarter of the parliamentary party?" said Anthony Steen, MP for Totnes, who was named by Tory aides last week as one of the plotters. Mr Steen added: "I have come to the conclusion he is murally dyslexic. It means he doesn't see the writing on the wall. The party and the MPs desperately want him to succeed but everything he is doing is actually making that not possible."

Francis Maude, a close ally of Mr Portillo who voted in favour of adoption by gays and unmarried couples, said: "Iain himself knows, because he voted against the government many times during the 1990s, that there are times when you feel that the obligations of what you believe in are very strong."

Mr Duncan Smith will face an important test today when he faces Tony Blair at Prime Minister's Questions. Another big hurdle will be his response to the Queen's Speech a week today. And the leadership crisis is almost certain to be discussed at the weekly meeting of Tory MPs.

Monday's rebellion sent the Tory high command into crisis mode. There was embarrassment after Mr Duncan Smith cancelled plans to launch the party policy that would allow housing association tenants to buy their homes. Instead, he met members of the Shadow Cabinet. But some Tory frontbenchers had not been told of their leader's U-turn and arrived for the policy launch.

Tory officials said the party's grass roots had rallied behind the leader. They released a series of e-mails in which Tory members sent messages of support. Jim Kelly, the chairman of Brent East constituency, said: "If we do not show a semblance of unity, we will be seen as having lost the plot and not fit or worthy to govern."