Living in Ghostland: the last Heygate residents

It was like a deserted scene from an apocalyptic movie: not a soul in sight. No hooded teenagers or their dogs, no mothers pushing prams with other little ones in tow, no pensioners on benches watching life. Not even echoes. Just silence.

This condemned concrete edifice, once home to 1,200 families, is today populated by just 20 spectral households – 54 people in all. They are the refuseniks, resolutely barricaded in the homes they grew up in, or in other cases merely awaiting the removal van to take them to another council property. Everyone else started leaving three years ago.

The infamous Heygate estate in Elephant & Castle, south London, is depressing even by the standards of Britain's post-war architectural carbuncles. Just down the road the pavements heave with vibrant city life, a Saturday market: stacks of Turkish rugs, Colombian food stalls and Afro-Caribbean hair salons crammed together. But on the Heygate, the few remaining residents hide behind security gates, and every unoccupied property is boarded with steel shutters. Amid the rubble of abandoned gardens, one stumbles across a broken television, a faded football, a discarded yellow child's bike.

One of hundreds of social housing estates of its kind to be hastily constructed across the country in the Sixties and Seventies (the Heygate was completed in 1974), this crime-racked labyrinth of grey high-rise blocks and small terraced houses, linked by raised foot bridges and stone stairwells, stands as a monument to the failure of post-war mass housing.

For this reason, the Heygate's demolition cannot come too soon for some. The actor Michael Caine, who grew up in a nearby part of south-east London and starred in the film Harry Brown – just the latest to be set against the estate's bleak backdrop – has lent his voice to a longstanding campaign to flatten it. He described the Heygate as a "rotten place... which fortunately is being pulled down. It should never have been built." The plan is to embark on a £1.5bn regeneration project, but it has been beset by delays. Southwark Council says the developer Lend Lease is finally close to sending in the wrecking balls.

Six floors up, 17-year-old Laura Cross peered out from behind her front door, which, mid-morning, was blocked by heavy iron-bars. She has lived in this same flat all her life. Her family was offered a new home, in another high-rise five minutes away.

At first, Miss Cross said she didn't want to leave, that this had been a happy, safe place to live. "Every morning you'd come out of your door and there'd be people on the balcony and you'd chat," she said. But once her neighbours started filing out in 2007, she became a prisoner in her home. "There are gangs in the hallways at night," she said. "No one's watching out for us. You'd be mad to go out here after dark."

Kevin Watson, 43, has lived on the Heygate with his mother for 10 years. The community had been obliterated, he said. "This wasn't a bad place to live. There was a sense of belonging. We'd all meet up at the bingo. It's important for the older people to have that friendship network. Now my mum's friends have been shipped away."

A council source had a strong message for those reluctant to move from their homes: "Sticking it out is not an option. The estate is coming down. Demolition has begun. It was a crap building in the first place, and it became too costly to maintain. People's health, their lives, children's education will all be improved." There are 30 offers of relocation to those still on the estate, the source said, with a right to return, eventually.

Even for the Heygate's detractors, its imminent demise comes as a hollow victory. Like similar regeneration projects across Britain, the recession came at a bad time. There has been talk of 800,000 sq-ft of retail, five new "open spaces" and 5,300 replacement homes, but campaigners question how many will be luxury apartments (within easy reach of the City) and how many affordable housing.

Residents said, uniformly, that since the clear-out began, crime and violence had spiralled out of control. Mustafa Seed, 39, originally from Djibouti, a resident of 12 years and the only person still on his floor of a tower, had been burgled three times in a month: "The thieves see that the balcony is open [not shuttered] so someone's living there, and they know there will be no one around to catch them."

But for others it will still be "home". Amid the garbage and undergrowth there was a single manicured lawn gilded with hanging baskets, all bright pinks and reds. Through the curtains, the television flickered.

One pensioner, who declined to be named, said he would refuse to leave the flat he has lived in since the 1970s until his case was heard in court. "I'm not going anywhere because they haven't offered me a good enough reason," he said. "It's not the Heygate that's the problem. All the crime and problems we have, it's just a symptom of how the working classes have been left to rot. They don't care about us. We are the forgotten victims"

Modern living: An infamous five

*Trellick Tower, west London, 1972 Designed by Erno Goldfinger, and perhaps the best example of "brutalist" architecture in the UK.

*Park Hill, Sheffield, 1961 "Streets in the Sky" to Sheffield City Council, but to locals it is "San Quentin", after the notorious US prison.

*Byker Wall, Newcastle, 1969-82 Design incorporates colourful brick, wood and plastic panels, in a deliberate departure from brutalism.

*Quarry Hill, Leeds

Built in 1938 and largest social housing complex in the UK.

*Galton Village, Smethwick, 1960s

Known as the "Concrete Jungle". Demolished and rebuilt in the 1980s.