The man who built Brum: A lament for the demise of John Madin's Brutalist Birmingham

My palms are pressed against Birmingham Central Library's rough exterior, but my mind is elsewhere – sifting through memories of kisses and fights. In a recurring dream, I'm the silent spectator watching a blonde-haired heroine wake up at spots around Birmingham where modernism was at its wildest – a grassy roundabout overlooked by tower blocks, underneath a flyover. Mostly that heroine wakes up here, the Central Library. But soon this building might only exist in dreams. The council wants to erase it, and to erase the era it represents.

Birmingham is building a new library and wants to bulldoze its current one. That could happen as soon as this autumn. The current 1974 Central Library is by John Madin – who died last year. His buildings were supposed to leave an indelible, futuristic mark on his beloved hometown. But they're being inexorably torn down by city fathers whose fathers' generation clamoured for the concrete that's fallen from fashion.

My first journalism job brought me to Brum a decade ago. The mind-boggling ziggurat of the Library captivated me – so did the otherness of this city. A fantasy fomented: to write about this building; to write the great Birmingham novel; to begin it here. Isn't a building filled with books the perfect place to start a story? It was a flâneur's apprenticeship: wandering, composing a satire set among the skyscrapers.

Buildings bear witness to the tiny dramas that make us human. If we turn concrete to dust, where do those memories go? Madin's buildings were the backdrops for a million Brummies' lives – though as time slipped by, they forgot to notice. The Post & Mail House was built for the Birmingham Post by f Madin in 1966, and its demolition started in 2002. Its smoked-glass tower was a neat appropriation of Lever House in Manhattan and it was the best office block Birmingham ever had. There was a prosaic annex next door where Metro, the newspaper I worked for, was housed. Every morning and lunchtime for three years I watched the tower being very slowly taken apart, without much dignity in death. In Leeds, the Yorkshire Post quit their Brutalist Madin offices for something more prosaic last year.

Madin's buildings are vanishing. His BBC Pebble Mill Studios were demolished in 2005. His condemned NatWest Tower has shutters over the windows of the opulent banking hall where I drew my pitiful first salary. "It'd be good to see Birmingham appreciating the richness of the many centuries of great buildings it has," maintains Catherine Croft of the Twentieth Century Society. "And especially not to have it trying to completely erase the 1960s, when it was an exciting place to be." Croft and her friend Elain Harwood of English Heritage repeatedly tried to get the Central Library and Post & Mail House protected. But unlike many of his contemporaries, Madin never had a building listed.

Now it's Birmingham Central Library's turn on the block. Along with Jim Roberts's Rotunda and Future Systems' Selfridges, it forms the trinity of the city's most distinctive landmarks. Madin and colleague John Ericsson's design was inspired by the work of Leslie Martin and Denys Lasdun. Leeds University has a bold Brutalist library by Chamberlin, Powell & Bon, but only Boston City Hall employs that similar fully-inverted ziggurat form.

The Library today boasts a tourist information centre selling Birmingham Monopoly and Cadbury mugs. I meet Alan Clawley outside. Clawley wrote a book on Madin for the Royal Institution of British Architects. "It took a long time to gain his trust," he explains. Since Madin's death, Clawley is the person politely striving to save this building, to save Madin's legacy, to save a vision which once flamed bright – but now merely flickers.

Madin is a forgotten scion of provincial architecture – in 1965 his was the third largest practice in Britain. That year René Cutforth made a BBC documentary called Six Men, where Madin is profiled as a family man – grinning, with a son gurgling on his knee. He danced between the demands of capitalist clients and a benevolent need he nursed within – to create a better tomorrow. In 1951, at 27, he drew a Christmas card showing the Birmingham of the present – and future. "He was interested in comprehensive redevelopment – of a whole civic centre for Birmingham," says Clawley.

Madin later planned the new town of Telford and, partially, Corby. "But Madin's ambitious plans were never fully realised here," laments Clawley.

Traffic rumbles below. Brum's roundabouts are called 'circuses' and none is more incongruously named than Paradise Circus – which encircles Madin's Library. Paradise. This is the throbbing heart of Brum. Here is ATV's Alpha Tower, the setting for the astonishingly awful 1973 Cliff Richard musical movie Take Me High – which surreally lauded the city. Birmingham's Greek revival Town Hall squats here, next to Snobs – a legendary indie club where shots are cheap and snogs guaranteed.

Antony Gormley's Iron Man perches outside, a tribute to Black Sabbath's gloom, to the metal genre born in Aston which was a more successful export than Rover cars. Here is where Madin and Birmingham chief engineer Herbert Manzoni's visions collided in spectacular style, Manzoni's inhuman Queensway punching beneath Madin's Library. The vision: multiple levels, levels upon levels. Madin was an apprentice boy in Manzoni's office, taking the boss his lunch during the Second World War. After peace broke out, the two joined forces to rebuild Brum in a total style which f marked it out as Britain's most futuristic city. Madin was the diplomat; Manzoni the obsessive. The dream: eternal modernism.

As a boy, I would guiltily pore over road maps. Maps of Birmingham were the best. Manzoni's hatchet handiwork made you gleefully dizzy. Britain's Motown had an automated car park at the Bullring, union boss Red Robbo agitated at Longbridge, novelist Jonathan Coe's dad worked as a motor-industry scientist – and Coe captured car-mad Brum with a precision Leyland designers didn't always deploy on their own products. Manzoni and the car lobby transformed Brum into a paean to autophilia.

The Central Library was the silent fulcrum around which the motorway madness swirled. Inside though, it could be a hoot. "We got up to all sorts of tomfoolery as teenagers – someone put my shoes in a lift and sent them to the basement," laughs Anne Heaton over drinks in the Yardbird bar next door. Heaton used to work on the newspaper too, but is now a teacher. The Cambridge graduate has a hypothesis: "Our university library was phallic, but I think Birmingham Central Library is feminine." She's right. It entices you in, it looks like an abstract Aztec depiction of a woman, boasting a sensual curved lending library.

Niky Rathbone spent her life inside Madin's ziggurat working as a librarian. She remembers the end of the millennium, when spectaculars celebrating each decade of the 20th century were directed by Sir Simon Rattle. "For the 1920s we staged the General Strike, dressed as oppressed workers, singing the Internationale off the balcony," she beams. Rathbone reflects on history repeating itself: the demolition of the previous Victorian library in the 1970s. "I was shocked by it." But she grew to love the ziggurat and believes that a way could have been found to ensure the building's survival.

On a grey day, Birmingham broods. This singular atmosphere has been mined in local literature – much of it coming via the exceptional publisher Tindal Street Press – all of it diligently catalogued by Rathbone and her colleagues and available in Madin's Library, until it closes in June. Female authors like Clare Morrall, Lynsey Hanley and Catherine O'Flynn have written tightly-coiled books which bring this mystical, masculine city to life. A distinctive, melancholy branch of writing has emerged in Brum.

The city's overwhelming scale can promulgate an ache. Buildings threaten to crush softer hearts. Ponderous shapes and blank, yawning spaces scare some people stiff. Concrete can oppress, motorways can segregate – Hanley talks of "the wall in the head" psychologically separating those who live on Birmingham's sprawling edge estates like Chelmsley Wood. And yet, the city bursts with creative promise and creative people speaking myriad languages. Within the weird vision there is stark beauty. I left – yet Brum still comes to me at night.

O'Flynn's The News Where You Are is the kind of book you snap shut after devouring in a weekend – and wish you'd written. It is the great Birmingham novel. It's about an architect whose buildings are all being demolished. You think of Madin. O'Flynn is a devotee of oddness, black humour, Brutalism. She remembers: "I spent a lot of my teenage years in the Central Library, failing to revise and wasting hours in the music library. It was incredible – all those albums to take home for free and illegally record!"

O'Flynn understands memories – and the illusory Brum mindset. She reminds me: "Birmingham's motto is 'Forward'." A city-wide £10bn, 20-year redevelopment is undoing a past that is no longer liked. The new library (built, incidentally, as Birmingham's libraries budget is slashed) is clad like a Christmas decoration. "I call it the lacy panty building," deadpans Rathbone.

"I like much of Madin's work," says Glenn Howells, who will redesign Paradise Circus if Madin's chef-d'oeuvre is put to the sword. He surprises me with his affection for Madin: "He was hugely influential in reshaping the city after the war. There are some gems he designed."

Incredibly, Howells even bought a house Madin built – near waterskiing-mad Madin's own home in Edgbaston. But it doesn't make Howells remorseful about overseeing the end of the ziggurat.

The death of a building or the death of an age – locals here take it on the chin. Their city is always in flux; eras erased, buildings replaced. Brummies lionise black humour. Archivist Richard Jeffs gave me a trippy 1980 Harold Baim film called Telly Savalas Looks at Birmingham. In it, the Library appears as the rising of a new republic; of a new Eden almost. In youth and on a sunny day it's a stunner. The commentary meanders. Savalas – who died without ever visiting Brum, and recorded his voiceover in Soho – bullshits that: "I took an express elevator to the top of Birmingham's highest building… and this was the view that almost took my breath away!" BBC online producer Jon Bounds told me that when he screened the film, "the audience burst into uncontrollable fits of laughter at this line".

Jonathan Meades got to the elemental essence of Birmingham, with his 1998 film Heart By-Pass. He opened up the second city with the satirical precision of a surgeon's scalpel, describing the architecture as "all vigour and no finesse". At one point Brum was a bolshy fiefdom – a city state controlled by Manzoni's progress-crazed Corporation, with a vocal local paper, and fearless television station which made documentaries by John Pilger and a soap opera set in a motel.

Brum is an adorable oddball; an eccentric void at the centre of the nation and yet the centre of nothing. Birmingham transforms. Birmingham forgets. Buildings, dreams and memories live in the same unreal world here. Birmingham Central Library is a dreamlike building: Madin's fantasy made real. Sometimes it's hard to know where dreams end and reality begins. Paradise Circus boasts a warren of hidden squares, haunted passageways, dead ends. The scent of Nando's hangs in the air as night falls. A busker plays a lament on a saxophone. A thud behind me. Human or ghost? Reality or dream? The shape of a cyclist materialises from the darkness. "D'you know how to get out of here?" he laughs.