Goya's Witches and Old Women Album go on show for the first time at the Courtauld Gallery

"Reject the angel, and give the muse a kick," the Andalusian writer Federico García Lorca said in a lecture in Buenos Aires in 1933. "The true struggle is with the duende." According to Lorca, it is the duende that rules the art of Spain, "a country of death, a country open to death". The duende is an elusive spirit with dark power that emerges in the moment of artistic creation and possesses the artist. In the shuddering body of the flamenco dancer and the murderous grace of the matador, it is the duende that "burns the blood like powdered glass… shatters styles and makes Goya, master of the greys, silvers, and pinks of the finest English art, paint with his knees and fists in terrible bitumen blacks."

Not surprisingly, the duende has no direct translation in English. We do not live in a culture where expressions of intense feeling are celebrated. This was Lorca's point. His reference to the great Aragon-born painter Francisco José de Goya y Lucientes (1746-1828) as "master of… the finest English art" points to the restrained palette and delicate sentiment of portraits such as The Marquesa de Pontejos (1786). The young Marquesa is shown in a sheperdess-style frilly dress, made fashionable by Marie Antoinette, who was about to have her head cut off. In this way, Goya made his name by flattering the Spanish aristocracy, and became the foremost court painter in Spain.

However, there were two Goyas. In 1793, at the age of 47, he suffered a near-fatal illness and lost his hearing. His art changed dramatically. He began to create a series of works that would culminate in the astonishing Black Paintings (1819-23), which were unlike anything that had come before. They were visions of hell, painted straight on to the walls of his Deaf Man's Villa on the outskirts of Madrid. Their subjects included cannibalism, devil worship, and death by drowning. This was the Goya whom Lorca described as painting with "his knees and fists". Here his duende emerged, "not a question of skill but of a style that's truly alive".

At the same time, he was working on the Witches and Old Women Album, which has just gone on display at the Courtauld Gallery, London. More than a scrapbook, the album was a way for Goya to document his visual ideas. It was cut up after his death into around 23 small black and grey ink drawings, which are shown as individual works. Like the Black Paintings, they convey a hallucinatory darkness. Goya made eight albums in total, and the curator has taken the wise decision of including earlier and later drawings too, which show his artistic development. These are some of the most riveting works in the exhibition.

The Witches Album was not intended for public consumption. This is the first time its pages have been shown together. What is fascinating and thrilling about the drawings is the way they show an ecstatic release from genre. The rules of the portraiture in which Goya excelled are here joyfully trashed in favour of total creative freedom. They are unbound from all market imperatives. Goya is one of my favourite artists and I was enthralled by this exhibition; others may be repulsed.

Marriage is revealed to be a depraved institution. Sure and Natural Union (1825-28) is a black crayon drawing that shows a man and a woman bound together at the waist. They have only one arm and one leg each, together forming an amputated whole. They smile with eerie contentment. Goya's titles are wonderfully wry – they seem contemporary.

Femininity, or its disintegration, is his theme. One of the more playful, albeit unsettling, ink drawings is Old Women Fight Too (1819-23). It shows two old women wrestling, their bodies locked together in an equal balance of power, their heads bowed like bulls, their legs open. Many of the drawings are theatrical scenes of one or two figures, often engaged in power struggles. They claw and caress one another by turns, falling or rising through the air. They seem both wildly autonomous and trapped by fate. Far from court painting, these scenes appear inspired by the everyday life of the pueblo, though touched by psychosis.

Celestina is the archetypal madam of Spanish literature; she organises and controls her younger prostitutes. Hush (1797-98) is a subtle etching that shows Celestina bent over, a crone, while a young prostitute in a traditional mantilla, or black veil, whispers a secret into her ear. Behind, a black tree echoes the form of the prostitute's figure. Goya's dense and shimmering aquatint marks make the tree look like a shadow. It seems even solid things are chimeras in his world.

This is Goya's power. Rather than overt horror, some of the most affecting works depend on suspense. They hint at obscure business transactions and weird private rituals. Here the women have a lot of agency, but it is mostly malevolent.

A later work, Mother Celestina (1819-23), shows the madam sitting alone in a chair with a bottle of what appears to be alcohol in one hand and a rosary in the other. Next to her is a table, with more bottles: possibly vials of drugs. It's likely that the rosary symbolises her trips to church, where she picks up new prostitutes and clients. Here, Celestina appears an almost sympathetic figure, a fading star of the underworld. The drawing conveys the loneliness of old age.

The witchcraft is explicit. Wicked Woman (1819-23) shows a witch about to eat a baby, while Dream of a Good Witch (1819-23) shows a witch carrying a bundle of babies on her back, ready to be used in spells. Unholy Union (1819-23) is notable for the humanity in the face of the witch. Rather than a crone, she is middle-aged and fierce, striding forward in long skirts. She carries an adult-size victim on her shoulders, bound together with live snakes and screaming.

Witchcraft was persecuted during the Inquisition of the 17th century and black magic retained a hold over the popular imagination. But Goya was not merely reproducing the kitsch iconography of a national panic. His witches give human form to the darkness of the imagination. Indeed, it was the very human capacity for cruelty tha terrified Goya. In a 1784 letter to his friend Martín Zapater, he wrote: "I'm not afraid of witches, hobgoblins, apparitions, boastful giants, knaves or varlets, etc, nor indeed any kind of beings except human beings."

The terror of a man emerging from a nightmare is conveyed through the fluency of Goya's draftsmanship in He Wakes Up Kicking (1819-23). Startled out of sleep, his face expresses horror at an invisible object. He appears half falling, half levitating. The bed beneath him is barely outlined; he is surrounded and engulfed by empty space. His fear is existential rather than occultist.

Madness (1819-23) is one of the most moving drawings. A man stands in what appears to be an animal pen, wearing a court jester's hat. His mouth gapes open and he seems to beseech the viewer for help. He retains an aura of greatness, though he is ruined. He recalls King Lear.

Goya made The Witches Album when he was in his seventies and contemplating his own mortality. What underlies these extravagant grotesques is vulnerability to the passing of time. There is tenderness beneath the satire. There is also a sense of the artist's struggle with his own, seemingly anarchic, imagination. In Who Would Have Thought It! (1797-98), two gnarled, naked witches fight on the edge of a precipice. A dark monster reaches up from the depths to claim them.

In an echo of Goya's image, Lorca said: "The duende delights in struggling freely with the creator on the edge of the pit… the duende wounds, and in trying to heal that wound that never heals, lies the strangeness, the inventiveness of a man's work."

Goya: The Witches and Old Women Album, Courtauld Gallery, London WC2 (020 7848 2526 ) to 25 May