Books of the year 2014: The best debuts

The year began with Nathan Filer's inventive and moving The Shock of the Fall (Borough Press, £7.99) winning the Costa. Filer also deserves a prize as Blurbist of the Year for his use of "Dazzling". Generous to other debutants, he praised Jessie Burton's slick, successful, if slightly over-egged The Miniaturist, which Waterstones has named its book of the year.

Just about every other award was carried off by 2013's major first-timer, Eimear McBride. The elliptical, staccato erotics of A Girl is a Half-Formed Thing (Faber & Faber, £8.99) were literary Marmite, but the judging panels of the Goldsmith's, Bailey's, and the Geoffrey Faber Memorial prizes found them in good taste.

January also witnessed Hanya Yanagihara's extraordinary The People in the Trees (Atlantic, £12.99). Anthropologist Norton Perina discovers a Micronesian tribe rumoured to hold the secret of eternal life, before plummeting, via colonialism, child abuse, moral relativism and male homoeroticism, towards a startling climax that should leave book groups in a lather.

Vanishing people of different sorts drive two impressive literary mysteries. Emma Healey's Elizabeth is Missing (Viking, £12.99) is a troubling, heart-felt depiction of failing memory and stubborn imaginative fixation. Celeste Ng's elegant Everything I Never Told You (Blackfriars, £8.99) provides a persuasive exploration of death, race, sexism and parental pressure in a 1970s Chinese-American family. Ng has just won Amazon.com's Book of the Year.

Unease was very much in the air for 2014's debutants. Fiona McFarlane's The Night Guest (Sceptre, £8.99) offers a hallucinatory depiction of a lonely woman being menaced by a larger-than-life care-worker and stalked by a nightmarish Blakean metaphor. I am still unnerved by Sarah Perry's After Me Comes the Flood (Serpent's Tail, £11.99), which kneaded Beckett, Hammer Horror and Old English into a moreishly gothic but humane concoction.

When John Darnielle isn't writing intense, skewed lyrics for The Mountain Goats, he is crafting intense, skewed and lyrical prose. The weirdly wonderful Wolf in a White Van (Granta, £12.99) is melancholy, clever, obsessive and even turns apocalypse into gameplay.

Laline Paull's The Bees (4th Estate, £14.99) buzzes between Kafka and Bee Movie to explore individuality, sex and the meaning of a life, apian or otherwise. Paull's prose oozes weird atmospheres: "A golden mist and soft harmonic chord shimmered from the centre of the great atrium, whose six towering walls were made of interlocking chalices of honey..."

Zoe Pilger's exuberantly seamy Eat My Heart Out (Serpent's Tail, £11.99) is just what a fledgling writer's first book ought to be: a funny, intimate and impolite whizz through being young, pretentious and lost. Ann-Marie's pursuit of love and feminism in a preposterous hipster London made me laugh out loud: "Everyone on the dance floor seemed to be suffering a simultaneous epileptic fit."

Another closely observed tale of youthful zeal propels Jonathan Gibbs's Randall (Galley Beggar Press, £11), which contained two of 2014's more appealing fictional premises: Damien Hirst's imaginary death in 1989, and a stash of paintings depicting the art world's major players in pornographic splendour. This portrait of a Young British Artist as a Deceased Man is vibrantly assured.

English-reading audiences have had to wait almost three decades for Tomas Gonzalez's In the Beginning was the Sea (Pushkin, £12) – a terse but poetic re-casting of his brother's real-life murder in a seemingly idyllic Colombian village. J's escape is by turns beautiful, hubristic and tragic. Gonzalez writes gloriously about nature for good measure.

Honorable mentions go to Meena Kandasamy's self-consciously ambitious The Gypsy Goddess (Atlantic, £12.99), Rob Doyle's unblinking depiction of male desperation Here Are The Young Men (Bloomsbury, £8.99), Simon Sylvester's polished The Visitors (Quercus, £16.99), and Iraq veteran Michael Pitre's well-crafted, semi-autobiographical Fives and Twenty-Fives (Bloomsbury, £16.99).

If I had to choose one first novel, it would be the addictive bedlam of Bonita Avenue by Dutch novelist Peter Buwalda (Pushkin, £12.99). Siem Sigerius is an intellectual icon and patriarch of a family for whom "dysfunction" seems an understatement. The twisted central twist – how Sigerius's taste for pornography explodes close to home – is typical of Buwalda's messy marriage of realism and chaos theory. Bonita Avenue deserves to be a book, not just a debut, of the year.