Seeing but not believing: The photography of Susan Hiller

The poster campaign for atheism, promised in the new year, seems mild-mannered enough. "There's probably no God. Now stop worrying and enjoy your life": that's the slogan to run on buses in London and perhaps elsewhere. Its off-hand tone may or may not be effective – to me it seems rather well judged. At any rate, it puts the issue of belief out on the street. These questions should be public questions.

They already arise in art galleries. Go the Rothko exhibition at Tate Modern, and you'll meet a famous quotation from the artist, captioned on the wall, to assist your looking. "If people want sacred experiences they will find them here. If they want profane experiences they'll find them too. I take no sides."

Well, if he says so. But obviously you're being invited to take a side, as you gaze into those deep red abstract portals. Sacred? Profane? What's your line? Or if you baulk, at least you can observe your fellow visitors gazing, and try to guess whether or not they're finding transcendence in the paint. Art as a spirituality-test, a spirituality-sorter: it's another use for art, and a good one.

And there's more around at the moment. Go to the Susan Hiller show at Timothy Taylor Gallery. It's called Proposals and Demonstrations. That sounds rationalistic, and so is the look of the work here: documentary displays, geometrical projections. But, in content, the otherworldly rules.

Hiller was born in 1940. She trained as an anthropologist. She was part of the first wave of conceptual art in Britain in the 1970s. Ever since, her art has been entertaining paranormal possibilities. It's invoked trance states, psychic powers, UFO visits, hauntings, speaking in tongues, spontaneous combustion, near-death experiences. This new show offers auras, levitation, mystical mazes, automatic writing and the voices of the dead.

Of course, many people do believe in this kind of stuff. What side Hiller takes herself, is never quite clear. She's plainly very interested. She's almost certainly not a believer. But she doesn't declare. She presents, confronts. A lot of her work involves found material. A typical trick is to take some weird piece of "evidence" and transform it, to imbue it with resonant visual authority, to make it hold the mind.

So there's a beautiful work here called From India to the Planet Mars, based on pages of automatic writing produced by a medium (who thought she was writing in Indian and Martian). They're turned into photo-negative, and shown as light-boxes. The lines of barely articulate scribble and doodle glow out, white on black. Sometimes they surface into half-sense: "Eavae museum / Anahmes us / Ubarr for me mes plu / As poturat menhu? / Ye pretty ones." The meaning is beyond one, though with an inkling that some sort of message coming through. The form is like a solemn showing. The effect is a mind-block, a piece of frozen weirdness.

That's how Hiller makes use of the paranormal: not to persuade you one way or the other, but to create something uninterpretable, impenetrable, a "maybe" we don't know what to make of. The piece about auras is a layout of 50 aura photographs – people whose heads are haloed with multicoloured glowing blurs. There are special cameras designed to pick up these supposed spiritual emanations, popular at psychic fairs. It is easy to debunk them. But again, the visual treatment saves the dodgy evidence. Arranged into a tight oblong, these photos become a field of intense fluid colour, and the images are re-invested with a mysterious power, as if they do indeed hold something "unexplained".

The best work here is a Hiller classic from 20 years ago, Magic Lantern. You sit on a bench wearing headphones facing a wall. On to the wall, circles of different coloured light are projected, getting bigger, sometimes overlapping like Venn diagrams, and their colours mixing. It's an almost meaningless visual sequence, a hypnotic focus for your eyes, as you listen to what's coming through the phones. You hear an account of an experiment conducted decades ago by a Latvian psychic researcher, purporting to record the speech of the dead.

An incredible noise. Words, names, phrases (in several languages) are just barely discernible, garbled, abrupt, and beneath such heavy interference they might be bits of interference themselves. You sit there, straining to hear them, to hear if there's anything there to hear, trying somehow to grasp these ungraspable faraway noises that seem to be talking. What on earth...? You come away with something, nothing, an absurdity, a dizziness.

That's where the power of Hiller's work is. It blanks. It amazes. It forces a mental interruption. It brings you up against something irrational, inexplicable, indigestible – and makes it undeniable, a voice you have to listen to. It takes somebody else's bizarre belief, and your own unbelief, and jams them.

But you can't put the question of belief to one side. The whole jamming effect depends on the audience's scepticism and resistance. Hiller's work operates on the understanding that the contemporary art world is a pretty secular place. If things were different, if the people who came into this kind of gallery were also people who regularly went to séances, clairvoyants, channelling, psychic fairs etc, then her work would have a quite different impact. It might seem more simply celebratory, giving an artistic affirmation to things the audience anyway affirmed. Or it might seem rather superficial, happy just to conjure with this material, for the sake of its thrilling weirdness.

For it's true, Hiller may be very taken with the supernatural, but she doesn't get stuck into it. It remains a glimpse of wild possibility, something on the very limit of experience. It's a romantic notion, a disorienting marvel, a pure shot of the "beyond" breaking into normality. Too pure, I feel. Her art can create a sense of unutterable strangeness, but it doesn't imagine what it would be like to inhabit this world every day.

In Hiller's world, the paranormal has glamour, the glamour of strangeness. But if this stuff were true, it would be a messier truth. A good comparison is Hilary Mantel's novel of a couple of years ago, Beyond Black. This is a superbly de-romanticising story about a spiritualist medium who lives in daily familiarity with the dead, and finds them as depressing, horrible and stupid as – well, as the living.

In Beyond Black, there's Alison, the obese clairvoyant with her professional troubles, which include Morris, her repellent spirit guide, whom she can't get rid of. "It was almost the worst thing, having him around at times like these, in your dressing room, before the show, when you were trying to calm yourself down and have your intimate moments. He would follow you to the lavatory if he was in that sort of mood." And the comedy gets much blacker than that.

Which side is Mantel on, belief-wise? She imagines so well into this belief-world that you can almost suppose she shares it. But her general moral is an antidote to all temptation to put our faith in the beyond. The otherworldly would be no escape. On the other side of the grave lies only more of the same crap.

Susan Hiller: Proposal and Demonstrations, Timothy Taylor Gallery, London W1 (020-7409 3344), to 20 December