Waltz with Bashir, Ari Folman, 91 mins, 18

The new Israeli feature Waltz with Bashir defies classification as thoroughly as any recent film you can name. It's at once war memoir, documentary investigation, historical essay, psychotherapeutic narrative.

Writer-director Ari Folman begins by questioning the unease experienced by some of his friends who, like him, served in the Israeli army in the Lebanese war of 1982. Pursuing their memories – and significant absences of memory – Folman investigates the trauma that has been eating at his generation, and at Israel, for more than two decades.

That trauma is the massacre of Palestinian refugees by Lebanon's Christian Phalangist militia in the camps of Sabra and Chatila, a horrifying episode in which, it has been estimated, anything between 700 and 3,000 men, women and children were slaughtered. The Israeli army stood by – whether knowingly or not, Folman investigates – while the Phalangists carried out their killings in revenge for the death of Lebanese president Bashir Gemayel. Folman's film puzzles over this troubling material, digging slowly but deep – and even the nightmarish tone that runs throughout this bitter, unflinching narrative can't prepare you for the horrific outcome.

Now, did I mention that Folman's film is also an animation?

Recent years have demolished the lingering idea that animated cinema is strictly for children, or that it trivialises its material. Persepolis, based on Marjane Satrapi's graphic novels about growing up in Iran, mixed political observation with wry self-portraiture. Richard Linklater's Waking Life and A Scanner Darkly used animated treatments of filmed actors to explore visual metaphors of the mind, in a manner unthinkable in live action.

Waltz with Bashir starts with a bravura sequence: a pack of ravening dogs gallop through a city, under a toxic yellow sky (the film's leitmotif colour), before stopping under the window of a man named Boaz Rein Buskila. A real-life friend of Folman, he was long haunted by this recurrent dream: he explains to the director that, while a conscript in Lebanon, he was ordered to shoot 26 dogs whose barking might have revealed the Israeli army's presence. When Buskila asks Folman about his own memories of the war, the film-maker is shocked to realise that he remembers nothing.

Setting out to explore this blank, Folman finds it is shared by his peers, who seem to have repressed their memories of the massacre in particular. Nevertheless, some have visions to narrate, remembered or imagined. Carmi Cna'an, now a falafel magnate in Holland, recalls falling asleep on a troop ship and dreaming of a naked giantess carrying him to safety. Ronny Dayag recounts his six hours swimming for his life after an attack on his tank. Shmuel Frenkel used to be identifiable by the reek of patchouli oil, which ensured that his troops would never lose track of him, even in the dark; Frenkel's crazed reaction to enemy fire, dancing frantically as he fires, is the "waltz" of the title.

Waltz with Bashir proposes, conventionally enough, that war is a nightmare, yet few films have evoked the texture of nightmare so specifically. One such work, of course, is Apocalypse Now, which Folman alludes to in the general hallucinatory tone, in images of Israeli soldiers riding surfboards and in the colours of a repeated vision of orange flares over a city at night. There's a dream-like quality to the sheer confusion: an Israeli tank advances like a clueless juggernaut, crushing cars and smashing chunks out of walls; young soldiers fire relentlessly, with no idea why, or at whom.

Folman's psychoanalytic investigation of repressed memory reveals something about cinema itself. An expert in post-traumatic stress disorder – one of the real-life interviewees in cartoon form – talks about soldiers experiencing war as cinema, a defensive mechanism that fails only when the mind's imaginary camera breaks down. In using animation, Folman directly depicts that process at work. It's only at the end that he acknowledges that animation cannot equal the hard evidence of filmed document: the film concludes with news footage that leaves no doubt about the enormity of what happened at Sabra and Chatila.

There's some political cartoon mixed in with the stylised realism. One soldier remembers his commanding officer sitting in a Beirut mansion watching German porn; Ariel Sharon, then Israel's defence minister, is seen tucking into a breakfast of fried eggs.

The animation mixes traditional, 3D and Flash techniques: sometimes beautiful, highly coloured and densely textured, but often functional and rough-edged. Humans become shifting flattened figures, their strangely mechanical movements emphasising that they are sleepwalkers. This roughness reminds us that we're not watching an illusory fiction made out of beautiful movement, but a film of images, which aim to capture an otherwise unthinkable reality.

Among these images – variously savage, absurd, beautiful – there is only one that I wish Folman hadn't animated. It is a shot of a dead child's angelic face amid rubble: painful precisely because of its unacceptable beauty, the picture is unforgettable, yet unavoidably kitsch. Folman may be making a point about cinema's aestheticising effects, because that image is soon contrasted with the dreadful photographic evidence of the real thing, after which Folman can only fade to black. But I still wish that that earlier image hadn't been there.

Folman has inevitably been criticised for making his film from the point of view of traumatised Israelis, rather than directly about the victims of massacre. But he's talking about what he knows, from the perspective directly available to him. Waltz with Bashir is anything but a denial of guilt: given authority by veteran war reporter Ron Ben-Yishai, the film unambiguously points the finger of accusation at Ariel Sharon. Folman directly implicates himself and his army comrades, however unaware of events they were at the time. And he explicitly establishes parallels with the Holocaust, notably in the images of Phalangists herding Palestinians on to trucks. Waltz with Bashir comes across as a sincere, personal and deeply painful undertaking that lets neither its maker nor his nation off the hook. It's a brave and necessary film, remarkably inventive and, against all odds, grimly entertaining too.