Ian Hamilton Finlay, Victoria Miro Gallery, London

Letters carved in stone conjure up a bygone handicraft used to dedicate memorials, signpost our way round gardens or else record pithy aphorisms or zen bites of wisdom such as: "To know yet to think one does not know is best" (from Lao Tzu's Tao Te Ching).

Such chiselled inscriptions seem to be as far removed from the materials of contemporary art as the ancient cultures that first marked their tablets with script, and yet Ian Hamilton Finlay found room to embrace all these anachronistic and bucolic elements – headstones, outdoor furniture, Roman remains, head-scratching one-liners – in his peculiar and arresting art.

Before Finlay's death five years ago, he'd become best known as an avant-gardener, for his highland sculpture park in miniature, Little Sparta, filled with stone quotations and temple follies. It is still a site of summer pilgrimage, close by Edinburgh, for Finlay followers and green-fingered enthusiasts, but his first love was undoubtedly poetry and language, which is where this puzzling show also begins.

A black slate from 1990 is engraved with its title, Laconic, and the alarming shape of a guillotine, proving that brevity was indeed the soul of Finlay's sharp wit. A neighbouring wall text, however, gives the meaning of guillotine as "the highest peak of the Mountain; and its deepest ravine", which doesn't much help, especially when you consider that the exhibition is called Definitions. It turns out that Finlay's dictionary-in-his-head was one of historical allusion, erudite punning and linguistic tricks, albeit tinged with menace. Zimmerit, we learn, was a military paint for disguising Panzer tanks, thanks to an explanation next to a green neon "Zimmerit-haunting wood nymph" (1992). Panzer – nothing to do with Pan – is the German for armour. Another wall definition runs: "Nature, n. the universal camouflage".

While, alas, the Victoria Miro gallery's floors are not about to sprout grass and return these totems and recent ruins to their natural landscaped habitat of Little Sparta, there is at least a garden where you can glimpse a slice of quintessential Finlay at play in the open air. On the wall is a cast iron plaque, Evolution of the Boat (1995), which simply reads "Long Boat, Narrow Boat, Short-flying Boat". As well as the sign of a keen sailor, this is concrete poetry of the highest order, deftly summing up how mankind's idea of progress – ever more descriptive names, technological gimmickry – leads only to non-essential advancements that are generally not fit for purpose, in this instance having first been mastered, and never bettered, by the Vikings in their plain longships.

In the duck pond opposite is what should be the very apotheosis of maritime evolution, an armed aircraft carrier, here rendered ridiculously as a scaled-down ornamental fountain for birds to alight on (before the opening, a moorhen briefly set up home on its narrow landing strip). This humorous, if too obvious, conflation of militaria and statuary is rammed home upstairs in a line of columns topped by ever more florid finials, the final one a hand-carved, faux-crenellated grenade. Elsewhere, Finlay's commentary on war is better served through words. Four life-size sentry boxes are manned by stonemasonry plaques carved with emasculating jokes about neo-classical grandeur and modern weaponry (neither statues nor soldiers should lose their arms, geddit?).

The last work Finlay made before he died in 2006, the installation Four Muses, is even tougher to unravel, unless you know your Platonic pronouncements from your Apollonian myths. There were traditionally only three muses, so the mystery fourth bronze bust here cheekily gazes face-forward to an unknown future, while the others look demurely to the past.

If all this can feel like being beaten over the head by a classics teacher, then the unflattering image of Ian Hamilton Finlay as visionary loner and dour Scot is perhaps not so far from the truth: he battled with Strathclyde Council over his land, fell out with fellow poets, and suffered from agoraphobia. Yet these philosopher's stones do speak to us from beyond the grave, requiring only a mental leap of faith to start the process by which his dark and fertile ideas take root and begin to grow in the mind.

Next Week:

Charles Darwent snaps up 25 years of photography by Paul Graham at the Whitechapel Gallery, London

Art Choice

Head to Margate for a view of David Chipperfield's bracing new Turner Contemporary gallery building and its first exhibition, Revealed, until 4 Sep. At the Royal Academy, London, you can hear the rustling silk and feel the downy skin in the bewitching drawings of Watteau, chronicler of the doomed French aristocracy, until 5 Jun.