Bella vista: Stuart Wheeler's model mansion

How have I managed to stagger this far on in life without understanding spread betting? Now I do, thanks to Tessa Wheeler of Chilham Castle in Kent. She and her husband, Stuart, bought the castle in 2002 with some of the proceeds he made from selling his company IG Index, initially set up in the Seventies to provide punters with the means to bet on the shifting price of gold. They've spent the past couple of years setting the property back on its feet, with Christopher Gibbs in charge of the house and its furnishing and Mary Keen and Pip Morrison providing a thoughtful and sympathetic masterplan for the garden.

Walking up The Street into Chilham's little square is like walking into a film set: the half timbering, the wonky red-tiled roofs, the flint-faced church. But most unexpected is the gateway that takes up the whole of one side of the square, with a view through the top half of the gates to Chilham Castle beyond. Most of the actual castle, except for the keep, has disappeared and what you see is a wonderfully handsome brick house, early 17th century, with twin cupolas either side, looking down over the village square over a long grassy slope.

It's an unusual situation – of the village, but separated from it; on view, but at the same time not showing off its secrets. One of the garden designers' most radical suggestions was to get rid of the lime avenue that, with a wide path between, once stretched from the gateway straight up to the front of the house. Those limes have been in and out of the view many times since Sir Dudley Digges first built the house in 1616, but it's absolutely right that they should be out. It leaves the house floating on top of a plain green foreground, with the drive curling off to the side following a dark shrubbery, and not in the way of the views from the house.

The views from the terraces round the house are superb. Ashford, the M2, the busyness of traffic in those parts, just disappear. In every direction you look out over a pastoral landscape. Capability Brown, brought in to advise in the 1760s said "there is little to do in the park", which covered more than 300 acres. "The truth is," the then owner wrote sourly to his brother, "he had little time for it."

Tessa Wheeler, bound up with her horses and a house in Tangier, built in the 1920s by her grandparents, says she is short of time too. She is "on the verge of becoming a really involved gardener" but keeps getting diverted by other things. But she's got a good eye (she spent her working life as a photographer and has just produced a nostalgic book about Tangier), and she's immersed in the history of the place, which has seen so many owners come and go, each leaving a different mark.

The house sits on a platform of ground on top of its hill, with new planting of tenderish shrubs frothing round its walls. The Keen/Morrison plan suggested myrtles, pale-lemon flowered Coronilla valentina 'Citrina', rosemary, phlomis, Gaura lindheimeri, and abelia flowering now in palest pink, with patches of dark-leaved, lemon-flowered dahlias scattered through the shrubs. The plants have settled in swiftly, with late-flowering ceanothus and trachelospermum already reaching up to the eaves on the pretty terrace outside the kitchen.

From here you get a great, head-on view of the monumental evergreen oak (Quercus ilex) that dominates the lawn. Defying gravity, six great branches spread out to make a huge canopy, chained and braced for so long that the chains themselves have become almost covered with bark. Beyond is Capability Brown's ha-ha, an ancient avenue of superb sweet chestnuts and, to the side, the shallow brick steps that lead to the most dramatic feature of the garden.

This is the series of 17th-century terraces to the south-east of the house, shored up by ancient brick walls that fall away down the slope to a paddock, soon to be planted up as an orchard. Only when you are at the bottom, looking back up to the house, can you see how staggeringly steep the drops are and how high the walls. But they make a wonderful background for training climbers and fruit trees, and when the borders have filled out, these long walks will be a phenomenal feature, rather like the terraced borders under the steep walls of Powis Castle. The other unmissable feature here is the topiary: cones (from the 1920s) lined out along the edge of the first terrace, huge great bulging shapes (from the 1870s), each now with its own personality, leaning out over the 15ft retaining wall between the final terrace and the croquet lawn below. These must be hell to clip, but they are superb things, the best kind of sculpture that a garden can have.

From the lawn by the house, brick steps connect the various levels, the steepest set at the bottom, jutting out at a right angle, with the corners of the steps carefully mitred to make sharp edges. The bricks are old, rubbed, and a rich warm colour.

The topmost terrace has a relatively narrow border under the wall because it has also to accommodate the first line of clipped yew topiary, with grass now replacing the gravel between the one and the other. The wall is warm enough for tender climbers such as mandevilla to flourish (Tessa Wheeler has it in her Tangier garden where it grows even more profusely); flat fans of fig provide a sympathetic backdrop for mounds of mauve-flowered Aster frikartii 'Monch', huge and handsome Eucomis pole-evansii, red salvias, fuchsias, schizostylis and nerines. The middle border has the oldest Judas tree I've ever seen, sprawled out against its brick backing, and the planting suggested in the Keen/Morrison plan is for a big, lush, late-summer extravaganza: dahlias, salvias and penstemons in deep rich colours with quantities of foliage between. The melianthus, in superb condition, is already showing that given time, this border has the potential to be the garden's great grand slam.

If you go, don't miss The Quiet Garden with its tall limes, like the pillars of a cathedral, or the kitchen garden beyond with the biggest wisteria you'll probably ever see in your life. Here, too, is a fine frameyard and vinehouse with grapes oozing like juice out of the ventilators.

The garden at Chilham Castle, Chilham, Kent is open today (2-5pm), admission £4. Tessa Wheeler's book, 'Spirits of Tangier', written under her maiden name, Tessa Codrington, is published by Arcadia Books (£25)

Also open this month: Robin and Edwina Hill's garden at Andrew's Corner, Belstone, Devon with organic kitchen garden and spectacular autumn colour, open tomorrow 2.30-5.30pm, £2.50; Tinpenny Farm, Fiddington, Glos, a windswept garden with an extraordinary collection of plants, open Tues-Thurs (9am-1pm) until the end of October, £2.50; Stonehill Quarry Garden, Great Gate, Croxden, Staffs, where Caroline Raymont has filled a quarry with acers and other trees and shrubs, open 20 & 27 October (2-5pm), £2.50.