Heaven is Hvar when the hordes have gone

If you stayed in Britain this summer to reap its bumper harvest of Queen's Diamond Jubilee, Wimbledon's nearly-made-it Andy Murray moment and the country's hosting of the Olympic Games and Paralympic Games, you doubtless felt blessed and proud – but possibly a little cheated by the debit column in the summer's sunshine balance sheet. As the Paralympic Games finally draw to a close today – and with the school holidays well and truly over – perhaps it's time to address that deficit. For, come autumn, while many European destinations are drained of tourists, there are certain places that come into their own once the crowds have left.

One such is Hvar, that little Dalmatian island an hour's ferry south of Split. Every time I leave Hvar, I get a sinking feeling in my stomach. And last time I departed the splendour of its neoclassical harbour I promised myself it was a secret I had to share.

When we – my wife, our eight-year-old son and three-year-old daughter – return, it's a golden morning, the water silk calm, Hvar's wharf (burnished by footfalls of Greeks, Illyrians, Venetians and Napoleonic soldiers, and latterly the tottering stilettos of oligarchs' wives) shiny in the dawn light. I'd timed our visit to coincide with the departure of summer's hedonists – and it's as if the old town is breathing a sigh of relief.

In July and August's high season, Hvar nets one of the most gorgeous itinerant populations you've ever seen: imagine a cast of raven-haired Italians, wolf-eyed Russians and rugged American yachtie types, all looking as if they've been torn from the pages of Cosmopolitan, and all towered over by Croatian colossi – at an average of around six feet tall, Croatians (along with the Dutch) are among the tallest in Europe. Meanwhile, Croatian women, especially those in Hvar, it seems, appear to be almost uniformly stunning: broad shouldered, strong featured, olive skinned and dark eyed. It's as if the gods spent a little longer in their making, and were I a model scout, I'd ditch the King's Road and come straight here for my next discovery.

Then there's the unusually pretty township of Hvar itself, a maze of cobbled streets and shuttered villas radiating from a Renaissance harbour and central piazza. Beauty begets beauty, and late afternoon when the breeze blows and sun slinks behind the nearby Pakleni Islands, the light takes on a honey hue and makes – to paraphrase a line from Shakespeare's Antony and Cleopatra – the wind seems to glow the delicate cheeks which it cools. For during "magic hour" in Hvar's exquisite piazza, everyone looks like an Antony or Cleopatra: wrinkles are chased away and couples wander arm in arm.

But don't be too swiftly sold on summer's skin-deep appeal; there are plenty of reasons not come to Hvar then: hotels are heavily booked, restaurants and cafés are over-crowded, and the town feels overly self-conscious – and in some sense stripped of its authenticity. Not to mention the fact that its population of 6,000 has swollen to 30,000. Wait until September, however, and those rocks around the harbour chocka with lounging ladies reading Fifty Shades of Grey are now deserted, and the nearby Pakleni Islands mysterious and enticing once more.

On our first morning, we drop our bags at the luxurious Amfora Hotel. It's an imposing flat-topped building with a distinct signature of the 1970s. Indeed if, back in his Bond days, Roger Moore had designed a hotel, it might have looked something like this. Inside, all is chic and recently revamped; outside is a private beach and lagoon-style swimming pool sprinkled with a few nut-brown fashionistas. The air is piped with lounge music.

We head for lunch in the piazza and devour a plate of calamari and thin-crust pizza at the al fresco Mizarola Bistro. Meanwhile, I try (and fail) to explain to my family just what makes this place so special. It's as if, I say, elements of Greece's Symi and the elegance of Venice have been pressed together to produce the essence of the Mediterranean. They nod, just to humour me, then get back to their squid. But every visitor here, from Napoleon to Kevin Spacey, Bill Gates to Prince Harry, must surely have been as agog as myself.

Over a frappé in the piazza I meet Janko, a retired teacher whose been coming here every year since he can remember, but always at this particular time. "It's the light," he tells me. "It's softer than high summer. And the town, now the noise has gone, you can feel its history and hear yourself think." Nina, a local, has her own take on the charms of being here in late summer: "It's when the real people come, the ones who know the secrets of this place but keep it quiet."

Whatever time of year you visit, you can't come to Hvar and not explore its back streets, for to do so is to miss its hidden treasures of quirky galleries and off-the-wall clothes shops. And if for some strange reason that doesn't entice, head past the elegant bas-reliefs and neoclassical loggia of the Palace Hotel – the former residence of a 16th-century duke – down a gauntlet dripping with turquoise and amber jewellery. Beyond, we watch as a fisherman heaves a five-foot giant tuna on to the quayside. Enraptured by the leviathan, my son gets all Hemingway-esque and declares he wants to go fishing, so I pop into the local supermarket and buy him a cheap telescopic rod. As we're wandering back to our hotel he points to a nearby island and asks: "Can we go fishing over there?"

We rise early the next day and hire a boat from Luka Rental on the corner of the harbour, which offers a four-man, 15-horsepower craft (for £40 per day) to get there. It's the ideal water-horse for exploring the Pakleni Islands on our own terms; we can drop anchor in any bay of our choosing, or putter about from cove to cove looking for a konoba (taverna) we like the look of. (If you don't fancy playing captain for the day, it's almost as easy to catch one of the water taxis that run regularly throughout the day from outside the arsenal on the harbour.)

The Pakleni Islands are largely uninhabited and are often described as being among the most beautiful in the world. More than two millennia ago the Illyrians believed them to have restorative powers for the soul; later still Jacques Cousteau, the marine biologist, commented he'd never seen such clear blue water.

The islands of Jerolim and Stipanska (the latter long fashionable for naturism) are a chance to wander through sun-dappled pine forests deafening with crickets, with the scent of honeysuckle in your nostrils and turquoise coves flashing through the trunks of twisted trees. September is also perfect for nearby Palmizana island, as the flashy sailing brigades have gone and there are two great restaurants to lunch at: Toto's, which sits on gently shelving Palmizana beach, or its nearby sister restaurant, Palmizana, which offers a menu boasting authentic Dalmatian dishes such as chicken with rosemary, gregada (seafood stew), and shellfish risotto, all washed down with a glass of locally produced posip (white wine).

After three days of relaxing by the pool, we move for the rest of our trip to the contemporary charms of the Adriana, arguably Hvar's most stylish hotel. With its glass façade and trendy rooftop bar, it smacks more of South Beach Miami than neoclassical Hvar, and yet somehow it blends rather nicely.

From there, we took a taxi for a day trip to the pocket-sized seaside village of Milna, three miles outside Hvar town. It turned out that this is a favourite spot for local subathers, drawn by twin bays and a handful of restaurants. We headed to the cosy Moli Onte restaurant, well known locally for its seafood and spit-roasted pig. Working our way through a carousel of anchovies, tuna, squid and dorado (spear-fished the day before), it was easily the best seafood I'd ever tasted.

On our final evening, as the sun was turning the bay a rich amber, we splashed out on dinner at Hvar's new rising star, the aptly named Divino. Occupying the former 19th-century mansion of one of Hvar's last great noblemen, its al fresco terrace enjoys one of the best views in town at the tip of the wharf. The menu, too, is suitably regal; a fusion of French and Dalmatian cuisine with a contemporary twist. As my son surreptitiously abandoned his burger and started to pick at my tenderised steak with sautéed mushrooms, rosemary and jus, I was busy looking through the restaurant's Ionic pillars at the fading light. Already I had that sinking feeling in my stomach.

Travel essentials

Getting there

Richard Waters and family were guests of Transun (01865 265200; transun.co.uk) which offers a week-long "Heavenly Hvar" trip, staying at the Adriana hotel with breakfast, flights from Gatwick, Birmingham or Manchester to Split, excursions and transfers. You can fly to Split from a range of UK airports on Croatia Airlines, easyJet, Flybe and Jet2.

Where to stay

The Adriana (00 385 21 750 200; suncanihvar.com). Doubles from €156 (£123), B&B.

Amfora (00 385 21 750 555; suncanihvar.com). Doubles start at €156, B&B.

Eating there

Mizarola Bistro (00 385 9879 9978; mizarola.com).

Palmizana (00 385 21 717 270; www.palmizana.com).

Moli Onte (00 385 21 745 025).

Divino (00 385 21 717 541; divino.com.hr).

Getting around

Luka Rent (00 385 21 742 946; lukarent.com; waterfront Hvar Town), hires cars from 450 kuna (£48), scooters from €34 (£27) and boats from €180 (£142) a day. Water taxis sail 7am-7pm from outside the arsenal for the Pakleni Islands.

Hvar Adventure (00 385 21 717 813; hvar-adventure.com) offers sea-kayaking, rock-climbing, walking, cycling and sailing.

More information

hvarinfo.com; croatia.hr

Click here to view Croatian tours and holidays, with Independent Holidays.