Earls of Paradise, By Adam Nicolson

After the execution of Charles I, John Milton sneered that the king's final prayer had not been an original composition, nor even Christian, but part of "the vain amatorious poem of Sir Philip Sidney's Arcadia". Sidney's nostalgic dream of England, Milton believed, deluded the people and empowered tyrants. Now the axe had fallen, the nation would awake.

Arcadia was written in the 1570s at Wilton House, Wiltshire, seat of the earls of Pembroke. The first earl was a warrior turned power-broker, rewarded by Henry VIII with a dissolved abbey, where he built a lavish but architecturally illiterate house. Wilton became a symbol "of enormous riches, profound culture, extensive political and military control", and in the 1630s completed its transformation from Tudor mansion to Palladian "palace in the trees". Van Dyck was commissioned to paint a family portrait, a sublime icon of magnificence.

Awestruck coach parties might not appreciate what once really mattered at Wilton: intense and intricate relationships between house, land and people; mentalities and emotions effaced by time. This sensibility is Adam Nicolson's forte. As traveller and historian he escorts us backwards, encountering evanescent beauty and strangeness and absorbing both. We are shown the brutality of dominance and ambition, alongside the frailty of feeling and imagination. Earls of Paradise is an elegy aching with longing and loss, capturing a way of life in a turbulent era. Nicolson is a terrific writer. The countryside scenery essential to his drama, is described with transcendent sensitivity.

For Nicolson the Pembrokes embody Arcadia, an ideal realm "of bliss and beauty... driven by a hunger for the past and a fear of the future". Fantasy was rooted in political fact. Manorial life was governed by hierarchy, tradition and community – an organic ordering of duty and deference. Modernity – mercantilism, liberalism, statism – was anathema, but loomed like a storm-cloud over a perfect summer's day. Classical laments for the bucolic resonated in a society destabilised by over-population, inflation, religious turmoil and urbanisation. According to Virgil, "beauty on the point of disappearance had the added beauty of evening light, of impending loss as a glow on the untouched cheek".

This is a clever take on the early modern period: here, roughly 1540-1660. Every event from the Reformation to the Revolution has a Pembroke in attendance. Most startling is Mary Sidney, wife of the second earl, the poet's sister, and a Jacobethan culture-vulture of staggering ability. The Sidney siblings made Wilton a screen onto which Arcadia was projected, power encoded in the pastoral.

Their salon included dabbling scientists (like spider expert Dr Muffett) and aspiring literati. Such was Mary's influence, it seems she persuaded Shakespeare to rewrite As You Like It for a royal performance at Wilton, thus moving James I to pardon her ex-lover, Sir Walter Raleigh. She continued her brother's translations of the Psalms, an eloquent expression of Wilton's marriage of Protestant piety and Renaissance court culture.

For all its airy humanism, the Pembroke philosophy rested on inequality and authoritarianism. Nicolson sometimes seems too misty-eyed, although he properly describes the misery of the poor. This was an England where fewer ordinary people owned land and so fell upon the vicissitudes of the market. The Wilton estate covered 50,000 acres, and the family spent £800 on shepherds' costumes for a masque when a real shepherd earned £12 a year. Nicolson observes that the place of the term "privilege" was then occupied by the purer "nobility". Yet nobility led to cruelty and resentment, and the Pembrokes must share the blame for the dissipation of their dream. Rents rose, pasture was enclosed and dissent silenced. The pleasure park "erased the custom of the manor", and when in 1549 tenants tried to restore it – part of a wider rebellion – the earl chased them like game and slew them like a foreign foe.

This was the first wave of a long farewell. Within a century family fortunes had peaked, and for all its grandeur van Dyck's portrait of 1638 was a prophetic tableau of "transience and fragility, of failure and disconnection". Four years later, the question of the relationship between monarchy, aristocracy, legislature and commons demanded an answer. In the 1620s, the grasping pro-Catholic, pro-business culture of the duke of Buckingham had undermined the Pembrokes' "model of conservative wholeness", a depredation continued by Charles I's absolutism.

As Wilton was sucked into civil war, dividing kinsmen and breaking hearts, Arcadianism was appropriated by Parliament as a defence of the ancient constitution against a king who had desecrated the sacred bond. "If the lord had betrayed the copyholders of the manor, the copyholders no longer owed him any allegiance." This development makes Charles's scaffold prayer all the more ironic.

The destruction of the 1640s represented everything the guardians of Arcadianism had feared. Communities were morally adrift, country houses became strongpoints, and in Winchester, Cromwell's troopers fashioned kites from the cathedral's muniments, seizing the present, mocking the past. Most Pembrokes had backed the winning side in order to reform the monarchy and reinvigorate the nobility. These outrages suggested they had failed.

Individualism poisoned pre-lapsarian Wilton, consumerism replaced custom, and "Arcadia in the 18th century became décor, not a hope for society". Rural poverty deepened, and the dark satanic mills inspired new generations of Arcadians from Blake to William Morris to New Age pagans.

"The elite dream of happiness" in England evaporated, and found its best opportunity abroad. In America, wholesome exceptionalism was pitted against European decadence, although here too Arcadianism would be trampled by the leviathans of state and market. Milton understood what Sir Philip Sidney did not: after Eden, man's desire would bedevil his dreams of paradise forever.

Malcolm Gaskill is reader in history at UEA, Norwich; his book 'Witchfinders' is published by John Murray

Harperpress £25 (298pp) £22.50 (free p&p) from 0870 079 8897