Turf wars: Tales of fashion, greed and violent disorder that lie beneath the grass and behind the railings of London's elegant squares

London may be famous for its glorious parks, but it is unique for its squares. The history of the square charts the capital's development and reflects enormous changes in fashion and society. During Open Garden Squares Weekend (12-13 June), there is a rare chance to see many squares that are otherwise for residents' eyes only.

Most Londoners will have passed a private square, perhaps near where they live or on the way to work, and wondered what lies inside. I've often tried to peer through their railings and impenetrable ancient trees, yet the gardens remain a mystery. The desirability of gardens kept under lock and key has resonated in literature through the centuries, from Chaucer's The Merchant's Tale to The Secret Garden and Alice's Adventures in Wonderland, where Alice gazes longingly through a tiny gate at a lovely garden which she has to shrink to enter.

Since private gardens in squares in London are the exception rather than the rule, their exclusivity largely goes unquestioned. The very existence of such gardens makes a striking contrast with New York, where there is only one exclusive garden square, Gramercy Park. Few people may realise, however, that the land occupied by famous squares such as Leicester Square has been contested and even fought over. These communal oases of peace and tranquillity have a turbulent history, encompassing natural disaster, crime and riots.

The original London squares were luxury developments in the Italian style. In the 1630s, the architect Inigo Jones created the capital's first square in Covent Garden for the 4th Earl of Bedford, taking inspiration from French and Italian sources: the Place Royale in Paris, now the Place des Vosges, and Italian piazzas focused on a church, such as those in Venice and Florence. Before the dissolution of the monasteries, the land had belonged to Westminster Abbey – early maps show it as "Convent Garden" – and early squares owed much to the medieval cloister. The south side of Jones's square, known as the "piazza", bordered the Earl's own garden, creating an extra dimension for his property and a feeling of airiness, as well as beautiful views for residents. Three sides of the piazza had houses fronted by arcades, and the chapel stood on the west side. At the centre stood a platform with a small tree, surrounded by benches.

Around the same time, Robert Sidney, 2nd Earl of Leicester (the brother of the poet Sir Philip Sidney), set about remodelling the land near his new house and garden. Once a popular duelling ground, the neighbouring fields where locals dried their clothes on bushes were transformed into leafy avenues with grass plots. Access to the newly formed Leicester Square came with conditions. People were still allowed to do their laundry and had "free use of the place", but grazing was banned.

After the Great Fire of London destroyed many of the capital's markets, a new site for a grand central market was found in Covent Garden's piazza, leading to its expansion. The fire created numerous opportunities for developers. Sir Christopher Wren designed Golden Square in 1673, and one of the city's first garden squares, Soho Square, also rose from the ashes. Soho was a hunting ground ("Soho" was the cry that went up on catching sight of prey) until it was built on to create housing for aristocrats who had lost their homes in the fire.

Ordinary people's rights of access to the gardens of Soho and Leicester squares did not last long into the 18th century, when there was widespread enclosure of common fields. John Strype described King Square, as Soho Square was also known, in 1720 as "a very large and open place, enclosed with a high pallisado pale, the square within being neatly kept, with walks and grass-plots, and in the midst is the effigy of King Charles the 2nd, neatly cut in stone, to the life, standing on a pedestal". The gravel paths were lined with lime trees, symbolising royalty, and the platform for the King's statue was ornamented with river god fountains, representing the Thames, Severn, Tyne and Humber. In 1748, a gilded statue of George I was placed at the centre of the newly railed Leicester Square.

The great squares chart London's westward expansion. In the 18th century, the West End was viewed as the capital's third-most healthy area, after the royal riverside, which encompassed the palaces of Westminster, St James and Somerset House, and the City's Square Mile. Elegant squares were clearly viewed by investors as a good bet. "New squares and new streets are rising up every day to such a prodigy of buildings," marvelled Daniel Defoe, "that nothing in the world does, or ever did, equal it except old Rome in Trajan's time."

But London was a divided city, and it was a battle for developers to protect their land. By 1726, the garden of London's most aristocratic square, St James's, home to the Prince of Wales (the future George II), had fallen into a "filthy condition" and was described as "a common dung hill". At the centre stood a carriage builder's shed and a few trees. The residents made history when they petitioned Parliament to create legislation enabling them better to maintain their property. The garden designer Charles Bridgeman, later the creator of the ultimate landscape garden, Stowe, transformed the space in just two years from a rubbish dump into the most celebrated square of its day. The area was entirely paved, and its centrepiece was a large oval basin surrounded by a gravel walk and iron rails forming an octagon, with stone obelisks surmounted by lamps at each angle. Bridgeman's fee was a colossal £5,630: gardening – or hard landscaping, in this case – had become a highly lucrative occupation. His choice of paving stones over plants might seem strange, but there was a widely held belief among gardeners that the city air was so unhealthy that flowers could not thrive in it.

Developers had plenty of money to play with, judging from the number of garden makeovers. Soho Square underwent two transformations within the century. In 1748, it was redesigned in octagonal form, with an iron railing, gates and lamps. The basin was reshaped into a circle, and the flowers and dwarf trees were removed, leaving lime and philadelphus trees and plain grass plots. In 1790, the garden was remodelled with naturalistic planting, in line with the vogue for the picturesque. The residents' committee included Sir Joseph Banks, the great plant collector and founder of the Royal Botanic Gardens at Kew.

Towards the end of the century, the safety of London's latest prestige developments was threatened by a series of popular revolts. During the anti-Catholic Gordon Riots of 1780, squares were prime targets since so many establishment figures lived there. Some 12,000 soldiers and policemen were drafted to restore order; foot patrols surrounded Red Lion, Bloomsbury and Queen squares and Lincoln's Inn Fields, and horse patrols guarded Leicester, Soho, Cavendish, Portman, Hanover and Manchester squares and the neighbourhoods of St James's, Grosvenor, Berkeley and Golden squares. In 1792, John Wilkes's decorative windows at 35 Grosvenor Square were broken by the Mount Street rioters. During the Corn Law protests of 1815, rioters invaded the Lord Chancellor's house in Bedford Square, smashing windows, tearing up railings and destroying the ground-floor apartments.

Squares inhabit two worlds: although private in terms of housing and, in many cases, access to their central gardens, they are also public, as part of the fabric of London. By the 19th century, the scales had tipped too far in favour of exclusivity, according to the writer and designer J C Loudon, who looked forward to a time when the gardens of the great squares would be open to the public, envisaging layouts which would "interest and satisfy the mind".

Numbers had increased dramatically, from 13 squares at the beginning of the 18th century to 114 by the mid-19th century. But the square's popularity would lead to its downfall. With the proliferation of new squares, people inevitably tired of the form. Although they still saw its value in central London, as a source of light, space and better air, in the suburbs, many preferred informal styles of housing and private gardens of their own.

The riverside, with its dramatic, London Eye-dominated skyline, has replaced Trafalgar Square as the most spectacular site for national celebrations. The many transformations of the capital's squares, however, suggest that they might recapture the public imagination. The pedestrianisation of the north side of Trafalgar Square is a step forward. But what might really revitalise this celebrated square could be to carry out Richard Rogers's proposal to link it with the river and the South Bank through the creation of walkways, a waterside park and islands.

London's most famous square, after Trafalgar and Leicester squares, is of course fictional: EastEnders' Albert Square, centred around its tiny garden and all-important pub, helps to create a sense of community (no matter how dysfunctional) that is still valued. Successive mayors of London have pledged to make the city more "liveable", a priority being to improve air quality. Two hundred years ago, faced with similar problems, planners built square upon square, creating a distinctive cityscape celebrated by foreign visitors such as the German author of a guide to England: "the squares in London offer such objects to the eye as announce the opulence and good taste of the inhabitants: those who reside there, besides this, have the advantage of breathing a pure air, and are never disturbed by any noise". London Parks & Gardens Trust's Open Garden Squares Weekend presents visitors with the opportunity to have a fresh look at those idiosyncratic green spaces, public and private, which have so enhanced the city and the lives of its inhabitants.

Trea Martyn is the author of 'Elizabeth in the Garden' (£18.99, Faber and Faber). To read her blog visit Gardengoer.blogspot.com

One ticket allows entry to all venues over entire weekend. Tickets bought in advance cost just £7.50 and £9 during the weekend. These can be bought from the ticket hotline on 020 8347 3230 (Mon to Fri 9am to 6pm) or through www.capitalgardens.co.uk - both provided by Capital Gardens, as well as the Britain and London Visitor Centre on Lower Regent Street.