The Science of Success: Hamish McRae on what works – and why

By any rational calculation, the world is better placed now to make good choices about the future of our species and our planet.

We are better educated than ever before; have better health and better information; technology continues to leap forward; and, of course, we are – notwithstanding the odd bump – richer than ever before. But if we are to face the many challenges we have to try to learn from each other. We all have to do it. It is the millions of acts by ordinary people that will eventually make a difference.

Several years ago, I embarked on a project to pinpoint the secrets of human success; those groups and organisations that were capable of succeeding in good times and bad. I found 20 examples, from every corner of the planet – three of which, in Edinburgh, Copenhagen and Hong Kong, are noted here (full versions of these, and the 17 other examples are in my new book, What Works, published by Harper Press).

The list encompasses organisations and communities as diverse as the world's best university and the world's best slum. Of course, none of these stories is a tale of perfection and each example has flaws. But they are examples of collective human endeavour that I have found both humbling and inspiring and which I believe have powerful common messages for all of us. By understanding what works, we can make other things work better – the aspects of our daily life that we can improve.

My project began as a desire to report, to share stories. Of course, there have been many studies of success, particularly business books promoting apparently successful forms of organisation, usually with an ideology attached. I worked from the other end. I observed success and tried to draw lessons from it.

What became clear pretty early on was that for organisations to work really well, they needed to combine two features.

* They had to have a deep-seated sense of mission – a vision, drive and commitment to do something that is worth doing even better.

* They had to be acutely sensitive to the market. They worked because they went with the grain of the market, listening to its signals and being guided by them, and applying its disciplines and adapting to whatever these required.

In this article (and at greater length in my book) I describe how I came to include each example, and try to capture and explain its special features, seeking to draw out some general lessons. Then, because nothing progresses in a straight line, I have to acknowledge what may go wrong, and in some instances has gone wrong.

Two other points emerged during the research and writing. One was that there has been a host of 'near misses' – things that ought to work very well but have in some way fallen short. The other point that became clear is that what happens in a global economic downturn is more interesting than what happens in the good times.

I have tried to look through the current difficulties evident in several of the case studies, so as to make a judgement about the future. If I had not been confident, I would have dumped the example and chosen another. As it has turned out, I have not had to do this in any instance – for what seem to me to be sound reasons that I try to explain. Of course, I face the charge of being an eternal optimist, but I would rather answer to that than be a Dr Doom.

The world is at one of those inflection points that historians will look back upon with awe. Economic power is rebalancing away from Europe and, to some extent, North America and principally towards Asia. It is the biggest shift of power since the Industrial Revolution, which enabled Europe and then North America to leap ahead of the more populous Asia. Most people in the West are vaguely aware of the rising power of the so-called emerging economies, particularly of China, but have hardly begun to think through the consequences of this.

This shift is not just about economics; it is also about ideas. We are moving from a period when most of the ideas that have driven the world economy have come from the West, to one where many will come from the East – and the rest of the emerging world. My aim here, in starting from a series of stories, is to illustrate how there are great ideas and innovations in Western societies but also similarly great ideas and innovations in the emerging world.

That surely is the future. Good ideas will increasingly come from anywhere and everywhere. We need to learn from each other. You cannot, of course, lift one way of doing things, transplant it to the other side of the earth and expect it to flourish. You could not create an Edinburgh Festival in China and you could not transport Hong Kong's lean government to the welfare states of western Europe – though there are some things we could learn from that. Not every city could tame the motor car on the Copenhagen model. Yet I hope we can all learn a bit from each of these stories, and try to apply some elements of their success. We can also learn from the weaknesses. While every single example in the book is one of something that has been successful, every single story in some way encompasses threats that must be overcome or flaws that must be fixed. One of the great puzzles is the way in which clever, thoughtful and decent people can get things wrong.

What interests me most about these examples is partly the extent to which the baton of success can be passed on from one generation to another, but it is also the extent to which an entity has the capacity to correct its course when the winds change. The winds changed sharply during my research. It has made for a much more interesting world, a much more challenging one, and one in which we should dump ideological explanations and responses – and simply build on "what works".

The Edinburgh Festival

What's the story?

A shouted warning for the audience to stand back, and a burst of fire from a human flame-thrower surges towards the crowd of onlookers on the Mound. A troupe in silver body-paint hands out leaflets for the night's performance. Down the High Street, a jazz band bangs out the sounds of New Orleans. For it is August in Edinburgh and the prim grey capital of Scotland is once again home to the largest arts show in the world.

It is an extraordinary, if improbable achievement – and one that many other cities would love to emulate. How has Edinburgh done it?

The short answer is slowly. This is not one festival but – depending what you include – 10. Each reinforces the others, giving the city an artistic critical mass that makes it impossible to topple. There is the original arts festival, organised, like so many others, by the city authorities. There is a jazz and blues festival. There is the book fair, the largest of its kind on the planet. There is a film festival, the longest continually running one in the world. There is a television festival. In 2003, a video games festival joined the clutch. There is the Mela, a celebration of life in the Indian sub-continent, run by Edinburgh's Asian community. In 2004 the city added a visual art festival for the first time. In a slightly different category from all the rest, there is the Edinburgh Tattoo, where military musicians – again from all over the world – put on a show.

But Edinburgh's special feature, the thing that distinguishes it from every other celebration of artistic endeavour, is the Fringe – the open access given by the city to the thousands of events that take place in August. Others have tried to copy it. None has really succeeded.

The story, though, offers a lesson for anyone wanting to run an arts event. Back in that drab aftermath of the Second World War, many cities sought to recapture the life and joy of pre-war Europe. In 1946 and 1947 respectively, Avignon and Edinburgh both started arts festivals – the pattern being the classic one where a group of civil and artistic leaders invite companies to bring their acts, organise venues – and usually offer subsidies to get them to come. The original Edinburgh International Arts Festival was exactly that. But in the first year something happened that changed Edinburgh and the arts world for ever.

Eight groups that had not been invited, six from Scotland and two from England, decided to gatecrash the show. They found their own venues, stumped up their own money and put on a performance. That first Fringe has defined the movement ever since: no performers are invited – there is complete open access; they use unconventional theatres; and they carry all the financial risks themselves. More came the following year and an Edinburgh journalist pointed out that interesting things were happening on the fringe of the main festival – and so coined that expression to describe them.

Since then, the Fringe has gradually acquired a modest infrastructure. The first programme to bring the various independent acts under one loose umbrella was put together on the initiative of a local printer in 1954. A box office run by Edinburgh students followed in 1955, and the Festival Fringe Society in 1958. The event became famous across the UK in 1960 after the success of the comedy show Beyond the Fringe, but the first full-time paid employee was not appointed until 1969.

The Fringe raced on, getting into the Guinness Book of Records as the globe's largest arts festival in 1992. In 2009, an estimated 19,000 performers took part in more than 34,000 performances at more than 2,000 shows in 265 venues. The event generated £75m for the economy. Those figures beat all records by a huge margin. But how? Once the Fringe was established as the premier showcase for British, later world, theatrical talent, it was natural that Edinburgh in August should attract other festivals too. The market was largely ready-made: people who are interested in new experimental theatre are probably also interested in more conventional drama, in classical music and jazz, in new books – in all the other experiences that Edinburgh offers. For a young performer to get noticed at Edinburgh can be a life-changing experience. Win one of the top awards and nothing will ever be the same again. And for the more mature critics and impresarios, as well as the ordinary punters, it is "the chance to see it before it happens".

What are the lessons?

Edinburgh has long had a lot of things going for it. It is, physically, the most beautiful city in Britain, with its castle, its gardens, its medieval Old Town and its Georgian squares. It is a capital city and – important in the entertainment world – an English-speaking one. But none of this, of course, would have been enough. There are at least three special features about the Edinburgh Festival that carry a message for other cities seeking to develop their own special face to the world.

Lesson one is the willingness to create and permit a completely open marketplace. This means accepting that what happens cannot be controlled. Edinburgh has tended this marketplace wisely, not by piling in huge amounts of money or building infrastructure, but rather by clearing bureaucratic blockages that might stifle it. For example, one of the keys to the Fringe's success is the use of unconventional performance spaces, often in old buildings designed for another purpose. That means applying sensitive fire and access regulations – to make sure audiences really are safe – rather than insisting that venues should fit box-ticking requirements.

It also means accepting that the city will, for one month, be a quite different place from what it is during the rest of the year. Residents and businesses alike in effect lose control of the centre of their city. Were it badly managed, the disruption could damage the core activities that drive the city through the rest of the year. All tourist centres have to cope to some extent with surges of visitors with different values to the locals, but this is extreme stuff. The lesson therefore is not just to permit the creation of a market but also to relish it.

Lesson two is to blend top-down and bottom-up. There is no single mind planning what happens in Edinburgh; there are and always have been lots of minds, which work in different ways.

Some of these, such as the director and governing body of the International Festival, have to exert a top-down discipline. The companies performing have to be invited. Funding has to be found, venues secured and the events publicised. To get the right mix, there has to be some artistic direction.

At the Fringe, by contrast, the minds have to focus almost exclusively on logistics. They do not concern themselves about the artistic merit of the performers; all they have to do is make sure that anyone who fills the basic requirements is able to set up a show, for this is entirely a bottom-up exercise. There is, however, one crucial function that the Fringe performs beyond logistics. This is teaching.

Every year it holds a series of seminars to show would-be performers and promoters how to put on a show. These include: how much the different venues will cost; how to manage publicity; the timescale for decisions; the need to go for as long a run as possible to cover costs, and so on.

The trick, which the various organisers of the Edinburgh festivals have managed to pull off, is to achieve balance – to plan but not to over-plan, to lead but also to follow the demands of the market.

That leads to the third lesson: the need to listen. This has been central to Edinburgh's development at three stages.

First, what started as a conventional arts festival, and might have remained so, was swiftly transformed by the demands of the market into something much bigger.

Second, in the middle years, Edinburgh allowed market forces to develop the Fringe, rather than trying to stifle it.

Third, whenever a new festival wanted to tag along, it was welcomed. So films and TV, jazz and books were all grafted onto the official and Fringe core.

Edinburgh creates a platform not just where people can feel free to experiment but also one where they do not need to worry if it does not work. There is surely a wider message there: individual failure is an essential part of the wider success of almost all enterprises – and absolutely essential to a venture as huge and amorphous as the Edinburgh Festival.

Copenhagen city centre

What's the story?

Cities are for people. Walk out in Copenhagen on a summer evening and the streets are full of ... people. But Copenhagen has also managed, despite its northerly latitude – the same as Glasgow – to become a nine-month summer city, where the café society spills onto the squares and streets to all but the very darkest of winter days and nights.

How has it done it? Well, it could not tame the weather but it tamed the motor car.

All cities of the developed world face traffic problems. But Copenhagen is special because its medieval core is no longer a place where people drive or take buses; they walk. 80 per cent of city-centre journeys are on foot, another 14 per cent by bicycle.

This transformation has brought a string of benefits. One is economic. Walking is the most efficient way of using crowded city centre space: without cars or buses, thousands of people can navigate a narrow street. Another benefit is social. As cars were discouraged, the heart of the city was used by a much wider band of ages – more children and older people. And with the streets evolving into centres of recreation, rather than transit zones, so they have come to feel safer.

How has all this been achieved? It took a vision. Enter Danish architect and professor of urban design, Jan Gehl, who has become the global leader of a movement to improve the quality of life in city centres. But it has also been driven by long-term backing from Copenhagen's city authorities since 1962, when the main shopping street, Strøget, was pedestrianised. Successive municipal leaders have cautiously made the series of incremental changes to streets and buildings transforming, over a generation, the city. The car-free area was 15,000 square metres in 1962, 95,000 by 1995; now the two-kilometre-long Strøget is the central artery for a network of pedestrianised blocks, with over 100,000 square metres set aside for walkers.

As this has happened, not only has the number of people shopping increased so, too, the number of people carrying out some kind of stationary activity; having a cup of coffee at a pavement café, watching buskers, sitting on a bench or simply standing chatting. This has all been measured. So we know that on a typical summer day in 1968, 1,750-odd people were involved in such a stationary activity at any one time; by 1986 it was over 4,500 and by 1995 to nearly 6,000. As a result, the streets here are being used in a different way. No longer just a way of getting from one place to another; they are in themselves a destination.

Just shutting off streets from cars, however, does not in itself guarantee a safer, or more pleasant urban environment. There are plenty of examples of pedestrianisation creating urban wastelands. Getting rid of cars has sometimes meant getting rid of business activity. Copenhagen has taken a number of steps to avoid this fate. First, attention to detail in the planning means traffic is not simply pushed from one street to another. Second, street furniture and paving have been improved. Third, Copenhagen encouraged owners and tenants to make street facades more people-friendly. Finally, it has encouraged the use of bicycles.

No city is perfect. Cyclists can behave as selfishly as other citizens, and Copenhagen has its social and economic problems. The key point is that by thinking carefully about the purpose of a city centre as a cultural and social magnet, as well as a business one, it has managed to cope more effectively with traffic pressure than any other city on the planet.

What are the lessons?

The first conclusion is that it is possible to make radical improvements in the quality of city life by looking at the space between buildings rather than the buildings themselves. Unsurprisingly, Copenhagen is studied by urban planners the world over.

But all cities are different. Copenhagen has a medieval core, with a street pattern originally based on pedestrian traffic and carts. It is largely a single-level city: no underground railway system, no underground shopping malls, no pedestrian footbridges. It has few privately-owned shopping arcades drawing people off the streets and squares. It is also flat, and so cycle-friendly.

In a sense Copenhagen has been forced to use its street-level public spaces better – but it has also taken the opportunity to do so.

For lessons, though, it is best to start with the medieval core – even though few non-European cities have a similar central zone laid out hundreds of years ago. The most important lesson here is to understand the advantages in turning that core back into a place where people walk. For 10-minute journeys, most people are happiest on foot.

But if they are to walk, there has to be enough space not to feel crowded. When they cross a road, the lights have to be designed to give them – not other traffic – priority. They must not be forced up or down steps, into underground passages or onto pedestrian bridges. So it is not enough to get rid of cars. Lots of design details have to mesh together.

Some of these changes can be made easily, but others imply radical rebuilding. For example, the standard tall, slab-sided office block of the developed world can whip up vortex effects, increasing the winds at street level: not pleasant for walking. They also make it impossible to create places where people will linger, such as outdoor cafés. But you cannot tear down every office block.

Copenhagen starts with the advantage of being largely on one level; its old core mainly made up of four- and five-storey buildings. But it suffers from slab blocks outside the centre and not much can be done about that, except not to repeat those mistakes in the future. Designers and planners now at least, though, have the advantage of having learnt the lesson about these building's impact on the spaces in between.

That leads to a more general point about coping with weather. Copenhagen has sought to turn the city into a place where people will use outdoor public space in winter as well as summer. Outdoor cafés supply blankets for their guests and there are space heaters under large umbrellas. And the city is looking at other ideas: big entertainment screens that emit light on dark evenings; heated benches; skating rinks and so on. But the important change – and this applies to cities in all cold locations – is not to fight the winter but to celebrate it.

Perhaps the biggest lesson of all from Copenhagen is that city centres work best as places where a mixture of functions can flourish. They have long been centres of commerce as well as leisure and entertainment; but they also work better if they are residential centres too, particularly with young student populations.

The thing to avoid is letting any one function dominate or being too rigid when new projects are on the drawing board. The parts of Copenhagen that work least well are the planned blocks and streets of the 1960s; since then, the market has been allowed to signal what people want, not what planners think looks good on paper. The authorities have worked with the grain of demand to decide on the best use for the different parts of an ancient city. This has worked remarkably well.

The Hong Kong Jockey Club

What's the story?

If you haven't been to Hong Kong, do go. The view across the water to the island is one of the world's great urban sights; the food, hotels and transportation are first-class.

But, arguably, more impressive are the people: successful, hard-working, and rightly self-confident. There is, though, something more: a sense that this is a society that is honourable, honest and free.

The 7 million residents of Hong Kong enjoy a greater degree of economic independence than any other people on earth: the city-state regularly comes top of the world on this measure, and did so again in 2009. There is a standard rate of 15 per cent for income tax in Hong Kong, while the top corporate rate is 16.5 per cent. It takes less than half the world average of 38 days to start a new company there. Barriers to trade are low, with an average tariff rate in 2006 of zero per cent, and Hong Kong encourages foreign investment by not applying ownership restrictions.

What I find particularly intriguing is a government that controls some things extremely closely, yet allows huge freedom elsewhere. One example is state-run gambling. The government is behind the Hong Kong Jockey Club, which provides 6.5 per cent of the city-state's revenues – quite some contribution.

And the money is used well: Hong Kong is second only to Japan in terms of life expectancy, and has the fifth-lowest infant mortality rate. It scores highly with education, too – far ahead of the US and UK, and second only to Finland in science scores. And yet it finances a significant proportion of this spending from a gambling monopoly. With good management, the Jockey Club has managed to bring in this revenue since 1884.

There is a certain air of calm at Hong Kong's two race courses, Happy Valley and Sha Tin: all betting is on the tote. There are no bookies; no one is making the odds; everything is electronic. All off-course betting is run by the Jockey Club. In one sense, perhaps a monopoly is too easy; on the other hand, were it run by the private sector there would surely be more excess, probably more corruption, maybe more misery.

That leads to the much bigger story. Just as the Jockey Club gives freedom within defined limits for people to enjoy themselves, so Hong Kong's government does the same for people's livelihoods. Economic freedom is about doing the things you want to do, but within the limits that society has to set out.

Maybe most notable of all is the upward direction Hong Kong has taken in terms of economic freedom since it became a special administrative region of China. If anything, it has improved its position in this regard since the handover in 1997.

I am not suggesting that the Hong Kong government is an ideal model for the rest of the world. Rather, that there are elements which carry messages for the rest of us.

What are the lessons?

Hong Kong's attitude to governance is almost Victorian in its self-confidence, sense of morality and partial authoritarianism. I have chosen its management of gambling as a microcosm of the way it runs the whole economy, controlling some things very closely though seeking to deliver quality and value in the facilities it provides.

However, in terms of government spending as a percentage of GDP, Hong Kong's is very low, at around 15 per cent. How does it manage to provide the social and other services at that level of spending? The answer is that people pay a contribution, even for services such as healthcare that are state-supported. The negatives? People who lose their jobs will suffer, as will those who fail to save for old age. There is some safety net, but not at the level of western Europe or North America: this is a different deal. In Hong Kong the state pushes the responsibility of providing for themselves onto the people, only stepping in when absolutely necessary.

At a time when most countries in the West are seeking to push more responsibility back to their citizens, it seems to be improving its overall performance. There is surely a message here. Western governments find themselves unable to generate sufficient tax revenues to sustain the state infrastructures they have created. Demographic change has made the arithmetic more alarming, as each generation of old people relies on a smaller cohort of people of working age to pay for their pensions and healthcare.

And in one regard this problem is even greater in Hong Kong: its fertility rate, at an estimated average of 1.02 children per mother in 2009, is lower than in any western European country. But its government model is ultimately more robust. That is going to matter more and more as this century unfolds. How do governments meet the aspirations of their people without getting bogged down in the conflicting charges that they are not providing good enough services, or providing them at too high a cost?

You might say it is rather eccentric that the Hong Kong government should supply a service most people would not regard as essential – gambling on the horses – but be rather thin by developed world standards in its provision of state support for education, healthcare and pensions. But governments often take commercial activities under their wing if they feel their people need particular protection – take Sweden's monopoly over the retail sale of most forms of alcohol. So if a society judges that its public needs shielding – be it from excessive drinking or excessive gambling – then these governments' response is rational and right.

One broader conclusion is that there is no universally appropriate boundary between the state and the private sector, either in the provision of services or in the financing of them. Lots of different models work perfectly well. Yet, on the other hand, there is a case for nudging people to take responsibility as far as is practicable for their education, healthcare, pensions and so on. It preserves state resources for helping the people who really cannot help themselves, while also making it easier to deliver economic freedoms.

Those freedoms, or at least most of them, surely matter – especially if they help deliver greater growth and hence higher living standards. And it must surely also be worth making it easier for people to start new businesses, protect their savings, or to spend more of the money they earn in the way they want to.

My point is not that Hong Kong is the right form of government for the world. Rather that there are elements of the Hong Kong model, both its restrictions and its freedoms, which other governments should take very seriously. This system has performed effectively under such different political umbrellas as British colonialism and Chinese Communism. So it must have something going for it.

'What Works' by Hamish McRae is published this month by Harper Press (£12.99). To order a copy for the special price of £11.69 (free P&P) call Independent Books Direct on 08430 600 030, or visit www.independentbooksdirect.co.uk

McRae's 10 Laws

1. Optimism – balanced by realism: Pessimism paralyses

2. Excellence – tempered by decency: If you neglect your wider responsibilities, you're liable to end up in trouble when you meet headwinds

3. Community works – If it is allowed to: Look at things from the ground level up and mobilise community

4. Government works too: Compare like with like

5. Become a true magnet for talent: Put out the welcome mat

6. Be honest about failure: Keep learning, keep making mistakes

7. The need for humility: Be as sensitive to success as you are to failure

8. Be nimble: Make sure you are quick to adapt

9. Listen to the market: Remember, it's about more than money

10. Have a sense of mission: Keep the long game in view and do right by those who share your objectives