Linda Grant: The last time so many people enjoyed themselves this much in London must have been VE Day

There is a large group of people I would describe as Olympic miserablists.

These passionate Olympics obsessives take no pleasure in the Games except to pore over Panorama specials on the negative impact they will have on London, the stymieing of free expression by the sponsors, the corporate junketing, the corruption of the IOC, the siting of surface-to-air missiles on roofs, and the iniquity of the Zil lanes. The Olympics have seemed like an elite event, serving a few overprivileged athletes, the holders of corporate junkets, and the tiny proportion of the population who can afford the exorbitant ticket prices in this ugly, frightening recession. It has been a miserablist bonanza since the announcement of our winning bid seven years ago. To read the tweets of the miserablists, you would think that a convention of international despots and asset-stripping rogue traders had come to town.

I too was opposed to the Games, but in the sense of complete indifference and apathy. For the athletically inept, the playground phrase "Now everyone, pick teams" is one of the most stomach-churning in the English language. I don't care about any sport, and I could not really see what the Games, way out in east London, somewhere in Stratford, had to do with me. I don't live near the stadium. I didn't apply for tickets so I wasn't disappointed by the allocation fiasco or initially outraged at the sight of banks of unused seats. The Olympics were going to come and go without me really noticing, part of the unending summer of sport. There's always something else to watch on TV if only it didn't infect the news agenda so extensively. Those who whinged last year about the royal wedding, welcome to my world.

So all that was true until Thursday. I had an appointment in St John's Wood. Who were all these people streaming out of the Tube? What were those pink-clad individuals doing, milling around with clipboards? Why were so many people walking around with South Korean flags? An hour later in Harry Morgan's kosher-style nosherie, slurping that most unathletic of lunches, chicken soup with knaidlach, I found myself sitting next to the Mexican archery squad eating falafel and chips, on a break from the heats that were going on at Lord's cricket ground, round the corner. A South Korean Olympian came in and ordered two coffees to take away. Farther along the road, a Mongolian and a Belarusian archer were enjoying a beer at a table outside in the sunshine. Later that day, a photo appeared on Twitter of the Venezuelan fencing champion, on the Tube with his gold medal.

I had wrongly assumed the Olympians would be sequestered away in that far-off Olympic village, but here they are, all around us. Suddenly, sports clothes are being seen on the streets of London worn by world-record holders, not teenage looters. Something has come to town and it turns out that it's an extraordinary, democratic spectacle of passionate enthusiasts, mostly amateur, who will return home to day jobs in Latin American small towns rather than Cheshire mansions and pay cheques of £50,000 a week. Reading the biographies of the Olympians, I was struck by how much they were like the rest of us, except with that odd gene that gave them the right kind of body for rowing or judo and the single-minded will to pursue it.

The Olympics have not taken over the whole of London. In swathes of the city, they exist only as some Team GB flags in the windows of pubs and shops, but wherever you go you see people in Team GB kit, or that of other nations. Children seem particularly excited and caught up by it. Where events are concentrated, they have become mini Olympic villages, spilling out on to the surrounding streets. Suddenly, interested in sport or not, the penny drops. Something special is happening. We are in proximity to a few dozen people who are the best in the world within their particular sport, the fastest, the most flexible, or with the best hand-eye coordination. To the millions who follow archery, or athletics or diving or basketball, these people are world-class heroes, and they're just there, eating lunch.

What has happened to London is that, in the space of a few days, it has become filled with hordes of individuals and families who are happy. Who are delighted to be in London, delighted to watch their sport played the very best it can be played, delighted to wave a national flag with the enthusiasm that comes of knowing that, unlike our English football squad, we're actually going to win something. The last time so many people were enjoying themselves as much in London must have been VE Day.

Gradually, I have been drawn in to watching events on TV. I can swim, after a fashion. Seeing what Michael Phelps does on the underwater cameras makes you understand that his idea of the breaststroke and mine are two entirely different entities. Stripped, the swimmers reveal what the human body is supposed to look like if you don't lie around on the sofa watching Michael Phelps: the perfection of the human form that is Michelangelo's David. Look at the writhing flexibility of the gymnasts who turn their bodies into rubber bands on speed. Marvel at Bradley Wiggins's aerodynamic sideburns.

After Wiggins received his gold medal, he described the sense of melancholy he felt, standing on the podium, knowing that he had already achieved his goal; there was no topping this. I wonder if, in September, we will find that we too miss the Games, understanding that it is unlikely to happen again in the lifetimes of the adult population of this country. So many of the miserablist predictions turned out not to be true. The stadium was built on time. Londoners were not in many ways practically inconvenienced. The opening ceremony turned out to be a triumph.

In a month's time, it will be all over. The winners will have gone home. So many events – Diana's funeral, the Jubilee, the royal wedding – are ephemeral. You can't remember what it was you were caught up with, apart from the pleasure or emotion of the spectacle. The Olympics is different. Watching this month are future medal winners, who will go down to their local club and say, "I saw that at the Olympics. I'd like to give it a try." The rest of us will push on for another five minutes on the treadmill having learned a lesson about what the human body can do.

Linda Grant is an Orange Prize winner. Her latest novel, 'We Had It So Good', is published by Virago