Augmented reality: Even better than the real thing?

It's an average city street: buildings, buses, people with shopping bags. Suddenly information bubbles pop up in your field of vision. The woman over there is called Jane and right now she's listening to Florence and the Machine on her iPod; you don't know her but you have five mutual friends on Facebook. The building opposite was built in 1932, is 31m tall and down the road is the boarding school Russell Brand went to. In the alleyway 10m behind you, there was a mugging at 1.15am last Wednesday, two streets away TopShop is having a sale and their summer range is reduced by 20 per cent, there are 14 coffee shops in a one-mile radius, and the flowerbed to your left has won "the most imaginative civic planting" award four times in a row.

If Arnold Schwarzenegger's Terminator had walked through, say, Bishop's Stortford, this is the sort of information his computer-enhanced vision might have provided. But "Terminator vision" is no longer just in the realm of science fiction films. It's called augmented reality and it is on its way to a smartphone near you. Complete Terminator vision would require bionic contact lenses ... but from this month, anyone with an iPhone will be able to peer at the world through the phone's camera and will see layered on their phone screen extra information about the physical things in front of them. Phones with a phone that supports Google Android can already do this.

Virtual reality used to be technology's Holy Grail. It let you create new worlds where you can jack cars, date babes and win wars. In contrast, augmented reality – in which computer graphics are layered onto a real world image – was the boring sub-technology confined to sports footage replays and technical engineering. That's what shows you the path of a tennis ball after a player has hit it, or demonstrates to an engineer how to piece a complex machine together by modelling it in 3D.

But with augmented reality about to be opened up to the mobile phone-owning masses, it has become an exciting field for development. Developers are racing to find useful and interesting ways that computers can enhance our interaction with the real world. And that could be by superimposing reviews on restaurants, directions on streets and Facebook profiles on people. It could be trivial, it could be fascinating. Perhaps the most useful application hasn't been figured out yet.

The technology that has brought augmented reality to mobiles is called Layar. As the name suggests, it layers computer information on top of "reality" as seen through the phone's camera. Layar uses the phone's Global Positioning System (GPS) to work out where you are and what you are looking at. Different types of information appear on different layers: there's a sightseeing layer called Wikitude which contains tourist information about what's around you – the posts are linked to specific buildings and points of interest (the Tate Liverpool, the Belfast docks, Big Ben) and pop up, like virtual blue plaques, as you pass by them.

There's an estate agent layer which flags up properties for sale or rent: as you scan your camera down a street, available properties are highlighted. Of course, a board outside a house would tell you as much, but the layer, called Trulia, will also tell you how much the house is going for, give you pictures of the inside and a phone number that you can ring to arrange a viewing. You don't only see things in your viewfinder – properties two streets away will show up too. There's a layer which tells you where the nearest council facilities are and plots them on an image of the street.

More interesting than the location of your local dump is information about your friends. And that's what applications like Brightkite offer – by bringing social networks into augmented reality. Partnered with Layar, it offers an app which lets you see where your friends are on a real-time map. Like the magic map in Harry Potter, they appear as moving dots. Even more excitingly, you can create posts – pictures, messages, videos – and then anchor or "geo-tag" them to the places you made them. They stick in that virtual space like graffiti or post-it notes and you can view them later. You can see your friends' posts or other peoples'.

Say you're in Trafalgar Square – by looking at Nelson's Column with your phone's camera, pictures of friends posing with it from three months ago will swim up and fix on your phone's screen, and so will a tweet you wrote there last year. Depending on what information you want, you can also call up reviews of nearby restaurants or of current plays showing down the road.

Gradually, layer upon layer of this virtual content will build up. Some posts will be more interesting than others, but, like with the rest of the internet, we'll get used to searching and filtering for what interests us. Other applications of augmented reality lead us right back into Terminator territory: "There was a project," says Dr David England, computer science lecturer at Liverpool John Moore University, "where people wearing visors and backpacks re-enacted a virtual Pacman outside, so they ran down the road shooting the enemy they could see in their visors," he says, referring to the popular computer game.

"Human Pacman" took place in Singapore University, but similar though less-exciting experiments with augmented or mixed reality gaming have taken place around the world. "The main problem is visor quality," Dr England explains. "When mass-produced visors improve, you'll see a lot more of this kind of game."

It's called pervasive gaming and a group set up to explore it, IPerG (Integrated Project on Pervasive Gaming) describes it as "a radically new game form that extends gaming experiences out into the physical world". Which sounds dangerous. However, Dr England says the risks are similar to driving when using a mobile phone: "Of course there will have to be basic common sense." He adds that AR can be put to much more co-operative and educative uses: letting people leave virtual treasure trails, or plant imaginary gardens together using the Facebook app Farmville, for example. "You get more co-operative games in mixed reality, people in different places can come together and share tasks". He also mentions a mixed reality table that sketches science diagrams and can be used as a teaching tool. There's also a belt that can help its wearer avoid collisions in the dark by vibrating when it senses an approaching object.

Rob Hale – games designer and author of the Games Design Blog – thinks augmented reality gaming will be different from its virtual equivalent: "Getting hit by a car in reality hurts a lot more than it does in a videogame," he says. But he sees possibilities as well. In sport for example: "AR could allow us to bring the best bits of video games to real-life sports such as feedback on your performance and positions of your team mates ... If you were going for a run, you could bring up a display of your previous best times, heart rate and the locations of other joggers nearby."

Slipping out of the gym and standing back to consider the philosophical implications of augmented reality, Japanese net culture expert Toshinao Sasaki has a few things to say. He thinks that the boundaries between real and virtual, public and private will disintegrate further as augmented reality spreads. "While the internet sphere and the reality sphere had previously been completely separate as 'virtual and non-virtual', mobile phones have rapidly shrunk the distance between them ... Eventually, it seems possible that mobile phones might play the role of a kind of supplementary brain," he told The Technological Innovation and Social Development Blog. So perhaps augmented reality will make cyborgs of all of us, although we'll have our prosthetic brains in our pockets rather than welded to our skulls, Schwarznegger-style. Until the batteries run out at least. And for the sake of our sanity, being able to tune out is going to be every bit as important as tuning in. As Rob Hale wisely points out, "the most important thing about augmented reality will be the ability to turn it off."

Augmented reality: How it can be used

Surgery: can be simplified by inserting extra information like labels onto the surgeon's field of view.

Venue reviews: compare the price of beer in different pubs on the same street, and from just looking at a venue see menus, prices, reviews and real-time information about what's on now. Pre-recorded bursts of music from bands at gigs that are on later would start playing.

Person recognition: point your phone at a stranger and icons they've selected bob up around their heads – last fm, Facebook – with links to their profiles.

Computer control: with AR embedded into visors or contact lenses and motion-sensitive cameras, we'll be able to wave, flick or nod and control over a hundred simulated PC screens as projected into our eyes by computerised eyewear.

Curated walks: New York company Soundwalk "curate" walks that fuse treasure trails with interactive films and tourist info, feeding walkers with audio and visuals as you walk a variety of set paths. Soundwalk claims it feels like being in a film. It's running its first AR version in Paris this October.