Psychologist Gary Klein has helped the US President make critical decisions and analysed the 'pathways' to Eureka moments

Scientists call it the Eureka moment, that flash of insight leading to a profound discovery. Some describe it as instinctive intuition, perhaps built on years of experience. Others put insightful brilliance down to pure chance – or perhaps the hand of God.

Whichever way we describe insight, we know it when we see it. It's that moment when Albert Einstein developed his theory of relativity after imagining a trip on a light beam, or when Alexander Fleming discovered penicillin in a discarded Petri dish, or when Jim Watson and Francis Crick came up with the idea of the DNA double helix after seeing someone else's data.

Insight is not, of course, the sole preserve of science. People in a wide range of professions experience insights all the time, and sometimes these flashes of inspiration save lives.

Take the case of Michael Riley, a British naval officer on board HMS Gloucester in 1990 during the Desert Storm operation. He was monitoring the ship's radars when he saw something odd on his screen and within a minute his insight led to the destruction of an incoming Silkworm missile.

It turned out that Riley had made the right call. The blip he saw on his radar screen, which looked identical to the blips caused by American war planes returning to their aircraft carrier, was indeed a Silkworm rather than a friendly aircraft.

When experts later rewound the radar tapes to study Riley's decision, they were dumbfounded by how he did it. Riley himself said that it was because the missile was travelling faster than the aircraft, but this was impossible to determine in the short time that he had to study the data.

Only after further analysis did the truth come out. The Silkworm was flying lower than the aircraft and so it took longer for it to clear the ground "clutter" over the land that Riley routinely saw on the radar screen.

open image in gallery

The result was that the missile flew further over the sea before the clutter disappeared, which made it seem as if it was travelling faster. Because Riley had built up so much experience in looking at returning US jets, he had the insight that this particular blip was an incoming Silkworm – so saving the lives of many of his colleagues.

Gary Klein, an academic psychologist who once helped to redesign the Situation Room at the White House to make it easier for the US president to make the right decisions, has made a special study of insight. After analysing more than 120 cases where insight played a crucial role in decision-making, he believes he has come near to a rational understanding to explain how insight comes about.

"There is no simple formula. What I found was that there were three pathways that lead to insight and they reflect different types of forces," Dr Klein now says in his first interview with a British newspaper. He calls these three pathways: contradiction, connection and creative desperation.

In the case of Michael Riley, his insight came about as a result of him identifying an anomaly on his radar screen. In other words, he noticed a contradiction, which was too small for other people to see.

"Typically, this path gets initiated when you notice an anomaly – something that doesn't fit your established beliefs," Klein says.

"What we usually do is explain the anomaly away, which makes sense most of the time but leaves us with our pre-existing belief set. Insights occur when we take the anomaly seriously, go on to explore its implications, and discover that some of our existing beliefs are wrong," he explains.

"We only have an insight if the anomaly means something and I pay attention to it and I start to play through my mind what are the implications. To have what I call an insight stance is driven by curiosity."

open image in gallery

Another pathway to insight is to make connections. By putting one piece of information together with another, you end up with an insight. One of the most famous examples in science was when Charles Darwin read Thomas Malthus's treatise on the problem of dwindling resources from an expanding population – which led Darwin to the insight of evolution by natural selection, Klein explains.

"Variations of this path include noticing coincidences and just getting curious about an odd observation. This path, which is by far the most common in my sample, is about adding a new idea and belief to the ones you already hold," he says.

And then there is the third path, which Klein calls "creative desperation". This is when insight comes about under extraordinary conditions, sometimes involving life or death decisions.

In his new book, Seeing What Others Don't, Klein cites the example of the 15 "smokejumpers" parachuted into the hills of Montana in 1949 to control a forest fire. Suddenly, the firefighters were caught by a fire moving fast up a grassy hillside. They tried to outrun the flames, but only two made it to the hilltop because these fires travel faster uphill than most people can run.

The only other man to survive was Wagner Dodge who quickly realised he was not going to outrun the fire. His insight was to set fire to the patch of grass ahead of him.

He realised that grass was fuel for the fire and if he got rid of it, he would be able to walk into the smouldering ashes before the main fire caught up.

"This path is about dropping an unnecessary or flawed belief," he adds. "When good people do stupid things, sometimes there is an interesting reason."

Dodge tried to encourage his colleagues to come with him into the smouldering patch of grass. Unfortunately for them, they thought he was doing something stupid rather than insightful.

Seeing What Others Don't, by Gary Klein, is published by Notable Books, £12.99