The ultimate answer

Cosmological thought stretches back for thousands of years, but the conceptual excitement has never been more intense than it is at the start of the 21st century. Recent technical advances have enriched our cosmic perspective. Space probes have beamed back pictures from all the planets of our Solar System: new technology enables a worldwide public to share this vicarious cosmic exploration. Pictures of a comet crashing into Jupiter, made with the Hubble Space Telescope, were viewed almost in real time by more than a million people on the internet. During this decade, probes will trundle across the surface of Mars; they will land on Titan, Saturn's giant moon; samples of Martian soil may be collected and brought back to Earth.

Our universe extends millions of times beyond the remotest stars we can see – out to galaxies so far away that their light has taken billions of years to reach us. Bizarre cosmic objects – quasars, black holes, and neutron stars – have entered the general vocabulary, if not the common understanding.

Classically, everything on Earth was thought to be made of earth, air, fire and water: the firmament was made of a different substance, the "fifth essence". In modern guise, this disjunction has reappeared. We've learnt that all the stars and galaxies in the universe, made of atoms, are a trace constituent – 95 per cent of the stuff in the universe is not in the form of ordinary atoms at all, it consists of mysterious dark particles or energy latent in space.

Through the efforts of geologists and biologists we understand, at least in outline, how our Earth and its biosphere have evolved over their 4.5 billion-year history. We now envision our Earth in an evolutionary context stretching back before the birth of our Solar System. By observing the most distant galaxies whose light has spent 10 billion years in transit, we can probe the past, and see what galaxies looked like when they were newly formed. We can confidently trace cosmic history back to a universal fireball far hotter than the centre of the Sun – this much is now as well established as anything geologists can tell us about the early history of the Earth, because we can detect the faint afterglow of this intense heat.

In an earlier book, I expressed 90 per cent confidence in the evidence for this one-second-old fireball with a temperature of 10 billion degrees. Recent measurements have firmed up the theory, and I'd now raise my confidence level to 99 per cent. But I would still prudently leave the other 1 per cent for the possibility that our satisfaction is just as illusory as that of a Ptolemaic astronomer who had successfully fitted some more epicycles into his theory.

Cosmologists are sometimes chided for being "often in error but never in doubt". But when we try to probe back further still, into the first tiny fraction of a second, we enter the realm of conjecture. The temperatures and energies would then have been so high that they are beyond anything we can study in our laboratories, so we lose a firm foothold in experiment.

According to the favoured current guess, our entire universe started off as an infinitesimal speck. It underwent a period of "inflationary" expansion, driven by energy latent in space. The scale doubled, then doubled, and then doubled again. Within about a trillionth of a trillionth of a trillionth of a second, it is claimed, an embryo universe could have inflated to become large enough to encompass everything that we now see. And then the fierce repulsion switched off; some of the energy converted into heat, initiating the more familiar expansion process that has led to our present habitat.

Indeed, the process would be likely to overshoot, inflating by far more than is needed to account for the 10 billion-light-year dimensions of our observable universe: the distance to the edge could be a number with millions of zeros. In this expanse of space, far beyond the horizon of our observations, the combinatorial possibilities are so immense that close replicas of our Earth and biosphere would surely exist, however improbable life may be. Furthermore, even this stupendous expanse of space may not be everything there is. Some theories suggest that our Big Bang wasn't the only one. This line of speculation dramatically enlarges our concept of reality – from a universe to a multiverse.

We are flummoxed about the very tiniest scales, and the very earliest times, because we still lack a consistent theory that unifies all the forces of nature. The 20th century gave us two great concepts. One was Einstein's relativity, which deepened our insights into space, time and the force of gravity, and accounted for the motions of planets, stars and galaxies. The other was the quantum theory, which governs the microworld of atoms.

Quantum theory, however, doesn't yet incorporate gravity. In most of science, this doesn't matter. The gravitational pull between the atoms in a single molecule is so weak that chemists can neglect it. On the other hand, stars and planets, whose motions are controlled by gravity, are so large that the fuzziness and uncertainty induced by quantum effects can be ignored. But quantum effects could shake the entire universe when it was squeezed smaller than a single atom. A new theory, unifying the quantum and the cosmos, will be needed before we can understand these mind-bogglingly extreme conditions.

Einstein himself spent his last 30 years seeking a unified theory of the physical laws. His efforts were premature; not enough was then known about atoms and the forces within them. But interest in unified theories is no longer restricted to established dignitaries like Einstein, who can afford to risk over-reaching themselves and achieving nothing. Hundreds of young scientists are engaged in the quest, each hoping to discover a "final theory" compact enough to be written on a T-shirt. The smart money is on superstring theory, or M-theory, in which each point in our ordinary space is actually a tightly-folded origami in six extra dimensions, wrapped up on scales perhaps a billion billion times smaller than an atomic nucleus.

There's still an unbridged gap between this elaborate mathematical theory and anything we can actually measure. Nonetheless, its proponents are convinced that it has a resounding ring of truth about it and we should take it seriously.

A unified theory of all cosmic forces would, if it were achieved, be an immense intellectual triumph of all time – the culmination of an intellectual quest that started with Isaac Newton. He showed that the force keeping the planets in their orbits was the same force of gravity that pins us firmly to the Earth. In the 19th century, Faraday and Clark Maxwell showed that electric and magnetic effects were intimately linked, and 20th-century scientists linked these forces with the others that govern atoms.

A unified theory would elucidate the basic stuff that everything is made of, and the bedrock nature of space and time. It would exemplify what the great physicist Eugene Wigner called "the unreasonable effectiveness of mathematics in the physical sciences"; also the remarkable contingency – and it surely would be a contingency – that human mental powers could grasp the bedrock of physical reality.

But phrases such as "final theory" or "theory of everything", often used in popular books, have connotations that are both hubristic and misleading. I hope it's not curmudgeonly to point out what such theory wouldn't do. First, it would still leave the enigma of existence unsolved. Cosmologists sometimes claim that the universe can arise "from nothing". But this is loose language. Right back at the beginning, our universe may have been an infinitesimal speck, but it was latent with particles and forces: it had far more content and structure than what a philosopher calls "nothing". Why there is something rather than nothing remains a mystery beyond science.

Secondly, such a theory would not signal the end of challenging science. Indeed, it would actually have minimal impact on most science. Even if we knew the basic laws, we still wouldn't understand how their consequences have unfolded over the last 12 billion years, into stars, planets and life. We're baffled by our living world because of its complex patterns and interconnections – incomplete knowledge of the microworld isn't the impediment.

The sciences are often likened to different levels of a building – logic in the basement, mathematics on the first floor, then particle physics, then the rest of physics and chemistry, and so forth as we climb upwards. But the analogy with a building breaks down, because the superstructure (the "higher-level" science dealing with complex systems) isn't imperilled by an insecure base.

Almost all scientists believe that every lump of material, whether living or inanimate, is an assemblage of atoms that is governed by the equations of atomic physics. In practice, though, we can't solve these equations for anything more complicated than a single molecule. The complex motions in the air and in the oceans, and the far greater emergent complexity of the biological world, require different concepts. And, as Steven Pinker, the linguist, points out, "Human behaviour makes the most sense when it is explained in terms of beliefs and decisions, not in terms of volts and grams."

There are "laws of nature" governing our everyday environment that are just as fundamental as anything in the microworld, and are conceptually autonomous from it – for instance, general laws that describe chaotic behaviour apply to phenomena as disparate as dripping taps and animal populations.

The most complex things are neither on the cosmic scale, nor on the atomic scale, but in between. We ourselves – the most complex entities we know of – are midway between stars and atoms. It would take as many human bodies to make up a star as there are atoms in each of us.

Each science – chemistry, meteorology, biology, or social psychology – has its distinctive concepts and laws that cannot be reduced to physics. Nonetheless, as the physicist Steven Weinberg has emphasised, some sciences can claim a special "depth". If you go on asking "why? why? why?" you end up with a fundamental question either about subnuclear particles or about cosmology: the sciences of the very small and the very large. This is an important feature of our universe. We seek unified theories of cosmos and microworld, not because the rest of science (or even the rest of physics) depends on them, but because they deal with deep aspects of reality.

The lure of the "final theory" is very strong. Ambitious students want to tackle the No 1 challenge. But an undue focus of talent in one highly theoretical area is likely to be frustrating for all but a few exceptionally talented (or lucky) individuals. I advise my own students to multiply the importance of a problem by the (small) probability that they'll solve it, and maximise that product. I remind them also of Peter Medawar's wise remark: "No scientist is admired for failing to solve a problem beyond his competence. The most he can hope for is the kindly contempt earned by Utopian politicians."

Despite Einstein's aphorism that "the most incomprehensible thing about the universe is that it is comprehensible", it would be astonishing if human brains were "matched" to aspects of the external world. Elucidating some of nature's complexity may have to await the emergence of a more intelligent species than ourselves.

Sir Martin Rees is Royal Society Research Professor at Cambridge University. This article is based on his latest book, 'Our Cosmic Habitat', published this week by Weidenfeld and Nicolson