What happened when Albert Einstein met Charlie Chaplin?

What exactly happened when Chaplin met Einstein? When Gladstone met Tennyson? When Picasso met Stravinsky? What kind of sparks do we expect to fly when globally famous figures meet? Do we assume there'll be a blinding flash, a nuclear synergy, a Hadron-style collision of celebrity atoms that will register on some celestial Richter scale? Or are such encounters always doomed to failure – not to a flaming row, or a falling-out between the sainted participants, just a moment of blank social awkwardness, a mundane exchange of meaningless murmurs?

We have Norse mythology to thank for the concept of Valhalla, the "slain-hall," the great wine-bar of the afterlife wherein the souls of dead heroes congregate and carouse and try on each other's horned helmets. It's only a step from there to imagining a Valhalla of historical figures, where Napoleon and the Duke of Wellington do some back-slapping and pay each other guarded respects on soldiering, or where Cromwell and Robespierre compare notes on the overturning of monarchy. A little downmarket from such a dream are the fantasy dinner parties people have imagined, where (for instance) Casanova, Elizabeth I, Goliath, Cleopatra, Voltaire and Frank Sinatra converse brilliantly on love, patriotism, philistinism, style, democracy and cocktail harmonies, and every conversation is freighted with magic.

Readers of literature have always been prone to such imaginings. Spending so much time closeted with words, lives, conversations and situations, they like to imagine all their literary heroes and heroines getting on really well together – under their roof, naturally. Over here, Lord Byron whispers something mildly shocking into the ear of Jane Austen (or is she only pretending to be shocked, the minx?), while over there William Shakespeare complains to Harold Pinter about the monstrous new discovery of pipe tobacco which is making theatre punters cough their lungs out, mid-play, night after bloody night.

What really happens when Titans meet is a little more mundane. A new book, Peter Pan's First XI by Kevin Telfer, tells the startling story of the cricket team put together by JM Barrie at the turn of the 20th century. Barrie's squad of gentlemen was christened the Allahakbarries, after the Arabic phrase "Allahu Akhbar" ("God is Great") plus the playwright's surname. The team featured an extraordinary line-up of literary talents: Barrie, Sir Arthur Conan Doyle, PG Wodehouse, AA Milne, AEW Mason, author of The Four Feathers, EW Hornung who wrote the Raffles stories about a dashing jewel thief, Jerome K Jerome, the humorist behind Three Men in a Boat, and a dozen lesser talents. (Among those was Augustine Birrell, MP for West Fife, barrister, author and First Secretary for Ireland at the time of the Easter Rising in 1916. He wrote, and spoke, in a comically ponderous and wool-gathering style known as "birrelling". A complete novice at cricket – on the train to his first match he had to have the rules explained to him by Barrie, including which side of the bat to wave at the ball – when he arrived at the crease, he summoned his fellow batsman and explained: "Should I strike the ball, to however small an extent, I shall run with considerable velocity.")

They weren't the greatest cricket team in history. In their first game they scored a pathetic nine runs. But they improved and (helped by the ferocious bowling of the inventor of Sherlock Holmes) crushed rival teams of artists and village locals all over the south coast. The book has a charm, however, beyond cricket; it offers a sighting of late-Victorian and Edwardian summers when the nation's leading authors blithely played games together, in an atmosphere of manly endeavour, and met their heroes in a kind of earthly Valhalla.

AEW Mason introduced Barrie to Robert Falcon Scott, of Scott-of-the-Antarctic fame, and the two became close friends; Barrie was potty about explorers. Conan Doyle made the acquaintance of Oscar Wilde and, whatever one might think would happen at the convergence of the great aesthete with the nation's most four-square heterosexual, they got on very well. They met at the Langham Hotel in London in August 1889, both invitees of JM Stoddart, the publisher of Lippincott's Magazine, who commissioned Doyle to write The Sign of Four, and Wilde to write The Picture of Dorian Gray. Stephen Fry has speculated that Doyle, the failed eye surgeon with one book published, was inspired to get back to literature by meeting this extravagantly cultured literary supernova, five years his senior; while Wilde may have been emboldened by Doyle's praise of Dr Jekyll and Mr Hyde to try a novel with a supernatural theme. What a result! Not only did they become friends, they were mutually helpful and made literary history.

Most convergences between tremendously distinguished writers, however, tend to end in bathos. Take the head-spinning evening of 18 May 1922, at the de luxe Hotel Majestic in Paris, where a moneyed couple of London arts patrons called Sydney and Violet Schiff hosted dinner for 40 people to celebrate the first performance of Stravinsky's ballet Le Renard, performed by the Ballets Russes under the great impresario Serge Diaghilev. The Schiffs had a reputation for pulling diverse but brilliant people together; their guest list on this night, however, was exceptionally ambitious. Along with Diaghilev and Stravinsky, they'd invited Picasso, Proust and James Joyce. A perfect quintet of the arch-modernists of the 20th century, five men at the cutting-edge of innovation, the "breaking of forms" and the jettisoning of the past. Would they like each other? Would they strike sparks? Would they agree to collaborate? Would they chat in ordinary human words?

Joyce arrived drunk at 11pm. He'd failed to rent or borrow a dinner suit for the glittery occasion and was, reportedly, embarrassed about being under-dressed. For a time he sat with his head in his hands, gazing at his glass of champagne. Marcel Proust swanned in at 2.30am, having just got up (he was writing the Sodom and Gomorrah chapters of A La Recherché du Temps Perdu, and working through the night in his sealed-off, cork-lined room.) He homed in on Stravinsky and asked, "Doubtless you admire Beethoven?"

"I detest Beethoven," said Stravinsky shortly.

"But cher maître," Proust protested, "surely the late sonatas and quartets ... "

"Are even worse than the rest," said Stravinsky.

Joyce, meanwhile, had fallen asleep. When he woke, he found Proust standing before him, asking, "Do you like truffles?" "Yes I do," said Joyce. History does not record if the two literary Titans munched their way through a box of chocolates together, but it's pleasing to imagine the sight.

How elevated was their conversation? Apparently Proust said, "I have never read your works, Mr Joyce," and Joyce replied, quick as a flash, "I have never read your works, M. Proust." So there. Joyce later claimed that he tried to talk to the Frenchman about the allure of chambermaids (clearly Joyce didn't know his interlocutor very well) but Proust wanted to talk about duchesses, and Joyce didn't know any. To change the subject, Joyce complained about his eyes and how they were giving him headaches. "But my stomach!" said Proust. "My stomach!"

And that was it, except for an ill-tempered cab-ride home, when Joyce lit a cigar and opened a window. Proust, allergic both to cigar smoke and open windows, talked non-stop, while Joyce glared at him and finally took the cab grumpily on to his home. "Of course the situation was impossible," Joyce later reflected. "Proust's day was just beginning. Mine was at an end." Actually Proust's was more truly at an end – he died in November that year.

The conversational shards of these epic meetings formed the beginning of Richard Davenport-Hines's book A Night at the Majestic (2006), one of a number of books that record the convergences of the great at a fancy meal, and the consequences thereafter. Others include Richard Baldick's Dinner at Magny's, about a wild evening with Flaubert, George Sand, the Goncourt brothers and Turgenev; and Penelope Hughes-Hallett's The Immortal Dinner, which describes the evening of 28 December 1817 when the artist Benjamin Robert Haydon threw a dinner party at his house in Lisson Grove for some friends: Charles Lamb, author of Lamb's Tales from Shakespeare, Joseph Ritchie the explorer, Edward Landseer the engraver and two poets, William Wordsworth (then aged 47) and John Keats (aged 22).

Haydon's plan was merely to introduce the precocious young Keats to Wordsworth; but to the awestruck eye of hindsight, it represents a meeting between the first generation of Romantic writers and the second. How did the conversation go? Apparently the poets discussed Virgil and Homer at length, and the puckish Charles Lamb was the cause of "much laughter" despite the humour-free presence of Wordsworth. Beyond that we just don't know what was said. But how we would wish to have been a fly on the mutton, to hear how these people talked without any distinction between science and the arts – since both were regarded as fields for creative endeavour and boundless imagination. These would have been exchanges worth travelling through time to hear.

A century later, art, science and the press converged when Charlie Chaplin threw a dinner party for Albert Einstein, to introduce to, among others, William Randolph Hearst the newspaper magnate. It wasn't a success: Einstein wasn't disposed to explain his theories to the uninitiated, the boffin and the mogul failed to hit it off. Things might have frozen up completely had not Hearst's girlfriend twined her fingers through Einstein's barnet and cooed: "Albert, why don't you get your hair cut?"

Meetings between great men don't always result in elevated colloquies; sometimes they tend towards the crudely basic. When Ernest Hemingway and Scott Fitzgerald met in Paris, at Le Dingo Bar on the rue Delambre in April 1925, Hemingway was disconcerted to be asked: "Did you have sex with your wife before you were married, Ernest?" They became friends, however. Their most intimate conversation (as reported by Hemingway) was also about wives. One evening, Scott Fitzgerald confessed to his friend that his wife, Zelda, had told him his penis was unusually small, and that he could never satisfy any woman. Hemingway said it was just typical of Zelda's undermining ways, but Scott wasn't reassured. So Hemingway asked him to come to the lavatory, where he inspected his friend's lance of manhood. Back in the bar, he explained:

"You're perfectly fine," I said. "You're okay. There's nothing wrong with you. You look at yourself from above and you look foreshortened. Go over to the Louvre and look at the people in the statues and then go home and look at yourself in the mirror in profile." Now there was an act of friendship between creative giants, if not an especially artistic conversation. You can almost see the superior smirk on Hemingway's face as he presents himself as the macho guy who was able to reassure his less macho pal about his physical shortcomings (and then tell the world about it).

Another significant meeting was that between the great British novelist Anthony Burgess and the globally revered writer and fabulist, Jorge Luis Borges. Borges delighted the younger man by referring to him as "my Mancunian namesake". Burgess could speak a dozen languages, including Malay, and could read and understand four or five others, but he deferred to the awesome omniglossolalia of the Argentine sage. When they met at a literary party, they were surrounded by journalists, anxious to catch what they assumed would be wit and repartee of a high order. In which language should they converse? Borges muttered a few words in some guttural dialect. Burgess replied in what seemed to be the same tongue. One of the more learned hacks present recognised the profusion of H-noises and told the others: "My God, they're chatting in Old English."

Soon the half-truth became a headline across the bookish world: "Literary Titans Converse in the Language of Beowulf." The truth, as offered in Burgess's memoirs, was a little more prosaic: unsure which language to speak in, Borges had quoted the first line of the old Norse poem "The Seafarer" – and Burgess had simply supplied the next. It was a high-table moment of showing off, rather than a revelation of linguistic brilliance.

What a potent image is offered here: the gentlemen of the press, the eyes of the world, crowded around two elderly gentlemen, striving to hear what they're saying to each other. They can hear some kind of noises emanating from these distinguished lips, but they cannot make head nor tail of it. What does this remind us of? I suggest it's a throwback to the Sacra Conversazione paintings of the Renaissance, when Veronese or Filippo Lippi, or Bellini or Piero della Francesco would paint the Virgin and Child among an informally arrayed group of saints, apparently just hanging out together and passing the time. Art critics will explain that the figures in these paintings seldom look as if they're actually speaking – they're merely communing, on a higher plane of existence than the rest of us.

Of course the Virgin and Child and their attendant saints do not need to speak (What would they say? "I'm gasping for a cup of tea – St George? Would you mind?" "See the match last night? West Ham were rubbish ... ") but if they did, the conversation would be so elevated, so full of divine concepts, inexpressible virtues, Heavenly visions, we could hardly bear to hear it.

That, I suspect, is what we expect to hear when we think of great artistic figures colliding: some beautiful noise, hardly accessible to human ears, the emanations of genius in stereo. Unfortunately, we never quite get it. What we find instead is what the great photographer Julia Margaret Cameron found when she lured Alfred Lord Tennyson to be photographed at her house in the Isle of Wight. As they stood under a tree in the garden, she saw that William Gladstone, then prime minister, had appeared in the house, and went to greet him. She brought him outside, introduced him to Tennyson and dashed indoors for some new photographic plates. Rain began to fall. She spent an age finding an umbrella. When she returned to her camera, she found the two men – two of the three most famous people in the country, the third being Queen Victoria – standing back to back under the tree, quite unable to find any subject suitable for discussion, once they had exhausted the rain.