No more heroes: Why no one else can make films like Steven Spielberg

If George Lucas had had his way, Indiana Smith would have sported a moustache. Until three weeks before principal photography began on Raiders of the Lost Ark, Lucas, its producer, still hoped to cast Tom Selleck as the bull-whipping archaeologist. The hirsute Selleck, however, was contractually bound to his hit television series, Magnum PI. Eventually, the film's director, Steven Spielberg, persuaded Lucas to let him cast Harrison Ford instead. Oh, and to change the character's name – to Indiana Jones.

If Ford hadn't been suffering from dysentery, then Raiders would have lost one of the best of its many brilliant jokes: during a chase through an Egyptian bazaar, Indy is confronted by a flamboyant, scimitar-wielding assassin. Sick, and reluctant to film the scripted fight scene, Ford supposedly turned to Spielberg and said: "Let's just shoot the fucker." So Indy pulled out his pistol and blew the swordsman away.

And if every other studio in Hollywood hadn't first rejected Raiders for being too expensive and ambitious, then Paramount would probably not have relented and funded the movie, giving the picture its enticing opening shot – the Paramount logo fades to a distinctive Peruvian mountain; into the frame strides a silhouetted figure wearing a fedora; etc... – and forcing its creators to work to a brisk schedule, with a tight budget and no time for unnecessary retakes.

Raiders celebrated its 30th birthday this summer, and it seems almost unbelievable now that this near-perfect adventure movie (near-perfect? No, just perfect), should have been the product of a series of happy accidents, financial limitations and creative short-cuts. According to Spielberg himself, "The line in Raiders that most typifies the production of that movie was when Harrison says, 'I'm making this up as I go along'."

The film was a rare and precious collaboration between the two great popular filmmakers of their era, perhaps of any era. It was nominated for eight Academy Awards, including Best Picture, and remains one of the top 20 grossing hits of all time. Moreover, it was made at the peak of a golden age of classic family blockbusters, from Star Wars and Close Encounters of the Third Kind in 1977, to Back to the Future (produced by Spielberg) in 1985.

Audiences still hold those films in disproportionate affection: in Los Angeles on Monday night, fans gathered for a sell-out 30th anniversary screening of Raiders, just days since countless proud Star Wars nerds – including British filmmaker Simon Pegg – had taken to Twitter to rail against Lucas's latest alterations to a new Blu-Ray version of Return of the Jedi. (You want the details? Google them. I can hardly bear to type the words.) Last week, the web exploded with news that Nike plans finally to produce a limited-edition pair of trainers like those worn by Marty McFly in Back to the Future II.

Meanwhile, the children who sat awed in cinemas three decades ago are now making blockbuster movies themselves. As a boy, the precocious J J Abrams was hired to restore some of Spielberg's teenage Kodak Super 8mm films; this year Abrams paid tribute to that time with his own nostalgic sci-fi blockbuster, Super 8, which Spielberg produced. Pegg recently wrote and starred in Paul, a pastiche of (among other things) Spielberg's sci-fi classics Close Encounters and ET. Both Paul and Super 8 climaxed with explicit, and almost identical, homages to their hero.

Joe Johnston, who started out filming special-effects shots for Star Wars, this year directed Captain America, which borrows not a few familiar Indy tropes. And yet it feels, doesn't it, as if nothing in today's summer release schedule can quite match the brilliance of those early blockbusters. The characters were more vivid, the storytelling so much smarter. Are we watching Indiana Jones, Marty McFly and Han Solo through rose-tinted 2D spectacles? Or has Hollywood genuinely failed to find their like again?

Spielberg was a singular talent, whose personal successes altered the entire business. Summer was thought of as a fallow season for Hollywood until Jaws chewed up the 1975 box office, creating the blockbuster market that we live with today. Geoff Boucher is the man behind Hero Complex, the Los Angeles Times's renowned "fanboy" movie blog, which hosted this week's 30th anniversary screening of Raiders. "If you really love the big summer adventure movies," he says, "then the run of films released between 1977 and 1985 is just staggering. Those were the first generation of blockbusters, and maybe it's a sector that could only have so many years of real creative invigoration. They went down a list: 'we can do a World War Two movie, a space movie, a time-travel movie...', but there aren't a whole lot of genres to go through before things start to feel derivative."

"It seems as if summer blockbusters have got worse," agrees Tom Shone, the author of Blockbuster: How Hollywood Learned to Stop Worrying and Love the Summer. "You compare the artistry and craftsmanship of a movie like Close Encounters to the machine-tooled franchise films that get churned out now, and it seems a galaxy away. But even back then there were only a couple of guys capable of making those movies: Spielberg and Lucas. The competition for Raiders in 1981 was cheesy things like The Cannonball Run and For Your Eyes Only. Before Spielberg, blockbusters came once in a generation. Then there were one or possibly two a year after Jaws and Star Wars. Now, they're every week, banked up behind each other like aircraft waiting to land. So there are many more bad movies made. But the regularity of the really good movies is the same: about once every year."

Since Tim Burton's Batman in 1989, audiences have notoriously been fed a diet of superheroes, sequels, remakes and adaptations of classic television shows, games or theme-park rides. "Batman was when the studios figured out how to manufacture Spielberg-like success," Shone explains. "Now they'd rather do anything but invest in an original idea for a movie. Their business model is pre-saleability: anything that can achieve brand recognition before you go into the cinema. When the second wave of blockbusters arrived, there were some legendary flops like Waterworld and Last Action Hero. But now they've learned how to eliminate certain strata of failure; how to make money back for any movie that we don't actively dislike."

A similar lament can be found in The Good, the Bad and the Multiplex, the new book by film critic Mark Kermode, who explains that even films perceived culturally as failures, such as Pearl Harbor or Spider-Man 3, make their money back easily at the box office and beyond. And meanwhile, claims Kermode, "We've all learned to tolerate a level of overpaid, institutionalised corporate dreadfulness that no one actually likes but everyone meekly accepts because we've all been told that blockbuster movies have to be stupid to survive."

Indiana Jones, an original character inspired by Lucas's love of old television adventure serials, appealed to each of the industry's target demographics; Raiders and its sequels were genuine family films. The only studio committed to consistently cracking that nut nowadays is Pixar. But if all blockbusters are going to make money, regardless of their quality, then why don't more of them look more like Raiders? "Raiders ages well," says Boucher. "People are always pleasantly surprised when a movie successfully mixes action and comedy. It's mature, with believable adult relationships. The time period gives it a rollicking sensibility. And whatever they paid [composer] John Williams, it wasn't enough."

Like J J Abrams, Damon Lindelof was one of the co-creators of the television sci-fi drama Lost. In a love letter to Raiders published this month on Hero Complex, he identifies one of the crucial elements of its appeal. Yes, the script is a masterclass in narrative economy; yes, it features an abnormally well-developed and feisty female lead; yes, the final image of the film – that crate disappearing into that vast warehouse – is indelible. But most importantly, "Indiana Jones is a nerd... an academic who's motivated purely by his desire to find and retrieve really cool stuff so he can put it in a museum where other nerds can appreciate it. Also, he wears glasses and gets nervous when hot female students write the words 'Love You' on their eyelids... He's actually scared of stuff. It's quite rare for the hero of a movie to be scared of anything."

As Shone explains, "Those early Spielberg movies were underdog stories. The heroes of Jaws were cowards and landlubbers. But now a lot of movies are made by the bully in the sandpit. The Transformers movies are full of goliaths beating the crap out of other goliaths. They're bullies' movies. And that says something about how the film industry has changed."

Most of today's better blockbusters lack the lightness that characterised Raiders and its contemporaries; the very best big movies of 2010 and 2011 were, by common consensus, Inception and Rise of the Planet of the Apes – both of them dark, pessimistic and adult. Of course, they weren't the only big summer releases of recent years to be worth the cost of admission. The superhero genre may be predictable, but it manages its share of deserved critical successes: Iron Man, The Dark Knight, Captain America. Avatar and the first Pirates of the Caribbean were a cut above what tends to pass for entertainment in Hollywood nowadays. Until Cars 2, Pixar's reputation was unimpeachable. The recent blockbuster that most successfully recaptured that Raiders mix of action, comedy and adventure was Abrams' re-boot of the Star Trek franchise – featuring, among others, Simon Pegg.

But it seems unlikely that any of these will still have a hold over our imaginations, three decades hence, as strong as Star Wars or Back to the Future or Indiana Jones. During the 1980s, three teenagers from Mississippi spent seven years remaking Raiders in its entirety on VHS; Raiders of the Lost Ark: The Adaptation earned them the priceless praise of Spielberg himself, and the story of its production is set to be made into yet another movie. Are there young boys somewhere in the world right now, reshooting Pirates of the Caribbean? Somehow, I doubt it.

And what of the two men who first discussed The Adventures of Indiana Smith while building sandcastles in Hawaii in 1977? Lucas is obviously busy adding unwanted extras to the original Star Wars trilogy. Spielberg has two promising films coming out within weeks of each other this winter: The Adventures of Tintin, a motion-capture adventure based on Hergé's classic graphic novels; and War Horse, an adaptation of Michael Morpurgo's devastating First World War tale. Hopefully they'll help to dispel the whiff of his previous directorial effort.

"The key to Raiders was economy," says Shone. "At one point they were going to film with a biplane, but Lucas snapped one set of wings off the model and said 'OK, we'll do it for this much money'. Those constrictions really benefited the film. If Spielberg and Lucas collaborated now, you can only imagine the amount of latitude they would have." Actually, we needn't imagine; we have the proof: Indiana Jones and the Kingdom of the Crystal Skull. Unlike its predecessors, it's probably best forgotten.