Apple's Tim Cook on iPhones, augmented reality, and how he plans to change your world

Earlier this year, video emerged of a new iPhone feature, long before it was released. It showed the phone creating a magical portal in the middle of a city street. And now that’s arrived.

Apple’s new technology, named ARKit, seemed to arrive at the perfect time. It works by imposing bright virtual objects into the real world – at a time when reality has never seemed darker.

The feature is the kind of world-changing technology that’s on a par with the introduction of the iPhone 10 years ago, Tim Cook tells The Independent. And Apple is standing at the front of it, he says – as another, perhaps more literal, aspect of its long-standing mission to make the world better and a bit more magic.

Because for all the discussion of AR and VR, reality isn’t really being escaped so much as improved. And Apple isn’t running away from reality either – during our interview, Cook speaks passionately about his belief that the real world is getting better, as well as the virtual one.

Despite a focus on new technology, he makes clear that Apple is a company that will not stay quiet and will fight for human rights across the world. And he is optimistic that the world is gradually improving – an optimism that is driven by a sense of history, he says.

But today, Mr Cook – voice soft with Alabama politeness, wearing blue jeans – is demonstrating his plan for augmented reality to take over the world. Or, rather, show how it already has.

Augmented reality works by examining, mapping and understanding the real world and then laying virtual objects on top of it. Apple’s brand new ARKit works by doing that first bit – it uses the iPhone to get a highly detailed picture of the phone and its surroundings, and serving that up to developers so they can put virtual objects on top of it.

One particularly popular example since ARKit launched, for instance, is IKEA’s new app. That allows people to stand in their living room and look at a virtual recreation of the sofa they want to buy – and then they can check how it fits, move around it, look how it goes with the wallpaper, and so on.

Cook claims that shopping will be changed “entirely” by augmented reality, and says that he doesn’t think “anything will be untouched”. It brings together the two difficult forms of shopping today, both brick and mortar and online, in a way that could let people see the entirety of a dress by walking around it, as if they were in a shop, but from the comfort of their own iPhone.

Similar apps are being made across education, business and navigation, as well as shopping. Cook was joined by developers who showed The Independent their new software, which included hiking apps that overlaid information about the landscape, educational apps that did the same for the solar system, as well as games that could be virtually played on a real tabletop.

And Apple believes it will be huge. Cook says that it will end up not simply being used by some kinds of people but for “everyone”; like the App Store when it launched, it might not be huge right now, but AR will go on to have the same “dramatic” climb to take over the world that did.

Cook – whose responses have the same incredibly thoughtful, intentional quality as his company’s products; the sentences are designed with the precision of iPhones – was speaking as part of Apple’s launch of ARKit. That is a framework in the iOS software that powers its iPhones, but it has been accompanied by sensors inside the iPhone X and 8, as well as those that went before.

But he was also notably humble about what that actually is: Apple has created not the augmented reality apps themselves, but a vast framework of sensors and software so developers can build their own.

“The way that you get lots of great ideas is for us to do the heavy lifting of the complexity of locational things and software, and put those in the operating system,” says Cook. “And then you have all the developers that are able to put their energy into their passion.”

open image in gallery

Every developer that creates apps for the iPhone can now use those features, and with an ease that has allowed them to pull together stunning virtual worlds in just a few weeks. That means that Apple can “plant a lot of seeds”, says Cook, and since there’s 15 million or so developers in the world then at least some of those seeds will grow into the stunning, flowering app ecosystem of the future.

And it’s all those things that show why exactly Apple is feeling so confident about its lead in augmented reality.

“Our competitors are trying to mimic what we’ve done,” says Greg Joswiack, Apple’s vice president for iOS, iPad and iPhone marketing. “But they just don’t have that scale we bring to it.”

Android is bigger, but fragmented into different parts so that Google can’t be sure how long it will take for new features to make phones. And while Apple and Samsung battle for the top spot in the phone market, each of Apple’s individual new phones sell far more than their rival’s.

What’s more, the fact that only really Apple makes both the software and hardware for its phones puts it in a uniquely strong position – in both AR and elsewhere. Google has been doing this with its Pixel line of phones, and has integrated good virtual reality features, but those phones don’t sell in large numbers.

That gives Apple an especially strong position because its competitors “don’t control the hardware and software”, Cook says. “It goes to what Apple is about – the integration of those two things, with the App Store on the server side. I think it’s going to be hard for other folks.”

The comparison to the App Store is a key one for Cook. He described the slight scepticism it was greeted with at its launch – and the speed with and scale on which those people were proved wrong.

“Think back to 2008, when the App Store went live. There was the initial round of apps and people looked at them and said, ‘this is not anything, mobile apps are not going to take off’.

“And then step by step things start to move. And it is sort of a curve, it was just exponential – and now you couldn’t imagine your life without apps. Your health is on one app, your financials, your shopping, your news, your entertainment – it’s everything.

“AR is like that. It will be that dramatic.”

He then goes on to compare it to another technology that was pioneered by Apple, and which went on to transform the entire world.

“People initially didn’t think multi-touch was very profound,” he says. “But you think about how we all interface with software today, and we do it by touch. The point-and-click and buttons that we all used for so long are fading.

“And you couldn’t imagine now interfacing in that kind of way. So I think it’s like that – it’s that big.”

For now, ARKit lives in iOS and so can only be used on the iPhone and iPad. But an obvious question is where Apple goes from here: will there be a time in the future that Apple makes products where AR isn’t a feature but is the entire product?

Because AR is “horizontal”, he says, it can apply across the range. But that’s limited by the fact that understanding the world requires an awful lot of technology to work well, technology that at the moment can only be crammed into advanced devices like the iPhone or iPad.

“The products themselves have to have a lot of processing power, and a fair amount of different sensor technology in order to do this locational stuff,” says Cook. “So having it on iPhone changes the game for developers, because instantly they had hundreds of millions of potential customers.

“If it were on a different device then you would never have a commercial opportunity, and without the commercial opportunity you’d never have 15 million people that say, ‘I want to design my passion with AR’.”

By putting it on iPhone, Apple was able to “instantly overnight become the largest AR platform”, Cook says. And all without actually requiring anyone to buy anything, if they already have an iPhone.

But there’s no doubt that new, AR-focused products will come, from someone else if not Apple – the most obvious application of this technology is in glasses. The idea of eyewear that includes computers was undermined quite a little by Google Glass, and the “glassholes” that wore them, but once you start using augmented reality it’s clear that it would work brilliantly if it were strapped to your face.

“There are rumours and stuff about companies working on those – we obviously don’t talk about what we’re working on,” Cook says. (Patents have shown that Apple is at least looking into such wearables, and reports have suggested that the ARKit introduction on the iPhone is partly a way to help build that interest and ecosystem.)

“But today I can tell you the technology itself doesn’t exist to do that in a quality way. The display technology required, as well as putting enough stuff around your face – there’s huge challenges with that.

“The field of view, the quality of the display itself, it’s not there yet,” he says. And as with all of its products, Apple will only ship something if it feels it can do it “in a quality way”.

“We don’t give a rat’s about being first, we want to be the best, and give people a great experience,” he says. “But now anything you would se on the market any time soon would not be something any of us would be satisfied with. Nor do I think the vast majority of people would be satisfied.”

He doesn’t deny that future is coming. “Most technology challenges can be solved, but it’s a matter of how long,” he says – before moving on to talk about the fact that the achievement or possibilities of the phone shouldn’t be underestimated.

Whether or not glasses come, Apple will have succeeded in giving hundreds of millions of people a piece of augmented reality hardware – without them even knowing it. If Apple or anyone else does go on to make a piece of augmented reality hardware, it’s clear that the world will be far more ready than they would have been if the iGlasses had appeared alongside the iPhone X last month. There’s already a huge number of apps, and a vast amount of people who have now seen the possibilities of AR for the first time.

I asked Cook whether he saw Apple’s AirPods – the wireless earphones that also allow their wearer to talk to Siri and hear directions – as a kind of augmented reality technology. He didn’t, but said that he can “envision audio becoming a key part of the AR experience”, referencing a game we had played that was soundtracked by the beautiful and dynamic twinkling of a Japanese rock garden.

The Apple CEO is speaking to The Independent in London about the development of the ARKit products. But he used his trip to Europe to take in other visits – including an emotional trip to a cemetery in Normandy, to mark the lives of soldiers that were lost and destroyed there.

“From a personal point of view, I have a deep connection with men and women that served in the military. I came from a military family. My father fought in the Korean war, my brother served in the air force. Many people in the extended family – there’s a deep belief in serving country.

“So that’s a huge link for me personally. What happened there in particular was the key turning point, in that awful war.”

A tweet he posted after appeared to indicate not only the importance of that moment but the way that it echoes into our current time – amid the rise of the far right and dissatisfaction around the world. “At Normandy, honouring those who made the ultimate sacrifice,” he posted, before quoting a message written on a chapel there: “Think not only upon their passing. Remember the glory of their spirit”.

That quotation – and its reminder of the continuing importance of the spirit of those soldiers – seems to have taken on new edges in the past few months. Cook says that he hadn’t taken the trip to make any “current day statements”, but as a reflection of his “love of people that served and gratefulness that they did”.

But, he says: “You think about what happened in that war and what people were fighting for, and it goes to the basics of human dignity.”

“Today at Apple we still fight for this, and advocate for human rights, and we believe that every generation has a responsibility to enlarge the definition, not move inward.

“And so you can see, we’ve been very clear and straight, we don’t believe in being silent, we think silence is sort of the ultimate consent.”

Cook has certainly not been quiet about some of the recent decisions of Donald Trump and the US government. He objected forcefully and publicly to the suggestion that the US could end protections for Dreamers and kick people out of the country, and is known to have called the president and told him not to pull out of the Paris agreement on climate change – while ultimately failing to stop that decision.

But amid those worrying events, does he still believe that the progress of the world is towards more respect for more people?

“I think that history sort of ebbs and flows at time, but the arc always goes in a certain direction. And I think that will happen now as it happened in the 1960s and 1970s, and has in a lot of ways continued to happen.

“Sometimes I think being in the midst of it, it doesn’t feel like it. But looking back, particularly for me, I saw the way that African-Americans were treated in the 1960s and into the 1970s – and still today in too many places. But then arguably the laws also not only allowed it but facilitated discrimination.

open image in gallery

“So I’ve seen massive improvement. And my optimism stems from that history – I do think that is the arc across the world. What each of us has to do is do everything we can to hit the accelerator key.

“Life would be so much easier if we just treated everybody with dignity and respect. You think about all the problems in the world – half of them would be solved with just that! Life would be so much better.”

As Cook talks about the importance of speaking out, his commitment to improving the real world is clearly undimmed. His and Apple’s work isn’t limited only to the political, and has also taken in commitments to power its stores and data centres sustainably alongside new concentration on improving the diversity and accessibility of both its products and its business.

But while it might not be so pressing, Cook is a little more able to change the entire world with ARKit. It does that literally, by laying magical and strange virtual objects – or domestic ones – onto the real world.

But it does it in a broader way, too, and one that explains Cook and Apple’s focus on the applications of ARKit and other technology in education. Augmented reality apps are already allowing children to fly through virtual stars and mountains – and perhaps encourage them to change the world beneath.