How Mercedes is making cars that drive themselves – and let you Zoom

Katharina is driving. We’re in a Mercedes-Benz EQS on Highway 101, just outside Sunnyvale, California. But Katharina is not like the rest of us. Where we yearn for open roads with few other vehicles, she is looking for, hoping for, traffic.

That’s because she’s a test engineer for Mercedes, and we’re in a car equipped with Drive Pilot. This means there are a bunch of features, like automated lane changes, automatic braking and more, which come into play when the car is at Level 2, where the driver is responsible for the drive. But then there’s Level 3, which means the car is doing the driving, and that’s what Katharina wants to show me.

For this, the car needs to be going at no more than 40mph and there needs to be a car ahead that our car can see. This being California in the late afternoon, we’re quickly into heavy traffic and the car is at Level 3.

Not only does this mean nobody is touching the steering wheel, a smart console springs into life, showing video that the driver can watch. The driver can also do their emails, for instance, something which startles me at first, but which Mercedes says is part of the gift of time that they’re giving back to the driver.

The drive is super-smooth, braking gently as the car ahead slows, and speeding up to match it when it does. There are lots of sensors on this car, including cameras, LiDAR and radar. The car ahead picks up some speed and shoots off, so the car sends a warning that the driver needs to re-take control.

A little later, a car just ahead slows down. Quite suddenly. I glance at Katharina, but when our far begins braking, firmly but not heavily, I ask her if the car is braking or she is doing it. The car is doing it, I’m told, using automatic braking. It’s very effective and feels very safe, though, truth be told, it doesn’t happen until quite a few seconds after I’ve been pumping an imaginary brake in the passenger footwell.

Level 2 is in most Mercedes cars, and plenty of cars from other brands, but Level 3 is much rarer, and debuted last year on two Mercedes vehicles, the S-Class and EQS cars. It’s available in Germany, in Nevada in the United States, and is expected to be approved in California soon.

The Sunnyvale trip has also seen remarkable and impressive safety demonstrations plus automatic parking — both done using cars which don’t have Drive Pilot on board.

But the main event is the unveiling of the new operating system for Mercedes vehicles, called MB.OS, which is coming in 2025. This is a comprehensive new system, and while there will be visual similarities to what’s in Mercedes now, it will go much further.

There will be an Android-based app store, including games, optimised for the touchscreens on the seatbacks which rear passengers can use, and the screen the front-seat passenger can use.

The dashboard screen will stretch across the entire front of the car in some newer models, but clever filters mean that the passenger content will be invisible to the driver, so they won’t be tempted to watch.

There will be plenty of other options beyond games. For instance, in-car Zoom and Webex calls will be possible. In Level 3, a dashboard camera will stream video of the car’s occupants, but at other times the camera will be off, lest the driver becomes more consumed with how they’re looking on camera than they should be.

This is a strategy that is not without risk. Apple and Google have come to dominate in the car with Google Auto and Apple CarPlay, so how does Mercedes make its mark? As Magnus Östberg, chief software officer at Mercedes tells me, “When it comes to the operating system what the customer cares about is that it is a nice experience. So ultimately it’s the apps that the customer wants, the features, the services, the content.

“And they want it to be an experience that is very pleasant. And the thing that we have seen is that when you’re in a Mercedes, you know that it’s a luxury object but the customer wants to have these digital services and features. So how do you marry luxury with digital?

“This is where we differentiate. We’re co-operating with Google. We’re not trying to beat them. We want their services. We want their contacts, for instance. We’re integrating Apple music and Dolby Atmos, so it’s not trying to beat them. It’s more like what is aspecial when you come to a Mercedes?”

The operating system will benefit from collaborations from different partners. Mapping, for instance, will be provided by Google in some countries, A Map in China, T Map in Korea and so on. Here, the mapping company part-owned by Mercedes, will continue to be involved.

Other partners include YouTube for video content and Antstream for games.

Devising an operating system for a car is very different from one for a phone or tablet, for instance, where you aim to capture the user’s attention entirely. The driver, of course, needs to keep their eye on the road most of the time.

“We are using scientific methods and models to see what is the driver’s cognitive load. What is the customer usage? Where are they paying attention? Where are their eyes of the road? And for how much time do they do certain things so that when we design each element, we do it in context, to make sure that we have at least amount of cognitive load,” Östberg says.

“Unless you’re in Level 3, because then the computer is taking over the driving activities, and we can concentrate on making the operating system completely immersive. We have new technology to make sure that as a driver you cannot be distracted by what the passenger is watching.

“If you try to look over there, you actually don’t see anything. We can make sure that this is a safe experience but at the same time a very rich experience.

Mercedes has time to finesse the experience before MB.OS goes live. By which time, perhaps we’ll all be wanting to drive a bit more like Katharina.