LIDAR and Edge computing: Where do smartphones go next?

With smartphones sporting specifications and technology that would put broadcast cameras from 10 years ago to shame, where can they go next? 

Smartphones are a bit of a conundrum for manufacturers. It all started with the first "modern" smartphone, the iPhone, in 2007. Yes, it changed everything, but it also doomed everything to stay the same. 

The specific problem is that the phone was dominated by the shape of its screen. To someone who didn't know what it was, it was a black glass and metal rectangle. And nothing much has changed, apart from a few embellishments here and there, like curved screen edges and wrapping the screen around the front camera. 

Except that everything has changed. Today's smartphones are literally a thousand times more capable than the first ones. Even that figure could be low when you consider the advances in photography and video that are evident even in phones that cost less than US$200. 

For marketing departments, phones are both a delight and a nightmare.  One pretty basic problem is that while one smartphone company might bring out a new model that looks amazing and has stellar performance, all the other companies will probably bring out a similar phone at the same time. It's easier if you're Apple, when users are committed to (you could say, unkindly, "locked in to") your platform, but even Apple suffers from the second issue, which is pretty intractable.

Exponential rise of technology

Consumers don't study graphics showing the exponential rise of technology. They don't follow scientific trends. Worst of all, they typically don't have any appreciation of the extreme amounts of research and development that have to go into even a small improvement in smartphone performance. To a  consumer, the process is that it gets to September, and there's a new phone in the shops, and it's better than the last one. 

The original iPhone has a lot to answer for, design wise. Image: Apple.

Effectively - and especially with yearly or faster upgrade cycles - consumers have become normalised to the rate of change.  Another way to say this is that they expect phones to get better quickly. You might even say that this expresses itself as almost a sense of entitlement: "This phone is rubbish because it's only ten percent faster than the last one..."

Here's a quotation from an article in The Verge about a new smartphone:

"While the phone can shoot 8K at 30 frames per second, and 4K at 30 or 60FPS, it can’t manage shooting three simultaneous 4K HDR video streams or shoot three 28-megapixel photos at once"

The excerpt looks like it's complaining that a phone with the quite  frankly astonishing ability to shoot 8K video can't also shoot 3 simultaneous 4K streams or three 28 megapixel still photos at once. At which point you're tempted to wonder why a phone that can shoot video at a resolution that many professional cameras can't is worthy of criticism? To be fair to The Verge - an excellent website - the context was that the phone contains a chip that the manufacturer claims has the capacity to process three 4K streams at once. The phone simply doesn't use that capability. 

But what this does show is that phones have reached a stage where it's not at all obvious where they should go next. You could reasonably say about today's smartphones that they're good enough. They're certainly good phones: they have been for a decade. The screens are fantastic - the first 4K phone screens appeared years ago, but didn't catch on, because they weren't really necessary: you'd need bigger screens to appreciate that resolution. And they're powerful enough. This year's iPhones will be as fast as today's Macbook Pros.

• Smartphones are now a serious photography tool
• Powerful computational photography tech comes to smartphones
• Smartphone and lightfield cameras: Why "That will never happen" is wrong

Where do we go from here?

My inner conservationist wants to say that we should pause phone development and work to make them more sustainable. I do think we should pursue that latter goal, but I think there are directions we can also go with phones that will make the world a better place. 

Before we get into that, though, I would suggest that we slow down product cycles. That doesn't mean R&D efforts need to be cut back - far from it. What it means is that we shouldn't bring out a new model until it is radically different, and to keep customers happy, release significant upgrades to firmware and software to keep their existing phones feeling fresh. 

There are some developments external to smartphones that will make phones much more capable. One of these is Edge Computing. 

If you haven't come across the concept of Edge Computing, you're not alone, but it's easy to understand. 

Edge Computing brings the cloud closer to its users. This has multiple benefits, but almost certainly the biggest one is that it makes substantial cloud computing power accessible to any one and with a very low latency. In other words, if your phone is connected to an Edge computer, it's almost as if your phone becomes that computer. 5G technology will help here, because it has very low intrinsic latency. 

Here's a practical example. Let's say you're a basketball coach and you want to improve the performance of your players. One way to do this is to analyse their movements. You can do this today by shooting a video of the players in action and playing it back in slow motion. But what if, in real time, you could see their skeleton and musculature overlaid on the players as a moving, real-time graphic? 

Phones don't have enough computing power to do that. But in the future, they will be able to send the video in real time, over 5G, to an Edge computer, which will carry out the calculations necessary to show the analytical overlay. And they will do it in real time. The Edge computer will be able to get all the calculations done in less than the duration of one frame. With the latency of 5G as little as a few milliseconds, the end result will be that you see a perfect overlay on a live action game.

The iPhone 12 Pro has LIDAR capabilities. Image: Apple.

LIDAR

The top end iPhones and iPads now come with a LIDAR sensor. This captures volumetric information - essentially a 3D map of the world around the device. iPhone apps that use this capability are already available and the results are amazing. Need a plan of your house? Scan it with LIDAR. Want to see if your new sofa will fit in the living room? LIDAR is your friend. 

Extrapolate this ability into the future and you can see that the iPhone will be an important portal into the metaverse. The metaverse is what the internet will become when AR (Augmented Reality) and VR (Virtual Reality) become pervasive and ubiquitous. This may sound far fetched but companies like Nvidia are literally basing their future on the metaverse, and almost every trend, from smartphone developments to 5G, Edge computing and AI, are set to converge on this new domain.

Other trends may or may not lead somewhere. I like the idea of foldable phones, but I have yet to see one that I would actually buy. Cameras in phones will continue to improve and will do so at an increasing rate thanks to AI. We will reach the point where it's simply not possible to improve the optical path into a phone, but that won't matter, because our ability to infer better images will grow exponentially. 

Finally, and in order to bring us back to Earth for a minute, I would like to see smartphones become more repairable. I'm not sure how you could ever repair a chip with 18 thousand million transistors on it, but that's not normally the bit that goes wrong. More likely your battery, screen or even power connector will give up first. These should be easily replaceable. I would certainly trade my phone for one that was a little bit bulkier but which would last for years longer.