What does the future hold for compressed codecs?

HEVC was supposed to be the compression of the future, but could it be unsurped by an even better alternative with none of the licensing issues?

Five or so years ago, there was much discussion of HEVC, the video encoding technique that was intended to do a better job than H.264, the codec currently used by Netflix and Youtube et. al. The intention was pretty clear: we can get more picture quality for less bandwidth by using a better encoder, or, to put it another way, your fast cell phone lets you spend less money on monthly megabytes. That line of advance stretches all the way back to Video CD, with its MPEG-1 codec, and it's got us to the point where we can stroll down the street watching HD video on a pocket computer.

HEVC was intended to double the performance of H.264; to achieve the same image quality in half the bitrate with a limited increase in complexity. It's difficult to measure the performance of a codec in a way that won't provoke an argument, but in general, HEVC does what it was intended to do.

But HEVC hasn't been adopted as widely as we'd have liked and there are already discussions about alternatives. The Alliance for Open Media released the initial version of the AV1 codec (not to be confused with the AVI file format) in March 2018, with the idea of bettering HEVC. Background work on the specification and testing of video codecs has been done by the Internet Engineering Task Force with input from people like Netflix who have a very serious business interest in getting better pictures to people for less bandwidth and, therefore, less money. But why is all this necessary when a lot of work has already been done for HEVC, which is widely agreed to be successful?

The trouble with patents

The reason, sadly, is not technical. The situation is similar to the one which existed with H.264, where a group of companies which owned patents that they felt were relevant to H.264 formed a licensing organisation called MPEG LA. Confusingly, MPEG LA has no connection to the Motion Picture Experts Group, but it does coordinate patent activity with regards to H.264 and now HEVC, and charges licensing fees for making products that implement these standards.

Until August 2010, MPEG LA had suggested that it would start charging royalties for video broadcasts – all video broadcasts, even those given away for free – made using H.264 at the end of 2015. Needless to say, this would have involved an amount of money large enough to risk collapsing into a black hole and, essentially, the world said: “You've got to be kidding.” With Google waving its WebM codec around as an alternative – which had been developed with the specific intent of avoiding patent issues – MPEG LA essentially backed down, though it still charges for encoders and decoders.

The bad news is that a similar situation pertains with HEVC, only more so. MPEG LA has skirted any idea of charging for content, but fees for HEVC devices are considerably higher, taking into account the per-device cost and the way the caps on total fees work. Also, another entity, HEVC Advance, has also been formed by organisations (including AT&T, Microsoft and Nokia) that for various reasons weren't content to be part of MPEG LA. The HEVC Advance licence was initially much more expensive than MPEG LA's and vastly more expensive than anything which had ever been proposed for H.264.

These fees were revised in 2015 and 2016 but remain steep. Worse, Technicolor withdrew from HEVC Advance in 2016, meaning that people interested in implementing HEVC would need to separately negotiate with Technicolor. And HEVC Advance. And MPEG-LA. And possibly some others too because it's hard to know who might have a patent covering a particular idea.

That's a very quick overview of the situation, but it should be pretty clear why HEVC uptake has been on the slow side. The legitimacy of software patents is a huge debate in the field and some jurisdictions don't allow them, but there's instinctively something wrong to most people with a situation where engineering has been done, shown to work well, then left fallow because of red tape and – to at least some degree – greed.

Asymmetric numeral systems

If that seems like a combative stance, consider the work of Jaroslaw Duda, a mathematician at Jagiellonian University in Krakow, who developed a new mathematical compression technique called asymmetric numeral systems (ANS). The technique is an improvement over previous ideas (principally Huffman coding) and has been used in data compression since 2014. Duda elected not to patent his discovery, both out of a sense of altruism and based on the fear that legal entanglements would (as with HEVC) stunt the uptake of ANS. So, in 2017, Google filed paperwork  aimed at protecting the use of ANS for video compression.

The Alliance for Open Media doesn't claim that AV1 can be implemented without requiring patents, but they have designed it to be royalty free. The idea is to require that “technology contributors”, that is people who own relevant patents and are presumably members of the alliance, will allow the use of those patents to anyone who needs them. An intellectual property review is part of the development process, analysing the standard to be as sure as anyone can be that it doesn't infringe on any patents that are outside the organisation's reach.

It is impossible to know whether other patents exist that another company might try to enforce against the new codec. In essence, though, this is a battle of ideas, not technologies, that's been going on at least since someone realised that the GIF image format included a patented compression algorithm in the mid-90s. It's not even a battle of engineering ideas; it's a battle of moral and legal considerations and, at least in the case of HEVC, it's one major reason why we can't have nice things.