A Turing Test for cameras?

Replay: How do we deal with the fact that you can make convincing-looking films with a smartphone?

Our recent piece about a 4K film made with an iPhone 6S camera has set thinking about how we can meaningfully talk about the performance of cameras that are unfeasibly good for their size, cost and packaging.

The point is that a typical viewer watching the above film on a computer screen or a tablet on a computer monitor or a tablet screen, will not stop to consider the camera or the way the film was made, any more than when I say a sentence, you will stop to think about how my brain just did that.

Essentially, we should test cameras the same way as we test for artificial intelligence: with a Turing Test.

I'm sure you know already that the Turing Test is a method devised by the English Mathematician Alan Turing to determine whether a computer running suitable software is "intelligent". The basis of the test is that a human interrogator asks the machine questions, and if the tester is unable to determine whether he or she is talking to a machine or a human, then the machine has passed the test.

How would this work with cameras?

Almost the same way. You'd show a film to a viewer and ask them whether it was shot on expensive equipment or, let's say, an iPhone. But you'd only ask them this after they've seen the film.

The reason is that if you "prime" viewers to look for defects, then they're not watching the film as a neutral viewer.

You see, as long as I wasn't told in advance that this film was shot on an iPhone, I don't think it would have ever occurred to me that it was made with anything other than a professional camera.

You can certainly argue that this isn't a valid test. For example, watching films on the web means that they've been heavily compressed. You point out that the iPhone film was made with high production values and that it's not therefore a scientific evaluation of the quality of the camera.

But that's exactly my point. A true evaluation should be made in the context in which a film will typically be seen. If you take it out of context then it inevitably draws attention away from the film itself and towards potential defects in the camera.

So, what am I really trying to say here?

I'm saying that we're now in an era where a camera attached to a portable telephone can be good enough to make a film that people will watch without being aware that it was made with a smartphone.

And perhaps we should use this type of testing as a way of testing the ultimate quality of cameras - away from harsh specifications.

Of course we still need detailed, repeatable testing if we're going to compare cameras objectively. But I do think that camera designers should use "in context" testing to inform them while they're designing new cameras. This is entirely consistent with the idea that we need better pixels, rather than just (simply) more pixels.