The virtual future of cinematography

Perhaps one day most films will be made like Avatar, actors performance-captured on a featureless stage and placed into realistic 3D-rendered environments. Who knows? But either way, it seems inevitable that virtual reality and augmented reality will play increasing roles in a DP’s process.

One innocuous example of AR is the live view mode of a sun-tracker app like Helios Pro from Chemical Wedding. Many DPs use such apps in pre-production and on set to show us how the sun will move around a location during the day. Simply input your intended shooting date and hold up the phone, and the app superimposes graphics on the live camera view showing the position of the sun at different times of day, graphics which update in real time as you pan your phone around. 

Chemical Wedding’s other popular app, Artemis Pro, a virtual director’s viewfinder, allows you to insert a stock 3D model of an actor into your shot if there are no real people conveniently around to frame up on. These models do not currently stay fixed in space as you move your phone around, but that’s coming, and much more besides. The AR toolkits built into modern devices have opened up all kinds of possibilities for previsualising scenes.

Helios Pro from Chemical Wedding.

The latest Pro versions of the iPad and iPhone also contain LiDAR technology, bouncing light beams off objects to measure and reproduce them virtually. With an app like Polycam LiDAR & 3D Scanner you can move around a location with your device and watch it build a photo-mapped 3D model. That model can be easily shared with other members of the crew, who can inspect the space from all angles.

Virtual lighting setups

Other software like Blender and Cine Tracer enable you to manually construct a 3D representation of a set or location and then light it with virtual lamps. You can set up a virtual camera, input your sensor size, lens length and aperture, and render out frames or moving shots to show your collaborators. You can even add virtual haze.

Cine Tracer is based on Unreal Engine, and gaming engines like this and Unity would seem to be integral to the future of filmmaking. I recently had the chance to play with Unreal’s MetaHumans and try lighting and framing them virtually while a friend puppeteered their expressions using a face-tracking app on a phone. It may be a while before this technique emerges from the Uncanny Valley, but it seems certain that the boundaries between reality and virtual reality will only continue to blur. The realism of the environments that these engines can render make it easy to see how whole films can be shot with the cast on a green screen, perhaps using volumetric capture to record the performance from all angles as a moving 3D image.

Virtual set.

But a green screen is not even necessary. The virtual production technology made famous by The Mandalorian sees the cast perform surrounded by large LED screens displaying 3D backgrounds. The camera’s position is tracked and used to adjust the perspective in the backgrounds in real time, and the brightness of the screens means that realistic lighting and reflections automatically happen on the actors and their costumes. We will undoubtedly see this technology creep into more and more productions as time goes on. Did you know, for example, that scenes on the Bat Signal rooftop in The Batman were shot on an LED volume?

The rise of VFX in filmmaking worried some DPs, feeling that authorship of the image was being taken away from them. As VR and AR become more accessible, however, this tide may well turn. Virtual production technology allows the DP to see the whole shot they’re filming, not just a green screen which may or may not be replaced by the background they were expecting. And if a DP can quickly previz a shot in 3D on their phone, it opens up many more possibilities for collaboration than the current norm in which previz tools are only available to VFX teams.

A virtual future for cinematography doesn’t have to mean endless green screens, unconvincing digital humans and a loss of authorship. It can mean better communication, stronger collaboration and more creative control.