Lytro debuts holographic video camera for cinema

Camera maker Lytro is launching a first of its kind light-field video system intended for feature film with significant implications for the future of production.

The system combines the highest resolution sensor ever developed with specialised software to enable frame rate, aperture and focal length to be determined in post-production.

The prototype is being unveiled at the Society of Motion Picture and Television Engineers’s SMPTE Future of Cinema conference in Las Vegas on April 16, part of the NAB Show (Apr 16-21), along with a short film showcasing the technology, and will be launched commercially in Q3 this year.

While conventional cameras record light as a two dimensional image - a mechanism that hasn’t changed since the invention of photography - Lytro’s camera is able to capture multiple viewpoints on set in a single shot, including the direction and intensity of the light rays.

This data can be used to rebuild detailed pictures of the depth and colour of objects in a scene enabling traditional craft decisions such as focal point, camera position and even lighting to be reframed after the event.

“Currently, key creative decisions such as the frame rate are ‘baked in’ at the time of capture. Light-field means you can computationally change the shutter time, frame rate and aperture as a virtual camera process,” said Jon Karafin, head of light-field video, product management, Lytro. “This is a disruptive technology and we need the feedback of the studio, cinematographer and VFX community to make sure that what we are creating meets their needs.”

One advantage of the technology, according to Lytro, is that studios will be able to create versions of the same film at 24 frames per second (fps), 48 fps or 120 fps as desired for theatrical releases and at 60 fps for broadcast transmission all from the same source material, rather than limiting such decisions to the time of shooting.

Other benefits of recording include making Virtual Reality and VFX production less resource-intensive and enabling live action to be treated in the same way as computer generated imagery. Lytro has developed a plug-in for The Foundry’s software Nuke giving the raw data a route into post-production suites where image manipulations such as a change in the depth of field, a shift in zoom or matting for virtual backlots is possible.

The unit will output 4K resolution but in order to do that, alongside computing information such as the depth of objects in a scene, Lytro has developed the highest resolution video sensor ever made at 755 Megapixels (MP) shooting 300 frames a second.

This is more than three times the resolution of Canon’s highest rated 250 MP sensor, which is still in development. In comparison, High Definition is the equivalent of 2 MP.

Before light hits the sensor it will pass through a micro-lens array comprising over two million hexagonal lenses each with a slightly different perspective on the scene. The captured information is otherwise known as a hologram.

“We are leapfroging the leapfrog, if you will,” said Karafin. “We are massively oversampling the 2D image to be able to change the optical parameters in post. Everything that makes the image unique is retained, but you are able to rephotograph every single point in the field of view.”

Challenges

He admitted that the technology is “very challenging” and that work needs to be done on the software algorithms, compression system and hardware.

Too large to be handheld, the studio camera needs to be physically mounted to a dolly or crane and be cabled to a server up to 100 metres away. The server itself is being designed to throughput data at speeds up to 300GB/s (it currently works at 100GB/s).

The server will sit in a “video village” off-set and be supervised by Lytro technicians. The camera requires “a non-standard optical format” and Lytro will offer the unit with custom focal lengths. The whole system, including operators, will be offered for rental. Lytro says some studios are running trials.

Anticipating negative reaction from cinematographers, Lytro has incorporated a physical focus control and aperture check on the camera plus keyframe metadata to provide reassurance to the DoP that the decisions they make are preserved into post.

“When the software is opened in post then whatever decision the DP or assistant or focus puller has made on set will be the same,” explained Karafin. “We are trying to be sensitive to that workflow.”

Advancements in tech

Light-field was first theorised 125 years ago but research only took off in the 1990s. While Lytro uses a single camera, German researcher Fraunhofer IIS has employed a multiple camera array to achieve a similar effect.

Developments such as these have put the technology on the industry’s map. Light-field will be discussed by SMPTE at the Future of Cinema Conference. In addition, a new working group exploring light-fields and soundfields (the audio equivalent) has been established by the MPEG JPEG committees chaired by Fraunhofer’s Head of Moving Picture Technologies, Dr Siegfried Foessel.

The Lytro Cinema Camera builds on a decade of R&D at the company which brought the first consumer light-field stills camera to market in 2012. Lytro made a move into light-field video last year after receiving a $50 million investment led by GSV Capital. It also made a quarter of its staff redundant in order to hire specialists in video and virtual reality. In November 2015, it announced Lytro Immerge, a light-field video system designed for VR acquisition which is yet to be released. The Cinema Camera is the company’s debut professional acquisition equipment.

Lytro was founded by Standford university math professor Ren Ng in 2006 and has been helmed by CEO Jason Rosenthal since 2013. The company received a total of $150 million in funding from backers including Netscape co-founder Marc Andresssen.

Audiences will not notice any difference in content shot with a light-field video camera while presentations are made on flat screens. However several companies, including Leia3D and Zebra as well as tech giants like Microsoft and Samsung, are working on holographic or augmented reality displays.

“I have no doubt that light-field is the future of imaging,” added Karafin