Liv partners with Meta to capture mixed reality videos on Meta Quest

Liv, a pioneer in mixed reality capture, has partnered with Meta to become the official capture solution for Meta Quest wireless virtual reality headsets.

Founded by AJ “DrDoom” Shewki, Liv has inked a multi-year partnership with Meta to bring Liv’s mixed reality capture and virtual camera solutions to developers publishing on Meta Quest and also creators who wish to use those features in Meta Rift and Quest apps. Developers are adding the capability now for standalone headsets and it should be available for consumers to use in the coming weeks in VR games.

A big part of game marketing today combines trailers and video content made by the creator and player community inside games, and Liv wants to continue supporting developers in their mission to reach as many creators, players and fans as possible in XR. If you think about it, Liv is probably going to be the way that we’re going to record our first experiences in the metaverse.

“The first and most important thing is that it’s going to run natively on a Quest,” Shewki said in an interview with GamesBeat. “This will work on a standalone device, on the Quest — meaning the Quest, Quest 2 and Quest 3 and all the upcoming Quest devices. It’s part of our partnership with Meta.”

Shewki added, “We spent seven years building technologies. Our mission has always been to empower people in VR and AR to play and share their favorite moments with their friends, family and fans. This will be the first time that we can do this directly on standalone devices like the Quest.”

The Liv capture will work with Gorilla Tag by Another Axiom; Penguin Paradise (and their new game Skelly), by Sava Studios; and Scary Baboon, by Flixzy.

“Another Axiom builds fully realized spaces that are meant to be shared together, like in our popular game Gorilla Tag,” said David Yee, COO at Another Axiom, in a statement. “We’re always looking at new ways to give our players and creators a great experience they can share with their family and friends. This partnership between Liv and Meta provides access to best-in-class capture and virtual camera technology, introducing new ways to capture and share in-headset experiences. We can’t wait to see what the community does with these new tools.”

One of the key initial goals of the work was to make it easier for developers and creators to capture authentic mixed reality content using a PC with an external camera in both immersive apps and mixed reality apps that use the Meta Quest Presence Platform features. That work has been going on for a while and now it has moved on to standalone devices.

“It’s going to be a massive boost to user generated content inside the top VR applications,” Shewki said.

These capture possibilities include hand tracking, passthrough (where video cameras show you what’s happening in the real world while you’re wearing a headset), scene understanding, anchoring and occlusion. The latter refers to keeping a VR app grounded on the floor and able to figure out when a real-world object blocks the view of a mixed reality object in the virtual space.

As part of this partnership, Meta is deprecating its own mixed reality capture (MRC) tooling and Liv will take over as the official solution.

With Liv, VR players can do things like take a virtual selfie, operate first-person and third-person cameras and perform real-time mixed reality capture with physical cameras.

How it works

MRC has shown its power as a tool to market XR apps and games, and Liv believes it will continue to be an important tool to drive game awareness through video content and hence drive sales. It’s particularly good for filming trailers. A developer can set up the SDK to work with a game in perhaps 30 minutes, Shewki said.

Liv spawns a camera in a game. Its impact on performance is variable depending on the complexity and optimization of the game. In the wired version, the minimum spec for the PC the headset is connected to is a machine with an Nvidia 1080 graphics processing unit (GPU) and Intel i7 CPU.

If a PC crashes during the game, that’s OK. The Liv App and your game are decoupled. If Liv crashes, the game will continue to run and only the Liv features will stop working for the user.

Background

Liv has served the VR game dev community since 2016 and will continue to do so now with the help and support of Meta, Shewki said. But for most of that time, the Liv technology for capturing videos only worked with wired VR headsets that connect to a PC. The tech started there because it was easier to have a recording application running alongside an actual game using the power of a PC. But now the VR wireless headsets have become more powerful, as has the Liv tech.

Now, the difference is that it will work with the most popular VR headsets, the Meta Quest standalone wireless headsets, which don’t need to be connected to a PC. The company has 24 people and they’ve been working on this part for about a year.

“We spent the last seven years building camera technologies for app developers in VR, and content creators and players in VR. Historically, we’ve primarily been on Steam. When we started seven years ago, open VR, or Steam VR, was the only platform available,” Shewki said. “We made a whole bunch of assumptions back then about how the technology ought to work and how it ought to integrate with games. So what we’re releasing is effectively taking those seven years worth of learnings that we have learned alongside developers as we’ve been building this and we’re releasing a new SDK.”

The company has a patent for its volumetric capture and replay system.

So far, roughly 50 to 100 developers have been downloading the SDK every month. Most of them are making games, and they’re developing for VR systems that are connected to the PC. Many of these are for educational users at schools and universities.

The Quest market, for wireless standalone headsets, is an order of magnitude bigger than the PCVR market, thanks to games like Gorilla Tag.

“We expect our monthly creator numbers to go up,” he said. “We are going to roll out with tons of games. Our goal has been to be on every device on every platform. “

And there are some games where this works now.

“As part of this announcement, we’re also excited to share that we’ve got tons of new games getting Liv support, including Gorilla Tag by Another Axiom, and Racket Club by Resolution Games,” Shewki said.

The rush to the Quest

With the mixed reality capture, Liv can film real people composited into the game world, which before now has been primarily used for high-end company production. And Liv also has a trailer production studio in Australia that uses its own tooling, and Liv makes trailers for some of the biggest game developers and platforms in the world.

And then there is the Liv App, which people use for mixed reality capture and a virtual camera. But the limitation has been it had to be wired to a PC. Now, it will be available on standalone devices without the need for a PC or high-powered GPU in your PC, and will be natively available in the game.

“You won’t need to download a Liv app and run the Liv app in parallel. You will spawn the camera. You will film with the camera, and you will save the content and soon also stream the concept natively from the device without ever leaving outside,” Shewki said. “So it is a solution built for people who are primarily in Quests and don’t have additional tooling on their PC.”

To clarify, he said that if you’re on a Quest and you’re primarily playing Quest games without a PC, you can finally create really high-quality and rich video content from your favorite applications using a combination of selfie cameras, third person cameras, drone cameras and first-person view cameras, with all the bells and whistles that you need to make great content, he said.

The rollout plan for the new SDK

The beta release of Liv’s new software development kit (V1.6 SDK) for Unity-based apps that support Presence Platform features is available now. The goal is to unlock the ability for developers who publish on Quest, but build on PC, to capture high-quality video content using real and virtual cameras.

Later this year will mark the arrival of the beta release of the V1.6 SDK for Unreal-based apps that support Presence Platform features. That’s for apps built in Unreal Engine 4.27 or after.

All Liv SDK features are included in V1.6 SDK including the regular virtual cameras (first person, selfie, third person, drone) and avatars.

Beyond 2024, Liv is looking at improving the tooling for creators on the Liv App. Liv’s backlog has grown big over the years, Shewki said.

The Liv SDK for Unity and Unreal are both MIT licensed. It is free for game developers. You can only use Liv with games running on PC using SteamVR and Oculus PCVR/Rift.

As for trailers, the company used Liv for the raw captures, and with a bunch of editing and post-production magic courtesy of Liv Productions.

Sava, the creator at Sava Studios, said in a statement, “I am proud to say we added LCK into Penguin Paradise and our new game Skelly because we want to give our community a way to make the best and high quality content in our game.”

“I am proud to say we added LCK into Penguin Paradise and our new game Skelly because we want to give our community a way to make the best and high quality content in our game!”

Asked if the capture tools will work natively on a wireless Quest headset without a PC, Shewki said this will happen.

“We are releasing our Quest SDK in a few weeks,” Shewki said.

Creators will be able to immediately use the new Presence Platform features immediately. Mixed Reality capture for passthrough applications has one complication in that it’s typically filmed at a real location versus in front of a green screen, and that means that the final composite happens in post-production since the dev needs to cut out the human subject.

The Liv capture features are being built into the Liv App on Steam, and so the company said it is committed to helping creators across platforms. The company will invest more into the Liv App on the PC on Steam. A large part of the work with Meta is focused around bolstering the PC app on Steam.

Shewki said he wants these cameras to feel like they belong in the game world.

“We specifically want to avoid people thinking of it as, ‘I have to go download an additional tool.’ I’m going to have a rich camera available to me that I can spawn at any time and record my favorite moments without ever leaving the headset. And that’s what this will unlock,” Shewki said. “We’re going to be rolling out with Gorilla Tag and some other big titles initially. Once we roll out, the SDK will be publicly accessible.”

In the app now, only one camera can be running at any given time. Rather than monetize game developers, Shewki said the company will monetize directly with Meta. The company will make sure its tech can work on all upcoming popular AR and VR devices when they launch, he said.

Streaming will be the next thing that Liv will work on. At that point, players will be able to upload and stream directly to Liv. But that’s not ready yet.