Blackmagic Cine Immersive Capture for Vision Pro 8160x7200 Resolution per Eye(newsshooter.com)
newsshooter.com
Blackmagic Cine Immersive Capture for Vision Pro 8160x7200 Resolution per Eye
https://www.newsshooter.com/2024/06/10/blackmagic-ursa-cine-immersive-capture-content-for-apple-vision-pro-with-8160-x-7200-resolution-per-eye/
80 comments
I have no use for this whatsoever, and it's probably eye wateringly expensive. But damn this is a cool piece of tech.
That has been VR for a while, but I think VR experiences are now truly worthwhile pursuing in 2024 and beyond. I think we've been there for a few years now.
It does depend a lot on what you prefer to do with your free time in the first place. But if you happen to be killing time on YouTube or something anyway, watching a VR travel walk for example is actually really worthwhile.
I also find that because it is so immersive, I will consume less content because I am actually satisfied earlier than if I had been killing time on 2D content.
It does depend a lot on what you prefer to do with your free time in the first place. But if you happen to be killing time on YouTube or something anyway, watching a VR travel walk for example is actually really worthwhile.
I also find that because it is so immersive, I will consume less content because I am actually satisfied earlier than if I had been killing time on 2D content.
The problem with existing VR tech is that it's still too cumbersome to use. Nobody wants to wear a hefty computer strapped to their face for casual use, let alone for prolonged periods of time. The form factor needs to be much more lightweight and unobtrusive for the product to have a chance at mass adoption.
There are AR devices in the form of sunglasses now, but the displays and experience pale in comparison to VR headsets.
We're still a few more generations away from VR headsets shrinking and AR glasses improving for this experience to be universally enjoyable.
There are AR devices in the form of sunglasses now, but the displays and experience pale in comparison to VR headsets.
We're still a few more generations away from VR headsets shrinking and AR glasses improving for this experience to be universally enjoyable.
I couldn't agree with you more. I love VR as a concept, but the giant headsets are just unusable. Hence, I research this frequently and someone appears to finally have realised this:
https://www.bigscreenvr.com/
Linus Tech Tips did a review of it, so it's real. I have high hopes.
https://www.bigscreenvr.com/
Linus Tech Tips did a review of it, so it's real. I have high hopes.
The caveat being that the Bigscreen requires a wired tether to a PC and external Lighthouse tracking beacons, so you trade the clunkiness of strapping a computer to your face for the clunkiness of only being able to use it in a designated VR room. It's more comfortable for sure, but it's not for everyone.
It’s not for eeeeeeeeeeeevryone. But, it’s really damned good. And, micro display tech is making huge leaps again this year in both resolution and brightness. Whatever the 2025 equivalent of the Bigscreen Beyond is, it’s gonna rock.
> Nobody wants to wear a hefty computer strapped to their face for casual use, let alone for prolonged periods of time. The form factor needs to be much more lightweight and unobtrusive for the product to have a chance at mass adoption.
I wouldn't mind a hefty computer myself. The problem is, apparently nothing can be made today if it doesn't aim for immediate mass adoption. Niches and gradual evolution don't fly anymore.
I wouldn't mind a hefty computer myself. The problem is, apparently nothing can be made today if it doesn't aim for immediate mass adoption. Niches and gradual evolution don't fly anymore.
> Niches and gradual evolution don't fly anymore.
How so? In the current VR resurgence of more than a decade now we've seen the field go through gradual evolution. The technology has been very niche, and still is today.
What headset manufacturers are betting on is that one day they'll come up with a winning formula that makes _everyone_ want their device. Similarly to what happened for smartphones with the iPhone. I think we're still a few years away from that tipping point, but I don't doubt we'll get there eventually.
How so? In the current VR resurgence of more than a decade now we've seen the field go through gradual evolution. The technology has been very niche, and still is today.
What headset manufacturers are betting on is that one day they'll come up with a winning formula that makes _everyone_ want their device. Similarly to what happened for smartphones with the iPhone. I think we're still a few years away from that tipping point, but I don't doubt we'll get there eventually.
Agreed, and in my opinion, Apple messed up with the Vision Pro with this. They took the battery and made it external, yet the head unit itself is heavier than most battery-included head units! IMO they should have made the compute unit external as well (you already have to have a brick attached via a cable, it's madness not to (technical details aside, potential added latency, etc)). I don't really know the details and I'm sure someone has weighed all the components...but if the processing was not in the head unit, they ditched the external facing display + glass screen cover, and went all in on being light weight (maybe a carbon frame vs aluminum? not sure how much that would save), it would be much more enjoyable.
Obviously the Vision Pro in a glasses format isn't possible now. However, the Vision Pro with a few hundred grams shaved off of it, plus being a little bit less "horizontally" bulky would result in a much more enjoyable user experience.
Obviously the Vision Pro in a glasses format isn't possible now. However, the Vision Pro with a few hundred grams shaved off of it, plus being a little bit less "horizontally" bulky would result in a much more enjoyable user experience.
I’m guessing that splitting the hardware between the face and the brick isn’t that simple and would require a lot of custom solutions.
If the headset were solely a display then it seems straightforward, but I don’t know what existing system could be used to transmit all of the various sensor data down to a brick, especially in a way that would enable quickly iterating on the sensor set. I’m not a hardware person though so if someone from hardware land could chime in on plausibility I would be curious to hear that perspective.
Once the sensor set is more locked in and/or the market is more proven I’m guessing they will go ahead and invest in a custom solution to split the face and the brick hardware.
It would for sure be an obvious win, I just think it’s probably not easy to do.
If the headset were solely a display then it seems straightforward, but I don’t know what existing system could be used to transmit all of the various sensor data down to a brick, especially in a way that would enable quickly iterating on the sensor set. I’m not a hardware person though so if someone from hardware land could chime in on plausibility I would be curious to hear that perspective.
Once the sensor set is more locked in and/or the market is more proven I’m guessing they will go ahead and invest in a custom solution to split the face and the brick hardware.
It would for sure be an obvious win, I just think it’s probably not easy to do.
Sony's PSVR2 is essentially just a display, is incredibly lightweight and comfortable, and sends everything down the cable to the console. I think it's possible.
Also, the field-of-view is too limited. And the problem with focus: nearby objects will make your eyes focus the same as far-away objects; this feels unnatural and trains your eyes to be lazy.
It’s Blackmagic so it’s going to be reasonably priced
In the post-production world blackmagic sets the price floor. None of their hardware is priced for consumer use.
For truly immersive VR video it seems we’ll need to support moving your head in 6 degrees of freedom, so you can peak around objects etc. We’re a long ways off from it being realtime, but I can imagine generative AI being used to interpolate frames from an array of cameras spaced fairly widely apart.
Lytro had this more-or-less figured out, before they ran out of money.
https://www.roadtovr.com/exclusive-lytro-reveals-immerge-2-0...
https://www.roadtovr.com/exclusive-lytro-reveals-immerge-2-0...
Google had a complete system for capturing volumetric light fields in 2020: https://augmentedperception.github.io/deepviewvideo/
There was a lot of work on big light-field capturing rigs in the past, but honestly, with the current advancements in neural radiance fields, gaussian splatting, meshing and other scene reconstruction techniques even monocular capture can generate pretty amazing results. I'm sure that having ultra-high res binocular video could be converted into some pretty high quality free-look output (that can at least be played back in real-time) on high-powered mobile devices could be possible today (w/ just some additional optimization/engineering work).
Recent research examples:
* https://vidu4d-dgs.github.io/
* https://fudan-zvg.github.io/4d-gaussian-splatting/
* https://guanjunwu.github.io/4dgs/
* https://aoliao12138.github.io/VideoRF/
Recent research examples:
* https://vidu4d-dgs.github.io/
* https://fudan-zvg.github.io/4d-gaussian-splatting/
* https://guanjunwu.github.io/4dgs/
* https://aoliao12138.github.io/VideoRF/
There's a cool VR demo experience showcasing this on Steam: Welcome to Light Fields[0].
[0] https://store.steampowered.com/app/771310/Welcome_to_Light_F...
[0] https://store.steampowered.com/app/771310/Welcome_to_Light_F...
Yes, this is the closest to what I would consider a true VR photo. Stereoscopic images are gimmicky and feels utterly restrictive compared to this.
From the same lab 2 years later, there's "Immersive Light Field Video with a Layered Mesh Representation" which is the same, but instead of photo it's video.
From the same lab 2 years later, there's "Immersive Light Field Video with a Layered Mesh Representation" which is the same, but instead of photo it's video.
Interesting! Thanks for the pointer.
4D Gaussian splatting will probably get us there, there are many research papers about it for example:
https://fudan-zvg.github.io/4d-gaussian-splatting/
But even with that, it would allow only limited movement because if you move to much in the reconstructed space then it would have to show you occluded areas that has been never recorded. But for that case generative AI could be used to fill in the the gaps, which means it wouldn't be true to life reconstruction anymore, but it could be still useful where that isn't that important.
https://fudan-zvg.github.io/4d-gaussian-splatting/
But even with that, it would allow only limited movement because if you move to much in the reconstructed space then it would have to show you occluded areas that has been never recorded. But for that case generative AI could be used to fill in the the gaps, which means it wouldn't be true to life reconstruction anymore, but it could be still useful where that isn't that important.
In the Vision Pro demo appointment reel (i.e. anybody can demo the headset for free), everything is 6DOF, most is just stereo. Immersive VR is interactivity, which is more about good UX / gamification and less about using a dual-8k stereo capture device.
To some degree it's already being done on the headset in realtime to produce a passthrough image with proper depth
Not really, that image is stereo but it isn’t 360 degree and it can’t support arbitrary viewing angles and positions after the fact. It works perfectly for passthrough because the headset cameras are physically locked to your head.
It does support arbitrary viewing angles in the sense that IPD is accounted for. It generates a view for your specific eye position. It handles in painting around the user's hands as well.
Of course it's not magic so the views it can generate are limited.
Of course it's not magic so the views it can generate are limited.
This is great but… isn’t there something fundamentally eye-straining about capturing media at a given 'interocular' distance and then displaying it to a person at their own individual and different 'interocular' distance?
Interocular, meaning distance between a pair of eyes/lenses
Interocular, meaning distance between a pair of eyes/lenses
From playing games that get that distance wrong, it doesn't give me a headache or anything, it just makes the whole world feel bigger or smaller than reality. Really frustrating in games that try to replicate a real feeling. In a driving sim, for example, it can make you feel like you're driving a toy car rather than a full sized one, or that you're a kid in the seat of your parents car. Obviously not a feeling you want to invoke unintentionally, but perhaps there's ways in software to adjust?
No. While not everyone's eyes may be spaced at the "normal" 64mm distance, delivering images shot with that spacing won't look significantly different to people with differing eye spacing. And even if they did, the distortion would be one of scale.
For example, if you wanted to shoot something from a cat's POV, you'd put the lenses closer together and shoot objects closer to the camera. That would make it easier to fuse images of a mousehole right in front of your face. Things farther away would simply look "less 3-D" and therefore abnormally far a way to a human viewer.
For example, if you wanted to shoot something from a cat's POV, you'd put the lenses closer together and shoot objects closer to the camera. That would make it easier to fuse images of a mousehole right in front of your face. Things farther away would simply look "less 3-D" and therefore abnormally far a way to a human viewer.
I don't think they're just taking the images and projecting them onto each lens of the vision pro. Forgetting about inter-ocular distance for a second - that just wouldn't work for head tracking and letting you move around the image.
Instead I assume they're using the pair of lenses to approximate a 3d model which allows for display at a variety of angles (all reasonably close to where the camera is), and then rendering from that model for the person looking at the image based on the angle they are actually looking at it from. Which solves the interocular distance problem, because you render for each eye based on where the persons eye is resulting in a distance between the two display images that doesn't have to match the camera.
Instead I assume they're using the pair of lenses to approximate a 3d model which allows for display at a variety of angles (all reasonably close to where the camera is), and then rendering from that model for the person looking at the image based on the angle they are actually looking at it from. Which solves the interocular distance problem, because you render for each eye based on where the persons eye is resulting in a distance between the two display images that doesn't have to match the camera.
When I watch videos in VR i feel more "crosseyed" than when looking at virtual enviroments. Specially when things are near the camera.
Not exactly the same, but I like making stereo photos. I just use my iphone to take one photo left, move it over by a bit, then take another photo. I don't think I would notice if I was off by a centimeter or so. But maybe in VR it's different.
No, but the scale feels wrong if it doesn't match.
https://xkcd.com/941/
https://xkcd.com/941/
I never liked the 3D era of TVs that were so popular in the CES some years ago, I found it gimmicky and not that immersive and the 3D effect requiere an almost perfect scenario to work. But then I got to play with a Meta Quest, my hype for 3D came back.
I wonder if we'll see more productions like this thanks to these cameras being more common.
I wonder if we'll see more productions like this thanks to these cameras being more common.
It's a super high end VR180 camera. Neat.
But are we going to see better true 360° 3d cameras than the Insta360 Titan?
But are we going to see better true 360° 3d cameras than the Insta360 Titan?
Most viewers just need VR180. Perhaps Apple produces a few movies using this device, and then we’ll know if it could be the new IMAX.
The Apple produced demo videos on the Apple Vision Pro are exactly this… and they’re amazing.
https://variety.com/2024/film/news/apple-vision-pro-original...
https://variety.com/2024/film/news/apple-vision-pro-original...
Or any live event like concerts and especially sports. When they can capture a live event well for the VP, it will sell a lot.
The pron made with this camera will be truly eye-popping. BMD knows its target market.
Can you imagine the file sizes, though?
Current VR porn is typically 8K 60 fps at the max, which already creates HUGE files, like 500 MB/minute.
Current VR porn is typically 8K 60 fps at the max, which already creates HUGE files, like 500 MB/minute.
Yeah it’s pretty remarkable that as soon as the bandwidth and storage requirements for video catch up, the industry quickly pushes past to the next level. I’m sure there’s a way that will be found to distribute the output, but we’re still talking veritable supercomputers needed to edit the raw files that this camera pumps out.
> edit the raw files
Shit, I didn't even consider the editing part.
A pair of 4K images (one 4K image per eye) with 8 bits per channel is nearly 50 MB per frame uncompressed. At 90 fps, it's a whopping 4.48 GB per second. 268.8 GB per minute.
I can't imagine they're actually doing the editing in uncompressed form. Even a modest 15 minute scene would be 4 TB, which after typing out, I suppose isn't actually THAT big. If they had an hour of footage, that's 16 TB, which is certainly big, it's not insanely big and could be done on consumer-grade SSDs.
Shit, I didn't even consider the editing part.
A pair of 4K images (one 4K image per eye) with 8 bits per channel is nearly 50 MB per frame uncompressed. At 90 fps, it's a whopping 4.48 GB per second. 268.8 GB per minute.
I can't imagine they're actually doing the editing in uncompressed form. Even a modest 15 minute scene would be 4 TB, which after typing out, I suppose isn't actually THAT big. If they had an hour of footage, that's 16 TB, which is certainly big, it's not insanely big and could be done on consumer-grade SSDs.
You edit using ProRes or some other kind of RAW proxy, which is then either high enough quality to be used directly, or at the end of the process what is basically a list of instructions can be issued by the software to perform all the operations required on the uncompressed file, offline/non-realtime. Then it doesn’t matter so much about the power of the machine because it just crunches the data, albeit for hours on end.
Watching video of 3d movie vs insta360 home made one in my meta quest 3, I wonder the definition of immerse experience. 3d is not 360. Not that immersive may I say. Obviously for cinema perhaps you have a limit to how wide the scene go as you need to direct the audience eye. And ears.
Hence there is a limit and that camera is very camera like. Anyway wait for a cheaper version so to use my cheaper meta quest 3. Both 3d and 360 had their role.
Hence there is a limit and that camera is very camera like. Anyway wait for a cheaper version so to use my cheaper meta quest 3. Both 3d and 360 had their role.
They could put fisheye lenses on it and get close 180°.
I don't see audio mentioned at all - No integrated ambisonics or add-on binaural mics?
- Can't Generative AI synthesize 3D from 2D footage?
It can, but it’s not good. 2D data doesn’t give you enough information to reconstruct it in 3D and then infilling the details with generative models leads to inconsistencies.
Why does this have to be so big compared to an iPhone camera which is like a square inch or so? Ok, so maybe you need two for stereo vision, but that's still a huge gap in size.
Bigger lenses capture more light and produce better quality video, especially in low light.
Good point. But the iPhone camera proves that you can get away with bad lighting conditions using post-processing. I think you can still make professional-looking video using just an iPhone.
If by videofile you mean professional, then sure, but if you mean chasing quality that exceeds human perception limits, then no, if anything, the lenses are likely a compromise in order to get them so close together.
iPhone video is impressive, but even moderately successful YouTube channels upgrade once they start making money. Some even use big cinema style cameras like red 8k cameras.
iPhone video is impressive, but even moderately successful YouTube channels upgrade once they start making money. Some even use big cinema style cameras like red 8k cameras.
Making quality 10% better doesn’t necessarily mean the device is 10% bigger. The device could be 10x bigger for a boost of 10% in definition. Law of diminishing returns.
[deleted]
Genuinely, what is the imagined use case for this?
Have you tried a Vision Pro? The immersive video was the highlight for me. It makes you feel like you’re really there, on a level I hadn’t experienced with a Quest.
Same here. I did the in-store demo and I wish the whole demo was just the immersive videos.
The Alicia Keys one was first and my first instinct was to move back a bit, because she was so close, but I didn't, because I was on a stool and probably would have fallen. She was in my personal space, it felt almost uncomfortably intimate, like someone was right there. Then there were the flying ones, and I had that feeling I get in my stomach when on a roller coaster or zip line. I didn't expect that at all. I could have spent hours watching that stuff. I loved it. If they had a bigger library of immersive videos with regular releases, I'd probably get a Vision Pro just for that.
The windowed stuff was fine, but the full immersion is what I'm after.
The Alicia Keys one was first and my first instinct was to move back a bit, because she was so close, but I didn't, because I was on a stool and probably would have fallen. She was in my personal space, it felt almost uncomfortably intimate, like someone was right there. Then there were the flying ones, and I had that feeling I get in my stomach when on a roller coaster or zip line. I didn't expect that at all. I could have spent hours watching that stuff. I loved it. If they had a bigger library of immersive videos with regular releases, I'd probably get a Vision Pro just for that.
The windowed stuff was fine, but the full immersion is what I'm after.
I remember the same experience watching a 3D video of Bjork on YouTube, back when 3D video was the big new thing that would take over the world (amazingly, YouTube still supports the UI to turn the video around). It was very cute and interesting, and novel, but like all novelties, it wore off quickly.
You could be right. I think the novelty wears off, in part, because there is too much friction relative to the benefit.
I’ve used YouTube with a phone headset. It’s cool, but kind of annoying. The Vision Pro was also kind of annoying to put on and set up. And with both, the content is limited. So how much annoyance is someone willing to put up with for something that doesn’t has much content and only provides are marginally better viewing experience?
Televisions, computers, and phones, have very little friction and more content than a person could consume in 100 lifetimes. This makes the simple screen hard to beat.
I’m also remind of something I heard a film director say when being critical of the 3D movie fad… all movies are already 3D. There is depth in the shots when looking into the screen, no one really needs stuff flying out at them.
I’ve used YouTube with a phone headset. It’s cool, but kind of annoying. The Vision Pro was also kind of annoying to put on and set up. And with both, the content is limited. So how much annoyance is someone willing to put up with for something that doesn’t has much content and only provides are marginally better viewing experience?
Televisions, computers, and phones, have very little friction and more content than a person could consume in 100 lifetimes. This makes the simple screen hard to beat.
I’m also remind of something I heard a film director say when being critical of the 3D movie fad… all movies are already 3D. There is depth in the shots when looking into the screen, no one really needs stuff flying out at them.
I see it the same way I see surround/spatial audio and HDR on an OLED. Most people won't get much out of it, but those that do notice will tend to prefer the option when available, even at substantial cost.
I think that's a little different. With spatial audio or OLED, it's a one time up front cost to get the better hardware, but turning on an OLED iPad or TV is no different than turning on an LCD. By contrast, Vision Pro requires extra work every single time it's used. I'm much more willing to pay a little extra or do a one time setup than I am to do extra work every time I want to do something.
Going for a walk in a nice park is a better experience than walking around the block, but if I only ever walked after going through the trouble of driving to a nice park, I'd almost never walk. That's more like VR to me. Even if the cost is removed from the equation, it creates enough friction that using it becomes an event, not an everyday default, even if it's preferred.
Going for a walk in a nice park is a better experience than walking around the block, but if I only ever walked after going through the trouble of driving to a nice park, I'd almost never walk. That's more like VR to me. Even if the cost is removed from the equation, it creates enough friction that using it becomes an event, not an everyday default, even if it's preferred.
Really? I had better immersion with the first Quest
Apple TV+ content like Adventure [1] is really compelling as the next generation evolution of video media. The highlining episode is very impressive. Close up views of Faith's face and eyes and expansive views of the mountains and rivers, it really needs to be seen to understand. I really look forward to new content shot with this.
[1] https://tv.apple.com/us/show/adventure/umc.cmc.5al10vz5fkqzx...
[1] https://tv.apple.com/us/show/adventure/umc.cmc.5al10vz5fkqzx...
I can think of at least one: porn.
Except Apple won't allow it, though I'm sure people will try to flit with that line with things like music videos.
You don't need a porn-specific app, just a general VR video viewer that can load video files.
For PCVR users, that's likely going to be DeoVR. SteamVR has a built-in media player, but DeoVR is far better. More flexible, lets you move the video around, etc. Like, one of the big things for me when doing VR porn is that the camera tilt needs to match my head tilt, or else I get motion sick. So if the camera is pointed down 45 degrees, then by default, that means if my head is pointed straight forward, my view is actually looking down 45 degrees, and I get sick. DeoVR lets me move the video down, so that looking downwards makes me look at the center of the video.
For PCVR users, that's likely going to be DeoVR. SteamVR has a built-in media player, but DeoVR is far better. More flexible, lets you move the video around, etc. Like, one of the big things for me when doing VR porn is that the camera tilt needs to match my head tilt, or else I get motion sick. So if the camera is pointed down 45 degrees, then by default, that means if my head is pointed straight forward, my view is actually looking down 45 degrees, and I get sick. DeoVR lets me move the video down, so that looking downwards makes me look at the center of the video.
Heresphere is even better as a VR player, if you buy or pirate it. It has decent scale and position adjustment.
Sure… they’re going to ban one of the killer applications for their expensive gadget. Same as they won’t allow pron in Safari?
Porn has always been banned from the App Store. Jobs specially said it years ago. The browser has been the one way to access it, which limits what can be done.
High quality stereoscopic content. During the pandemic, I had my Oculus Quest with a Woojer Edge and watched a music festival 360 video. It was amazing.
Full spectrum rush.
Full spectrum rush.
Absolute unit of a camera.
Now give me glasses with this resolution per eye…