Light Field Photography(news.kynosarges.org)
news.kynosarges.org
Light Field Photography
http://news.kynosarges.org/2014/05/11/light-field-photography/
9 comments
I took a class that explained light field photography, but I thought understanding the concepts through static diagrams was difficult. I made a little Mac app that simulates a 2D scene of light sources, lenses, and light-field sensors. The sensors shows their captured output in a graph.
The source is at https://github.com/bridger/optic-workbench
A bit of explanation (and a link to the pre-built binary) is at http://www.bridgermaxwell.com/index.php/blog/optic-workbench....
(Sorry it is only for Macs. I would love to redo it in Javascript someday, once I know Javascript.)
The source is at https://github.com/bridger/optic-workbench
A bit of explanation (and a link to the pre-built binary) is at http://www.bridgermaxwell.com/index.php/blog/optic-workbench....
(Sorry it is only for Macs. I would love to redo it in Javascript someday, once I know Javascript.)
The problem with lightfield photography, other than the inherent image quality calculus of splitting a sensor into a bunch of lower-resolution, lower-illumination, differently-focused subfields, is that it's so easily simulated if anyone actually attempts to do scene capture properly - through adaptations of stereo-vision algorithms for higher-N arrays of semi-calibrated sensors, or near-infrared structured light (like the Kinect) for a 4D scene, or structure from motion for a static 3D scene.
The new HTC One M8 uses crap smartphone sensors, only two of them, without any structured light, with a first-generation stereo-blurring algorithm dumb enough to run quickly on a smartphone, and it produces a rough approximation of the same product as Lytro.
The new HTC One M8 uses crap smartphone sensors, only two of them, without any structured light, with a first-generation stereo-blurring algorithm dumb enough to run quickly on a smartphone, and it produces a rough approximation of the same product as Lytro.
As a programmer who also happens to be a photographer, that's flat out wrong.
The M8 produces a very crappy imitation of out of focus blur of what the bokeh of an camera with a thin depth of field looks like. [1] The linked article is actually being "neutral" and fairly generous; the pictures, frankly, look like shit.
It is difficult to properly do DoF, even if you know the Z-distance of every object in the scene. It's not just a simple blur of things that are deemed out of focus; things like spherical differences in the lens to produce hard/soft bokeh matter. [2]
"So easily simulated" couldn't be further from the truth. A bunch of academic papers have been written about this, [3] which include ways to reproduce effects that are similar to the "thin DoF" effect in large sensor cameras, but few try to tackle _all_ of the issues and nuances present.
[1] http://www.trustedreviews.com/opinions/htc-one-m8-camera-vs-... [2] http://jtra.cz/stuff/essays/bokeh/ [3] http://dl.acm.org/citation.cfm?id=2389316.2389318
The M8 produces a very crappy imitation of out of focus blur of what the bokeh of an camera with a thin depth of field looks like. [1] The linked article is actually being "neutral" and fairly generous; the pictures, frankly, look like shit.
It is difficult to properly do DoF, even if you know the Z-distance of every object in the scene. It's not just a simple blur of things that are deemed out of focus; things like spherical differences in the lens to produce hard/soft bokeh matter. [2]
"So easily simulated" couldn't be further from the truth. A bunch of academic papers have been written about this, [3] which include ways to reproduce effects that are similar to the "thin DoF" effect in large sensor cameras, but few try to tackle _all_ of the issues and nuances present.
[1] http://www.trustedreviews.com/opinions/htc-one-m8-camera-vs-... [2] http://jtra.cz/stuff/essays/bokeh/ [3] http://dl.acm.org/citation.cfm?id=2389316.2389318
in [1]:
"Now, this simulation above may look too artificial and unintuitive. Real lenses do not behave like this, right? They actually do. Just look at this picture below."
So he produces an accurate convolution model of how bokeh looks in soft/hard configurations, and that's proof it's not easily simulated? If it's not a straight Gaussian blur (as the HTC is likely using), that's fine - it can still be simulated.
What plenoptic cameras do is use a microlens array to perform focus bracketing. One can use a macrolens array as well, if one lives in a rich enough 3D-interpolating image processing pipeline.
Take a brick-sized lump of plastic, apply, say, 20 tiny cameras (each of four corners and center get a sub-array of medium_NIR_filtered-far-medium-near), and four temporally staggered Kinect structured light projectors on side, and you get all your cues double-checked by redundant information. Your algorithms get to play with direct low-resolution texture-focus info, stereo distance for a bunch of long-baseline pairs in two dimensions, redundant short-baseline pairs, active focus returns on laser dots as an anchor, stereo returns on laser dots, everything. One checkerboard calibration and such a system cross-calibrates itself and characterizes a position, focus, & lens distortion model for every lens.
Add additional lenses if you want to play with more of or some other type of bracketting.
The point is, attempting proper computer vision scene capture involves a crapload of sensors checking their results against each other, ideally diverse sensors and code. We simply haven't tried that, outside perhaps professional motion capture suites. The sky's the limit on such things.
[1] http://jtra.cz/stuff/essays/bokeh/#what_affects_bokeh_shape
So he produces an accurate convolution model of how bokeh looks in soft/hard configurations, and that's proof it's not easily simulated? If it's not a straight Gaussian blur (as the HTC is likely using), that's fine - it can still be simulated.
What plenoptic cameras do is use a microlens array to perform focus bracketing. One can use a macrolens array as well, if one lives in a rich enough 3D-interpolating image processing pipeline.
Take a brick-sized lump of plastic, apply, say, 20 tiny cameras (each of four corners and center get a sub-array of medium_NIR_filtered-far-medium-near), and four temporally staggered Kinect structured light projectors on side, and you get all your cues double-checked by redundant information. Your algorithms get to play with direct low-resolution texture-focus info, stereo distance for a bunch of long-baseline pairs in two dimensions, redundant short-baseline pairs, active focus returns on laser dots as an anchor, stereo returns on laser dots, everything. One checkerboard calibration and such a system cross-calibrates itself and characterizes a position, focus, & lens distortion model for every lens.
Add additional lenses if you want to play with more of or some other type of bracketting.
The point is, attempting proper computer vision scene capture involves a crapload of sensors checking their results against each other, ideally diverse sensors and code. We simply haven't tried that, outside perhaps professional motion capture suites. The sky's the limit on such things.
[1] http://jtra.cz/stuff/essays/bokeh/#what_affects_bokeh_shape
To be fair, Bokeh on the Illum is pretty poor, too. The M8 bokeh looks fake. The Illum Bokeh looks noisy and pixelated (and the effective resolution looks far lower than 5MP to me). I wouldn't bother with either.
The thing that excites me most about Light Field Photography is that LF cameras are the only thing that could take a photo that would actually feel at home in an Oculus Rift. Binocular cameras fake the 3D experience by capturing two 2d images and fixing them to each eye. But in the Oculus you can move your head around so two 2d images aren't enough. You need an actual depth map in order to generate images in response to head movements. LF cameras can provide one.
http://www.cs.berkeley.edu/~ravir/lightfield_ICCV.pdf
http://www.cs.berkeley.edu/~ravir/lightfield_ICCV.pdf
Not really. Without the correct IOD (interocular distance) [1] you won't get "correct" stereoscopy, and the effect you get will not feel right at best, and make you sick in the worst case.
A LF camera only gives a stereoscopic effect proportional to the size of the sensor. So unless you have a sensor even bigger than a medium format camera, you are not going to achieve the effect you are describing.
[1] https://en.wikipedia.org/wiki/Stereoscopy
A LF camera only gives a stereoscopic effect proportional to the size of the sensor. So unless you have a sensor even bigger than a medium format camera, you are not going to achieve the effect you are describing.
[1] https://en.wikipedia.org/wiki/Stereoscopy
The premise here would be two LF cameras, mounted rigidly in an IOD-like frame.
Or ideally, more than two, for flexibility of the IOD and a better stereo-vision fix, potentially used in 2D-3D compositing.
Or ideally, more than two, for flexibility of the IOD and a better stereo-vision fix, potentially used in 2D-3D compositing.
I'm not sure that consumers don't care about bokeh, they just don't know how to achieve it. I've been using the Google camera app that does lens blur and everyone I show it to loves it. It's a little cumbersome to use though and require a still subject. Maybe light field could be one of the features that keep compact cameras around in the face of competition from phones.
I also wonder if there are computer vision applications. I don't know how accurate depth fields from stereoscopic images are, and maybe two cameras are not always practical...
I also wonder if there are computer vision applications. I don't know how accurate depth fields from stereoscopic images are, and maybe two cameras are not always practical...
Bokeh isn't just reduced depth of field. Bokeh is the quality of the out-of-focus areas. Shallow depth of field makes the bokeh more apparent, but the two are somewhat orthogonal.
Looking at Lytro's examples, e.g. [1], I would say that the Illum demonstrates poor quality bokeh as a result of the sensor limitations. The bokeh is quite noisy and pixelated (which is a strange thing to say about bokeh, but it's the result of the microlens sampling the Lytro sensor does).
The technology is very interesting, but looking at Lytro's pictures has convinced me that their camera will not suit my needs any time soon.
[1] https://pictures.lytro.com/lytro/albums/149429/embed?token=6...
Looking at Lytro's examples, e.g. [1], I would say that the Illum demonstrates poor quality bokeh as a result of the sensor limitations. The bokeh is quite noisy and pixelated (which is a strange thing to say about bokeh, but it's the result of the microlens sampling the Lytro sensor does).
The technology is very interesting, but looking at Lytro's pictures has convinced me that their camera will not suit my needs any time soon.
[1] https://pictures.lytro.com/lytro/albums/149429/embed?token=6...
I don't see the $1.5k Lytro is going to gain enough market share to be profitable too. The only option I see for Lytro is to miniaturize their technology so that it is significantly better than Google's software offering and to partner with a major phone manufacturer.
The Lytro gallery:
https://pictures.lytro.com/lytro/albums/149429/embed?token=6...
I want this.
https://pictures.lytro.com/lytro/albums/149429/embed?token=6...
I want this.
a photo where nothing is in focus. great.
lytro needs a different name than photography because it's something else.
lytro needs a different name than photography because it's something else.
Are you serious? Click where ever you want to focus the shot
Could LFP be used to 3D-map a scene - assuming you can extract the data from their proprietary format? It would be great to be able to turn a collection of photos into a virtual world.
> "Holographic Recording ... remains irrelevant to consumers until we have convenient holographic displays to match."
I'm not sure why Lytro hasn't touted this side of the technology... the first consumer holographic display (oculus rift) is due relatively soon.
I'm not sure why Lytro hasn't touted this side of the technology... the first consumer holographic display (oculus rift) is due relatively soon.
The rift is not really holographic, it's 'just' stereoscopic. Nvidia is working on light field glasses though, and I believe this might be the future, because it allows for much sleeker eyepieces, at a higher computational cost. https://research.nvidia.com/publication/near-eye-light-field...
I believe[1] that the head tracking qualifies it as holographic. That light field stuff looks super interesting...
1. http://doc-ok.org/?p=337
1. http://doc-ok.org/?p=337
It's not holographic because you can't focus your eyes at different depths in the image.
It's not holographic because it doesn't use diffractive / interferometric effects.
[1] https://en.wikipedia.org/wiki/Holography
[1] https://en.wikipedia.org/wiki/Holography
Holography is a specific thing, let's not confuse things by assigning magical powers to the Rift.
Holographic displays already exist, usually they're transparent, they work by letting you focus on things through the display and the display stays in focus, making things like AR and HUDs work much better than other types of displays.
Holographic displays already exist, usually they're transparent, they work by letting you focus on things through the display and the display stays in focus, making things like AR and HUDs work much better than other types of displays.
My mistake, but I'm not assigning magical powers. I'm aware of what a holographic display is, and the rift fulfils 3 of the 4 criteria. The lack of accommodation in my experience is not critical and the applicability to lytro is still valid.
Sorry if I came off as snarky, but the attributes that have been assigned to the Rift are getting kind of out of control.
It's just a pair of high DPI displays with nice low latency head tracking tech and better than average API support. Other than using photons, the Rift fulfills almost zero of the criteria for holography.
https://en.wikipedia.org/wiki/Holography
It's just a pair of high DPI displays with nice low latency head tracking tech and better than average API support. Other than using photons, the Rift fulfills almost zero of the criteria for holography.
https://en.wikipedia.org/wiki/Holography
I was going off http://en.wikipedia.org/wiki/Holographic_display FWIW
I could be wrong, but IMO you and bane are both confusing "holography" with "3D display", and/or "display that involves depth". The author of the article made a similar mistake, and unfortunately a lot of commercial products are mislabeled as "holographic", so this is unsurprising.
As bane said, holography is a very specific thing. Most of the products I'm aware of today that claim to be "holographic" are things like head-up displays and red-dot gun sights. They use optics to make it appear as though a 2D image is appearing some distance behind the display surface. It's a clever, useful effect, but it is not holographic.
I think the author of the article is also leaving out one of the main interesting potential uses of depth-sensing cameras, which is to feed the output into a 3D printer. You don't need a holographic display (or even a faux-holographic depth display) for that.
As bane said, holography is a very specific thing. Most of the products I'm aware of today that claim to be "holographic" are things like head-up displays and red-dot gun sights. They use optics to make it appear as though a 2D image is appearing some distance behind the display surface. It's a clever, useful effect, but it is not holographic.
I think the author of the article is also leaving out one of the main interesting potential uses of depth-sensing cameras, which is to feed the output into a 3D printer. You don't need a holographic display (or even a faux-holographic depth display) for that.
Actually the display I'm talking about wasn't a 3d display it was a wearable holographic HUD I saw at a DARPA demo day. The use-case for these kinds of displays is pretty much to ensure a graphic on the display stays in focus no matter where your eyes focus, or where they are relative to the display by producing a hologram of the display.
It's subtly different from similar see-through HUDs because you don't have to keep changing your focus back and forth to the HUD.
The DARPA guy I was talking to wasn't working on the display but on some other bits of the HUD and didn't know all the details about how it worked sadly. Looking for it later I think it was a holographic waveguide technology.
http://www.dodsbir.net/sitis/archives_display_topic.asp?Book...
It's subtly different from similar see-through HUDs because you don't have to keep changing your focus back and forth to the HUD.
The DARPA guy I was talking to wasn't working on the display but on some other bits of the HUD and didn't know all the details about how it worked sadly. Looking for it later I think it was a holographic waveguide technology.
http://www.dodsbir.net/sitis/archives_display_topic.asp?Book...
My use of the term "holographic" was taken directly from Ng's dissertation, near the end of section 3.7: "...Camahart showed that holographic stereograms can be synthesized from light fields." Whether he's strictly correct with his use of the term I don't know.
Ng also mentions working with Zebra Imaging on printing such "holograms" [sic], so that's certainly a possible application. I'm not sure a lot of consumers would be interested in that, though, compared to simply showing pictures.
Ng also mentions working with Zebra Imaging on printing such "holograms" [sic], so that's certainly a possible application. I'm not sure a lot of consumers would be interested in that, though, compared to simply showing pictures.
Quick, shots of street moments. Shots where you have no time to focus properly, but want to focus the field of view on a subject...a lightfield camera is great for these situations and I think isn't played up enough.