StyleNeRF: A Style-Based 3D-Aware Generator for High-Resolution Image Synthesis(jiataogu.me)
jiataogu.me
StyleNeRF: A Style-Based 3D-Aware Generator for High-Resolution Image Synthesis
http://jiataogu.me/style_nerf/
9 comments
While watching it, I wondered whether the "this x does not exist," GAN sites also implies a subset of "this x must exist" and if looking at it that way could yield discoveries in other areas. e.g. This X must exist, where x was a solar system, protein, wave function, chemical substance, cell, transcendental number, etc. Naive view, but when you're extapolating from a field of arbitrary 2D data, it seems there are a lot of ladders you could climb with that.
The fact that this was in part developed with the help of Facebook AI Research is kind of unsettling.
That said, the results are actually very impressive.
That said, the results are actually very impressive.
It's only unsettling if you are completely unfamiliar with the machine learning field. FAIR publishes research in many ML/AI fields. They're responsible for plenty of advances and contributions.
I’m not sure that’s what was meant by unsettling…
Didn’t Facebook just get sued for illegally harvesting faces & tags of people? If derivatives of that data make it into research like this, where does that stand from a legal perspective?
Kinda puts a spin on the "face" in "facebook"
It literally puts a spin on it, actually!
I see there are a lot of questions/doubts in the comments about the usefulness of such research. This type of research is the precursor to gaining the ability to generative photorealistic 3D worlds. What is that useful for? Anywhere we currently generate 3D environments, including movies, games, VR/AR.
GANs are getting creepier by the day.
Do you mean their potential (mis)use is creepy, or are you genuinely scared by their "abilities"?
This whole research domain is confusing to me. I do get that it's a hard and cool problem to solve and I do see why researcher could work on it on those grounds.
But I'm concerned about what solving this is useful for, especially with the focus on generating human faces. Obligatory Jurassic Park quote: "Your scientists were so preoccupied with whether they could, they didn't stop to think if they should.".
I can imagine this technology to be used in the entertainment industry (movies, video games) as a way to save money on art. But it's quite obvious that this technology can be weaponized and used in the propaganda industry (if it isn't already).
So, this could save money in fun applications on one end, and provide means for mass opinion manipulation weapon on the other end. I am certainly biased but I'm not sure the recreative use case is worth the risk of the weaponized one (a bit like using tactical nukes for a firework show).
What am I missing then? Are there any other use cases that would make this technology actually desirable for the greater good or humanity, or is it just a weapon with some potential recreational use cases?
But I'm concerned about what solving this is useful for, especially with the focus on generating human faces. Obligatory Jurassic Park quote: "Your scientists were so preoccupied with whether they could, they didn't stop to think if they should.".
I can imagine this technology to be used in the entertainment industry (movies, video games) as a way to save money on art. But it's quite obvious that this technology can be weaponized and used in the propaganda industry (if it isn't already).
So, this could save money in fun applications on one end, and provide means for mass opinion manipulation weapon on the other end. I am certainly biased but I'm not sure the recreative use case is worth the risk of the weaponized one (a bit like using tactical nukes for a firework show).
What am I missing then? Are there any other use cases that would make this technology actually desirable for the greater good or humanity, or is it just a weapon with some potential recreational use cases?
I don't think of this as "we really wanted the ability to synthesize faces, this was very important to us as a civilization."
Rather, I think of it as "if we create an AGI, it will have required the ability to imagine things. Also, we have huge datasets of faces that all sorta point the right way and have a good medium of feature complexity, and we're extremely specialized in spotting visual errors in faces."
Consider the similar if slightly smaller research in creating photos of living rooms. (Or OpenAI's famous "comfy chair in the shape of an avocado.") Same reasoning. It's not about faces, faces just have beneficial properties.
Rather, I think of it as "if we create an AGI, it will have required the ability to imagine things. Also, we have huge datasets of faces that all sorta point the right way and have a good medium of feature complexity, and we're extremely specialized in spotting visual errors in faces."
Consider the similar if slightly smaller research in creating photos of living rooms. (Or OpenAI's famous "comfy chair in the shape of an avocado.") Same reasoning. It's not about faces, faces just have beneficial properties.
I guess the non-creepy application I'm seeing for the future of this research is the seamless conversion of a 2D image into 3D for when 3D projectors become more commonplace and perhaps overtake the ubiquity of flatscreens on our devices. And I am not sure, but it might also be applicable to the video-cam issue where we want to make it look as though the person we're having a video conference with is looking at us instead of the screen image of us and vice-versa.
So you're not terminator-worried, just jurrasic park-worried then. (that was what my question was about)
>I'm concerned about what solving this is useful for
As for all research, it is pretty hard to predict what and what for something will be useful (despite researchers being pushed to say stuff like "my research is going to cure cancer" to get funding). But about neural networks for image processing, they certainly start to be useful for radiology.
>I'm concerned about what solving this is useful for
As for all research, it is pretty hard to predict what and what for something will be useful (despite researchers being pushed to say stuff like "my research is going to cure cancer" to get funding). But about neural networks for image processing, they certainly start to be useful for radiology.
> I do get that it's a hard and cool problem to solve and I do see why researcher could work on it on those grounds.
There are quite a lot of harder and potentially even more useful if solved problems that receive significantly less attention and funding in this domain. Vast majority of GAN papers nowadays are just different applications of them, without significant contributions to underlying theoretical foundations.
There are quite a lot of harder and potentially even more useful if solved problems that receive significantly less attention and funding in this domain. Vast majority of GAN papers nowadays are just different applications of them, without significant contributions to underlying theoretical foundations.
If it's "just code" someone somewhere is going to do it eventually, restrictions notwithstanding. Might as well do it in the open and publish about it so the problem space and its implications are (at least potentially) the object of further research.
Oversimplification: Jurassic Park is about the lack of robust peer-review before going to prod!
Consider the case of how to create a very strong facial recognition system, without relying on photos of actual people. This system provides a data generation capability suitable to produce a training set with billions of "people", none of whom have had their privacy wounded by having their specific face in the training set.
That's not exactly a happy use case. Why would we even want very strong facial recognition systems?
Yea I mean the most eminent applications for this stuff is, putting unconsenting peoples likeness into porn, black mail/framing people, facial recognition, deceptive propaganda(because it's not deceptive enough already), weakening criminal evidence, etc. Then comes stuff like cgi for movies/games. Seeing a lot more of the former use cases not much of the latter. Pretty sure most people doing modern cv don't really realize where it's going, and people aren't publishing on the hazards so a lot of the criminality is and will remain novel until someone works it out..
So short sighted. A strong facial recognition system can locate lost people in a wilderness from drone footage. A strong facial recognition system can discriminate between twins, near age siblings, and ethnically similar but unrelated individuals. A strong facial recognition system retains reliability when test imagery is poor quality. A strong facial recognition system can identify kidnapped children being trafficked, even after they have aged. A strong facial recognition system compensates for the complete lack of quality operator training within the facial recognition industry.
Cool idea :)
1. They generate a low-resolution position map using NeRF => Good camera and 3D control
2. They then upsample that with StyleGAN2 => Good features in the final 2D image.
So in effect, this is a StyleGAN2 network where the latent space has the 3D-related components split off into a different generator network.
1. They generate a low-resolution position map using NeRF => Good camera and 3D control
2. They then upsample that with StyleGAN2 => Good features in the final 2D image.
So in effect, this is a StyleGAN2 network where the latent space has the 3D-related components split off into a different generator network.
that's weird that they mostly ignore the comparison to stylegan3:
https://nvlabs.github.io/stylegan3/
it's not a nerf based solution but since the goal is high resolution image syntesis..
it's not a nerf based solution but since the goal is high resolution image syntesis..
subtitle: an amazing trip through the uncanny valley
The twinkling children's TV style background music in that video is not making it any less creepy.
Interesting that the reflections in each eye vary so much between each other.
This might fix that: https://dorverbin.github.io/refnerf/