Generative Image Dynamics(generative-dynamics.github.io)

346 pointsby hughesvor 3 Jahren29 comments

generative-dynamics.github.io

https://generative-dynamics.github.io/

10 comments

This is super cool. Cinemagraphs have always been a bit of a passion of mine, and I try to bring that feeling of subtle-stillness in a lot of the work I do, whether it’s marketing or shooting, so i can see this becoming a regular tool.

The trick to a 10/10 cinemagraph is the more subtle, the bigger the impact. You almost want the viewer to think it’s a still photo before their brain clicks in thinks “wait, something isn’t normal here, this isn’t a photo, it’s a video”

albert_evor 3 Jahren

any good examples you can share please?

nextaccounticvor 3 Jahren

there's a subreddit about that

check out www.reddit.com/r/cinemagraphs/top

nicovor 3 Jahren

Not op, and not sure if these qualify as cinemagraphs, but they kinda fit the “subtle image movement” description

https://twitter.com/nicobrenner/status/1685754265393041408?s...

https://twitter.com/nicobrenner/status/1686074457159274496?s...

https://twitter.com/nicobrenner/status/1685875401351217152?s...

https://twitter.com/nicobrenner/status/1686060465783443457?s...

https://twitter.com/nicobrenner/status/1685170395086045185?s...

vsskanthvor 3 Jahren

These just look like vibrating images on my phone

[deleted]vor 3 Jahren

nicovor 3 Jahren

Impressed and sad with how toxic HN can be sometimes

Just sharing some art here, and getting downvoted without even a comment about it. And even getting flagged

@dang care to check what’s going on here?

hashharvor 3 Jahren

Maybe because it could be construed as "self-promotion" especially because you didn't bother to tell that you are the author of those tweets.

puffybloggervor 3 Jahren

[deleted]vor 3 Jahren

juunppvor 3 Jahren

The tree has severe distortion when dragged from the edge. Still an interesting idea.

Timon3vor 3 Jahren

You'd probably have to combine this with segmenting and generative infill for the background layers, but luckily there's been a lot of progress there!

divyajgvor 3 Jahren

I wonder why in the first picture (red rose) the flower in the bg also moves, but we don't see the same affect in the third picture (tree). I also find it impressive that the amount of motion differs in the first and the second picture, could it be because the density around the pointer is considered?

The slo-mo ones are super relaxing to watch!

actionfromafarvor 3 Jahren

I don't know why but I reacted with slight fear to the rose ones.

waffletowervor 3 Jahren

Nice to see Google researchers continuing to publish open papers with bonus demos. Won't beat a dead horse about Google failing to productize or open source their AI research.

crazygringovor 3 Jahren

This is so cool. Not earth-shattering or productivity-enhancing, but still really cool.

I could definitely see this becoming a standard feature on desktop and phone wallpapers.

Could also see it being applied selectively to photos in things like historical documentaries -- especially if it can handle the gentle movement of water and clouds as well.

CSServor 3 Jahren

They used webGL for the demo. Nice!

sva_vor 3 Jahren

This would be crazy in a video game. Walking through a bush and dragging the plant with you

dragonwritervor 3 Jahren

> This would be crazy in a video game. Walking through a bush and dragging the plant with you.

But game physics can handle stuff like that, there's no need for GenAI.

esperentvor 3 Jahren

I've been playing Red Dead Redemption 2 recently. When it comes to 3D nature scenes, this game is state of the art, and incredible in its attention to detail.

Still, one of the things that stands out to me is how fake the small movements of leaves, grass, and flowers in the wind still feels. The actual physics of tree branches as you push past them isn't bad, but you can't use the physics engine for every leaf. The wind in trees and grass is still just a randomized oscillation of some kind that doesn't quite feel real.

So I was immediately struck that this technique would be an improvement, perhaps using animated billboards. But it would come with its own issues - can't be done in real time any more than physics, for example, so it would be hard to use it for real time weather systems. And anyway, in that case since you're using a transparent video, there's probably easier ways to make it.

solardevvor 3 Jahren

RDR2 was good for its time, but is it really state of the art these days? Seems like other games have pretty good grass...

Horizons Forbidden West: https://youtu.be/COY-pKl1BJY?si=uDrWYAqAlAiNVKZ8&t=16

Crysis 3: https://www.youtube.com/watch?v=4_WwYMFS8EM

Zelda BOTW: https://www.youtube.com/watch?v=1T7Z-u_JfmE

Or in UE5:

https://youtu.be/Dj60HHy-Kqk?si=-iLHHWEH3cOEo-2L&t=42

https://www.youtube.com/watch?v=AShGmWyFamY

https://www.youtube.com/watch?v=ZP_9DUhXVbk&t=2s

behnamohvor 3 Jahren

I'm still waiting for video games to adopt stable diffusion, GPT, and other GenAI models. the tech is there, but I guess the inertia in the industry doesn't allow us to have nice things yet.

a_wild_dandanvor 3 Jahren

You'll see those things! We already have DLSS, for instance. But unfortunately we can't simply glue expensive black boxes onto games and ship them. Wrangling performant, richly interactive media is difficult enough without these models. This modern ML + gaming fusion space is barely in its infancy. We need to explore what's practical, and discover patterns to do it.

Even without further breakthroughs, the next 5 to 10 years will be incredible. I'm so excited.

wayfindervor 3 Jahren

Wouldn’t say tech is there yet. It still needs a lot of human input and direction so slapping it into a video game would just be immersion breaking when it generates something out of character randomly.

There’s less impactful ways to implement it like generating art paintings in a museum dynamically but that is in the “a little gimmicky” territory.

orbital-decayvor 3 Jahren

Temporal stability has been solved for a while [1], there's just nothing of that grade for diffusion models currently. From above, tagged geometry guiding a temporally stable neural renderer seems to be the way to go for games, but this needs to be confirmed in the trenches. The industry has a lot to digest, and of course the hardware should improve quite a bit.

[1] https://isl-org.github.io/PhotorealismEnhancement/

sva_vor 3 Jahren

I think quite a bit of stuff like texture generation has already been used for some time - just on the developers machines pre-release. I suppose using these technologies client-side in real time will be something else entirely. Consumer hardware is lagging behind for that though, even with optimizations such as quantization tweaks etc.

peloratvor 3 Jahren

What do you mean, lots of games have physics that interreact with flora?

CSServor 3 Jahren

Wow, that's a neat idea! That could potentially be pretty cool. It'd almost be like a form of photogrammetry but for physics. Kinegrammetry, maybe? I wonder what the storage efficiency and performance would look like. Perhaps something like ths could be adapted into a framework for object modeling.

Hard_Spacevor 3 Jahren

This suffers from the same low-vector movement requirements as EbSynth.

GaggiXvor 3 Jahren

I think the achievement here is mostly about generating the image dynamics, so for example there is a cat in an image, the model understand that cats need to breathe so the dynamics show the lungs contracting, then the paper covers how to traslate the image dynamics and the image itself into a seamless video. I could be wrong tho

[deleted]vor 3 Jahren

FrostKiwivor 3 Jahren

That's one step away from Harry Potter style photo frames for static photos.

mdrznvor 3 Jahren

Wow! This seems surreal, can't wait to have it integrated into Photoshop.