What happens if we don't add any moments matching objective? e.g. at train time just fit a diffusion model that predicts the target given any pair of timesteps (t, t')? Why is moment matching critical here?
Also regarding linearity, why is it inflexible? It seems quite convenient that a simple linear interpolation is used for reconstruction, besides, even in DDIM, the directions towards the final target changes at each step as the images become less noisy. In standard diffusion models or even flow matching, denoising is always equal to the prediction of the original data + direction from current timestep to the timestep t'. Just to be clear, it is intuitive that such models are inferior in few-step generations since they don't optimise for test time efficiency (in terms of the tradeoff of quality vs compute), but it's unclear what inflexibility exists there beyond this limitation.
Clearly there's no expected benefit in quality if all timesteps are used in denoising?
Fascinating article. I wonder what made her do the steps in finding what's "different" with her, and most of all, why the need to fix it arose. Is it to "fit in", understandably? It somehow felt alien to me, and this shows my ignorance on the topic, that people lacking in empathy department would attempt to understand the reasons and act "good" towards others even if this feeling is only understood intellectually.
Yes it makes sense a bit. Many popular convents operate on 3x3 kernels. But the number of channel increases per layer. This, coupled with the fact that the receptive field increases per layer and allows convnets to essentially see the whole image relatively early in model's depth (esp. coupled with pooling operations which increase the receptive field rapidly), makes this intuition questionable. Transformers on the other hand, operate on attention which allows them to weight each patch dynamically, but it's clear to me that this allows them to attend to all parts of the image in a way different from convnets.
Can you give some example usecases of your application? I wonder how it scales to complex (in terms of structure) information processing, e.g. digesting scientific topics
I've got good results with white noise, not sure if that's something that would work for you. And of course I didn't suggest that this eliminates other distractions.
A bit surprising to hear that you weren't swimming in offers to hire you all the time. Any idea why that was? Maybe you're somewhat picky about the job and responsibilities that come with it?