(Author here.) Maybe it's worth moving to a GitHub issue. Try `--model=small`. The demo server limits the number of pixels to around 320x200 or 256x256 and can do only 4 at the same time to fit in RAM.
(Author here.) Unlike most other non generative adversarial network (GAN) approaches to super-resolution, it does try to inject high-frequency detail; see the faces example on GitHub. But I tuned down that parameter in the released models a bit so it performed better generally.
(Author here.) If you have the luxury to train on domain-specific textures, the results will definitely be better. That's why I included all the training code in the repository as well—to allow for this kind of solution.
If you scroll down on GitHub to see the faces examples, those are achieved by a domain-specific network. I suspect you'll similarly get extremely high-quality if you have good input images.
(Author here.) Did you see the faces example on the GitHub page? It was a domain-specific network trained adversarially for that purpose, but I have yet to see any super-resolution of that quality with or without machine learning.
Most other approaches don't even try to inject high-frequency detail into the high-resolution images because the PSNR/SSIM benchmarks drop. Until those metrics/benchmarks are dropped, there'll be little more progress in super-resolution.
(Author here.) Yeah, I knew this would come up but decided to proceed with the pixelated comparison anyway. I couldn't get the GIFs to reflect the results because of 8-bit quantization/dithering. The images show the neural network inputs and outputs, not a comparison with other super-resolution algorithms (still fascinating :-).
I'm working on the Docker instance now, that should help anyone with interest/experience in the field compare results easily.
(Author here.) Absolutely! Using multiple super-resolution networks, not only continuity would present problems, but also blending between different regions. I agree there's a lot of value for domain-specific networks here, as you can see from the faces example on GitHub.
I'd be curious to see an ensemble-based super-resolution, where each model can output the confidence of a pixel region, then have another network learn to blend the result.
Conversely, these results are achieved using a single top-of-range GPU. Everything fits in memory for a batch-size 15 at 192x192. By distributing the training somehow, you could make the network 10x bigger and train for a whole week and likely get much better general purpose results.
(Author here.) My biggest insight from this project is that super-resolution with neural networks benefits significantly from being domain specific. If you train on broader datasets, it does pretty well but has to make compromises. Many recent papers do a comparison in terms of pixel similarity (PSNR/SSIM), and using those metrics the quality drops because high-frequency detail is punished under those criteria (even though it may look better perceptually). Reference: http://arxiv.org/abs/1609.04802
On GitHub, below each GIF there's a demo comparison, but on the site you can also submit your own to try it out (click on title or restart button). Takes about 60s currently; running on CPU as GPUs are busy training ;-)
Almost nobody in deep learning uses OpenCL. All the DL frameworks primarily focus on CUDA and that's where you get the best performance. OpenCL is off the beaten path and you pay for it in every way: support, performance, reliability.
AMD is going to support CUDA somehow too, I think that's a sign they admitted defeat on OpenCL for this.
You can use any image as source, but to create annotations you have to do that yourself currently. Using simple segmentation libraries (or clustering) can do a good job for certain images, or look at better solutions for semantic segmentation: http://gitxiv.com/search/?q=segmentation
From that perspective, this research is two steps further than Neural Style, I wrote about it yesterday here: http://nucl.ai/blog/neural-doodles/
First, the paper I call "Neural Patches" (Li, January 2016) makes it possible to apply context-sensitive style, so you have more control how things map from one image to another. Second, we added extra annotations (which you can specify by hand or from a segmentation algorithm) that helps you control exactly how you want the styles to map. We call that "semantic style transfer" (Champandard, March 2016).
You're right about it being hard otherwise, it was for many months and that's what pushed this particular line of research! Try it and see ;-)
The semantic map remains static during the optimization, so it can be provided as a pre-computation (e.g. pixel labeling, semantic segmentation, etc.) or done by hand. The ones in the repository are done manually, but now experimenting with other algorithms. Anything that returns a bitfield or masks can be used!
The algorithm does the same thing every time (it's triggered on request), only the input is changed by the human modifying the doodle—as shown in the video.
The output gets better because through iteration the glitches are removed incrementally, and it converges on a final painting that looks good!
The research is based on work I did writing and improving @DeepForger (http://twitter.com/deepforger), an online service for "basic" style transfer. The GitHub is a standalone version for learning and education, which doesn't do HD rendering as well yet and uses a bit more memory. The positive side, however, is that opening up the source code makes these ideas progress faster!
We'll try to integrate the idea of semantic style transfer into @DeepForger in the future, but this require quite a bit of work to get it to reliably understand portraits or landscapes without anyone's intervention. The fact it does require these semantic maps for all images makes it less straightforward to release as a service.
Actually, the code does none of that ;-) All of the semantics are provided by the users: either as manual annotations or by plugging in an existing architecture for semantic segmentation / pixel labeling. It's designed to be independent of the source of the semantic maps, so we can continue to work on both problems separately.
Thanks for clarifying, I'll update the README. The research paper does a better job of explaining this with its figures!
The algorithm can only reuse combinations of patterns that it knows about, it can do extrapolation but it often ends up being just like a blend. However, you can give it multiple images and it'd borrow the best features from either—for example drawing from all of Monet's work. (Needs more optimization for this to work though, takes a lot of time and memory.)
As for the images, as long as the type of scene is roughly the same it'll work fine. The fact it can copy things "semantically" by understanding the content of the image makes it work much more reliably—at the cost of extra annotations from somewhere. With the original Deep Style Network it's very fragile to input conditions, and composition needs to match very well for it to work (or you pick an abstract style). That was part of the motivation for researching this over the past months.
You can specify two pairs of images (content+annotation) and it'll transfer the style from one to another as consistently as possible. The down side is that you need to find an algorithm, neural network, or person to create the annotations. (We're working on training one for portraits only.)
These examples are in the paper above, direct link for convenience:
No, do you have any good ones? As long as entire sections are colored (not just lines), and those colors match with the annotations of another image, it should work fine!