Depixelizing Pixel Art(research.microsoft.com)
research.microsoft.com
Depixelizing Pixel Art
http://research.microsoft.com/en-us/um/people/kopf/pixelart/supplementary/multi_comparison.html
163 comments
A problem with this algorithm is that it's pretty slow. It would be hard to use it for real time scaling.
Even if it's too slow for realtime on today's hardware, I'd love to see some prerendered sample videos.
Even if it's too slow for realtime on today's hardware, I'd love to see some prerendered sample videos.
Here's a video: https://research.microsoft.com/en-us/um/people/kopf/pixelart...
Am I crazy for preferring the one on the left (nearest-neighbour)? The other two really mess up certain details such as the point numbers, the stem of the fruits and pokey's eyes.
They also tend to cause a very subtle distortion of the outlines of foreground sprites as they move over the background.
They also tend to cause a very subtle distortion of the outlines of foreground sprites as they move over the background.
I preferred hq4x for much the same reason. Removed the large blockiness while still preserving the pixelated feel (and staying truer to the shapes in the original).
No, you aren't. I prefer nearest-neighbor too, I think it preserves the "spirit" of the game. Generally speaking, I prefer it for any kind of "pixel art".
That said, those algorithms are impressive, and I can see why someone could prefer them over a simple nn.
That said, those algorithms are impressive, and I can see why someone could prefer them over a simple nn.
> I think it preserves the "spirit" of the game.
The art-direction of games from this era wasn't done with the intention that these games would be played on pixel-perfect computer monitors, though. The art was designed and tested for CRT TVs, which would generally give you something like http://i385.photobucket.com/albums/oo299/muddi900/LTTP3xNTSC..., or even http://4.bp.blogspot.com/_Kzdww8T9fUA/TRplyGIj6PI/AAAAAAAAAt...
The art-direction of games from this era wasn't done with the intention that these games would be played on pixel-perfect computer monitors, though. The art was designed and tested for CRT TVs, which would generally give you something like http://i385.photobucket.com/albums/oo299/muddi900/LTTP3xNTSC..., or even http://4.bp.blogspot.com/_Kzdww8T9fUA/TRplyGIj6PI/AAAAAAAAAt...
This reminds me of staring at those tiny screenshots on the back of NES boxes with sweaty palms as a six year old.. lol.. Pixel art was not yet embraced as an art form, I guess. (well, not by the adults)
edit: proof: https://www.google.com/search?q=nes+box+back&tbm=isch
edit: proof: https://www.google.com/search?q=nes+box+back&tbm=isch
The one on the left is effectively Super Mario in its original pixelated glory, so it'll certainly look the most familiar.
As interesting as these smoothing algorithms are, they can't add any information to the sprites that wasn't already there. Why make everything look smooth and plasticky when the original game was carefully hand-drawn, pixel by pixel?
I think there is a purpose for this in video, but I'm not sure it's an all or nothing game. Especially here. For starters, the gameplay looks smoother in nearest neighbor because your eyes defocus while playing. Hence, you need sharp pixel edges to dilettante and your eyes do the rest. In a still shot, that's not the case at all. You want focus and smoothness.
Interesting though...
Interesting though...
> They also tend to cause a very subtle distortion of the outlines of foreground sprites as they move over the background.
I'm surprised none of these algorithms are made to keep state between frames, so they can keep things like this consistent.
I'm surprised none of these algorithms are made to keep state between frames, so they can keep things like this consistent.
Is the video based on applying the algorithm to the entire screen, or is it applied to the sprites individually, then they are plotted to the screen?
Just wondering, because the samples in the paper are all lone objects against a clear background. If the same object is drawn in front of a detailed background and then the upscaling is applied, this would affect the patterning. It's a little tricky to tell from the video, since the graphics (especially the backgrounds) use large blocks of plain colours.
Worst case, you could imagine Mario walking in front of the background and his shape shifting madly with every step he took?
(Edit: From freeze-framing the video, I'm guessing that the algorithm is being applied to the whole screen. The scaling works great when Mario is in front of the pale background. There are small artifacts when he crosses the black outline of the green bushes. Very minor in this video, I wonder how it would affect more intricate backgrounds or those that aren't such contrasting colours to the foreground?)
Just wondering, because the samples in the paper are all lone objects against a clear background. If the same object is drawn in front of a detailed background and then the upscaling is applied, this would affect the patterning. It's a little tricky to tell from the video, since the graphics (especially the backgrounds) use large blocks of plain colours.
Worst case, you could imagine Mario walking in front of the background and his shape shifting madly with every step he took?
(Edit: From freeze-framing the video, I'm guessing that the algorithm is being applied to the whole screen. The scaling works great when Mario is in front of the pale background. There are small artifacts when he crosses the black outline of the green bushes. Very minor in this video, I wonder how it would affect more intricate backgrounds or those that aren't such contrasting colours to the foreground?)
> Worst case, you could imagine Mario walking in front of the background and his shape shifting madly with every step he took?
This brings up something interesting: In terms of media containing sprites and minimal layering (e.g. NES-like games), wouldn't it be computationally cheaper to perform the scaling on the sprites and textures independently of one another, instead of post-processing? I wonder if there is any emulator that does this. I'm thinking not, as the NES wouldn't be capable of layering such detail (large sprites + textures). Upsampling emulators.. Hmm...
This brings up something interesting: In terms of media containing sprites and minimal layering (e.g. NES-like games), wouldn't it be computationally cheaper to perform the scaling on the sprites and textures independently of one another, instead of post-processing? I wonder if there is any emulator that does this. I'm thinking not, as the NES wouldn't be capable of layering such detail (large sprites + textures). Upsampling emulators.. Hmm...
It'd be kind of hard, as what makes up a "sprite" lives in two different places - one that stores all the tile data, and another that basically describes which tiles make up which sprites. The latter is likely to change every frame, and potentially the former, too, so you'd still be rescaling some stuff every frame. You'd also have to rescale some things that are affected by palette changes, which sometimes change every frame (popular way to animate water, for example). Would you end up saving time this way? I'm not really sure. It's definitely an interesting idea.
i dont understand, as long as you figure out how to get the sprites into the right formats this would be immensely computationally cheaper ("time-saving"?) than post-processing every frame
The thing is, there's no "right format" for sprites, as they're used in the game. See http://benfry.com/deconstructulator/ for an example of how sprites are handled by the NES for Super Mario Bros. Mario is split into 8x8 pieces that are swapped out as needed: how do you determine which pieces are "Mario's sprite" and which are "that coin's sprite"? Remember that some pieces will change while others won't.
yea i dont doubt it would be complex & to build an upsampling sprite rendering engine you'd have to understand all this stuff. With old games like this it would probably involve some manual work cuz you'd have to recompose images, smooth as a unit, then decompose somehow back to the original tiling. I mean.... I'm not gonna do it & wouldn't bother trying
the computational price will be much much cheaper tho. swap in upsampled sprite (36 tiles to every 1 in old format, let's say) rather than post-processing every single frame in real-time with a smoothing/upsampling algorithm.
it would also keep the pieces distinct from each other rather than having objects/characters/backgrounds morph in & out of one another
the computational price will be much much cheaper tho. swap in upsampled sprite (36 tiles to every 1 in old format, let's say) rather than post-processing every single frame in real-time with a smoothing/upsampling algorithm.
it would also keep the pieces distinct from each other rather than having objects/characters/backgrounds morph in & out of one another
I did a GPU version in 2012. Unfinished though. http://yzhwang.github.io/depixelization/
[deleted]
Well SuperEagle seems even slower for me, and I know for a fact that I can emulate with SuperEagle scaling on and it works fine (>30 FPS). So I don't think their algorithm is out of the question.
> If you had a program that let you author your sprites under the same conditions, but let you preview an upscaled version
Combine Atwood's law with a proposed law of mentioning ideas on HN being implemented shortly thereafter and there may be a HTML5+JS implementation popping up as a "Show HN" soon.
Combine Atwood's law with a proposed law of mentioning ideas on HN being implemented shortly thereafter and there may be a HTML5+JS implementation popping up as a "Show HN" soon.
I wish they would use this in Windows 8. As far as I can tell, their scaling algorithm for 4k displays is: scale up 2x using nearest neighbor, then randomly drop pixels to get back to 125% or 150%. The result is horrifyingly bad. Not just kind of bad, but unusably bad. (Adding insult to injury, they ClearType the text before scaling.)
While something similar would probably be better, this is heavily optimized for pixel artwork. Lots of curves, shading, and most importantly - limited colors. UI is generally much straighter, less shaded, with far more colors.
Here's an example of UI that's scaled with potrace (a similar algorithm): http://i.imgur.com/jDq4M7e.png (left side is the converted side, right side is nearest-neighbor. Top is 3x, bottom is 2x)
That's with the highest amount of colors/depth that potrace allows. On an i7 processor, it takes a LONG time to compute all of that. Each icon is no longer just a couple hundred pixels, it's a couple hundred vector paths. Even rendering it takes a number of seconds to refresh the screen in inkscape.
Here's an example of UI that's scaled with potrace (a similar algorithm): http://i.imgur.com/jDq4M7e.png (left side is the converted side, right side is nearest-neighbor. Top is 3x, bottom is 2x)
That's with the highest amount of colors/depth that potrace allows. On an i7 processor, it takes a LONG time to compute all of that. Each icon is no longer just a couple hundred pixels, it's a couple hundred vector paths. Even rendering it takes a number of seconds to refresh the screen in inkscape.
1. That looks awful.
2. We just need vector-based icons already.
2. We just need vector-based icons already.
I think the ideal would be something similar to fonts: there's the vector definition for big sizes, and optional pixel-perfect definition for small sizes, because rendering vectors on very small sizes (16x16, 32x32) might look worse than hand-crafted pixel work.
These days, font rendering does not often rely on bitmaps, even at small sizes. Instead, they typically use hinting and subpixel rendering. I would love to see these effective techniques applied to vector graphics too.
[deleted]
Linux has had vector icons on the desktop for over a decade now. It truly boggles my mind how Apple (especially once they realized Retina was coming) hasn't gone fully resolution-independent on the UI side by now…
> That's with the highest amount of colors/depth that potrace allows. On an i7 processor, it takes a LONG time to compute all of that.
OTOH, for icons, you just need to compute that once and then cache the png.
OTOH, for icons, you just need to compute that once and then cache the png.
To be fair, that would look a little better with ClearType off.
Do you know if MS is working on Mac-Style resolution independence for W9? I'm currently running a Retina MBP, at 2x res I think, and it works _perfectly_. The first few months some apps didn't work well, but the community adapted very quickly.
Windows has had support for resolution independence where DPI is configurable across a continuum (rather than only available as 1x or 2x scale like on the Mac) for quite some time - the issue is that with a huge legacy application base, a giant morass of drawing technologies, and no shipped "HiDPI" or "Retina" Windows hardware until recently, there's been no real incentive for app developers to support it.
In Windows 7 and 8, WPF apps are already supposed to be resolution independent (vector-based).
Old-style GDI+ apps which aren't marked as "resolution aware" (i.e. almost everything) are poorly scaled by the window manager and hideously ugly.
In Windows 7 and 8, WPF apps are already supposed to be resolution independent (vector-based).
Old-style GDI+ apps which aren't marked as "resolution aware" (i.e. almost everything) are poorly scaled by the window manager and hideously ugly.
Old style GDI+ apps that are incorrectly marked as being resolution aware even though they aren't is a bigger problem than programs not marking it at all. If it's not marked you get simple 2x scaling but if it's marked and not implemented properly anything can happen.
What is needed for this to work is a user setting to override the programs own preference. There is just too much legacy cruft that accidentally marked themselves as being resolution aware even if they're not. Or drop the old flag and create a new one called something like "really really resolution aware"
What is needed for this to work is a user setting to override the programs own preference. There is just too much legacy cruft that accidentally marked themselves as being resolution aware even if they're not. Or drop the old flag and create a new one called something like "really really resolution aware"
Didn't there used to be a user setting in the "Compatibility" tab?
I haven't played with this stuff in a few years - my housemates and I tried to use a 46" TV in the basement as our web browsing machine for a while and ended up frustrated every time we tried to scale the UI on any platform (OSX, Ubuntu with both Unity and GNOME2, Windows).
I haven't played with this stuff in a few years - my housemates and I tried to use a 46" TV in the basement as our web browsing machine for a while and ended up frustrated every time we tried to scale the UI on any platform (OSX, Ubuntu with both Unity and GNOME2, Windows).
> (rather than only available as 1x or 2x scale like on the Mac)
It isn't continuous, but there are more than two scaling levels on OS X:
https://blog.qt.digia.com/wp-content/uploads/2013/04/qtRetin...
More details: https://blog.qt.digia.com/blog/2013/04/25/retina-display-sup...
It isn't continuous, but there are more than two scaling levels on OS X:
https://blog.qt.digia.com/wp-content/uploads/2013/04/qtRetin...
More details: https://blog.qt.digia.com/blog/2013/04/25/retina-display-sup...
Right, but as noted in the blog you link the application is still only exposed to 1x or 2x scale (where 1pt = 2px), and then everything is scaled back down.
On Windows the app is exposed to the scale factor (for better or worse): https://blog.qt.digia.com/blog/2009/06/26/improving-support-...
On Windows the app is exposed to the scale factor (for better or worse): https://blog.qt.digia.com/blog/2009/06/26/improving-support-...
Prior to retina display Macs, OS X implemented resolution independence as an arbitrary scale factor, with most of the UI elements being vector rendered. You could test it out with Quartz Debug. TextEdit, for example, scaled very nicely and remained crisp at arbitrary scale factors.
They didn't end up using this approach because the 2x mode is so simple, and will likely be the last scale factor really needed (especially with the additional scaling modes).
They didn't end up using this approach because the 2x mode is so simple, and will likely be the last scale factor really needed (especially with the additional scaling modes).
@DougBTX: Technically, there are 2 scaling levels. There's 1x for normal screens and 2x for retina. If you want one of the intermediate resolutions, it will actually render at 2x, then scale down.
>Windows has had support for resolution independence where DPI is configurable across a continuum (rather than only available as 1x or 2x scale like on the Mac) for quite some time - the issue is that with a huge legacy application base, a giant morass of drawing technologies, and no shipped "HiDPI" or "Retina" Windows hardware until recently, there's been no real incentive for app developers to support it.
The same was true for even longer (IIRC) on the Mac. You could enable it with a hidden preference, but lack of app support, compatibility with existing stuff and lack of hidpi displays at the time made it take the back seat. And vector drawed apps only go that far, you need pixel assets at some point or another.
The 2x scale was just the solution finally used by Apple because of the same kind of issues as you describe for Windows.
The same was true for even longer (IIRC) on the Mac. You could enable it with a hidden preference, but lack of app support, compatibility with existing stuff and lack of hidpi displays at the time made it take the back seat. And vector drawed apps only go that far, you need pixel assets at some point or another.
The 2x scale was just the solution finally used by Apple because of the same kind of issues as you describe for Windows.
It's not just the "legacy application base" or "app developers" which are at fault; even most of Windows itself is still using the old drawing libraries. As soon as you start digging into system properties, MMC, etc, everything is horribly blurry.
But yes, apps are also broken, even very popular ones like Chrome and Steam.
But yes, apps are also broken, even very popular ones like Chrome and Steam.
Fair point - even some of the shallowest corners of Windows itself are broken too (like Windows Media Player), although most of the other big user-facing shell apps like Explorer and IE are great.
Google almost had support in Chrome 34 after a few false starts, but it looks to have been disabled yet again in current releases.
Now that Windows users are trying to use Retina MacBooks via BootCamp and 2x-resolution Windows 8 tablets like the IdeaPad Yoga2 are out, the DPI scaling issue is finally starting to get mainstream exposure. Most reviews of the IdeaPad called out Chrome specifically and once 4K displays come down in price the gamer crowd will probably jump on the bandwagon as well.
Google almost had support in Chrome 34 after a few false starts, but it looks to have been disabled yet again in current releases.
Now that Windows users are trying to use Retina MacBooks via BootCamp and 2x-resolution Windows 8 tablets like the IdeaPad Yoga2 are out, the DPI scaling issue is finally starting to get mainstream exposure. Most reviews of the IdeaPad called out Chrome specifically and once 4K displays come down in price the gamer crowd will probably jump on the bandwagon as well.
There are parts just a couple clicks deep where you can see W2k-like controls.
It works perfect on the Mac but I wonder if that would be the case if you could still run old 'classic' apps.
You'd think Microsoft of all companies would work in compatibilty fixes for at least the major apps. Like how they worked in so many fixes to make them compatible from 98-XP and such.
You'd think Microsoft of all companies would work in compatibilty fixes for at least the major apps. Like how they worked in so many fixes to make them compatible from 98-XP and such.
The classic apps actually work perfectly at their original resolution, or higher in places if they used Apple APIs for things like text.
I think they mean carbon apps, which are no longer supported on OS X.
"Classic" apps will be scaled (using the normal bicubic filter) to twice their resolution on Mac. If they were implemented using Apple APIs, there's a good chance that text will be at a higher resolution and will appear clear.
I don’t think there is any interpolation happening, actually. The display just uses four pixels to display one pixel, that’s all.
Even when there were a few apps that didn't work well, in my experience at least, it was no one near as bad as W8's HiDPI features continue to be today. Apple did a really good job with that.
Unfortunately I think this is just the price of being an early adopter.
I would guess it's for performance reasons.
I have 4 i7 cores and a $500 graphics card. I feel like it's possible :)
TF2, for example, plays beautifully at 3840x2160@60Hz. Scaling up the graphics for an 800x600 visual novel that plays at 1 frame per 30 seconds is computationally possible.
(The good news is that someone has written an emulator for one VN that I'm working through, so I should be able to play with scaling in that way. If I can get it to compile.)
TF2, for example, plays beautifully at 3840x2160@60Hz. Scaling up the graphics for an 800x600 visual novel that plays at 1 frame per 30 seconds is computationally possible.
(The good news is that someone has written an emulator for one VN that I'm working through, so I should be able to play with scaling in that way. If I can get it to compile.)
hq4x looks better (IMHO), sadly the website is offline (default blank wordpress site): http://hiend3d.com/hq4x.html
backup on archive.org: https://web.archive.org/web/20120630044334/http://www.hiend3...
http://en.wikipedia.org/wiki/Hqx
Source: https://code.google.com/p/hqx/source/browse/trunk/src/hq4x.c
backup on archive.org: https://web.archive.org/web/20120630044334/http://www.hiend3...
http://en.wikipedia.org/wiki/Hqx
Source: https://code.google.com/p/hqx/source/browse/trunk/src/hq4x.c
Only if you look at the first ones. All of Boo, Dolphin, all of SoM, Invaders, Golden Axe, Windows assets and Lost Vikings look great and have vastly reduced artifacts compared to hqx. Also, according to the paper abstract[0], it outputs vector, not scaled up bitmap.
[0]: http://research.microsoft.com/en-us/um/people/kopf/pixelart/
[0]: http://research.microsoft.com/en-us/um/people/kopf/pixelart/
No, hq4x is still better on almost all of them. Microsoft's makes everything too round and blobby. It pretty much only works really well when everything is round, like in Boo. But look at what it does to the Golden Axe characters' faces.
hq4x keeps angles when it should not, degrading many features in the process, and that bothers me much more than the overall change to a bit more blobbiness. Look at the outline of Bowser's horns and mouth, Toad's cap outline, Fake Sage's moustache, or Axe Battler 2 sword. The mouths of invaders are also faithfully reproduced. In the Ports icon, the cable is correctly represented, and the port itself has volume instead of becoming two planes as in hq4x. On the Keyboard one, hq4x produces a good icon, although it rounds the keys a little, while "ours" produces a non-faithful in style but nonetheless correct result. Control Panel is interesting, as "ours" looks like the hq4x version has melted, but the hq4x version produces a result that is semantically similar (piano keys, weird mouse shape, angles in letter) leading me to think the original icon is not exactly the best pixel art material (maybe due to palette constraints).
Maybe the results of "ours" aren't exactly faithful in style in some cases, but they're globally consistent, and I find the shapes semantic more truthful. As for hq4x by itself, I actually prefer the original sprites overall.
Maybe the results of "ours" aren't exactly faithful in style in some cases, but they're globally consistent, and I find the shapes semantic more truthful. As for hq4x by itself, I actually prefer the original sprites overall.
I think we're trying to optimize different functions.
For Bowser, I think hq4x is fine (the mouth needs not have a smooth contour like a human's - compare with a dinosaur or a crocodile), but you are giving more weight to the fact that the shaded part on the bottom right of the mouth is broken in several pieces, instead of being a single piece like in ours.
For Fake Sage, I think hq4x does better. A pointy moustache is fine. The "ours" version is practically melted, and hardly recognizable as an old man - it looks more like some sort of fish shaman.
For Axe Battler 2, the shaded part of the sword is jagged in hq4x, but that doesn't bother me. Same thing for the broken black outline in the leg. I am much more bothered by how "ours" messes up his face.
In general, I would say that "ours" is better at putting together some parts that are supposed to be a single line, while hq4x separates them (especially with thin shadows). But graphically, hq4x's result is much more faithful to what the sprite is supposed to represent. With "ours", everything looks like melted ink, or like it's been seen through a wet pane of glass in the rain.
It would be nice to see if it's possible to combine "ours"'s better detection of continuous shades with hq4x's better preservation of straight lines and details.
For Bowser, I think hq4x is fine (the mouth needs not have a smooth contour like a human's - compare with a dinosaur or a crocodile), but you are giving more weight to the fact that the shaded part on the bottom right of the mouth is broken in several pieces, instead of being a single piece like in ours.
For Fake Sage, I think hq4x does better. A pointy moustache is fine. The "ours" version is practically melted, and hardly recognizable as an old man - it looks more like some sort of fish shaman.
For Axe Battler 2, the shaded part of the sword is jagged in hq4x, but that doesn't bother me. Same thing for the broken black outline in the leg. I am much more bothered by how "ours" messes up his face.
In general, I would say that "ours" is better at putting together some parts that are supposed to be a single line, while hq4x separates them (especially with thin shadows). But graphically, hq4x's result is much more faithful to what the sprite is supposed to represent. With "ours", everything looks like melted ink, or like it's been seen through a wet pane of glass in the rain.
It would be nice to see if it's possible to combine "ours"'s better detection of continuous shades with hq4x's better preservation of straight lines and details.
I noted the same issue with round vs sharp sections. Honestly, what this needs is a blending function, kind of like how they do intense shocks in fluid dynamics, where it would use MS's algo in areas where it detected significant roundness, and then blend into the hq4x algo in areas where there's more angular sections, like in the treasure box.
I think the lesson here is that it is subjective and the ability to use multiple such tools together would be best.
I agreed with you, but then I realized they're representing hqx deceptively.
There was a video posted here, with them rendered at the same resolution side by side.
https://news.ycombinator.com/item?id=7307975
It's really a matter of taste in this video.
There was a video posted here, with them rendered at the same resolution side by side.
https://news.ycombinator.com/item?id=7307975
It's really a matter of taste in this video.
> they're representing hqx deceptively.
A big difference in the techniques is that hq4x is a fix-rate upscaling, whereas Microsoft's can handle arbitrary target resolutions. At the same rez, they are pretty equivalent. But, MS has the option to go higher. HQ8x has not yet been developed. I doubt that HQ(arbitrary)x is possible without completely changing the algorithm.
A big difference in the techniques is that hq4x is a fix-rate upscaling, whereas Microsoft's can handle arbitrary target resolutions. At the same rez, they are pretty equivalent. But, MS has the option to go higher. HQ8x has not yet been developed. I doubt that HQ(arbitrary)x is possible without completely changing the algorithm.
I agree, that's huge advantage. That's why I said it was a deceptive representation, but not an invalid one.
Not just higher, but non integer/power of two.
I took great care to compare 4x, only considering 16x as a zoom, especially since "zoomed in", it blows out the details too much so you lose the global picture. Still, 16x means HiDPI will make it looks crisp as hell for the same physical size.
As for the video, the dent in Mario's cap outline and the similar angles in the background outlines ruins hq4x for me, while I could live with the "200"/"400" lettering (which looks better on hq4x) because it disappears quickly (but does not bode well for other cases where text appears)
As for the video, the dent in Mario's cap outline and the similar angles in the background outlines ruins hq4x for me, while I could live with the "200"/"400" lettering (which looks better on hq4x) because it disappears quickly (but does not bode well for other cases where text appears)
As you say: It depends. E.g. the sword looks better with hq4x, because the result isn't rounded.
The Super Mario and Secret of Mana stuff works quite good, except it changes the facial expression on "Peach 2". I'd like to see Donkey Kong Country 1/2/3, Megaman (NES and SNES) and Chrono Trigger processed by this. And not just the sprites cut out of the background but embedded in a scene.
"ours" Peach 1 suddenly looks really sad.
Look at Axe Battler 2 - if the sword isn't vertical, then the result is not superior. Unless, of course, you think the guy is wielding a sawtooth blade.
Not to mention that muscles have a smoothly-graded tone under the 'ours' algorithm, whereas hq4x has an unlifelike blocky effect.
Not to mention that muscles have a smoothly-graded tone under the 'ours' algorithm, whereas hq4x has an unlifelike blocky effect.
Honestly I think hq4x and "Ours" both look like ass for both of the Axe Battlers. Nearest neighbor is a better choice for either of them. EPX on either is acceptable.
even so. I suspect a live rendering of the vector process would be fairly attractive. I'd love to see this technique used on the textures for n64 games. The rendering times wouldn't be such an issue in that case.
hq4x definitely looks the best and most accurate. If it only further smoothed the jaggies, it would look phenomenal.
The issue with depixelizing pixel art is that you need an AI that can identify semantically what the features in the sprite are. When you look at Mario you know you're looking at a stout little man with a hat, you can identify his eyes, ears, nose, hair, hat, moustache, shirt, gloves, overalls and shoes. Given that knowledge, if I were to depixelize Mario's sprite, I'll know the rough shape of his shoes and where the shading should go and what shape it should have, i.e. I can reason what the 3D shape is and how the 2D shape was derived from that. I'll also be able to identify which lines were put there to make sure separate elements can be legible and not make those thicker than needed. Or that his eyes blend with his hat simply because there aren't enough pixels to make them separate.
For simple shapes, like the background, their algorithm works really well, but for complex objects it fails, because it distorts details that were put in with very careful thought and completely depend on the resolution. Such small sprites rely a lot on being looked at by someone who can identify semantically what they're looking at, and any really successful depixelization solution will need to be able to understand what basic shapes the sprite is made of based on what it's supposed to represent.
For simple shapes, like the background, their algorithm works really well, but for complex objects it fails, because it distorts details that were put in with very careful thought and completely depend on the resolution. Such small sprites rely a lot on being looked at by someone who can identify semantically what they're looking at, and any really successful depixelization solution will need to be able to understand what basic shapes the sprite is made of based on what it's supposed to represent.
I've been thinking about doing something like this. Compress images down to half resolution, then train a neural network to blow it back up and guess the pixels as accurately as possible.
There has been a neural network scaler called NNEDI3 available as a plugin for avisynth for a few years. It was trained in exactly the way you are talking about and does an amazing job at both deinterlacing and upscaling video.
It does a great job of scaling pixel art too. I'm kinda surprised it hasn't been implemented in any emulator. With the newer opencl implantation it is finally able to run in real-time.
http://screenshotcomparison.com/comparison.php?id=46792
It does a great job of scaling pixel art too. I'm kinda surprised it hasn't been implemented in any emulator. With the newer opencl implantation it is finally able to run in real-time.
http://screenshotcomparison.com/comparison.php?id=46792
Whoa, that's pretty good.
I did something very much like this years ago (upscaling used to be an obsession of mine). The results were pretty disappointing.
But there's a lot of space to explore there in terms of choosing and designing the neural net, choosing the right training set, and figuring out an initial transform for data that is input to the neural net.
But there's a lot of space to explore there in terms of choosing and designing the neural net, choosing the right training set, and figuring out an initial transform for data that is input to the neural net.
New deep learning techniques might help this approach a lot.
A C++ LGPL implementation of this has been as been developed as part of last year Inkscape GSoC and is available as a standalone library: https://launchpad.net/libdepixelize
It will be released in the next Inkscape major version, expected soon :)
It will be released in the next Inkscape major version, expected soon :)
Perfect! Thanks for the link. I want to run this over the classic SimCity tile set (and the additional ones in the add-on graphics sets like the moon base, etc), to generate high resolution pixel and vector tiles for the JavaScript browser based version of Micropolis (GPL SimCity).
The authors of this paper also have other amazing image processing/optimization algorithms. Eg: Deblurring, Optimizing Color Consistency, Content-aware Automatic Photo Enhancement, Object Discovery and Segmentation in Internet Images and more:
Dani Lischinski's page: http://www.cs.huji.ac.il/~danix/
Johannes Kopf page: http://johanneskopf.de/
Dani Lischinski's page: http://www.cs.huji.ac.il/~danix/
Johannes Kopf page: http://johanneskopf.de/
Wow, the content-aware downscaling seems very useful!
http://research.microsoft.com/en-us/um/people/kopf/downscali...
> This paper introduces a novel content-adaptive image downscaling method. The key idea is to optimize the shape and locations of the downsampling kernels to better align with local image features. [...] Besides natural images, our algorithm is also effective for creating pixel art images from vector graphics inputs, due to its ability to keep linear features sharp and connected.
IIRC, this is one of the major reasons that vector graphics based icons have not taken off.
http://research.microsoft.com/en-us/um/people/kopf/downscali...
> This paper introduces a novel content-adaptive image downscaling method. The key idea is to optimize the shape and locations of the downsampling kernels to better align with local image features. [...] Besides natural images, our algorithm is also effective for creating pixel art images from vector graphics inputs, due to its ability to keep linear features sharp and connected.
IIRC, this is one of the major reasons that vector graphics based icons have not taken off.
Sometimes the Vector Magic results are truly magical; striking poses become even more striking when the figure is reduced to a collection of simpler shapes.
I like a lot of VectorMagic results... Very abstract art!
Yeah, I really like the VM ones - particularly how the space invader turns into a StarTrek badge and the Bombermans(men?).
It reminds me of old Microsoft Word clipart.
I thought this looked familiar! It is from 2011 https://research.microsoft.com/en-us/um/people/kopf/pixelart...
But the blocky versions actually look better than vectorized ones. Most of the vectorized sprites look hideous with the exception of Boo and dolphin.
Count me as another one who likes the pixel aesthetic. The title made me ask myself "why would you want to do this?"
(I also loathe most forms of antialiasing, including ClearType, so that might have something to do with it... I like sharp, clearly contrasted edges, including pixel edges.)
(I also loathe most forms of antialiasing, including ClearType, so that might have something to do with it... I like sharp, clearly contrasted edges, including pixel edges.)
they have a certain style to them that is lost at higher resolutions.
No, I think the vectorized versions are much better but yeah, Boo is the best one.
From when this first came out:
https://news.ycombinator.com/item?id=2601347
https://news.ycombinator.com/item?id=2578706
https://news.ycombinator.com/item?id=2601347
https://news.ycombinator.com/item?id=2578706
Does anyone know where I can find an implementation of this? I can't even figure out what to call it. They just refer to it as "our method."
Here's a pure Python implementation: https://github.com/jerith/depixel
Thanks!
See? Even he doesn't know what to call it!
See? Even he doesn't know what to call it!
A work in progress implementation, anyway.
I wonder why they all fall down on Peach and Peach 2? None of them look nearly as good (to me) as bicubic.
I think it comes down to your point of focus. With peach and peach 2, The eyes change shape quite a bit. Humans notice that much more than other aspects.
In general, I think the "ours" method could benefit by moving the boundaries between colours in and out until they get an average brightness match with the similar area in the nearest neighbour version. A lot of the "ours" images have a higher proportion of black.
In general, I think the "ours" method could benefit by moving the boundaries between colours in and out until they get an average brightness match with the similar area in the nearest neighbour version. A lot of the "ours" images have a higher proportion of black.
Super2xSaI looks better to me at 4x than bicubic, and EPX does an OK job but the others look pretty terrible.
I think the problem is that those sprites use multiple strategies for handling diagonal outlines. Those look fine when the image is blurred together, but when you try to detect the outlines and upscale them, the differences between techniques are exaggerated.
I think the problem is that those sprites use multiple strategies for handling diagonal outlines. Those look fine when the image is blurred together, but when you try to detect the outlines and upscale them, the differences between techniques are exaggerated.
Boo is the best example of 'ours'. Looks amazing.
I think bower looks badass! http://research.microsoft.com/en-us/um/people/kopf/pixelart/...
yeah, it seems to handle patches of complex colouring badly (especially where two tones of the same colour were used to good effect in the icon, but scaled up as messy blobs), but boo and the invaders were very impressive.
I chose to embrace the Pixels, myself: http://vulariter.wordpress.com/tag/jewel-punk/
Looks like a normalized version of the EPX algorithm.
HQ4X and Ours are the clear winners if you must resample like this; the downsides are that HQ4X occasionally has jaggies and Ours makes makes a lot of fine details look "blobby".
The Vector Magic results reminded me a lot of this: http://www.cowboybooks.com.au/html/acidtrip1.html
The Vector Magic results reminded me a lot of this: http://www.cowboybooks.com.au/html/acidtrip1.html
Regarding HQ4X, they used the algorithm to create 4X image but used Nearest neighbor to create 8X and 16X images. I think doing HQ4X two times would have resulted into a better 16X images without jags.
I did a GPU version in 2012 for my course project. For small size sprite, it runs roughly the same or even slower than the original paper. Anyone interested in finishing this and maybe try this on realtime emulator? http://yzhwang.github.io/depixelization/
Kind of ironic that MSR is using an unregistered version of PhotoZoom.
I think the most interesting aspect of this is the internal shading. I haven't seen any other algorithm that manages that while maintaining sharp edges overall.
I imagine it would end up with pretty weird results though when a sprite was on top of a background that with similar enough colors to trigger the shading detection at its edges.
I imagine it would end up with pretty weird results though when a sprite was on top of a background that with similar enough colors to trigger the shading detection at its edges.
I must be a luddite, I prefer my pixilation like my soup: chunky :)
to me, hq4x looks better every time.
I think it depends on what your end goal is. Are you trying to scale and hide the pixel-ness of the original? The "ours" does pretty darn well, especially if you look at the Boo example as mentioned above.
However, if you're looking at in the case of an SNES emulator or something.. then hq4x seems like the best choice. It keeps true to the original pixel art, while appearing significantly better.
However, if you're looking at in the case of an SNES emulator or something.. then hq4x seems like the best choice. It keeps true to the original pixel art, while appearing significantly better.
Linked earlier:
Video of Ours vs HQ4X. Ours seems to have a slight advantage to my eyes, admittedly it is a personal preference. Text has a distinct advantage on HQ4X but Mario and Yoshi look better even if less "accurate". https://research.microsoft.com/en-us/um/people/kopf/pixelart...
Video of Ours vs HQ4X. Ours seems to have a slight advantage to my eyes, admittedly it is a personal preference. Text has a distinct advantage on HQ4X but Mario and Yoshi look better even if less "accurate". https://research.microsoft.com/en-us/um/people/kopf/pixelart...
I was thinking that too, but then came to terms with 'ours' version
I wrote an implementation of this in common lisp when they published it:
http://dev.crazyrobot.net/2011/07/12/pixelart-to-vector/
http://dev.crazyrobot.net/2011/07/12/pixelart-to-vector/
Technology almost caught up with CSI.
There is a technique that might actually be able to do this called compressed sensing[1] that I first heard about it in a Wired article from several years ago[2]. The technique allows reconstructing images, music, etc. by sampling a portion of the original and applying an algorithm that reconstructs the missing information and is apparently used in MRI image processing to get by with shorter scans. Researchers from Rice University constructed a camera with only a single pixel image sensor that, using this technique, produced legible images with as little as a 2% sampling of the final image resolution, although more complex images required 10-20% or more[3]. I don't see why this technique couldn't be used to take already good quality images and "enhance" areas of them like in CSI.
[1] https://en.wikipedia.org/wiki/Compressed_sensing [2] http://www.wired.com/magazine/2010/02/ff_algorithm/ [3] http://dsp.rice.edu/cscamera
[1] https://en.wikipedia.org/wiki/Compressed_sensing [2] http://www.wired.com/magazine/2010/02/ff_algorithm/ [3] http://dsp.rice.edu/cscamera
Amazing image zoom algorithm, t creates a hires version of a low res image. I would love to see this image zoom feature implemented the in ImageMagick.
The Space Invaders "ours" were very impressive. The text was always really good too.
The algorithm works well on more regular shapes.
It would be interesting to see a pixelart/emulator scaler that was temporally aware - that is, able to use context from near or all relevant sprite frames to get additional detail/context. This would require a compositing emulator, as some others have discussed, but I could see it producing notably superior results, given it would have much more source data to work with.
Is there no 8x or 16x version of hq4x? I thought you can use it to get full HD resolution from 320x240 (in realtime using GLSL shader).
Is there a paper or code associated with this?
Which also has the original HN discussions
https://news.ycombinator.com/item?id=2673271
and links to +1000 day old threads:
https://news.ycombinator.com/item?id=2601347
https://news.ycombinator.com/item?id=2578706
yet those space invaders still haunt me.
https://news.ycombinator.com/item?id=2673271
and links to +1000 day old threads:
https://news.ycombinator.com/item?id=2601347
https://news.ycombinator.com/item?id=2578706
yet those space invaders still haunt me.
Thank you.
[deleted]
Something related that I stumbled upon some time ago (submitted to HN, got one upvote):
http://gpuhacks.wordpress.com/2013/07/08/signed-distance-fie...
I needed some depixelation software for some graphics awhile back, so I threw together a quick program that depixelates using various technologies.
https://github.com/kstenerud/depixelate
So far it supports HQX, XBRZ, and Scale2x.
https://github.com/kstenerud/depixelate
So far it supports HQX, XBRZ, and Scale2x.
Proper Video here :- https://research.microsoft.com/en-us/um/people/kopf/pixelart...
Am I the only one who prefers the Vector Magic versions? They're like mini Picassos.
Here's the home page: http://research.microsoft.com/en-us/um/people/kopf/pixelart/
It actually looks like as if you took the polylines from EPX and used the points as the CVs for a biquadratic or bicubic interpolating spline (e.g. Catmull-Rom), you'd be very close to their result already.
I think hq4x is the best. Microsoft's curves too much - and while it can look good/stylized for some pieces, it's ultimately inaccurate and warps many of the pieces from their original form.
Ctr+f "Invader 1" and choose the Vector Magic version. Star Trek!
Funny how I don't remember those vikings to be so low-detail and tiny.
Todo: Blend hq4x and the kopf techniques in a post processing step.
The Secret of Mana ones made my eyes leak, not gonna lie. Awesome graphics technology update not withstanding, that game was an integral part of my teenage years.
I have a hard time believing the use of Nintendo IP in this case (big corporation-backed research lab) falls under Fair Use.
Looks great :) Makes me a bit less tense about the appearance of older programs in the 4k+ display future.
This would be awesome if it was available as a scaler for high-DPI screens and old apps. Or even bundled into televisions for upscaling.
The paper does note that it is designed for hand-edited pixel art, where every different coloured pixel is treated as significant. So I wonder how it behaves when scaling photos?
(Edit: the paper does note that it doesn't do well on anti-aliased graphics...)
The paper does note that it is designed for hand-edited pixel art, where every different coloured pixel is treated as significant. So I wonder how it behaves when scaling photos?
(Edit: the paper does note that it doesn't do well on anti-aliased graphics...)
Televisions have upscale algorithms. But nothing like this. But they don't need anything like this because they don't scale pixle art with sharp borders and very low resolution. They scale video footage (which one could say is "anti-aliased") with about 640x480 resolution.
actually running the hqx algo on the art and then throwing it through ANY vectorisation engine with a slight roundness comes on top of any of these.
And Microsoft is not really being fair to Vector Magic here, there's a ton of settings there to improve that rendering depending on what you want...
And Microsoft is not really being fair to Vector Magic here, there's a ton of settings there to improve that rendering depending on what you want...
Doesn't work well with geometric objects. Look at the swords, shields, the letter 'A'.
They forgot natural neighbour interpolation... Should show better results then their research ;)
Well I guess it's retro week again on HN... Better check my bookmarks from 3 years ago...
This seems pretty awesome. Can someone please tell me where this could be useful?
The 'ours' filter (I guess it's the one from MS) is mind-blowing.
At first I thought this was a live JS-based demo. That would have been amazing.
Are there any ruby libraries that would depixel an image in a similar fashion?
[deleted]
I just learned that my favorite depixelizer is hq4x
This is pretty amazing of a demo.
Just awesome.
amazing.
I can't believe noone linked this so far. https://www.youtube.com/watch?v=3uoM5kfZIQ0&hd=1
I can't believe you are linking to CSI instead of Blade Runner, the grand daddy of all enhance scenes http://www.youtube.com/watch?v=qHepKd38pr0 makes me feel old...
A problem with this algorithm is that it's pretty slow. It would be hard to use it for real time scaling.
I think that it's pretty cool though for any precalculated conversion to vector format. It would also be pretty cool if there were software that allowed you to create a pixel art image with a live preview in this format and the ability to export into a vector format.
One of the benefits of pixel art IMO is that it's got a pretty distinct style and its simplicity and restrictions means it's easy to be aware of composition where if you are given an unrestricted canvas that freedom can make it hard to know where to start. If you are given a 8x8 pixel grid to indicate "happy" there's only so many ways to do it, a symbolic heart, a little smiley face, etc. Given a whole sheet of paper you'll have more difficulty.
With the high res displays, there's more to consider than just the upscaling of old programs, the fact is people (especially indie game devs) still want to work with pixel art, and the number of titles with the term "8-bit" in their name that have come out recently. Part of that's because it's in vogue now, but a big part of it is simply that a poor artist can make a passable 24x24 sprite.
But it's still a shame that on high res displays they end up looking like a blocky mess. If you had a program that let you author your sprites under the same conditions, but let you preview an upscaled version and left you with a vector image (or high res texture with mip maps) you can make use of the fidelity while maintaining the simplicity. I think that would be cool.