Building a New Image Format(richardwalshlabs.blogspot.co.uk)
richardwalshlabs.blogspot.co.uk
Building a New Image Format
http://richardwalshlabs.blogspot.co.uk/2012/11/building-new-image-format.html
13 comments
I don't see what a new format would bring to this problem, in the end you only want to send one image to each visitor, so you probably want a server side program that can transform one big image into various smaller size images. You don't actually want to be shifting all the various sizes around as a bundle.
(And WebP's real killer feature is lossy-photo + alpha at the same time, that's where there's a real gap in the market at the moment, in between PNG and JPEG.)
(And WebP's real killer feature is lossy-photo + alpha at the same time, that's where there's a real gap in the market at the moment, in between PNG and JPEG.)
I'm enthusiastic about animation support. It's mind-boggling that we are still using an image format from 80s with crappy compression and limitation to 256 colors.
What about the <video> tag? I guess it's nice that WebP support animation as well, but that's what the video tag exists for. Shoe-horning in frame-by-frame animation in an image format when you can just use video (or the canvas, if you want to do something key frame based) seems like a bad idea to me.
The only thing I can think of off the top of my head where it's preferable is if you needed to do a video with an alpha channel. But this is more codec related than a problem with the concept itself.
The only thing I can think of off the top of my head where it's preferable is if you needed to do a video with an alpha channel. But this is more codec related than a problem with the concept itself.
Have you seen how people use gif animations? Video is completely different format than animated images in today's internet. If video formats were preferable, things would have already changed.
Animated images have several advantages:
-Almost all places that support images support also animated image
-Can be downloaded and reposted (as opposed to canvas)
-No worry about audio, there is no audio
-Seamless looping
-Auto play everywhere
-Variable frame rate
These features could be achieved in a video format, but how would you convince websites that support images to support video with these characteristics?
For example, I would never let people upload autoplaying, looping videos as their avatars or as part of their comment unless I could be 100% sure that there will be no audio. I would have to inspect the video for audio and I would need different handling for images and videos.
Animated images have several advantages:
-Almost all places that support images support also animated image
-Can be downloaded and reposted (as opposed to canvas)
-No worry about audio, there is no audio
-Seamless looping
-Auto play everywhere
-Variable frame rate
These features could be achieved in a video format, but how would you convince websites that support images to support video with these characteristics?
For example, I would never let people upload autoplaying, looping videos as their avatars or as part of their comment unless I could be 100% sure that there will be no audio. I would have to inspect the video for audio and I would need different handling for images and videos.
Until Mozilla finally ships H.264 using <video> means you have to double-encode every single file, once using comparatively primitive tools, or use Flash as a fallback for Firefox and IE8. Until then, GIF files are much easier to work with.
What's the benefit of a photo-compression with animation support, compared to actual movie formats (H.264)?
Can be used in places that only support images. Can be used in places that don't want audio. Variable frame rate. Auto play enabled by default. Designed to loop continuously.
Have you seen how people use gif animations?
Have you seen how people use gif animations?
WebP is not a hybrid of GIF or JPEG or PNG. It's a subset of the VP8 video codec, which, if anything, is more similar to H.264.
What's needed is a downloadable JavaScript renderer which parses the webp file format and draws the resulting image in an HTML5 Canvas.
Then we don't have to wait for browser makers to implement it in their browsers. You just add a one-line script tag to load webp.js, a script that scans your DOM for img tags with a src that ends with ".webp", and your website has webp support.
If browsers later support webp natively, you can just add a check to webp.js so it turns itself off when a webp-supporting browser is detected.
It's really too bad that Java in the browser never took off; its performance characteristics are far better for writing image decompressors than JS. But JS with a modern HTML5 Canvas running in a modern JS engine on a modern CPU is good enough for the purpose.
Then we don't have to wait for browser makers to implement it in their browsers. You just add a one-line script tag to load webp.js, a script that scans your DOM for img tags with a src that ends with ".webp", and your website has webp support.
If browsers later support webp natively, you can just add a check to webp.js so it turns itself off when a webp-supporting browser is detected.
It's really too bad that Java in the browser never took off; its performance characteristics are far better for writing image decompressors than JS. But JS with a modern HTML5 Canvas running in a modern JS engine on a modern CPU is good enough for the purpose.
It's sad that Mozilla has not included WEBP. I cannot understand why since it seems to be a wonderful format.
Maybe they are waiting for all the promised features to deliver. So far I haven't seen working animations or lossy + alpha in Chrome so they probably aren't ready yet.
Maybe they are waiting for all the promised features to deliver. So far I haven't seen working animations or lossy + alpha in Chrome so they probably aren't ready yet.
I looked for a bug, and found https://bugzilla.mozilla.org/show_bug.cgi?id=600919.
The reason given (that I could find, it's a pretty verbose bug entry) is that WEBP doesn't support alpha. Sounds like a pretty weird reason to me, but I'm not a Mozilla developer.
The reason given (that I could find, it's a pretty verbose bug entry) is that WEBP doesn't support alpha. Sounds like a pretty weird reason to me, but I'm not a Mozilla developer.
WebP does support Alpha, now.
I think the true reason is that evaluating the quality of, and introducing widespread support for, a lossy image format are both really hard problems and Mozilla didn't want to go off half-cocked. Google on the other hand control both ends of the wire and so can see the benefits immediately and as a result are more keen to experiment.
I'm intrigued to see if they do the same with with WebM v2 including VP9 and Opus and just roll it out (in beta even, which makes much more sense for 'disposable' video conferencing use-cases than it does for long-term video storage or Youtube) rather than continue the usual 10-year upgrade treadmill for video codecs.
I think the true reason is that evaluating the quality of, and introducing widespread support for, a lossy image format are both really hard problems and Mozilla didn't want to go off half-cocked. Google on the other hand control both ends of the wire and so can see the benefits immediately and as a result are more keen to experiment.
I'm intrigued to see if they do the same with with WebM v2 including VP9 and Opus and just roll it out (in beta even, which makes much more sense for 'disposable' video conferencing use-cases than it does for long-term video storage or Youtube) rather than continue the usual 10-year upgrade treadmill for video codecs.
WebP only seems to support 8-bit integer component values, which makes seems pretty backward for a new format.
Higher-bit depths and floating-point components (ala OpenEXR) seem an important feature for any image format that hopes to become a future standard. 8 bits were maybe sufficient in 1990; not so much today. Even microsoft eventually got that clue with JPEG XR...
You only get so many chances to define a new standard, so going with something as lackluster as WebP would seem to be a mistake. JPEG XR, despite its MS origins, seems to be vastly superior. [JPEG XR has licensing issues with the reference implementation (it's from MS after all), but that's not a fatal flaw...]
Higher-bit depths and floating-point components (ala OpenEXR) seem an important feature for any image format that hopes to become a future standard. 8 bits were maybe sufficient in 1990; not so much today. Even microsoft eventually got that clue with JPEG XR...
You only get so many chances to define a new standard, so going with something as lackluster as WebP would seem to be a mistake. JPEG XR, despite its MS origins, seems to be vastly superior. [JPEG XR has licensing issues with the reference implementation (it's from MS after all), but that's not a fatal flaw...]
It look like Mozilla is considering JPEG XR support:
https://wiki.mozilla.org/Platform/GFX/2012-November-26
https://wiki.mozilla.org/Platform/GFX/2012-November-26
This makes very little sense. The article suggests that you bundle many sizes of the same image into the same file; say, a 1024x1024 version as well as a 512x512 version. This means that a browser that only wants the 512x image will also download the 1024x image whether it wants to or not. So why not just pick the 1024x image and resize it on the client size, if everyone is going to download it? This is something that today's chips do very fast. At 2x resolutions you can even skip interpolation and just go for nearest-neighbour scaling.
Maybe it could be in a stream [headers][512px data][1024px data] and you could stop after the [512px data]. And since 512px image is a subset of 1024px one, you could do [headers][512px data][1024px\512px data] and so on.
(Just a guess.)
(Just a guess.)
You've basically just described interlacing. For instance, interlaced PNGs start with a 1/8 scale pass/frame and fill in the remaining pixels with the following 6. (Frames 1-5 would be required for a 1/2 scale image.)
Unfortunately, because of buffering I believe this would still be inefficient. Also, it would screw up HTTP keepalive. A better way would be for the browser to fetch the header first by using the HTTP "Range" header, and then fetch the subrange it needs, but that adds one roundtrip just to get an image.
>At 2x resolutions you can even skip interpolation and just go for nearest-neighbour scaling.
I'm not sure what your reasoning is there, but it's incorrect. Nearest-neighbor looks really bad even when scaling down to half size.
I'm not sure what your reasoning is there, but it's incorrect. Nearest-neighbor looks really bad even when scaling down to half size.
Depends very much on the size. The resulting edginess becomes much more pronounced the fewer pixels you have. I'm just pointing out that in some cases you can conserve CPU usage by cheating. You could render a shoddy version first just to get the page up, for example, and then replace it with a high-quality version; some browsers already do this today.
"We need an image file format that is, in essence, a storage locker."
I disagree.
HTTP supports cache-able compression, and connection re-use. It wouldn't be any slower to download frames as needed (lazy = good)), using the existing transport protocol. All you need is an ASCII/UTF-8 index file for an animation. Getting browsers to support anything new that doesn't come from the W3C and contain the word "semantic" in the spec on the other hand... That's the hard part.
I disagree.
HTTP supports cache-able compression, and connection re-use. It wouldn't be any slower to download frames as needed (lazy = good)), using the existing transport protocol. All you need is an ASCII/UTF-8 index file for an animation. Getting browsers to support anything new that doesn't come from the W3C and contain the word "semantic" in the spec on the other hand... That's the hard part.
This is really one of the worst articles I've seen submitted here lately. I'm truly surprised that it is currently ranked up so high.
The presentation of the article is atrocious. The font alone renders extremely poorly in Firefox. It is just plain hard to read. Please just use a common font that will render well basically anywhere.
The inline "$3.99 ThanksGiving Offer" ad links peppered throughout the article are distracting, too. They really take focus away from the article itself (unless, of course, that is exactly what they were intended to do).
Then it has one part that goes, "It’s the WebP Image Format (see https://developers.google.com/speed/webp/) as shown here", with an image of the WebP logo that follows. The WebP logo image is a PNG, however! With a lead-in like that, I was sure it was going to be a WebP image demonstrating some of the format's benefits.
The various inline URLs that aren't hyperlinks are very annoying, too. They should obviously have been actual links.
Seeing stuff like "Microsfot" only makes the horrible article look even worse.
While I'm not expecting perfection, nearly everything about this article is sub-par. It's not what I wish to see when I come to HN.
When it comes to information, the Wikipedia article is so much better, yet it's still quite concise, too. For anyone interested, it is at:
http://en.wikipedia.org/wiki/WebP
The presentation of the article is atrocious. The font alone renders extremely poorly in Firefox. It is just plain hard to read. Please just use a common font that will render well basically anywhere.
The inline "$3.99 ThanksGiving Offer" ad links peppered throughout the article are distracting, too. They really take focus away from the article itself (unless, of course, that is exactly what they were intended to do).
Then it has one part that goes, "It’s the WebP Image Format (see https://developers.google.com/speed/webp/) as shown here", with an image of the WebP logo that follows. The WebP logo image is a PNG, however! With a lead-in like that, I was sure it was going to be a WebP image demonstrating some of the format's benefits.
The various inline URLs that aren't hyperlinks are very annoying, too. They should obviously have been actual links.
Seeing stuff like "Microsfot" only makes the horrible article look even worse.
While I'm not expecting perfection, nearly everything about this article is sub-par. It's not what I wish to see when I come to HN.
When it comes to information, the Wikipedia article is so much better, yet it's still quite concise, too. For anyone interested, it is at:
http://en.wikipedia.org/wiki/WebP
The presentation doesn't bother me much, but the argument doesn't make much sense.
> Asking web designers, bloggers, and non-techies to create multiple versions of their images in order to appease every Android, iOS device and desktop browser seems not only like a very non-standards approach, it’s also not a very practical one.
I've never heard of FlashPix, but it seems like just a wrapper around a bunch of variously sized images. The creator still needs to make them and save them or use a program that does it automatically.
And what happens when you request one? You either get them all and download a gigantic file, or the server has to somehow pull out just the part you need.
This problem could be much more easily solved by the extensions to the img tag that allow various src attributes (ignoring the fact that we might not end up with the right one), media queries and using vector formats where possible.
> Asking web designers, bloggers, and non-techies to create multiple versions of their images in order to appease every Android, iOS device and desktop browser seems not only like a very non-standards approach, it’s also not a very practical one.
I've never heard of FlashPix, but it seems like just a wrapper around a bunch of variously sized images. The creator still needs to make them and save them or use a program that does it automatically.
And what happens when you request one? You either get them all and download a gigantic file, or the server has to somehow pull out just the part you need.
This problem could be much more easily solved by the extensions to the img tag that allow various src attributes (ignoring the fact that we might not end up with the right one), media queries and using vector formats where possible.
"Then it has one part that goes, 'It’s the WebP Image Format (see https://developers.google.com/speed/webp/) as shown here', with an image of the WebP logo that follows. The WebP logo image is a PNG, however!" <-- LOL!
Is it because of 1:53AM?
Giving that according to a new graphics standard is surprisingly difficult, wouldn't it be a good idea to try to do a "container" format?
It seems that adding new video codecs is (relatively) simple, but adding a new encode for images is awfully difficult due browser issues.
I mean, I am not sure, it will open a can of worms with a lot of problems, but not sure if anyone is trying something in that direction...
I mean, I am not sure, it will open a can of worms with a lot of problems, but not sure if anyone is trying something in that direction...
I'm not sure about the responsive image format thing. The problem is : the restrictions of mobile today (both in term of dpi and in term of connection speed) are likely to change very quickly. This means we'll need to re-encode images as mobile devices change.
A server-side solution where you store the full resolution image only and then use a cache for lower resolution seems much more flexible.
A server-side solution where you store the full resolution image only and then use a cache for lower resolution seems much more flexible.
CSS+JS with a bit of PHP on the server can generate/cache appropriate image sizes for each device from larger source files, using current image formats.
This sounds almost like the Opus of images - a format that tries to combine all the others into one.
First off, the concept of using the intra-frame compressor of a video format to compress images is unsound. Video compression has constraints that don't apply in still-image compression, and you are dragging that baggage around for no reason. These constraints lead to my other two issues.
One of the biggest problems with JPEG is that block artifacts are often visible even at relatively high quality settings. Post-decoding deblocking algorithms have proven utterly inadequate for fixing this.
One simple and effective leap in mitigating block artifacts is to use larger block sizes. Modern computers are obscenely fast compared to when JPEG was introduced, and we could bump the block size up from 8x8 to 16x16 or even 32x32 without breaking a sweat. This reduces the number of pixels that are at block boundaries, and that means less block artifacts. WebP uses smaller blocks because it's a video format, not an image format.
Another way that you can avoid edge artifacts is to overlap the blocks so that the edge artifacts don't create discontinuities. This is called "time-domain alias cancellation" and it's used in the 1D case by every lossy audio codec you've ever used. It generalizes easily to the 2D case, and it would generalize easily to the 3D case, but for reasons that I don't understand because video compression is not really my field, nobody does video compression by taking the DCT of 3D blocks of video. Instead they use a more ad hoc method of motion compensation, which consists of brute force searching for where blocks in one frame have moved in the next frame. It's not clear how you'd reconcile that with overlapping blocks in the interframe compressor, so VP8 doesn't use overlapping blocks. And by extension, neither does WebM.
Beating JPEG with a new video format is not all that difficult. We have two decades of research between JPEG's introduction and today, and each of us routinely carries hardware in our pocket that would have been considered a major military asset in 1992. It isn't at all surprising that the key frame compressor of a modern video codec would beat JPEG by 25%, but we can do so much better than that. Once we manage to displace JPEG, we're going to be stuck with whatever displaces it for a long time. So we need to make sure we have the best format we can muster before we make that commitment.