Introducing Fresco: a new image library for Android(code.facebook.com)
code.facebook.com
Introducing Fresco: a new image library for Android
https://code.facebook.com/posts/366199913563917/introducing-fresco-a-new-image-library-for-android/
24 comments
I'm an engineer on the project. Would be happy to take questions.
Hey. Quote,
"Our breakthrough came when we realized we didn't have to do that. If we called lockPixels without a matching unlockPixels, we created an image that lived safely off the Java heap and yet never slowed down the UI thread. A few lines of C++ code, and we were home free."
That doesn't sound complete. It's off the Java heap, true but doesn't mean it's either safe or won't slow down the UI or is even home free. Allocation in ashmem will still count towards the PSS[1] of the app. The image data won't be unpinned right until it throws an OutofMemory exception. And if you have handled the exception, the UI thread will wait around for the unpin/unlock of data. This would be ok if you could make an educated guess of how much ashmem can be allocated to your app. You can't make that guess easily as it depends a lot on OEM implementation of Gralloc/HWComposer and the GPU. But since you are Facebook, :), you can probably test this on all device/gpu variants. And when you do please publish the results. And while you are doing that, please test this too: http://developer.android.com/reference/android/os/MemoryFile...
[1]https://developer.android.com/tools/debugging/debugging-memo...
Thoughts?
Edit: Incorrect link
"Our breakthrough came when we realized we didn't have to do that. If we called lockPixels without a matching unlockPixels, we created an image that lived safely off the Java heap and yet never slowed down the UI thread. A few lines of C++ code, and we were home free."
That doesn't sound complete. It's off the Java heap, true but doesn't mean it's either safe or won't slow down the UI or is even home free. Allocation in ashmem will still count towards the PSS[1] of the app. The image data won't be unpinned right until it throws an OutofMemory exception. And if you have handled the exception, the UI thread will wait around for the unpin/unlock of data. This would be ok if you could make an educated guess of how much ashmem can be allocated to your app. You can't make that guess easily as it depends a lot on OEM implementation of Gralloc/HWComposer and the GPU. But since you are Facebook, :), you can probably test this on all device/gpu variants. And when you do please publish the results. And while you are doing that, please test this too: http://developer.android.com/reference/android/os/MemoryFile...
[1]https://developer.android.com/tools/debugging/debugging-memo...
Thoughts?
Edit: Incorrect link
The key, explained in the blog post, is that every Bitmap that lives in ashmem needs to have an explicit .recycle call to unpin and free the memory. Fresco's DraweeViews automatically do this when they go off-screen.
Fresco limits the total amount of memory an app can allocate to bitmaps, even in ashmem. But although ashmem does count towards total PSS, it doesn't count towards the Dalvik heap limit, which is the bound that most Java apps have to deal with.
Fresco limits the total amount of memory an app can allocate to bitmaps, even in ashmem. But although ashmem does count towards total PSS, it doesn't count towards the Dalvik heap limit, which is the bound that most Java apps have to deal with.
As someone who has written a lot of code (including driver level) in the DirectX realm, that line (about not unlocking surfaces) gave me a minor aneurysm though I admittedly do not know enough about the Android ashmem internals to know if the reaction is actually warranted or not on the Android platform.
I have written my fair share of code in the Android realm and that line gave my an aneurysm as well.
To be fair, I think that Bitmap handling is one of Android's weaknesses and that it is far too easy to fragment or overflow your heap with Bitmaps.
Well allocating on ashmen does reduce the UI lag since its just a memory pool and espaces GC reprimand but it's definitely not safe. Such operations are better handled at Gralloc/HWComposer Layer where you have a better handle on what's going on. Someone's gotta unlock that surface!
Once you go ndk aren't you opening up the library to all sorts of compatibility issues with specific devices? The Java api always seems safer to me.
Fresco is compiled natively to support ARM, ARMV7, and x86 CPUs. Almost all Android devices are using one of those three.
Could you ask your marketing staff to stop spamming android developers on LinkedIn?
If the app crashes is the mem freed?
... and the good ol' Facebook PATENTS file.
The mandatory link to Facebook's stance on patents that make their "open source" a no-go for many:
https://github.com/facebook/fresco/blob/master/PATENTS
https://github.com/facebook/fresco/blob/master/PATENTS
so what's the layman description of this? Don't sue us and we won't sue you? That seems like a pretty good way of applying MAD to the patent space
It goes a step beyond and says not to even think about challenging one of their patents. So if facebook patents something stupid, and you call bullshit, you can't use their software.
Even worse: if you just say "Software Patents are invalid", you lose the right to all their products. Or if you claim to have prior art to any of their products, you lose access to all other of their products as well.
I wrote a painting app with layer support a while ago for Android that had to use similar tricks (the app is coincidentally called Fresco: https://play.google.com/store/apps/details?id=org.seanw.fres...).
Each image layer (like you would have in GIMP or Photoshop) takes a couple of MB to store in Java and given the roughly 16MB Java app memory limit you didn't have much memory left to play with after creating 5 layers. I found that apps that used C and the NDK had no such limit; you could allocate huge amounts of memory compared to Java without any issues.
To take advantage of this, I wrote code that would let me allocate C memory, copy a Java bitmap into it then free the Java memory. To make all the app features work without crashes due to Java hitting the memory limit, I would shuffle bitmap memory between Java and C as needed, while having to implement several algorithms in C that I would rather have done in Java. For example, the undo/redo feature needed to store bitmaps of what changed when edits were made so this was mostly kept in C. One tricky issue was I couldn't find a way to get the Java APIs to render image data that was stored in memory used by C; you had to copy the data back into Java first.
I found the above frustrating in that if you're writing an image manipulation app, it means there is a massive incentive to avoid the native Java UI libraries and write your app entirely in C as you don't have the same memory limitations for some reason.
Each image layer (like you would have in GIMP or Photoshop) takes a couple of MB to store in Java and given the roughly 16MB Java app memory limit you didn't have much memory left to play with after creating 5 layers. I found that apps that used C and the NDK had no such limit; you could allocate huge amounts of memory compared to Java without any issues.
To take advantage of this, I wrote code that would let me allocate C memory, copy a Java bitmap into it then free the Java memory. To make all the app features work without crashes due to Java hitting the memory limit, I would shuffle bitmap memory between Java and C as needed, while having to implement several algorithms in C that I would rather have done in Java. For example, the undo/redo feature needed to store bitmaps of what changed when edits were made so this was mostly kept in C. One tricky issue was I couldn't find a way to get the Java APIs to render image data that was stored in memory used by C; you had to copy the data back into Java first.
I found the above frustrating in that if you're writing an image manipulation app, it means there is a massive incentive to avoid the native Java UI libraries and write your app entirely in C as you don't have the same memory limitations for some reason.
AFAIK all the serious Painting apps on both Android and iOS are implemented in OpenGL.
The native UI frameworks of these platforms are not designed for this very specific usecase.
"you could allocate huge amounts of memory compared to Java without any issues". No sir. It
still comes under the process memory account. It works in practice because coincidentally other processes were not using that memory.
Well, I meant that if you hit the Java memory limit your app was guaranteed to crash but C allowed you to use an order of magnitude more memory without noticeable consequences when your app was in the foreground. Your app was probably more likely to be closed when it went in the background but that wasn't an issue.
[deleted]
...I couldn't find a way to get the Java APIs to render image data that was stored in memory used by C; you had to copy the data back into Java first.
I think you could use a GPU surface for that. On iOS there's IOSurface, which is basically a GPU-accessible memory buffer from the window manager that can be shared between processes. AFAIK there's something similar on Android, but I'm not sure off-hand what it's called.
The idea is to get a surface and blit your image there on the C side; then access that surface as a texture on the Java side and draw it directly into a view. That way the pixels are never touched by the Java code.
I think you could use a GPU surface for that. On iOS there's IOSurface, which is basically a GPU-accessible memory buffer from the window manager that can be shared between processes. AFAIK there's something similar on Android, but I'm not sure off-hand what it's called.
The idea is to get a surface and blit your image there on the C side; then access that surface as a texture on the Java side and draw it directly into a view. That way the pixels are never touched by the Java code.
This was around the Android 2.1 days; I considered switching to OpenGL but I think I was deterred by inconsistent device support at the time and thought the Java option was less of a headache.
I'd be very interested in more details about the solution you mentioned!
I'd be very interested in more details about the solution you mentioned!
I wonder if this work has any link to what Carmack and the Oculus Rift crew are doing with Android?