Very interesting presentation, really on a high level, I enjoyed reading it.
I noticed that many problems that you mention other uploading gems have are actually solved by Shrine:
- "Transformation juggling" -- Versions are processed at instance-level, so you don't need to remember a class-level DSL.
- "File path config" -- If you change the location where your files are stored, the existing files on old locations will still work normally
- "Form validation dance" -- Very simple with Shrine, you just add an "<attachment>" hidden field.
- "Schema pollution" -- Shrine stores uploaded files exactly in the way that you described, in a single column as JSON
The idea of an image server indeed looks very interesting, but I think you're forgetting that this solves only one part of file uploads (although I agree it's a big part, and imgix looks really sexy). If you're using an image server, you still need to do:
- Caching the file (this still needs a direct upload endpoint)
- File validations (most notably preventing big files)
- Storing the file to image server (this still needs a background job)
- Model attachment logic (all of the assignment/callback logic)
One problem that I always had with on-the-fly processing is, if you have a page with a lot of photos, and you want to change the URL so that they're processed differently, how will that page look when a first person visits it? I think it will be horrible, the user will have to wait for a really long time to actually see the images, which is not nice user experience.
Also, the "image server" strategy can be used only for images, so if you want to upload documents, audio or video, you still need do regular processing on the application side. What's nice about uploading gems is that they provide a general solution.
Yes, you got it right, this is exactly what it means. Yeah, there are some rare use cases for on-the-fly processing (think a WYSIWYG editor where the user can choose the size of the image), but for me upfront definitely works best.
The part that I mind about both delayed_paperclip and carrierwave_backgrounder is that it doesn't really allow you to write your own job classes. This can be quite limiting for example if you want your jobs to send information to some monitoring services.
Another part that I mind is the user experience. Ok, both of these gems allow you to know when the backgrond job is in process, and then you can display some placeholder image until the processing is done. But that's not really nice user experience. In Shrine the user immediately sees the image they uploaded, because it has that image already cached (usually on the filesystem), and then from the user's point of view the uploading is finished, before the background job even started. CarrierWave also has the cached image, but by design it's not possible to use it, while Paperclip doesn't cache images.
I had exactly this scenario in mind when I was building Shrine. It has a recache plugin which allows you to do additional processing after validation but before storing, and then you can do the rest of the processing in a background job as part of storing. https://gist.github.com/janko-m/cfc50049c360d5c8be34
I feel the same as you. Refile was first very exciting for me, because if you wanted to change a version, you don't have to reprocess them, you only have to change the URL in the HTML. But that means that if you have a page with lot of these images, it would be super slow and can really easily DoS your server. And CDNs also cache the images for like 4 hours, which is terrible. I realized it's much better to just have the thumbnails done immediately.
Yes, I was inspired by CarrierWave's uploaders, the goal of Shrine is to have all of the uploading logic in the uploader. And it even allows you too hook up to ".included" of the model that receives the attachment methods, from inside the uploader (http://shrinerb.com/rdoc/classes/Shrine/Plugins/Included.htm...).
"Upfront" processing means that you're doing the processing during upload, and on the "on-the-fly" processing means that you first upload the file, and then it's processed dynamically through the URL. For example, in Refile you make an URL to the uploaded file like this:
/attachments/sdkd/resize/300/300
That will make Refile trigger the specified processing at the moment the URL is requested. You should then put a CDN or a reverse proxy in front to kind of cache these processed files, so that they're not processed each time.
About the documentation, I wrote documentation for each of the plugins, and I linked them all on http://shrinerb.com. The plugin names should hopefully indicate what feature they accomplish, and since they're all individual, it's up to the user to choose what features they want (i.e. combination of plugins).
I was actually contributing actively to Refile before, so in a lot of ways Shrine is similar to Refile (the idea of backends is basically the same). However, the main difference is that Refile does processing on-the-fly, while Shrine (and CarrierWave) do it upfront, so it goes in that direction. Basically in Shrine I added all of the features I was missing and wanted to change, which I wasn't able to add to Refile in a compatible way. One major addition is the support for background jobs, which I just couldn't see how it could be added to Refile. Also, I wanted Shrine to be far less opinionated than Refile, that you can have much more options how to do uploads.
Yes, multiple databases are really useful for me, for some reason I often need to transfer data from one to another, e.g. legacy database to a new improved one (which I think happens often if you're in a non-startup).
In ActiveRecord you would have to specify for each model to which the database to connect. And then you have to figure out how to call a User model in one database, and how in the other (you have to call them differently, even if table names are the same). I think it can get pretty chaotic with something bigger.
I noticed that many problems that you mention other uploading gems have are actually solved by Shrine:
- "Transformation juggling" -- Versions are processed at instance-level, so you don't need to remember a class-level DSL.
- "File path config" -- If you change the location where your files are stored, the existing files on old locations will still work normally
- "Form validation dance" -- Very simple with Shrine, you just add an "<attachment>" hidden field.
- "Schema pollution" -- Shrine stores uploaded files exactly in the way that you described, in a single column as JSON
The idea of an image server indeed looks very interesting, but I think you're forgetting that this solves only one part of file uploads (although I agree it's a big part, and imgix looks really sexy). If you're using an image server, you still need to do:
- Caching the file (this still needs a direct upload endpoint)
- File validations (most notably preventing big files)
- Storing the file to image server (this still needs a background job)
- Model attachment logic (all of the assignment/callback logic)
One problem that I always had with on-the-fly processing is, if you have a page with a lot of photos, and you want to change the URL so that they're processed differently, how will that page look when a first person visits it? I think it will be horrible, the user will have to wait for a really long time to actually see the images, which is not nice user experience.
Also, the "image server" strategy can be used only for images, so if you want to upload documents, audio or video, you still need do regular processing on the application side. What's nice about uploading gems is that they provide a general solution.