Ask HN: Managing tons of parallel uploads on upload heavy site
1 comments
I'm not an expert on scaling uploads, but here's what I'd say off-the-cuff:
1. Try to save time on the UI if possible by reusing existing solutions for multi-file uploads: https://encrypted.google.com/search?q=upload+javascript+plug...
2. Queue up uploads so that users don't start uploading until the server(s) can take them.
3. Delay any processing of the uploaded files.
4. Throw some more cheap servers, additional VMs at the problem, or more dynos/workers if using Heroku, etc.
5. Ensure your pipes are large enough.
6. Or just use Amazon S3, CloudFiles, etc.:
- http://aws.amazon.com/articles/1434
- http://www.rackspacecloud.com/cloud_hosting_products/files
but this may be much higher burn rate, depending on your needs. See what people have written: https://encrypted.google.com/search?q=amazon+s3+alternatives
Scaling is difficult. Don't scale until you need to, because it gets expensive (higher burn rate is bad).
If you really have that many uploads, you may be ready to hire someone that has some proven experience in it, not just helping develop an application that was being scaled by someone else, but someone who was hands on, and preferably someone with recent experience working with S3, etc.
1. Try to save time on the UI if possible by reusing existing solutions for multi-file uploads: https://encrypted.google.com/search?q=upload+javascript+plug...
2. Queue up uploads so that users don't start uploading until the server(s) can take them.
3. Delay any processing of the uploaded files.
4. Throw some more cheap servers, additional VMs at the problem, or more dynos/workers if using Heroku, etc.
5. Ensure your pipes are large enough.
6. Or just use Amazon S3, CloudFiles, etc.:
- http://aws.amazon.com/articles/1434
- http://www.rackspacecloud.com/cloud_hosting_products/files
but this may be much higher burn rate, depending on your needs. See what people have written: https://encrypted.google.com/search?q=amazon+s3+alternatives
Scaling is difficult. Don't scale until you need to, because it gets expensive (higher burn rate is bad).
If you really have that many uploads, you may be ready to hire someone that has some proven experience in it, not just helping develop an application that was being scaled by someone else, but someone who was hands on, and preferably someone with recent experience working with S3, etc.
Html forms are cross-platform solution, but seeing many sites to use Flash, ActiveX, or Java for more functionality. Tips or articles about client side also very much appreciated.