CloudFront Uploads via POST and PUT(aws.typepad.com)
aws.typepad.com
CloudFront Uploads via POST and PUT
http://aws.typepad.com/aws/2013/10/amazon-cloudfront-content-uploads-post-put-other-methods.html
17 comments
Interesting, this definitely positions them as a competitor to CloudFlare. But the CDN-as-full-proxy has some distinct disadvantages: every POST to CloudFront will have to make an extra round-trip to retrieve the data before returning to the user. We ditched CloudFlare because of this -- simple requests would take 500ms longer via CloudFlare. Presumably CloudFront will run into the same issues...
This is super exciting. My only concern on cost relates not to CloudFront itself, but rather the burden of using your own SSL certificate, as mpetrov mentioned.
I wonder if this will be improved as well? I personally wouldn't want to accept user data without SSL.
> Pricing for Custom SSL Certificates is simple. We charge a fixed monthly fee of $600 for each custom SSL certificate you associate with your CloudFront distributions, pro-rated by the hour. For example, if you had your custom SSL certificate associated with at least one CloudFront distribution for just 24 hours (i.e. 1 day) in the month of June, your total charge for using the custom SSL certificate feature in June will be (1 day / 30 days) * $600 = $20. Detailed pricing information for the Custom SSL Certificate feature is available on the CloudFront Pricing Page.
http://aws.amazon.com/cloudfront/custom-ssl-domains/
I wonder if this will be improved as well? I personally wouldn't want to accept user data without SSL.
> Pricing for Custom SSL Certificates is simple. We charge a fixed monthly fee of $600 for each custom SSL certificate you associate with your CloudFront distributions, pro-rated by the hour. For example, if you had your custom SSL certificate associated with at least one CloudFront distribution for just 24 hours (i.e. 1 day) in the month of June, your total charge for using the custom SSL certificate feature in June will be (1 day / 30 days) * $600 = $20. Detailed pricing information for the Custom SSL Certificate feature is available on the CloudFront Pricing Page.
http://aws.amazon.com/cloudfront/custom-ssl-domains/
We use CloudFront (without SSL) for our public site (static on S3) and this definitely looks cool. Our app itself has SSL (via an ELB) but not our static WWW site. The only thing we have on there I'd consider sensitive is our PGP key for emailing us security issues and I have that also linked directly to the HTTPS S3 bucket. As there's no user input it didn't seem worth it to pay the $600/mo for CloudFront SSL.
Was a no brainer to setup and honestly I never even think about it. We just update the S3 bucket for the site and CloudFront picks up the changes. Support for these new HTTP methods makes CloudFront a lot more interesting for a dynamic app.
Regarding SSL, the $600 per month seems like a lot for SSL but I think (pure speculation) it's because of the individual IPs needed for each endpoint. SSL ports can't be shared with other hosts[1]. Since CloudFront has endpoints at multiple edge locations, they would need multiple IPs per SSL cert. Add CPU cost for SSL processing too and I guess that's where the $600 comes from.
[1]: Well technically they can using SNI[2] but some older browsers don't support it (mainly IE on XP).
[2]: http://en.m.wikipedia.org/wiki/Server_Name_Indication
Was a no brainer to setup and honestly I never even think about it. We just update the S3 bucket for the site and CloudFront picks up the changes. Support for these new HTTP methods makes CloudFront a lot more interesting for a dynamic app.
Regarding SSL, the $600 per month seems like a lot for SSL but I think (pure speculation) it's because of the individual IPs needed for each endpoint. SSL ports can't be shared with other hosts[1]. Since CloudFront has endpoints at multiple edge locations, they would need multiple IPs per SSL cert. Add CPU cost for SSL processing too and I guess that's where the $600 comes from.
[1]: Well technically they can using SNI[2] but some older browsers don't support it (mainly IE on XP).
[2]: http://en.m.wikipedia.org/wiki/Server_Name_Indication
Until we can forget about the following platforms not supporting SNI, http://en.wikipedia.org/wiki/Server_Name_Indication#No_suppo..., CloudFront will have to consume and instrument IPs per edge per client. Maybe if they gave the option of dropping support for non-SNI and/or IPv6 finally took off...
It is worth noting that Google Cloud Storage has had this functionality for well over a year (maybe since 2010), as well as proper support for CORS so you can use XHR2 for uploads. And read-after-write consistency. If you are a developer who remotely cares about this feature, my personal recommendation is to at least compare the options.
Some more details here
http://docs.aws.amazon.com/AmazonCloudFront/latest/Developer...
I'm curious if the edge receives the full POST/PUT first and then does a complete PUT/POST to the origin, or does it forward as it's receiving.
I'm curious if the edge receives the full POST/PUT first and then does a complete PUT/POST to the origin, or does it forward as it's receiving.
Decided to do a quick test..
//make yourself a file dd if=/dev/zero of=1m count=1 bs=1m
//post directly to origin
time curl -F "file=@1m" -X POST "http://up4.pinkbike.com/upload/t.php" -w %{speed_upload}Bytes/s
array(3) { ["HTTP_USER_AGENT"]=> string(81) "curl/7.19.7 (universal-apple-darwin10.0) libcurl/7.19.7 OpenSSL/0.9.8y zlib/1.2.3" ["REQUEST_METHOD"]=> string(4) "POST" ["CONTENT_LENGTH"]=> string(7) "1048770" } 230177.000Bytes/s real 0m4.560s user 0m0.004s sys 0m0.010s
//post via Cloudfront to same origin
time curl -F "file=@1m" -X POST "http://dhima35gjf4ct.cloudfront.net/upload/t.php" -w %{speed_upload}Bytes/s
array(3) { ["HTTP_USER_AGENT"]=> string(17) "Amazon CloudFront" ["REQUEST_METHOD"]=> string(4) "POST" ["CONTENT_LENGTH"]=> string(7) "1048770" } 227055.000Bytes/s real 0m4.623s user 0m0.005s sys 0m0.011s
So upload time is not much different from my location. The origin is in San Jose, and in my case I'm in Vancouver BC, and going through the Seattle cloudfront edge. Would be interesting to see what you get from other locations ( I'm occasionally getting "ERROR: The request could not be satisfied" with cloudfront post )
//make yourself a file dd if=/dev/zero of=1m count=1 bs=1m
//post directly to origin
time curl -F "file=@1m" -X POST "http://up4.pinkbike.com/upload/t.php" -w %{speed_upload}Bytes/s
array(3) { ["HTTP_USER_AGENT"]=> string(81) "curl/7.19.7 (universal-apple-darwin10.0) libcurl/7.19.7 OpenSSL/0.9.8y zlib/1.2.3" ["REQUEST_METHOD"]=> string(4) "POST" ["CONTENT_LENGTH"]=> string(7) "1048770" } 230177.000Bytes/s real 0m4.560s user 0m0.004s sys 0m0.010s
//post via Cloudfront to same origin
time curl -F "file=@1m" -X POST "http://dhima35gjf4ct.cloudfront.net/upload/t.php" -w %{speed_upload}Bytes/s
array(3) { ["HTTP_USER_AGENT"]=> string(17) "Amazon CloudFront" ["REQUEST_METHOD"]=> string(4) "POST" ["CONTENT_LENGTH"]=> string(7) "1048770" } 227055.000Bytes/s real 0m4.623s user 0m0.005s sys 0m0.011s
So upload time is not much different from my location. The origin is in San Jose, and in my case I'm in Vancouver BC, and going through the Seattle cloudfront edge. Would be interesting to see what you get from other locations ( I'm occasionally getting "ERROR: The request could not be satisfied" with cloudfront post )
This will be awesome for terminating SSL closer to the user and should make a big impact on latency in mobile apps on high-latency connections.
Combine that with a custom SSL certificate at all the edge nodes ($600/month) and this is a pretty compelling dynamic CDN offering.
Can't wait to try it out for our API.
Combine that with a custom SSL certificate at all the edge nodes ($600/month) and this is a pretty compelling dynamic CDN offering.
Can't wait to try it out for our API.
as someone who has an iPhone app that requires asynchronous high-resolution images uploads, hooray.
But won't the total time of transfer be higher if there is an additional hop? I understand that the time before being able to confirm the transfer to the user is slower, but then the photo wouldn't be immediately available to the user (it still needs to go from CF to S3). How is that useful?
I think now it would be: iPhone <----> Closest CloudFront Edge Server <-----> S3
The upload to the edge CF server could be ACK'd faster and at this point the user still has a copy and so does the edge server. If meanwhile he sends somebody a link to this photo, if they are nearby, they'll hit the same edge server and get it faster not slower.
If on the other hand they are far away they'll hit a different edge server that would have to grab it in turn from S3 but that would have happened anyway, no? As far as I understand it.
Also a trick that helps propagate things to CF edges faster just occurred to me: S3 can be setup to POST somewhere after every write. So maybe..
iPhone <----> Closest CloudFront Edge Server <-----> S3 ----> Central service (App Engine? :P ) ---> Cloudfront Edge{1,2,3..} <---> S3.
But that might be a tad overkill, heh. (Also I'm not sure how you can trick CF to direct to a specific edge without using proxies)
The upload to the edge CF server could be ACK'd faster and at this point the user still has a copy and so does the edge server. If meanwhile he sends somebody a link to this photo, if they are nearby, they'll hit the same edge server and get it faster not slower.
If on the other hand they are far away they'll hit a different edge server that would have to grab it in turn from S3 but that would have happened anyway, no? As far as I understand it.
Also a trick that helps propagate things to CF edges faster just occurred to me: S3 can be setup to POST somewhere after every write. So maybe..
iPhone <----> Closest CloudFront Edge Server <-----> S3 ----> Central service (App Engine? :P ) ---> Cloudfront Edge{1,2,3..} <---> S3.
But that might be a tad overkill, heh. (Also I'm not sure how you can trick CF to direct to a specific edge without using proxies)
CF can keep connections to the origin open, which saves several long round-trip times in TCP/SSL handshakes. Conceivably, the CF-to-S3 network might also be better than the you-to-S3 network.
I guess latency in connection setup isn't really important when transferring high quality pictures, the bandwidth is going to affect the transfer time much more.
I see your point about a better connection between S3 and CF, but it's a hypothesis, and the docs don't confirm it. My experience with CF is that the first fetch (for GETs) tends to be "slow"; I don't have enough numbers to check whether is as slow as S3 (whose speed is highly variant over time) or not. But I have always thought that the speed of S3 was impacted by "I/O" (virtual I/O) time, not network or bandwidth. EC2 servers which are in the same region are able to sustain much higher transfer rates.
I see your point about a better connection between S3 and CF, but it's a hypothesis, and the docs don't confirm it. My experience with CF is that the first fetch (for GETs) tends to be "slow"; I don't have enough numbers to check whether is as slow as S3 (whose speed is highly variant over time) or not. But I have always thought that the speed of S3 was impacted by "I/O" (virtual I/O) time, not network or bandwidth. EC2 servers which are in the same region are able to sustain much higher transfer rates.
my app is fire-and-forget for users. they don't actually need to re-download the image. so this is a clear win as it will allow them to get those bytes down the pipe to my backend faster.
Awesome! This could be really significant for us adding mobile event tracking with POST support to Snowplow. (Currently our CloudFront event collector uses GET + access log parsing.)
Does this mean I can basically use Cloudfront at the root of my site, à la Cloudflare?
Second, your users can now benefit from accelerated content uploads. After you enable the additional HTTP methods for your application's distribution, PUT and POST operations will be sent to the origin (e.g. Amazon S3) via the CloudFront edge location, improving efficiency, reducing latency, and allowing the application to benefit from the monitored, persistent connections that CloudFront maintains from the edge locations to the origin servers.
So you get to save on having to configure and serve multiple domain names (which isn't all that hard), but you have to add logic to your server side code and user exposed uploading UX to check and display to the user if content is finished being uploaded by the user to cloudfront and by cloudfront to the origin/backing store before using/processing/displaying it.