Ex-USDS here. Did 6 months at USDS, still federally employed as a USDS reserve.
Just wanted to chime in on "Who in their right mind would go to work for the government, for minimum wage, unless they were already independently wealthy?" GS-15 pay is nowhere near minimum wage. If you actually think so, I'm sorry to say that you are out of touch with society. And while some USDS employees may be independently wealthy, the majority certainly aren't.
There are also the nice perks of USDS, like working on things that critically matter to the lives of millions of the neediest demographic (both domestically and internationally), and collaborating with a bunch of A+ human beings who, despite the personal sacrifices, passionately want to do what they can to improve the welfare of others.
To answer your specific question about why not in TCP, it's not deployable. Here's why:
* In order to get multiplexing and other features introduced with HTTP/2, you need to change protocol framing. However, this means that the protocol is no longer backwards compatible. There are many ways to roll out non-backwards-compatible changes, but for TCP, they were deemed unacceptable. For example, you could negotiate the protocol change via a TCP extension. However, TCP extensions are known not to be widely deployable (https://www.imperialviolet.org/binary/ecntest.pdf) over the public internet. You could use a specific port, but that doesn't traverse enough middleboxes on the public internet (http://www.ietf.org/mail-archive/web/tls/current/msg05593.ht...). Yada yada.
* More importantly, TCP is generally implemented in the OS. That means, updating a protocol requires OS updates. That means we need both server and client OS updates in order to speak the new protocol. If you look at the very sad numbers on Windows versions uptake, Android version uptake, etc, you'll understand why many people don't want to wait for all OSes to update in order to take advantage of new protocol features.
"""
I understand the points you make, and am sympathetic. I feel the same way when I see people abusing HTTP to provide async notifications, etc.
The fact is, however, if it isn't deployed, no matter how nice it would theoretically be when it is, it isn't useful and people WILL work around the problem.
That is true regardless of whether the problem is at the application layer (e.g. HTTP), or at the transport layer (e.g. TCP), or elsewhere.
The primary motivation is to get things working, and making things that lack redundancy, or are elegant comes in at a distant second.
Deployed is the most important feature, thus things which quickly move protocols and protocol changes from theoretical to deployed are by far the most important things.
The longer this takes, the more likely that the work-around becomes standard practice, at which point we've all "lost" the game.
-=R
"""
Basically, this is another instance of Linus's quote (https://lkml.org/lkml/2009/3/25/632): "Theory and practice sometimes clash. And when that happens, theory loses. Every single time." Theoretically, it'd be far more elegant to fix this in the transport layer. But in practice, it doesn't work. Except, maybe if you implement on top of UDP (e.g. QUIC), since that would be in user-space and firewalls don't filter out UDP as much.
Er, I thought I was agreeing with you. What do you think I'm arguing? To be clear, I view these two statements as grounded in the same logic, although perhaps one is more strongly worded than the other:
Yours - "So unless someone has the resources to test and deploy unstable web servers, it effectively means you shouldn't bother with spdy at this point."
Mine - "I think it's a fair assessment that supporting experimental technologies requires more engineering resources. Everyone has to do the cost/benefit analysis themselves."
I think it's a fair assessment that supporting experimental technologies requires more engineering resources. Everyone has to do the cost/benefit analysis themselves.
Speaking as a Chromium SPDY & HTTP/2 developer, we are very much focused on standardization. SPDY is an experimental protocol meant to drive the standards process, not become a de facto standard itself. Therefore, it's critical for us to kill off old SPDY versions.
And note that your reference is slightly misleading (I don't think you intended this). It's true, the vast majority of hosts supporting SPDY are running nginx. In practice though, the vast majority of these hosts are Cloudflare or WordPress.com hosted sites. Both run newer versions of nginx with SPDY/3.1 support.
I don't see why you think this is a semantic requirement of HTTP. Perhaps there's some confusion over what HTTP semantics are. Let me refer you to http://tools.ietf.org/html/draft-ietf-httpbis-p2-semantics-2.... It doesn't discuss exposing all HTTP traffic to network intermediaries. Perhaps you're thinking of the HTTP messaging layer http://tools.ietf.org/html/draft-ietf-httpbis-p1-messaging-2.... Also, I think your statement about allowing an in-path intermediary to act as a CDN is weird, since a CDN is defined as "a large distributed system of servers deployed in multiple data centers across the Internet. The goal of a CDN is to serve content to end-users with high availability and high performance." [1].
It's true, HTTPS is full of tradeoffs. You've identified some of them.
What do you see in HTTP/2 that "codifies this workaround"? That wasn't immediately obvious to me. Recall that HTTP/2 is basically just multiplexing with prioritized streams. There's no requirement on TLS in the spec, although all current browser deployments (of SPDY) require TLS.
Only time will tell. It's all still in progress. Of all the major browser vendors (Firefox, Chromium, IE) present at the Zurich HTTP/2 interim meeting, only Patrick McManus (Firefox) has expressed interest. Notably, he's a co-editor of that Alternate-Services internet-draft.
The specification indeed is about proxying http resources, not https ones. So it's not initially as alarming as some other proposals discussing trusting proxies to intercept SSL connections. For more details, you can refer to https://insouciant.org/tech/http-slash-2-considerations-and-....
This specific proposal is interesting because it specifically is related to opportunistic encryption proposals, in particular, the one that allows sending http:// URIs over an unauthenticated TLS connection: http://tools.ietf.org/html/draft-nottingham-httpbis-alt-svc-.... The problem here for proxies is, if you mix http and https (authenticated) traffic on the same TLS connection, the proxy cannot tell if it can safely MITM the connection. The proxy vendor would like to know if it can do so, probably for network management / caching / content modification reasons. Of course, the point of the opportunistic encryption proposal is to increase security (although its actual effective impact is controversial: https://insouciant.org/tech/http-slash-2-considerations-and-...). But if you believe in opportunistic encryption's security purposes, then it doesn't seem to really make sense to make the MITM'able traffic identifiable so proxies on the network path can successfully MITM them without detection.
If someone can install a root cert onto your computer then you are already owned - there is no end to the other things they can do too. Call it a virus, call it an enterprise, but call it a day - you're owned and there is no in-charter policy this working group can enact to change the security level of that user for good or for bad..
The good news is not everyone is already owned and SSL helps those people today.
I guess I should out myself as a Chromium HTTP stack maintainer (since 2009, so this behavior predates me). One might consider me a domain expert here. I participate in IETF HTTPbis for the HTTP/2 work as the primary Chromium representative. I am not involved with the RFC 2616 revision work as that's tough, thankless work, that thank god we have Julian Reschke and Roy Fielding working on. As far as I'm concerned, I owe them a drink everytime I see them. They do an awful lot of legwork talking to various implementations and trying to build consensus on actually conforming with the standard and all its edge cases. It's really quite unfortunate to see this blog post author treat them so unfairly, although I can see how one might easily jump to his conclusion.
Now, as far as "SHOULD NOT", that's a reasonable thought for people not aware of what popular user agents currently do. The thing is, the majority of major browsers rewrite POST to GET on a 301. Here's my browser's code for it: https://code.google.com/p/chromium/codesearch#chromium/src/n.... Here's Firefox's code for it: http://mxr.mozilla.org/mozilla-central/source/netwerk/protoc.... To my knowledge, all browsers implement this behavior. We basically copied IE's behavior, because, IE did it and websites expected all user agents to do what IE did. Story of the web, sound familiar? :P
"SHOULD NOT" implies that our implementations are behaving badly. Now, it's true, our implementations may not be behaving ideally from a spec cleanliness point of view, but interop trumps spec cleanliness, at least from the perspective of anyone who actually deploys real software on the internet. So it's probably best for the spec to acknowledge this and officially allow this. Specs that don't mirror reality are...probably not just useless, but actively harmful.
I think the author of this blogpost has a few things off:
- HTTP2 != httpbis. Both work is being done by the same working group "httpbis". http://datatracker.ietf.org/wg/httpbis/charter/ covers this. httpbis (http://stackoverflow.com/questions/9105639/httpbis-what-does...) was originally chartered to revise HTTP/1.1 (RFC2616)
The working group will refine RFC2616 to:
* Incorporate errata and updates (e.g., references, IANA registries, ABNF)
* Fix editorial problems which have led to misunderstandings of the specification
* Clarify conformance requirements
* Remove known ambiguities where they affect interoperability
* Clarify existing methods of extensibility
* Remove or deprecate those features that are not widely implemented and also unduly affect interoperability
* Where necessary, add implementation advice
* Document the security properties of HTTP and its associated mechanisms (e.g., Basic and Digest authentication, cookies, TLS) for common applications
As for the HTTP/2 work, here's a snippet from the charter on that:
The Working Group will produce a specification of a new expression of HTTP's current semantics in ordered, bi-directional streams. As with HTTP/1.x, the primary target transport is TCP, but it should be possible to use other transports.
- He seems to think the httpbis folks gratuitously redefined 301. It should be noted that RFC2616 (which, by definition, predates the httpbis work since httpbis is defined to revise RFC2616) had already noted the issue with 301 (http://tools.ietf.org/html/rfc2616#section-10.3.2):
Note: When automatically redirecting a POST request after receiving a 301 status code, some existing HTTP/1.0 user agents will erroneously change it into a GET request.
- It's unclear to me whether or not the author acknowledges the existence of buggy implementations as noted in section 10.3.2. It's an open question as to what to do in the presence of buggy implementations. From a server standpoint, if the client is buggy, and you don't want to break the client (willingness to break clients probably depends on how many of the server's users use that client), then you will attempt to work around it, irrespective of what the standard says. Therefore, it's simply pragmatic to ignore the spec if it doesn't mirror reality, and pragmatic spec editors may update the spec to acknowledge this difference.
- As far as current status of the various 308 usages, Julian (author of the 308 draft) is lobbying major user agents to adopt this, and has written up a status update on the Chromium bug tracker: https://code.google.com/p/chromium/issues/detail?id=109012#c....
That's true, which is why I was careful to say in its original form :) Since the original attack was on cookies (request headers). To my knowledge, no other SPDY server defaults response header compression to off. But yeah, if your application does pass secrets in response headers, you should be careful.
Let's be clear that the original CRIME attack was against request header secrets. Therefore, disabling response header compression (as nginx defaults to) does not prevent that. SPDY/3.1 request header compression is a client-side choice, not server-side.
Head of line blocking is when something ahead in the line blocks things behind it. If browsers choose to use connection limits, then that means that things ahead in the line and get to use the connections are blocking things behind in line which have to stay queued up waiting for the connections to free up. So yes, that's head of line blocking.
As for pipelining working great in those browsers, you should note that no major desktop browser uses pipelining. Firefox does not enable it by default. See the Firefox bug thread where it's explained why it's not on by default, and also identifying its head of line blocking issues: https://bugzilla.mozilla.org/show_bug.cgi?id=264354#c30.
Microsoft did test pipelining. In their theoretical lab network, with ideal situations, they can match SPDY performance. But the sad reality is that real websites and networks don't match that. Please see http://www.guypo.com/technical/http-pipelining-not-so-fast-n.... And you have to ask yourself, if Microsoft really believed that result, why don't they enable pipelining in IE?
It's fascinating to me that you still seem to hold onto your claims that HTTP pipelining is feasible in real networks, even though IE, Firefox, and Chrome developers have all clearly tried and given up on it.
Head of line blocking is so well documented that there are many common web performance techniques to get around it, like domain sharding: http://www.stevesouders.com/blog/2009/05/12/sharding-dominan.... It'd be very interesting to see where you get your data that head of line blocking is not a problem.
I'm not sure where you are getting your info that SPDY was designed largely by one recent college grad. I don't think of Mike Belshe and Roberto Peon as the same person, much less recent college grads :) Maybe 40+ year olds still counts as "recent"?
I'd love to hear what's stopping you from forcing SSL site-wide. Is it cost? If so, what are the specifics if you don't mind sharing?
Also, I don't know if you're using different VIPs for load balancing or lack of SNI support reasons or what not, but if your certificate provides proof of authentication for all your hostnames (probably need to use SubjectAltNames and maybe wildcards too) and the VIPs match, then Chrome & Firefox will send requests for those different hostnames over the same SPDY connection.
I'm not going to argue whether or not other libraries/protocols (e.g. your example of protobufs) may provide higher value. I was simply questioning your assertion "SPDY has no benefit for mobile APIs."
That said, I've got some comments on your new points:
* "Almost all APIs are transactional. Your API should be returning all the data you need to render a single screen in your mobile app, or it is inefficient." - Addressing this completely would take awhile, and there's no good point in our discussing this exhaustively. I'll simply note that while an API response may return all data necessary to render a single screen in the app, it does seem nice to allow for prioritized out of order responses like SPDY does. There does not seem to be a good reason to have head of line blocking in the responses, since the app should be able to render incrementally.
* "Pings are not the correct way to deal with hangs. I don't want to spill any secret sauce here, but it should be obvious to anyone with low level TCP experience." - I think you must be misunderstanding this, or it must not be obvious Google's TCP team, since they agree with the usage of SPDY PINGs. Perhaps you think SPDY PINGs are a keep-alive mechanism? Note that the article I linked to identifies them as a liveness detection mechanism.
* "Pipelining also uses a single connection. Opportunistic FINs are not unique to SPDY." - I don't know why you bring up pipelining again when I've pointed out that multiplexing is superior. Why put up with response head of line blocking? And I don't know what you're exactly referring to with opportunistic FINs, perhaps you can explain in further detail?
And I'll throw in extra data points. Despite the fact that you don't feel like it's useful for mobile APIs, other non-Google parties clearly do:
Just wanted to chime in on "Who in their right mind would go to work for the government, for minimum wage, unless they were already independently wealthy?" GS-15 pay is nowhere near minimum wage. If you actually think so, I'm sorry to say that you are out of touch with society. And while some USDS employees may be independently wealthy, the majority certainly aren't.
There are also the nice perks of USDS, like working on things that critically matter to the lives of millions of the neediest demographic (both domestically and internationally), and collaborating with a bunch of A+ human beings who, despite the personal sacrifices, passionately want to do what they can to improve the welfare of others.