An Empirical Evaluation of TCP Performance in Online Games(iis.sinica.edu.tw)
iis.sinica.edu.tw
An Empirical Evaluation of TCP Performance in Online Games
http://www.iis.sinica.edu.tw/~swc/pub/tcp_in_games.html
37 comments
It's not that SCPT is less known by developers, I think most developers actively looking for a solution to the downsides of TCP will come across SCTP.
There are two real reasons, the first and most important is that Microsoft never implemented SCTP. Apparently there was not enough customer demand, but that's a lousy excuse, I'm not sure what the real reason is. Perhaps they just have no one thinking about networked applications besides the Web at the moment. Sadly lack of forward thinking is something that bugs the Windows team a lot I feel. The second reason can be partly blamed on Microsoft as well, but most network devices operating at the transport layer have difficulty with SCTP. Why implement it when no one uses it right? just like IPv6 support. Anyway, apparently the difficulty lies with checksumming packets, which requires some more memory than is necessary for TCP/UDP.
It's a shame really. I think you can safely say that SCTP is the perfect gaming protocol. It supports almost all features gaming networking libraries usually implement on top of UDP just by configuring your streams.
There are two real reasons, the first and most important is that Microsoft never implemented SCTP. Apparently there was not enough customer demand, but that's a lousy excuse, I'm not sure what the real reason is. Perhaps they just have no one thinking about networked applications besides the Web at the moment. Sadly lack of forward thinking is something that bugs the Windows team a lot I feel. The second reason can be partly blamed on Microsoft as well, but most network devices operating at the transport layer have difficulty with SCTP. Why implement it when no one uses it right? just like IPv6 support. Anyway, apparently the difficulty lies with checksumming packets, which requires some more memory than is necessary for TCP/UDP.
It's a shame really. I think you can safely say that SCTP is the perfect gaming protocol. It supports almost all features gaming networking libraries usually implement on top of UDP just by configuring your streams.
> Perhaps they just have no one thinking about networked applications besides the Web at the moment. Sadly lack of forward thinking is something that bugs the Windows team a lot I feel.
If you found out about the wide range of networking protocols and services they provide in Windows, you wouldn't feel that way. For example, P2P. A few years ago, while working on P2P applications, we discovered Microsoft has a lot of technologies in that area. Things like Teredo (IPv6 transition and NAT traversal) and a complete P2P networking API -- which gives you encryption, self-organizing overlays and DHTs out of the box -- have been around since around 2001, and have been built into Windows since Vista. They also had lower level protocols for things like link-layer topology discovery and automated device (printers, etc) configuration.
Almost all proprietary stuff, of course - except Teredo, which has an RFC and an open source implementation - but quite interesting nonetheless. We talked to the team working on this, and they claimed it was even being used by some customers in 1000+ machine deployments.
For some reason, these things are not widely marketed. Maybe because it's a niche market. However, I wouldn't say they're not forward-thinking, at least when it comes to networking.
If you found out about the wide range of networking protocols and services they provide in Windows, you wouldn't feel that way. For example, P2P. A few years ago, while working on P2P applications, we discovered Microsoft has a lot of technologies in that area. Things like Teredo (IPv6 transition and NAT traversal) and a complete P2P networking API -- which gives you encryption, self-organizing overlays and DHTs out of the box -- have been around since around 2001, and have been built into Windows since Vista. They also had lower level protocols for things like link-layer topology discovery and automated device (printers, etc) configuration.
Almost all proprietary stuff, of course - except Teredo, which has an RFC and an open source implementation - but quite interesting nonetheless. We talked to the team working on this, and they claimed it was even being used by some customers in 1000+ machine deployments.
For some reason, these things are not widely marketed. Maybe because it's a niche market. However, I wouldn't say they're not forward-thinking, at least when it comes to networking.
I remember the forward-thinking present in the NT kernel being been seen in a negative light around here a while ago.
One reason is quite possibly that SCTP is not supported by most home routers/firewalls.
>And here comes the problem: SCTP is not used, even for projects where those features are needed.
sctp's achilles heel is its inability to work with NAT. It was designed to not work with NAT. NAT has since become widespread especially for connecting homes and sctp ended up not being so.
sctp's achilles heel is its inability to work with NAT. It was designed to not work with NAT. NAT has since become widespread especially for connecting homes and sctp ended up not being so.
So SCTP will be useful the day after IPv6 has wide adoption.
Well, shit.
Well, shit.
Unfortunately, I don't see NATs going away even after IPv6 gets here.
Why do you say that? I had IPv6 native with Comcast and there was no NAT.
There is a school of thought that details of an internal network topology should not leak out [1] (ostensibly because obscuring that has security benefits), and so NATs should be used even if each machine can have a unique IP address. Even though I worked on P2P apps and NATs were the biggest pain point, I do kinda feel the same... Leaking internal details make me squeamish even if I can think of no obvious risks. I think there is a real chance that home routers will also NAT by default even when IPv6 is common.
And you can bet corporate networks will be NATted. Not to mention data centers.
Another concern is Carrier Grade NATs. These are already being deployed within ISP networks to alleviate the IPv4 address shortage. I worry that once this infrastructure is in place, not only will it delay IPv6 adoption, but the adoption will be uneven, as ISPs with CGNs may hold off transitioning much longer. So even if there is a part of the Internet that is all IPv6, the rest of it might still be behind CGNs, and to connect arbitrary peers we'd still have to deal with NATs.
1. http://lwn.net/Articles/452293/
And you can bet corporate networks will be NATted. Not to mention data centers.
Another concern is Carrier Grade NATs. These are already being deployed within ISP networks to alleviate the IPv4 address shortage. I worry that once this infrastructure is in place, not only will it delay IPv6 adoption, but the adoption will be uneven, as ISPs with CGNs may hold off transitioning much longer. So even if there is a part of the Internet that is all IPv6, the rest of it might still be behind CGNs, and to connect arbitrary peers we'd still have to deal with NATs.
1. http://lwn.net/Articles/452293/
> There is a school of thought that details of an internal network topology should not leak out
That's why we have Privacy Extensions
That's why we have Privacy Extensions
IP address exhaustion is not the only reason to use NAT.
There's also Google's QUIC:
https://docs.google.com/document/d/1RNHkx_VvKWyWg6Lr8SZ-saqs...
http://en.wikipedia.org/wiki/QUIC
https://docs.google.com/document/d/1RNHkx_VvKWyWg6Lr8SZ-saqs...
http://en.wikipedia.org/wiki/QUIC
SCTP is used in WebRTC DataChannels, so it will see use in web games hopefully. However I think Chrome and Firefox implement it on top of UDP at the moment (perhaps because not all OSs support it?)
If it is using native SCTP, I don't see how well it would work with firewalls and routers (both enterprise and home ones). It would have to fallback to user-space over UDP presumably.
Yeah, practically speaking, a remake of SCTP on top of UDP is what is needed if there's going to be any adoption.
The WebRTC approach of using UDP gives NAT resistance and OS independence.
We should probably accept that TCP and UDP are the only "natural" L4 protocols we get, build all new protocols on top of UDP, and then just abstract the difference at the OS level, such that those tunneled-over-UDP protocols appear as native L4 transports. Then when you say "I'm using SCTP", you really mean "I'm using SCTP-over-UDP" -- but since that's the only possible meaning, it's just taken as a given.
Heck, maybe we should even reimplement TCP on top of UDP (stripping out the redundancies, so UDP carries the ports and checksum), and then just consider UDP an L3 protocol--a bit of extra stuff you get with every IP packet, whether you want it or not.
If we can figure out a way to do implicit protocol negotiation, such that we can build both clients and servers that "prefer" TCPv2-over-UDP but will drop back to plain-TCP, this could really happen, and we might get to start writing new transport protocols again. (Heck, for that matter, DTLS might finally get used.)
See also: the comment by eternaleye on the above post http://250bpm.com/blog:22/comments/show#post-1759542
Heck, maybe we should even reimplement TCP on top of UDP (stripping out the redundancies, so UDP carries the ports and checksum), and then just consider UDP an L3 protocol--a bit of extra stuff you get with every IP packet, whether you want it or not.
If we can figure out a way to do implicit protocol negotiation, such that we can build both clients and servers that "prefer" TCPv2-over-UDP but will drop back to plain-TCP, this could really happen, and we might get to start writing new transport protocols again. (Heck, for that matter, DTLS might finally get used.)
See also: the comment by eternaleye on the above post http://250bpm.com/blog:22/comments/show#post-1759542
> due to firewalls and routers refusing to route anything that's not TCP or UDP
If "The world is on fire because we're almost out of addresses" doesn't get residential ISP's to support another protocol for home users (IPv6), somehow I doubt "We can improve the performance of online games that work reasonably well right now" will convince them to support SCTP.
If "The world is on fire because we're almost out of addresses" doesn't get residential ISP's to support another protocol for home users (IPv6), somehow I doubt "We can improve the performance of online games that work reasonably well right now" will convince them to support SCTP.
Re: last point - at the expense of 8 bytes any IP-based protocol can be moved to run over UDP.
It is also important to consider DDOS when evaluating protocols.
TCP has an advantage over UDP because data cannot be spoofed. There are many ddos protection services that will protect against spoofed TCP connections (SYN floods).
Applications must implement their own 3 way handshake in UDP in order to avoid these problems. Many applications did not do this which is why COD servers ended up being a source of attack traffic themselves.
TCP has an advantage over UDP because data cannot be spoofed. There are many ddos protection services that will protect against spoofed TCP connections (SYN floods).
Applications must implement their own 3 way handshake in UDP in order to avoid these problems. Many applications did not do this which is why COD servers ended up being a source of attack traffic themselves.
The problems raised here can be solved with parametrizing TCP itself. A few notes here:
(a) Packet reordering is highly dependent on the timescale that you use. Its far worse in micro/smaller timescales than to be seen as attached in the graph
(b) By using TCP URGent byte effectively you can hack some realtime control stuff onto a data stream
(c) The fast retransmit not kicking in can be resolved by lowering the number of dupacks from 3 to 1 and add a timer that kicks in the fast retransmit after the timer expiry. This isnt a new ide. BTW 50% is the standard number for the whole internet so your stats does not show anything different from the standard internet
(d) TCP slow start after idle can be disabled on server side
(e) Agreed all of these require some co-operation from the client side which might be hard to get (especially on windows) but can easily point to a http proxy for the MMORPG in question and usually the speed benefit will drive the adoption of such proxies before actually having someone sell a Gamer TCP Stack :-)
Long story short all the problems can be mitigated to some degree within the norms if you use the right proxy :-)
(a) Packet reordering is highly dependent on the timescale that you use. Its far worse in micro/smaller timescales than to be seen as attached in the graph
(b) By using TCP URGent byte effectively you can hack some realtime control stuff onto a data stream
(c) The fast retransmit not kicking in can be resolved by lowering the number of dupacks from 3 to 1 and add a timer that kicks in the fast retransmit after the timer expiry. This isnt a new ide. BTW 50% is the standard number for the whole internet so your stats does not show anything different from the standard internet
(d) TCP slow start after idle can be disabled on server side
(e) Agreed all of these require some co-operation from the client side which might be hard to get (especially on windows) but can easily point to a http proxy for the MMORPG in question and usually the speed benefit will drive the adoption of such proxies before actually having someone sell a Gamer TCP Stack :-)
Long story short all the problems can be mitigated to some degree within the norms if you use the right proxy :-)
This is a really interesting article, I love reading about game networking. Hard to find many resources on this type of stuff... the 'Age of Empires' Gamasutra one is awesome too.
We used Apple's Game Center API for the online portion of Stratosphere: Multiplayer Defense (iPad), and it was pretty nice that Apple did all the work as far as network packets go... but you do have a choice between reliable and non-reliable sending. It wasn't explicitly said, but we assumed one was TCP and the other UDP. We went with the reliable option for all of our packets and it didn't really prove to be a problem, even simulating under 3G. Granted, Stratosphere is a strategy game. The bigger problem ended up being just having enough people online at one time to even play lol.
We used Apple's Game Center API for the online portion of Stratosphere: Multiplayer Defense (iPad), and it was pretty nice that Apple did all the work as far as network packets go... but you do have a choice between reliable and non-reliable sending. It wasn't explicitly said, but we assumed one was TCP and the other UDP. We went with the reliable option for all of our packets and it didn't really prove to be a problem, even simulating under 3G. Granted, Stratosphere is a strategy game. The bigger problem ended up being just having enough people online at one time to even play lol.
Check out the Tribes netcode model, http://library.theexiled.pwnageservers.com/file.php?id=2694
Another good read on game networking:
http://gafferongames.com/networking-for-game-programmers/
http://gafferongames.com/networking-for-game-programmers/
If you're a web game developer reading this you should also know that WebSockets, XHR, and anything built on top of those technologies (socket.io etc.) are TCP based. Only WebRTC DataChannels and flash are able to do UDP based communication in the browser.
Perhaps the developers communicated with their netops teams before designing the game protocols? Maybe they came to the conclusion that it's better to run TCP from a security/DDoS perspective than UDP? In other words, the ability to shut off inbound UDP outweighed any theoretical gains from using it in the game?
TCP SYN floods are handled with ease today. If you're actually using UDP in your product, you might as well paint "DDoS me" on your homepage.
TCP SYN floods are handled with ease today. If you're actually using UDP in your product, you might as well paint "DDoS me" on your homepage.
HN ought to parse "http://www.iis.sinica.edu.tw" as "sinica.edu.tw" rather than "edu.tw".
Or more broadly, \w+.(com|gov|net|org}.countrycode should really happen. Probably a couple other standard ones.
I call that thing bullshit.
"Our analysis indicates that TCP is unwieldy and inappropriate for MMORPGs" yet World of Warcraft, Lineage I/II, Guild Wars, Ragnarok Online, Anarchy Online, Mabinogi all use TCP. Extremly successful. I using UDP would have help making World of Warcraft better they would have done it years ago. But they all probably have no idea right?
That TCP is ill-suited for this does not mean it cannot be used. It also does not mean that it is World of Warcraft's particular bottleneck and that it would be a good investment of development time to change it once it is in production.
You should probably resist the urge to call bullshit on something when your best examples rely on legacy concerns.
You should probably resist the urge to call bullshit on something when your best examples rely on legacy concerns.
Yes, they all manage it--but this is why "MMORPG network middleware engineer" has always been an extremely well-paid senior-level position. If it was as easy as "SCTP.connect(host: 'example.com', port: 3044)" to get something sensible going, I think a lot more games would do the kind of smooth network-latency-compensation-via-action-prediction that you only tend to see right now from the big players.
Fully agreed. I've implemented a bad equivalent on top of UDP myself. It should be easier given how common a use case it is.
The premise may be valid, however the authors only examined a network trace for one game. At best, they've proven only that game's use of TCP is unwieldy and inappropriate.
A more appropriate methodology would have been to examine traces of the top 5 MMORPGs in terms of active players.
A more appropriate methodology would have been to examine traces of the top 5 MMORPGs in terms of active players.
TCP only really breaks down when there's a nontrivial rate of packet loss. Otherwise, excepting very fast-paced action games where you don't want a single noticeable hiccup, the concerns about TCP are largely outdated.
I mean, I was reading this kind of material back when FPSes and even MMOs were played over dialup connections. The world has changed.
I mean, I was reading this kind of material back when FPSes and even MMOs were played over dialup connections. The world has changed.
You missed many points, like Nagle's algorithm, in/out buffers, etc.
Very commonly TCP was used for messages that had to be reliable, and UDP for the other stuff. A lot, and I mean a lot, of work was put in by devs in the early 00s trying to figure out how to do as well with TCP as they could with UDP.
Also, RO server code was and is a steaming pile of sadness and bugs.
Also, RO server code was and is a steaming pile of sadness and bugs.
http://250bpm.com/blog:22
Especially:
"Obviously, the right solution would be implement a new transport protocol directly on top of IP protocol, one that would provide desired functionality — failure detection and/or multiplexing — directly, without duplicating the features.
And, as a matter of fact, the above was already done. The protocol built directly on top of IP with both heartbeating and multiplexing is called SCTP and is available out of the box in most operating systems.
And here comes the problem: SCTP is not used, even for projects where those features are needed. Instead, they are lousily re-implemented on top of TCP over and over again."
(Not necessarily saying that SCTP would be the best fit for games, more the general point -- that there's room for more than just TCP and UDP on top of IP -- and also that that might not work, due to firewalls and routers refusing to route anything that's not TCP or UDP).