With our team, we also found shared folders performance to be too low. Our Python framework/app is very read-heavy and stat() a lot of files (the Python module loading system isn't your friend)
We ended up using the synchronization feature in PyCharm to continually rsync files from native FS into the VirtualBox instance. Huge perf improvement but a little more cumbersome for the developers. But so far it has been working good, PyCharm's sync feature does what it is supposed to.
There's also Locust, a Python powered load testing tool similar to this. It's focused on simulating user behavior rather than hitting specific URLs. This allows for much more true-to-life testing.
We used it to load test Battlelog (the online service for Battlefield 3) and successfully simulated over 2 million concurrent users hitting the service using EC2.
See http://locust.io for more info.
(Disclaimer: I'm one of the authors)
Protocol handlers are a pretty shitty way of interfacing with desktop apps. There's no two-way communication and no error handling. Lots of potential screw-ups and incompatibility issues will/can happen. Sure, they don't require a browser plugin but that's about the only advantage.
I would strongly advise not to base your push mechanism solely on Web Sockets. Despite advances modern browsers are making with Web Sockets support, the current compatibility is not there yet.
For starters, IE has no support, though that is starting to change with IE10. But the biggest issue is with personal firewalls / corporate proxies. These are unfortunately quite horrible at handling Web Sockets. Many popular firewalls such as avast! and Bitdefender can’t handle the WS protocol and completely blocks traffic despite browser support and it being valid HTTP.
The socket.io guys did some extensive research on what firewalls are causing these issues. I don’t think their list is exhaustive but it features most major personal firewalls out there. Have a look at https://github.com/LearnBoost/socket.io/wiki/Socket.IO-and-f... , it's no reading of joy.
I’ve deployed quite large installations of Beaconpush (a push server incidentally based on Netty as well). First going from Web Sockets with fallback, over to pure WS and we’re looking at going back to using fallbacks once again due to these incompatibility issues. This time we are looking at using SockJS instead of rolling our own (to be fair, there were no open-source libs doing this when we started). I’ve ported SockJS to Netty (https://github.com/cgbystrom/sockjs-netty) and it’s working although I wouldn’t say it is production ready.
I urge you to have a look at SockJS and socket.io. Personally I prefer SockJS since it aims at emulating the Web Socket interface and nothing more as compared to socket.io (which adds a pub/sub layer). Also, it does not rely on a Flash fallback (which is another story). The socket.io guys are however working on engine.io, which will split the transport fallback mechanisms of socket.io to a separate library. That's a good idea, but it still uses Flash and IMHO a more complex "transport upgrade".
Redis should get an an HTTP API. Would enable you to implement something like this purely in the browser.
UIs are really useful from a sys admin point of view. I can also imagine adoption increasing if Redis supported a UI out-of-the-box.
Planet Framework is designed from the ground up to be real-time and scalable, but it also uses a lot of existing open-source projects.
For Planet Framework customers, we offer "partner source". Meaning, that customers will get full access to source code and also submit their own patches.
The reason behind "yet another template language" is that we compile down to both Python and JavaScript. Allowing us to render templates both on the server-side and client-side.
Sharing templates between these two gets very important when you develop real-time web sites.
(I'm one of the developers behind Planet Framework)
If the startup isn't considering their first employee a crucial hire then they're doing it wrong.
And if you, as a job seeker, are dependent to land a potentially low-paying job at a startup to get a "real" job, then you probably aren't that "crucial hire" the startup is looking for.
Twitter's "commons" library also features a very good stat package. Presumably used by their own services. It's designed for the JVM, so should work with any JVM language.
See http://twitter.github.com/commons/
Have used it successfully implementing stats/metrics in Beaconpush, a real-time user presence server (http://beaconpush.com).
In search of performance we also developed a non-blocking MySQL driver for Python (and gevent). It's used by our real-time web framework, Planet (http://www.planetframework.com).
While we could've gotten better performance with more processes, we wanted to stick with one process per core. Given this and a non-blocking driver we naturally got a good performance gain.
The reason why GitHub has become so popular is due to the fact they put code and developers first. Surfing to a repo doesn't bring some overview or ads page.
Opening a repo actually show useful stuff for me as a developer. The source is there together with an optionally inlined README file, rendered with popular micro markups.
Then add the ease of forking and you clearly get something people really like.
This style of presentation isn't of course for everybody. But GitHub chose to target developers above end-users.
GitHub could have been "SVNHub" or "hghub" for all I care, the technology is secondary in this case (but close behind).
"Thrudb is a set of simple services built on top of the Apache Thrift framework that provides indexing and document storage services for building and scaling websites. Its purpose is to offer web developers flexible, fast and easy-to-use services that can enhance or replace traditional data storage and access layers."
We ended up using the synchronization feature in PyCharm to continually rsync files from native FS into the VirtualBox instance. Huge perf improvement but a little more cumbersome for the developers. But so far it has been working good, PyCharm's sync feature does what it is supposed to.