QMachine – a supercomputer made out of crowdsourced Web browsers(qmachine.org)
qmachine.org
QMachine – a supercomputer made out of crowdsourced Web browsers
https://www.qmachine.org/
5 comments
Sean is a really great guy--glad to see he got something like this working!
Example application? (excepted scientific computing)
Fun, probably. It's hard to imagine a workload in which this would be the truly best solution. Real supercomputers nowadays use immense intra-node bandwidth, and if you care about performance (and why are you using a supercomputer if you don't?), languages and environments that run around an order of magnitude faster than Javascript (on Javascript's best day) even before we talk about them running on a supercomputer environment, with the aforementioned bandwidth.
(Tack on another order of magnitude or two if we're talking about something like GPU computation, which I would not yet trust in WebGL, even ignoring the problem of bandwidth for the problem specification. If you work the math, it becomes difficult to overcome orders of magnitude in performance differences by throwing lots of slow resources at a problem, because you get to the point where it becomes impossible to make up the difference due to additional communication overheads.)
This basically limits this "supercomputer" to embarrassingly parallel problems that have very small specifications, for which performance is no particular concern, including the fact that one does not even know how much resources you will have for the problem on a second-by-second basis. This is not a large niche, but it can be fun.
(Tack on another order of magnitude or two if we're talking about something like GPU computation, which I would not yet trust in WebGL, even ignoring the problem of bandwidth for the problem specification. If you work the math, it becomes difficult to overcome orders of magnitude in performance differences by throwing lots of slow resources at a problem, because you get to the point where it becomes impossible to make up the difference due to additional communication overheads.)
This basically limits this "supercomputer" to embarrassingly parallel problems that have very small specifications, for which performance is no particular concern, including the fact that one does not even know how much resources you will have for the problem on a second-by-second basis. This is not a large niche, but it can be fun.
These are thoughtful comments, here's a couple more
> his basically limits this "supercomputer" to embarrassingly parallel problems that have very small specifications
note support for map-reduce so it goes a step beyond "embarrassing" (map).
> It's hard to imagine a workload in which this would be the truly best solution.
Biology offers a few examples - typically (alignment-free) sequence analysis is limited by memory, not by communication, so being able to reach a large number of volunteers may be more significant that the time it takes for each of them to communicate their piece of the problem. Here's an example: http://www.almob.org/content/7/1/12
> his basically limits this "supercomputer" to embarrassingly parallel problems that have very small specifications
note support for map-reduce so it goes a step beyond "embarrassing" (map).
> It's hard to imagine a workload in which this would be the truly best solution.
Biology offers a few examples - typically (alignment-free) sequence analysis is limited by memory, not by communication, so being able to reach a large number of volunteers may be more significant that the time it takes for each of them to communicate their piece of the problem. Here's an example: http://www.almob.org/content/7/1/12
> languages and environments that run around an order of magnitude faster than Javascript (on Javascript's best day)
I'd love to see the programming language that's an order faster than Google V8. See http://benchmarksgame.alioth.debian.org/u32/javascript.php and play with it a little bit ;-)
I'd love to see the programming language that's an order faster than Google V8. See http://benchmarksgame.alioth.debian.org/u32/javascript.php and play with it a little bit ;-)
Javascript's speed can be overstated, especially on benchmarks, where everything can be just so, and maximize the amount of time you spend in the C portion of the runtime. In practice, you do not get a "mere" 3x performance loss in Javascript code, especially for things like what supercomputers do. At least, I haven't seen anyone seriously report that result for anything like real code.
asm.js may overcome that to some extent, but as I mentioned elsewhere, the real problem is bandwidth. Frankly you could run things effectively instantaneously on the client side, and still end up with wildly worse performance on this sort of cluster than a real supercomputer.
asm.js may overcome that to some extent, but as I mentioned elsewhere, the real problem is bandwidth. Frankly you could run things effectively instantaneously on the client side, and still end up with wildly worse performance on this sort of cluster than a real supercomputer.
Google V8's regular expressions can outrun C for real bioinformatics workflows, and it's actually not even close. I routinely write JS that outruns C and C++, and you haven't seen it in reports because my papers haven't been accepted yet. I doubt that I'm the only person capable of writing "amazing" code. Do you write JS?
As for bandwidth, there is no reason to believe that bandwidth is an issue for all parallel architectures. Stream processing works very well under the given constraints, for example.
As for bandwidth, there is no reason to believe that bandwidth is an issue for all parallel architectures. Stream processing works very well under the given constraints, for example.
"Do you write JS?"
Yes, along with a ton of other languages. It is well known that if you use a slow language to set up problems for code written in fast implementations, the result can still be quite fast, but it's the fast bit, not the slow bit that's responsible. See also NumPy. It's not Javascript being fast, it's the regex engine. If what you're doing is regexes, it's almost certainly more efficient to just do the regexes than to try to ship the relevant strings down over commodity internet connections... which is, indeed, exactly why you don't take this approach in your code.
Yes, along with a ton of other languages. It is well known that if you use a slow language to set up problems for code written in fast implementations, the result can still be quite fast, but it's the fast bit, not the slow bit that's responsible. See also NumPy. It's not Javascript being fast, it's the regex engine. If what you're doing is regexes, it's almost certainly more efficient to just do the regexes than to try to ship the relevant strings down over commodity internet connections... which is, indeed, exactly why you don't take this approach in your code.
What about using asm.js?
I classed that in with doing GPU computations in JS; it may work in a carefully selected browser and GPU combo today, but it's going to be a while before it's out there.
And even if your JS is running at native speeds, it doesn't solve the bandwidth problem, which is nowadays arguably the distinguishing characteristic of a supercomputer, since the compute nodes of a supercomputer are hardly any faster than a commodity desktop anymore. (Some, yes, sure, but nowhere near the multiples there used to be.)
And even if your JS is running at native speeds, it doesn't solve the bandwidth problem, which is nowadays arguably the distinguishing characteristic of a supercomputer, since the compute nodes of a supercomputer are hardly any faster than a commodity desktop anymore. (Some, yes, sure, but nowhere near the multiples there used to be.)
Hello, this is Sean Wilkinson, the author of QMachine. I'm not sure what "excepted scientific computing" means, but there is an example study available at http://q.cgr.googlecode.com/hg/index.html, but it's probably going to fail if everyone on HN tries it because the data source (the NCBI) will probably crash ...
Bitcoin mining
A similar service: http://crowdprocess.com/
where are my QCoins?
Coming soon, if you'd like to help, actually :-)
:-D