Show HN: A simple scheduler for running jobs on GPUs(github.com)
github.com
Show HN: A simple scheduler for running jobs on GPUs
https://github.com/ExpectationMax/simple_gpu_scheduler
7 comments
Examples simulated with GNU Parallel:
Interesting! Is there a way to use this, or some other library, that can spin up a GPU instance, on AWS or GCE, if there are no GPUs available, and discard GPUs when no more tasks are left? Maybe also allow some "overtime", in case a new task may get registered in the time when the instance gets free and a new one starts up?
The package can simply be installed from pypi
$ pip install simple_gpu_scheduler
ERROR: Could not find a version that satisfies the requirement simple_gpu_scheduler (from versions: none)
ERROR: No matching distribution found for simple_gpu_scheduler
Sorry thats my bad. The package is written in python 3. Thus it is also possible that you have to install the package using `pip3 install simple-gpu-scheduler`. If the command pip3 does not exist try installing python3 using the ubuntu package manager apt.
$ python3 -m pip3 install --user simple_gpu_scheduler
/usr/bin/python3: No module named pip3
$ python3 -m pip install --user simple_gpu_scheduler
ERROR: Could not find a version that satisfies the requirement simple_gpu_scheduler (from versions: none)
ERROR: No matching distribution found for simple_gpu_scheduler
$ python3 --version
Python 3.5.2
$ python3 -m pip --version
pip 19.3 from /usr/local/lib/python3.5/dist-packages/pip (python 3.5)
Could you add an issue to the GitHub project? Shouldn't be a problem to add support for 3.5 and I'll take care of it soon.
Try with hyphens instead of underscores? https://pypi.org/project/simple-gpu-scheduler/ has it both ways which is very confusing.
Actually both should work. The problem in this case is probably that the package requires python3.6 and thus cannot be found using a python 2 pip. I changed the README for consistency though.
The README is incorrect, the PyPi package name uses hyphens.
Curious if anyone used a GPU-enabled Kubernetes cluster for something similar? I recently set one up. The only downside was an inability to assign fractional GPUs to tasks, so concurrent tasks were necessarily limited to the discrete number of GPUs available to k8s
This is one of the problems we solve at Algorithmia. We have a senior engineer set to do a lightning talk at Kubecon next month, on the topic: https://events19.linuxfoundation.org/events/kubecon-cloudnat.... If you’re interested in working on the problem, reach out to me.
We used to have a google doc where everyone would put name, machine, gpu(s), and estimated time to run, whenever they launched a simulation (longer than an hour). Low tech, but worked ok as long as people didn’t forget to do so.
Reminds me of how BOINC started out over a decade ago :)
A month in the lab saves a day in the library.
"Please don't post shallow dismissals, especially of other people's work. A good critical comment teaches us something."
https://news.ycombinator.com/newsguidelines.html
https://news.ycombinator.com/newsguidelines.html
Did you actually _look_ at the repo? The author does not claim that this is the only way of doing it; and in fact, they seem to be quite aware of the existence of other schedulers. This is just a very simple one that does not require a lot of overhead and installation time.
In fact, I would go as far as to say that this scheduler is written in the spirit of Unix: make a task easy and provide a simple, chainable interface.
In fact, I would go as far as to say that this scheduler is written in the spirit of Unix: make a task easy and provide a simple, chainable interface.
I also have the feeling that this should be "easy" with a traditional scheduler. But could you (or someone else) point me in the direction of something that's easy to set up for a quad-GPU machine? I spent two days trying to get SLURM to work, at one point. Currently I'm running with GNU Parallel, but it's much less flexible of course.
Slurm works well even for a single machine. It does require some configuration but you only need to do it once and then it's very robust. The packages are available in standard repositories (like Ubuntu's) and there's a web-based configuration generator available. And if you ever want to add a second machine or more, it will be very easy to expand.
What stopped you from using Slurm?
What stopped you from using Slurm?
Installing Slurm requires root access to the server/workstation, which is not always available. In academia the management of computational resources is often the responsibility of a separate IT unit which is not part of the research group itself. Further the IT unit might not be willing to implement desired changes to the server infrastructure or might take _extremely_ long to do so. Often the setup of a scheduler is also significantly more complicated in this context, for example due to authentication via a University LDAP or oddities of the server setup.
Luckily in our academic research group we administer our own workstations and servers ourselves. Has downsides as well of course, but at least we can fix or change anything we want hardware or software-wise.
I can imagine it would be excruciating to explain our requirements to an external IT department and then go back and forth clarifying what we want, make them learn stuff they'd otherwise not need, convince them that we really need it, wait until they have time...
The downside is that things are not "professionally set up" (we're computer scientists but not pro admins), but at least if there's a deadline and something isn't working, we'll surely fix it because we have skin in the game.
I can imagine it would be excruciating to explain our requirements to an external IT department and then go back and forth clarifying what we want, make them learn stuff they'd otherwise not need, convince them that we really need it, wait until they have time...
The downside is that things are not "professionally set up" (we're computer scientists but not pro admins), but at least if there's a deadline and something isn't working, we'll surely fix it because we have skin in the game.
Slurm is not great. I found recently that when submitting an array of jobs, it will reject you if the upper bound is too high, regardless of the size of the array. I filed a bug but received no response.
That's according to documentation (https://slurm.schedmd.com/job_array.html).
> Note that the minimum index value is zero and the maximum value is a Slurm configuration parameter (MaxArraySize minus one).
Note that they say "the maximum value is a Slurm configuration parameter", not "the maximum value minus minimum value is a Slurm configuration parameter". The name of the parameter is confusing though, I admit.
Also, they prioritize paying customers when responding to bug tickets. Slurm is free software but if you want to interact with them, you need to pay. Nevertheless, for simple use cases (just scheduling a bunch of jobs) it's very comfortable to use. If you get into more complex territory (like array jobs), you'll need to read more docs.
And it scales very nicely from just one workstation up to the largest HPC clusters on the planet with complex accounting, multi-node jobs and so on.
> Note that the minimum index value is zero and the maximum value is a Slurm configuration parameter (MaxArraySize minus one).
Note that they say "the maximum value is a Slurm configuration parameter", not "the maximum value minus minimum value is a Slurm configuration parameter". The name of the parameter is confusing though, I admit.
Also, they prioritize paying customers when responding to bug tickets. Slurm is free software but if you want to interact with them, you need to pay. Nevertheless, for simple use cases (just scheduling a bunch of jobs) it's very comfortable to use. If you get into more complex territory (like array jobs), you'll need to read more docs.
And it scales very nicely from just one workstation up to the largest HPC clusters on the planet with complex accounting, multi-node jobs and so on.
It is documented, but it strikes me as simply being a lazy implementation. The consequence of this is that you cannot submit jobs 0-999, then 1000-1999, etc. You must submit 0-999, 0-999, ..., with an additional argument. It doesn't really prevent you from doing anything, it just gets in your way.
(The name MaxArraySize definitely makes it seem more like this is a bug than intended, albeit documented, behavior.)
I looked into submitting a patch but unfortunately MaxArraySize is used in more places than I would have guessed, and lost interest in trying to test all of them.
(The name MaxArraySize definitely makes it seem more like this is a bug than intended, albeit documented, behavior.)
I looked into submitting a patch but unfortunately MaxArraySize is used in more places than I would have guessed, and lost interest in trying to test all of them.
We're talking about installing Slurm for one's own use, so you can just change MaxArraySize in the configuration as an easy solution.
This is fundamentally why people buy DGX machines from Nvidia. It's preloaded. You have to figure out how to use them though.
My favourite scheduler is still "Son of Grid Engine", the open source continuation of the "Sun Grid Engine". I never had much trouble setting it up, but it is not really maintained at this point. PBS as suggested by others never worked well for me. But Slurm didn't give me much trouble, certainly not 2 days worth.
Son of grid engine definitely shows its age but I never had troubles submitting jobs to my school's cluster.
try PBS Pro.
[deleted]