If you're the author, unfortunately I have to say that the blog is not well-written -- misinformed about some of the claims and has a repugnant click-baity title. you're getting the attention and clicks, but probably losing a lot of trust among people. I didn't engage out of choice, but because of a duty to respond to FUD.
> > torch.compile is 2 years old, XLA is 7 years old. Compilers take a few years to mature
> That was one of my major points - I don't think leaning on torch.compile is the best idea. A compiler would inherently place restrictions that you have to work-around.
There are plenty of compilers that place restrictions that you barely notice. gcc, clang, nvcc -- they're fairly flexible, and "dynamic". Adding constraints doesn't mean you have to give up on important flexibility.
> This is not dynamic, nor flexible - and it flies in the face of torch's core philosophies just so they can offer more performance to the big labs using PyTorch. For various reasons, I dislike pandering to the rich guy instead of being an independent, open-source entity.
I think this is an assumption you've made largely without evidence. I'm not entirely sure what your point is. The way torch.compile is measured for success publicly (even in the announcement blogpost and Conference Keynote, link https://pytorch.org/get-started/pytorch-2.0/ ) is by measuring on a bunch of popular PyTorch-based github repos in the wild + popular HuggingFace models + the TIMM vision benchmark. They're curated here https://github.com/pytorch/benchmark . Your claim that its to mainly favor large labs is pretty puzzling.
torch.compile is both dynamic and flexible because: 1. it supports dynamic shapes, 2. it allows incremental compilation (you dont need to compile the parts that you wish to keep in uncompilable python -- probably using random arbitrary python packages, etc.). there is a trade-off between dynamic, flexible and performance, i.e. more dynamic and flexible means we don't have enough information to extract better performance, but that's an acceptable trade-off when you need the flexibility to express your ideas more than you need the speed.
> XLA's GPU support is great, its compatible across different hardware, its optimized and mature. In short, its a great alternative to the often buggy torch.compile stack - if you fix the torch integration.
If you are an XLA maximalist, that's fine. I am not. There isn't evidence to prove out either opinions. PyTorch will never be nicely compatible with XLA until XLA has significant constraints that are incompatible with PyTorch's User Experience model. The PyTorch devs have given clear written-down feedback to the XLA project on what it takes for XLA+PyTorch to get better, and its been a few years and the XLA project prioritizes other things.
the author got a couple of things wrong, that are worth pointing out:
1. PyTorch is going all-in on torch.compile -- Dynamo is the frontend, Inductor is the backend -- with a strong default Inductor codegen powered by OpenAI Triton (which now has CPU, NVIDIA GPU and AMD GPU backends). The author's view that PyTorch is building towards a multi-backend future isn't really where things are going. PyTorch supports extensibility of backends (including XLA), but there's disproportionate effort into the default path. torch.compile is 2 years old, XLA is 7 years old. Compilers take a few years to mature. torch.compile will get there (and we have reasonable measures that the compiler is on track to maturity).
2. PyTorch/XLA exists, mainly to drive a TPU backend for PyTorch, as Google gives no other real way to access the TPU. It's not great to try shoe-in XLA as a backend into PyTorch -- as XLA fundamentally doesn't have the flexibility that PyTorch supports by default (especially dynamic shapes). PyTorch on TPUs is unlikely to ever have the experience of JAX on TPUs, almost by definition.
most of the GenAI players use both PyTorch and JAX, depending on the hardware they are running on. Character, Anthro, Midjourney, etc. are dual shops (they use both). xAI only uses JAX afaik.
How can you convince the world to use it (and pay you)?
Step 1: You need a 3rd party to approve that this model is safe and responsible.
the Purple Llama project starts to bridge this gap!
Step 2: You need to prove non-sketchy data-lineage. This is yet unsolved.
Step 3: You need to partner with a cloud service that hosts your model in a robust API and (maybe) provides liability limits to the API user. This is yet unsolved.
the absolute golden benchmarks are https://github.com/pytorch/benchmark
They are a diverse set of userland code taken from github as-is and made into benchmarks.
thanks for the heads-up, looks like we didn't yank the CPU wheels on those dates. will get to them in the next set of working hours, as its an unlikely scenario (not only do you have to install the wheel of a specific date, you also have to specify the undocumented feature flag [dynamo])
> So technically, if you are pulling the older version of pytorch-nightly (specifically 2.0.0.dev20221230), it will still pull that compromised dependency (because torch have explicit version lock to it).
All PyTorch nightlies with this dependency have been deleted
the version numbers are different, even though the package names are the same.
A stable version will have a version number such as `1.13.0`, where as a nightly version will have the date in the version number, such as `2.0.0.dev20221230`. You can check this either with `pip list | grep torch` or via `python -c "import torch; print(torch.__version__)"`
If you installed `torch` via the instructions to install the nightly version specifically, then you get the nightly version.
By default, you get the stable version of `torch`.
counter-example: CPython, NumPy, SciPy and many critical well-used open-source projects that have tiny volunteer teams and operational budgets that are minuscule compared to something like React / PyTorch
Whether using Lua or Python, for GPU-based scientific computing, the need to drop into a C call is the same. The overhead of Python vs Lua never really mattered.
While we were based on top of LuaJIT, we couldn't use the JIT for anything because we had to always call into the C library for GPU kernels (and LuaJIT can't JIT through an opaque C call, incase that C call changes the interpreter stack).
Where Python really helps is with its ecosystem. The entire data science and ML ecosystem is in Python.
The difficulty of calling out to C is not much greater in Python, things like PyBind11 make it pretty natural.
We ship everything needed for userland -- including parts of CUDA/CuBLAS and CuDNN that we need (which is why our binaries are so fat).
GPU drivers would be kernel-land and I don't think we actually can install GPU drivers as part of a `pip install`. Will look into what Flux is doing, but I doubt they ship GPU drivers.
Separately, thanks for flagging the Poetry issue, we might prioritize it, especially if the fix is easy.
I've been helping on the project, it's lead by Chris Cummins and Hugh Leather.
Just a heads-up for folks, we haven't fully cleaned up and gotten ready for public attention yet, we are 90% there.
Once we are golden, we're going to write a note with a way to submit the results of your own agents, compare with baselines (random, actor-critic), etc.
And as others noticed, in it's current form we're focusing on code size and phase ordering, but we will be expanding over time to other optimization problems like runtime.
> > torch.compile is 2 years old, XLA is 7 years old. Compilers take a few years to mature
> That was one of my major points - I don't think leaning on torch.compile is the best idea. A compiler would inherently place restrictions that you have to work-around.
There are plenty of compilers that place restrictions that you barely notice. gcc, clang, nvcc -- they're fairly flexible, and "dynamic". Adding constraints doesn't mean you have to give up on important flexibility.
> This is not dynamic, nor flexible - and it flies in the face of torch's core philosophies just so they can offer more performance to the big labs using PyTorch. For various reasons, I dislike pandering to the rich guy instead of being an independent, open-source entity.
I think this is an assumption you've made largely without evidence. I'm not entirely sure what your point is. The way torch.compile is measured for success publicly (even in the announcement blogpost and Conference Keynote, link https://pytorch.org/get-started/pytorch-2.0/ ) is by measuring on a bunch of popular PyTorch-based github repos in the wild + popular HuggingFace models + the TIMM vision benchmark. They're curated here https://github.com/pytorch/benchmark . Your claim that its to mainly favor large labs is pretty puzzling.
torch.compile is both dynamic and flexible because: 1. it supports dynamic shapes, 2. it allows incremental compilation (you dont need to compile the parts that you wish to keep in uncompilable python -- probably using random arbitrary python packages, etc.). there is a trade-off between dynamic, flexible and performance, i.e. more dynamic and flexible means we don't have enough information to extract better performance, but that's an acceptable trade-off when you need the flexibility to express your ideas more than you need the speed.
> XLA's GPU support is great, its compatible across different hardware, its optimized and mature. In short, its a great alternative to the often buggy torch.compile stack - if you fix the torch integration.
If you are an XLA maximalist, that's fine. I am not. There isn't evidence to prove out either opinions. PyTorch will never be nicely compatible with XLA until XLA has significant constraints that are incompatible with PyTorch's User Experience model. The PyTorch devs have given clear written-down feedback to the XLA project on what it takes for XLA+PyTorch to get better, and its been a few years and the XLA project prioritizes other things.