t-vi

2 か月前議論

Not sure about the link, the official landscape URL seems to be this (same data probably):

https://landscape.pytorch.org/

t-vi

2 か月前議論

We do try to check the pulse of projects there with a checklist, but I guess it started in a more uncoordinated way.

If you file an issue here, I think it would work to update things:

https://github.com/pytorch-fdn/ecosystem

t-vi

3 か月前議論

They have some pretty cool people, though, no reason not to think they'll catch up soon enough.

t-vi

8 か月前議論

- I don't think it hurts to learn PyTorch (and having learned JAX is good, too). I don't know if JAX + triton is as impossible as you make it out, but it seems that PyTorch integration is quite good for many things. - For pallas, triton and CUDA/C++, you probably want to know a bit about how GPU works. There is the GPU-Mode discord / lectures / ressources if you are looking for material https://github.com/gpu-mode/ . - In my experience how well Triton works varies depending on what you want to do (depending on the how well the programming model fits the task). If it does, it is quite nice to get something reasonably fast reasonably fast. PyTorch (in the inductor torch.compile backend) has made many things work well, so you could check that out if you run out of examples elsewhere).

t-vi

9 か月前議論

Note that the NVIDIA container uses CUDA+cuBLAS 13.0.2 which cites "Improved performance on NVIDIA DGX Spark for FP16/BF16 and FP8 GEMMs", which seems to be your use-case. In general, I would suspect that it mostly comes to versions of the libs.

Interestingly, there is a cuBLAS 13.1 whl on PyPI, not sure what that does.

t-vi

9 か月前議論

It seems to me that in 2016 people did (have to) play a lot more tricks with the backpropagation than today. Back then it was common to meddle with gradients in between the gradient propagation.

For example, Alex Graves's (great! with attention) 2013 paper "Sequence Generation with Recurrent Neural Networks" has this line:

One difficulty when training LSTM with the full gradient is that the derivatives sometimes become excessively large, leading to numerical problems. To prevent this, all the experiments in this paper clipped the derivative of the loss with respect to the network inputs to the LSTM layers (before the sigmoid and tanh functions are applied) to lie within a predefined range.

with this footnote:

In fact this technique was used in all my previous papers on LSTM, and in my publicly available LSTM code, but I forgot to mention it anywhere—mea culpa.

That said, backpropagation seems important enough to me that I once did a specialized videocourse just about PyTorch (1.x) autograd.

t-vi

昨年議論

Note that PyTorch's kernels are somewhat generic in shape. It has always been relatively easy to get speedups by specializing the shape, e.g. Apache TVM had that (back before it was "Apache" even).

t-vi

昨年議論

If you like JIT wrappers and Python interpreters:

In Thunder[1], a PyTorch to Python JIT compiler for optimizing DL models, we are maintaining a bytecode interpreter covering 3.10-3.12 (and 3.13 soon) for our jit. That allows to run Python code while re-directing arbitrary function calls and operations but is quite a bit slower than CPython.

While the bytecode changes (and sometimes it is a back-and-forth for example in the call handling), it seems totally good once you embrace that there will be differences between Python versions.

What has been a large change is the new zero cost (in the happy path) exception handling, but I can totally why Python did that change to that from setting up try-block frames.

I will say that I was happy not to support Python <= 3.9 as changes were a lot more involved there (the bytecode format itself etc.).

Of course, working on this has also means knowing otherwise useless Python trivia afterwards. One of my favorites is how this works:

  l = [1, 2, 3]
  l[-1] += l.pop()
  print(l)

1. https://github.com/Lightning-AI/lightning-thunder/

t-vi

2 年前議論

When I did a similar thing (but with less LLM) I liked https://github.com/coqui-ai/TTS but back then I needed to cut out the conversion step from tensor to a list of numbers to make it work really nicely.

t-vi

3 年前議論

The subtraction is because "is an example of constructing the “Inner Product” distance" per the text above it. That ymmone might not be one could be because they only need that up to a constant and so don't care, but it's probably not ideal to name the thing containing 0s ymmone.

t-vi

3 年前議論

> Is avoiding CF potentially just a matter of sheer scale ?

My intuition would be that you get more orthogonal directions to the gradient (of previous samples) if you have larger model.

t-vi

3 年前議論

After the first epoch, the average time since the present data item was last used for during training is small at the beginning of an epoch grows during the epoch. I'd expect that to positively relate to loss on the present iteration.

t-vi

3 年前議論

My former neighbors run https://justanotherfoundry.com/ and I like their work and bought some.

t-vi

3 年前議論

To be honest, I never get what people want with all that business and wonder if it is because the abstraction ("ordered derivatives") implied is not ideal.

If we follow the ordinary chain rule (for a single coordinate if you want) through the edges of the computational (DAG) graph, we get the right thing in each step.

The only other rule you need is that "if you use one variable several times in a calculation (i.e. several edges from(fw)/to(bw) the same node), you need to add the gradients computed for each", but IMHO that is pretty basic and intuitive, too. (So if you plug in z for both x and y into f(x, y), you have d/dz f(z, z) = f_x(z, z) + f_y(z, z), where the subscript indicates partial derivative.)

To me this seems both mathematically simpler than mixing the two into a "more than chain rule" thing and closer to what is actually going on algorithmically in a given implementation (the one I'm most familiar with is probably PyTorch's).

t-vi

3 年前議論

The PEP says that a naïve implementation would cause a 4% performance hit and have a list of various ways of speeding up cases where immortality makes sense / is guaranteed that they say can bring this to parity. Maybe they implemented part of the list.

t-vi

3 年前議論

This special casing just had not been implemented yet. But as it is an interesting optimization, more so with multi-interpreter or no-GIL Python, the developers will actually introduce immortal objects in Python 3.12 to avoid counting references on some objects (PEP 683 has been accepted):

https://peps.python.org/pep-0683/

t-vi

3 年前議論

Not saying that the vector space bit isn't neat, but it's called functional analysis because you can take limits of various forms and define (semi-) continuity, have completions of spaces, and all that has nice properties. So to me, a crucial thing is that these vector spaces are indeed topological.

t-vi

3 年前議論

It knows nothing of the modern stuff (because MacKay died too early), but skipping the first parts of David MacKay: Information Theory, Inference, and Learning Algorithms you get a very accessible course in (200x) Bayesian Inference that should cover most of what you need for diving into PPL applications.

http://www.inference.org.uk/mackay/itila/book.html

t-vi

3 年前議論

The logic error seems to be that you have the second for loop where you only want to destructure the splitted line into a tuple. Now writing that decomposition as a for loop over a one-element list (as in the corrected code) works, but seems "hmhm". In other words, the misunderstanding here is that you want a chain of generators, not a "nested for loop"-generator.

A more proper solution could be to chain the generator expressions: with

    lines = ["a,b", "c,d"]

you could do

    ((a,b) for a, b in (l.split(',') for l in lines))

t-vi

3 年前議論

I use Debian's system Python 3.10 install for most of my stuff and it works really well for me. Some things I install via pip but the key libraries (e.g. PyTorch) from source.

投稿

The fiery, deadly crashes involving the Tesla Cybertruck

AMD and OpenAI Announce Strategic Partnership to Deploy 6 Gigawatts of AMD GPUs

The Buchstabenmuseum Berlin is closing

[untitled]

Ask HN: Implementing Licensing for a Python Application

Optimizing models using the PyTorch JIT

コメント