Tensor Considered Harmful Part 2(nlp.seas.harvard.edu)
nlp.seas.harvard.edu
Tensor Considered Harmful Part 2
http://nlp.seas.harvard.edu/NamedTensor2
4 comments
Mathematically tensors are multilinear maps, not data structures. Typically tensors in deep learning represent multilinear mappings forwards and backwards. Scalars, vectors and matrices can also be tensors when they represent mapping.
If you think how normal fully connected feedforward network learns, its outer products of derivatives. Inference is affine transformations + nonlinearity sequence.
If you think how normal fully connected feedforward network learns, its outer products of derivatives. Inference is affine transformations + nonlinearity sequence.
> Mathematically tensors are multilinear maps, not data structures.
Yes, but any parameterized operator needs data for its representation, similar to how a closure is code + data.
Yes, but any parameterized operator needs data for its representation, similar to how a closure is code + data.
We've got a winner. Tensors "do" something; they're functions.
In the rank-1 and rank-2 case these functions can be represented as objects familiar to linear algebra. 3-tensors are not the same as stacked two-tensors, however.
I wonder if "neural networks as ODEs" will catch on. I could use some of my masters work and go get a PhD.
In the rank-1 and rank-2 case these functions can be represented as objects familiar to linear algebra. 3-tensors are not the same as stacked two-tensors, however.
I wonder if "neural networks as ODEs" will catch on. I could use some of my masters work and go get a PhD.
>We've got a winner. Tensors "do" something; they're functions.
And a neural network is just a stack of linear transforms (the "tensors") and nonlinear transforms (the activations). So in a lot of cases, these "tensors" do do something. Granted the abstraction for storing the active tensors is also a great abstraction for storing high-dimensional data (like a 128x128x4x60 2-second video).
>neural networks as ODEs
You're a few months late :P [0]
[0]: https://arxiv.org/abs/1806.07366
And a neural network is just a stack of linear transforms (the "tensors") and nonlinear transforms (the activations). So in a lot of cases, these "tensors" do do something. Granted the abstraction for storing the active tensors is also a great abstraction for storing high-dimensional data (like a 128x128x4x60 2-second video).
>neural networks as ODEs
You're a few months late :P [0]
[0]: https://arxiv.org/abs/1806.07366
> they're functions
Well, in math a lot of things we normally do not think of (or use) as a function can still be considered as such. For example, a vector "is a function." Why? Because it does something (to a covector, or a linear form - which, in turn, is a function, etc. etc.) Even a number "is a function" by a similar reasoning. Objects in math are very flexible conceptually.
Well, in math a lot of things we normally do not think of (or use) as a function can still be considered as such. For example, a vector "is a function." Why? Because it does something (to a covector, or a linear form - which, in turn, is a function, etc. etc.) Even a number "is a function" by a similar reasoning. Objects in math are very flexible conceptually.
Ehhh.
Take for simplicity a constant real number. A constant is not a function but there's a function from the reals to a space of real valued functions. Likewise every vector can be converted into a cove tot, but...
You can say these function-valued functions are bijective so you can work on either side of the fence. Your can also work with equivalence classes if you don't quite have a bijection. But a lot of mathematics is about being specific about what we want, even if we're lazy in writing later and smush related concepts together in the same symbol.
Take for simplicity a constant real number. A constant is not a function but there's a function from the reals to a space of real valued functions. Likewise every vector can be converted into a cove tot, but...
You can say these function-valued functions are bijective so you can work on either side of the fence. Your can also work with equivalence classes if you don't quite have a bijection. But a lot of mathematics is about being specific about what we want, even if we're lazy in writing later and smush related concepts together in the same symbol.
Once you put them into a particular coordinate basis, I'm pretty sure tensor algebra is just operation on slices of arrays. The magic happens when you change bases.
Its kinda like how a matrix is "just" a rectangular 2D array of numbers in a particular basis.
Its kinda like how a matrix is "just" a rectangular 2D array of numbers in a particular basis.
I think the reasoning behind the nameing convention is that tensors in DL are just an abstraction of multi-dimensional arrays and that the operations you do on multiple arrays such as sum or multiply are algebraic tensor operations due to their multi linearity.
It could also be dated to earlier work on computer vision where tensors were used to store a bunch of spatial coordinates so it might be that back then it was even more aligned with their mathematical definition.
That said the grief doesn’t seem to be with the naming convention but rather with how tensor classes are designed in common DL frameworks.
http://nlp.seas.harvard.edu/NamedTensor
It could also be dated to earlier work on computer vision where tensors were used to store a bunch of spatial coordinates so it might be that back then it was even more aligned with their mathematical definition.
That said the grief doesn’t seem to be with the naming convention but rather with how tensor classes are designed in common DL frameworks.
http://nlp.seas.harvard.edu/NamedTensor
Mathematica has used the term tensor for multidimensional arrays since 1988. Sergei Brin worked at Wolfram Research for a summer internship or something back before Google, so would have been exposed to that nomenclature, as many in Urbana-Champaign were.
probably to avoid confusion with numpy arrays. and to irritate physicists
Yes to irritating physicists, no to avoiding confusion with numpy arrays. Pytorch README:
- "Tensor computation (like NumPy) with strong GPU acceleration"
- "If you use NumPy, then you have used Tensors (a.k.a ndarray)."
Pure marketing drivel.
- "Tensor computation (like NumPy) with strong GPU acceleration"
- "If you use NumPy, then you have used Tensors (a.k.a ndarray)."
Pure marketing drivel.
i mean, they have different names when used in pytorch code, tensor vs ndarray.
To the best of my knowledge the operations in deep learning are very much tensor operations. In most cases they seem to boil down to matrix multiplicaiton on one of the indices, but it's still kind of useful to use tensor notation (especially using the Einstein summation convention) since IMHO T_abc M_ax = S_xbc is easier to read than the corresponding sum, or the corresponding pseudocode.
Not sure if the fact that tensors have slightly more structure than mere arrays w.r.t. e.g. a change of basis is that useful, but it could be useful in proofs, especially if a FFT is involved somewhere.
Not sure if the fact that tensors have slightly more structure than mere arrays w.r.t. e.g. a change of basis is that useful, but it could be useful in proofs, especially if a FFT is involved somewhere.
Some operations are quite tensor-ish. For instance the re-shaping which einops writes like this
Rearrange('b c h w -> b (c h w)')
Index notation is often less confusing than numbered dimensions, without requiring an you to use an explicit NamedTensor.
Perhaps of interest, here's a quick prototype of how this could be useful. If you use an index "n"-for-number on the dimension for which you previously used an index "c"-for-colour (when performing any einsum or slicing or re-shaping operation) then you will get a warning:
https://github.com/mcabbott/TensorSlice.jl#checking
Perhaps of interest, here's a quick prototype of how this could be useful. If you use an index "n"-for-number on the dimension for which you previously used an index "c"-for-colour (when performing any einsum or slicing or re-shaping operation) then you will get a warning:
https://github.com/mcabbott/TensorSlice.jl#checking
A couple critiques for this article’s argumentative approach, as well as the proposed library. My expertise is not in Torch, but I believe these thoughts are valid on a more general level.
Yes, the ability to add names to dimensions of Tensor objects would be useful. Accessing components by named dimension instead of numeric indices (assuming clean syntax) would reduce mental overload for both readers and writers of code. However, the examples provided do a poor job of making the case that people should use this library. I’ll try breaking the issues down into these buckets:
* Awkward UX
* Muddied and uninteresting examples
* Missing the point
---
Awkward UX
First and foremost, it’s hard to ignore that this API is not fully fleshed out. I appreciate that the author is explicitly seeking feedback and taking an iterative approach, but it feels like there needed to be some more time spent thinking about how to best approach fixing the problem at hand from a user perspective.
Since this post explicitly attempts to take a pragmatic approach, it’s important to understand that pragmatic users will only tolerate as much user-unfriendliness in a library as it provides value. Idealistic users will happily saddle themselves with difficult/painful libraries, but pragmatic users demand value for pain. And unfortunately, while it is useful to have named dimensions in a Tensor, it is not so useful that a user should be expected to completely shift the way they write their code. This library asks above-and-beyond what I’d expect a client to reasonably sign up for. A couple examples:
* Forcing users to use a vaguely named op() method to wrap functions, but only sometimes (for functions that “change dimensions”?). * Wrapping Torch library functions, but changing their signature along the way (e.g. the sample() method for distributions) * Incompatibility between NamedTensor and base Tensor with raw Torch (ie. if a client uses NamedTensor, they don’t get to use the rest of the Torch ecosystem as-is without manual access to the underlying Tensor object).
Note that I’m not making note about some of the controversial decisions here (such as not allowing for explicit dimension access by index). There’s discussion to be made there, too, but I want to focus on the decisions that were made implicitly, as that’s where many UX issues can arise.
The author seems open to taking advice and iterating from there. My recommendation would be to iterate the API even earlier: _before_ writing implementation code. It will be faster to type out some example client uses and see what people think than to implement a working version and find out it fails in some cases. You’ll also find it easier to take a step back, as you won’t have spent as much time in the code you’ve already written. This may allow for a fresh perspective on what would make the most useful/useable API.
---
Muddied and uninteresting examples
Several of the examples given make it hard to compare the original version of the code and the proposed version. This is most obvious in the MNIST example. The default version uses a “reassign to same variable” approach, while the proposed version uses method-chaining. Some of this may be due to some of the issues mentioned above, but I want to note that it detracts from the author’s argument as it distracts a reader who many otherwise be on board.
The examples given aren’t particularly inspiring, either. The “Experiments on Canonical Models” section includes toy examples, many of which are just as readable (if not more so) in the original format as they are in the NamedTensor version.
---
Missing the point
At the end of the day, there’s a large stack of reasons why working on adding named dimensions to Tensors is useful. After the first article came out, it seems that the author received a fair amount of comment on how pragmatic their proposed approach was. Unfortunately, I feel like the author took “pragmatic” to mean “how would one literally use the library as it stands” as opposed to a challenge to consider the practicalities of the end-user and the tradeoffs that need to be made between the ideals of the author and the needs of the client.
I think the main failing of this library/article is that instead of honing in on a narrow-scoped thesis of “how Tensors should have their dimensions named”, it ends up being distracted by the quirks of the way the named tensor library was implemented. I think the author may benefit from taking a step back from what they have and thinking about how they could achieve their goals in a more cohesive way.
Yes, the ability to add names to dimensions of Tensor objects would be useful. Accessing components by named dimension instead of numeric indices (assuming clean syntax) would reduce mental overload for both readers and writers of code. However, the examples provided do a poor job of making the case that people should use this library. I’ll try breaking the issues down into these buckets:
* Awkward UX
* Muddied and uninteresting examples
* Missing the point
---
Awkward UX
First and foremost, it’s hard to ignore that this API is not fully fleshed out. I appreciate that the author is explicitly seeking feedback and taking an iterative approach, but it feels like there needed to be some more time spent thinking about how to best approach fixing the problem at hand from a user perspective.
Since this post explicitly attempts to take a pragmatic approach, it’s important to understand that pragmatic users will only tolerate as much user-unfriendliness in a library as it provides value. Idealistic users will happily saddle themselves with difficult/painful libraries, but pragmatic users demand value for pain. And unfortunately, while it is useful to have named dimensions in a Tensor, it is not so useful that a user should be expected to completely shift the way they write their code. This library asks above-and-beyond what I’d expect a client to reasonably sign up for. A couple examples:
* Forcing users to use a vaguely named op() method to wrap functions, but only sometimes (for functions that “change dimensions”?). * Wrapping Torch library functions, but changing their signature along the way (e.g. the sample() method for distributions) * Incompatibility between NamedTensor and base Tensor with raw Torch (ie. if a client uses NamedTensor, they don’t get to use the rest of the Torch ecosystem as-is without manual access to the underlying Tensor object).
Note that I’m not making note about some of the controversial decisions here (such as not allowing for explicit dimension access by index). There’s discussion to be made there, too, but I want to focus on the decisions that were made implicitly, as that’s where many UX issues can arise.
The author seems open to taking advice and iterating from there. My recommendation would be to iterate the API even earlier: _before_ writing implementation code. It will be faster to type out some example client uses and see what people think than to implement a working version and find out it fails in some cases. You’ll also find it easier to take a step back, as you won’t have spent as much time in the code you’ve already written. This may allow for a fresh perspective on what would make the most useful/useable API.
---
Muddied and uninteresting examples
Several of the examples given make it hard to compare the original version of the code and the proposed version. This is most obvious in the MNIST example. The default version uses a “reassign to same variable” approach, while the proposed version uses method-chaining. Some of this may be due to some of the issues mentioned above, but I want to note that it detracts from the author’s argument as it distracts a reader who many otherwise be on board.
The examples given aren’t particularly inspiring, either. The “Experiments on Canonical Models” section includes toy examples, many of which are just as readable (if not more so) in the original format as they are in the NamedTensor version.
---
Missing the point
At the end of the day, there’s a large stack of reasons why working on adding named dimensions to Tensors is useful. After the first article came out, it seems that the author received a fair amount of comment on how pragmatic their proposed approach was. Unfortunately, I feel like the author took “pragmatic” to mean “how would one literally use the library as it stands” as opposed to a challenge to consider the practicalities of the end-user and the tradeoffs that need to be made between the ideals of the author and the needs of the client.
I think the main failing of this library/article is that instead of honing in on a narrow-scoped thesis of “how Tensors should have their dimensions named”, it ends up being distracted by the quirks of the way the named tensor library was implemented. I think the author may benefit from taking a step back from what they have and thinking about how they could achieve their goals in a more cohesive way.
(I'm the author)
Thanks this is really helpful. I agree with the points being made. The post started out with the hypothesis, "look this basically works as is", but the conclusion of the UX experiments seems to be "the idealistic API forces a jarring style change". I'll step back and think it through a bit more.
My sense from your comments is that the ideal useful/usable library would offer most of the benefits of the first post, while being basically be compatible with the torch API? Or at least that should be the number one priority to aim for from the pragmatic perspective.
Thanks this is really helpful. I agree with the points being made. The post started out with the hypothesis, "look this basically works as is", but the conclusion of the UX experiments seems to be "the idealistic API forces a jarring style change". I'll step back and think it through a bit more.
My sense from your comments is that the ideal useful/usable library would offer most of the benefits of the first post, while being basically be compatible with the torch API? Or at least that should be the number one priority to aim for from the pragmatic perspective.
> Accessing components by named dimension instead of numeric indices (assuming clean syntax) would reduce mental overload for both readers and writers of code.
Couldn't this be solved (at least for the reader) by using descriptive variable names when accessing the dimensions?
Couldn't this be solved (at least for the reader) by using descriptive variable names when accessing the dimensions?
This is exactly how I solve the problem for myself, however if this is a language feature it will be neater, more concise, automatically verifiable, and universally legible by everyone.
[deleted]
Perhaps I'm missing something, but (practical) deep learning seems to have little tensor algebra going on, just operations on slices of arrays.