At Manifest AI we have just released our open-source CUDA kernels to implement Symmetric Power Transformers, as described in our paper from back in August:
Since this is a variant of a linear attention, you get linear cost when training (as opposed to quadratic in regular attention), and constant when doing inference. This is especially attractive for longer contexts!
Have a look and play with it -- and of course contributions are very welcome! It's an early alpha!
Experimenting is good. I do it all the time. Just take a look at my repos on Github. Just a few of them are conceptually (and arguably) "new", while the others are just exercises to learn about new things. Experimenting about things, even about the basic ones, helps me a lot in my learning process. At a personal level.
But when you watch the bigger picture, with companies actually building software for clients that trust them, then choosing technology is a delicate, non-trivial matter. It's important, and I mean in actual dollars (or euros). There is where the expertise of older developers, people who have founded, sold or shut down profitable and unprofitable companies, who have worked in a gazilion of projects and made a lot of expensive mistakes, there is where that expertise comes to play. They can enlighten the sometimes childish, faction-like debates about technology that keep repeating themselves over and over. That can free us to think about new problems and new things, based on past experience. That's how science advances, and that's how software development advances too, in my opinion.
I chose that word on purpose because of both its semantic and emotional power. Besides, I'm pretty happy that all the critics have been focused on either using the term "cancer" or the swearing at the end, so I guess my points are valid to almost everyone, and you just didn't like the way I expressed them. That sound ok.
I'm really proud that so far no one has argued about any point I make in the article, but rather about the "cancer" reference and the swearing at the end. I guess that means I happened to write a pretty damn good article.
I am currently organizing the Barcelona Ruby Conference, and have attended many other conferences in Europe. I know a lot of people put much effort in doing things right. But that's not the majority of the community, unfortunately. I also agree that the Pareto principle (and many other points raised in my post) are a universal problem, not just Ruby's.
I did provide constructive advice. If you read the post thoroughly, you can see a call to improve our learning manners, stay away from hot trends, and even a simple pattern you can use in your day to day to compare technologies, or to understand better some debates (like threads-processes-reactors).
Heh, I really liked the analogy with Postel's law, and I guess you're right. The main criticising I've received about the blog post has been about the swearing, so I guess I could skip that next time.
I don't think that's Rails, or Ruby or whatever paritcular tech we're discussing. I think that's a common trait in human communities, and I'm worried about the one I feel mine, but I'm sure everyone can see a part of their community reflected on my post.
I didn't need to grow old to watch relatives and friends fighting cancer. I know how serious it is, and I really admire that you're working in that particular field. That said, I used the metaphor on purpose, because of its power, because I think it's a perfectly valid analogy.
I didn't mean to give a bad impression of Ruby. I think (and many people have pointed out) that the problems I outline are valid for many developer communities, if not all of them. In the end, those communities are human, and share a good amount of our defects.
It is meant for development environments only. Nobody would risk putting a Ruby console in production :)
In fact, when using it with Rails, it is loaded only in development environment. With other frameworks you should take care of what middlewares you use in which environment.
https://manifestai.com/articles/symmetric-power-transformers...
Since this is a variant of a linear attention, you get linear cost when training (as opposed to quadratic in regular attention), and constant when doing inference. This is especially attractive for longer contexts!
Have a look and play with it -- and of course contributions are very welcome! It's an early alpha!