Anecdotally the site we manage we are easily seeing 100x the traffic from bots than from humans in the past year. So much so that it is impacting our hosting costs.
My problem with the article is that they don't even mention this fact. The metaphors with Uber often is brought up but it breaks down at cost optimization. It also wouldn't be fair to say we are at the peak efficiency of LLMs and that there wouldn't be any improvements left.
It make sense if you account for cost of intelligence getting cheaper every year. Most of the models per unit of intelligence is getting far cheaper. We get better hardware, architecture, training techniques, inference optimizations and caching. All those improvements add up. In in early 2022 you were getting 10x cheaper annually now is closer to 2x - 5x cheaper annually. The cost is still dropping where as Uber can only get the cost down by so much.
The human brain also is a product of billions of years of evolution. We branched off from our common ancestor 7-9 million years ago. We encode quite a lot of structure and information that is essential for intelligence. The starting point of just our life time of training is incomplete.
If you calculate 100W * 7 million years * 365 = 255,500MW to train.
Yes definitely we were using our own budget and out of our own pocket and these model runs were getting expensive. Claude costed us around 200-300 dollars a 8 month run for example. We want to scale it and get more statistically significant results but wanted to share something in the interim.
Via api you can turn off websearch internally. We provided all the models with their own custom tools that only provided data up to the date of the backtest.
We used the LLMs API and provided custom tools like a stock ticker tool that only gave stock price information for that date of backtest for the model. We did this for news apis, technical indicator apis etc. It took quite a long time to make sure that there weren't any data leakage. The whole process took us about a month or two to build out.
OP here. We realized there are a ton of limitations with backtest and paper money but still wanted to do this experiment and share the results. By no means is this statistically significant on whether or not these models can beat the market in the long term. But wanted to give everyone a way to see how these models think about and interact with the financial markets.
! basically functions like @ in Slack and chat applications. The only difference in Linen is that we give you an @mention so that it doesn't necessarily need to send a push.
We actually tried shipping a desktop client with Tauri and Rust. It helped with initial bundle size but performance and resource consumption was kind of similar. We also ran in to issues with the api’s being bit limiting. Most likely we’ll revisit the desktop client with Tauri but depending on how mature the api is we may have to resort to electron.
Our bundle size has actually only gotten smaller since we’ve shipped our first version. We know it will gradually grow but we try to be disciplined in not using too many external dependencies
Thanks! Linen is secretly email in disguise. We have played around with syncing with email clients which would let you manage all your messages and notifications from one client.
We've actually done test on scaling. This is a fork of Linen.dev which had community of up to 50k members. So we've tested this well. Search is an on going feature we are working on. We're going to enhance it with more embedding like search functionality.
Definitely agree on developer community and third party integrations. That is our main focus after this launch. Building just a good quality chat tool took quite a bit of time but we understand how important the dev ecosystem is.
Thanks for checking the privacy policy will update to include AWS!
One of the pain point I've actually seen for Slack is Slack Connect becomes disorganized very quickly to the point where you need a custom solution to manage it. I think it is a symptom of the overall problem with Slack. Since Linen is thread first it makes manage these conversations much easier. We do plan on shipping our own version of Slack Connect or even integrating with existing channels.
Pricing wise we're still figuring out. I do think paying for active users is a likely direction though!