Before I buy, can you confirm the bridge is GDPR-compliant, AI-Act-ready, has a digital product passport, and passed its environmental impact assessment? Otherwise the local compliance officer will fine us before it even collapses.
I am working on post-training and evaluation tasks mostly, and I built Evalica as a convenient tool for my own use cases. The computation is fast enough to not bother the user, but the library does not stand in my way during the analysis.
I am thinking about this for a while and I think I’ll vibecode them. Not sure about WASM, though, as the underlying libraries should support it, too, and I am not sure about all of them at the same time.
You might also consider a fast implementation of Elo and Bradley–Terry that I have been developing for some time: https://github.com/dustalov/evalica (Rust core, Python bindings, 100% test coverage, and nice API).
GPT-2 follows the very well-studied architecture of Transformer decoder, so the outcomes of this study might be applicable to the more complicated models.
Thank you! I excluded the coding tasks as most annotators don't possess this expertise. I trust them in comparing pairs of dissimilar model outputs that don't require any specific skill but commonsense reasoning.
The only manual analysis was when I checked the passed/failed prompts of the top-performing model.
It's all about timing and communities. In my echo chamber, everybody is currently making them a part of the product, but I'm interested in a more representative sample. Hiring usually precedes shipping.
I found code LLMs to be very useful for rewriting nested for-loops in Python as nice vectorized operations in NumPy. One has to be careful about unexpected array materialization, but generally it works really well and saves a lot of time.
Discussions about LLM alignment often miss topics of data quality and quantity. It turns out that current models like Llama 2 use 10K+ prompts and responses for supervised fine-tuning (SFT) and 100K+ human preference pairs. While the preferences are pretty easy to annotate, producing a good SFT dataset is uneasy.
As this tutorial introduces the spectral clustering method that uses a similarity matrix between objects, I believe graph clustering methods are also worth mentioning. When I was investigating this topic, I released an implementation of a very efficient randomized clustering algorithm for graphs called Chinese Whispers: https://github.com/nlpub/chinese-whispers-python. Since it does not use matrices internally, it allows handling very large NetworkX graphs.
Great job! Gorse reminds me of the now-retired Apache PredictionIO recommendation engine: https://predictionio.apache.org/templates/recommendation/qui.... Have you evaluated why it took off and why it was later shut down, so that your project can avoid its mistakes?
I am using the former for some personal documents and found that it improves text selection and copying on Apple devices. (This could be related to how PDFKit handles text.)
I would say I really enjoyed it. Although the gaming experience in Safari with a Magic Mouse is odd, the connection is somewhat laggy, and people are constantly (dis)connecting, it worked really well and brought me some sweet memories about playing CS 1.6 a long time ago.