The model released is called Latam-GPT, the first large-scale open-source language model developed specifically for Latin America and the Caribbean.
Its size is 70 billion parameters (70B).
It is a continuous pre-training (CPT) adaptation of Meta’s Llama 3.1 70B base model, trained on a regionally-curated corpus of ~300 billion tokens that emphasize Latin American languages, cultures, and contexts.
This is great news for the community. We should measure a bit more closely how tool usage behaves in the wild. The gains you can get from different providers are not marginal.
Sometimes it's not just raw intelligence, but engineering as well
This is stupid because a UTF8 is a tokenizer that covers all Unicode with a vocab of only 256 (yes, without a K). This is the only way of scaling the bitter lesson with tokenizers. Also, with architectures that span +1M context windows, it’s no longer an argument/issue the reduced context windows.
This is one of the reasons I've been advocating to use UTF-8 as a tokenizer for a long time. The actual problem IMHO are tokenizers themselves, which obscure the encoding/decoding process in order to gain some compression during training to fit more data in for the same budget, and arguably gaining some better understanding from the beginning. Again just a lack of computing power.
If you use UTF-8 directly as tokenizer, this problem becomes evident once you fit it into the context window. Plus, you can run multiple tests for this type of injection; no emoji should take more than up to 40 bytes (10 code points * 4 bytes per code point in the worst case). This is an attack on tokenizers, not on UTF-8.
Plus, Unicode publishes the full list of sequences valid containing the ZWJ character in emoji-zwj-sequences.txt
Who would have known that BitTorrent, shadow libraries, and seeders will help to train the best AI models out there, that adds a whole new meaning to a "seed".
This is a wonderful submission to Anna's archive [1]. I really love people pushing the boundaries of shadow source initiatives that benefit all of us, especially providing great code and design. Can't emphasize enough the net plus of open source, BitTorrent, and shadow libraries that have had in the world. You can also make the case that LLMs wouldn't have been possible without shadow libraries; it's just no way of getting enough data to learn.