Nuclear-powered ion thrusters could solve this issue. They provide low acceleration for a long time consuming very little consumable. This would allow the telescope to stay at the right position for observation.
Cloud models can use batch processing which is significantly more efficient. A local model has basically a batch of one which takes as much time to process as a batch of 100 because the gpu is memory bound and spend most of its time loading the model from vram to the gpu cache while the gpu cores are idle. With a batch of 100 the model loading time and compute time are roughly similar. So local
Models have a first 100x lower efficiency. Secondly, local models are idle most of the time waiting for the user to write a prompt, so the efficiency gap is probably more around 1000x.
I have done that at meta/FAIR and it is published in the Llama 3 paper.
You usually start from a seed. It can be a randomly picked piece of website/code/image/table of contents/user generated data, and you prompt the model to generate data related to that seed.
After, you also need to pass the generated data through a series of verifiers to ensure quality.
For the knowledge preservation, I guess that a copy of deepseek has most of the required information. But, it would be hard to run it in a primitive world.
But we have landfills which are full of great raw materials. I would argue that it is easier to collect steel from a landfill than from a mine during the industrial revolution.
Come on... Meta has been refining pytorch for more than a decade. It basically contains all that you need to train LLMs, including the latest technologies. What more do you need? The part of the code that is specific to Meta infrastructure?
The reasoning happens in the chain of thoughts. But OpenAI (aka ClosedAI) doesn't show this part when you use the o1 model, whether through the API or chat. They hide it to prevent distillation. Deepseek, though, has come up with something new.
Training a 1B model on 1T tokens is cheaper than people might think.
A H100 GPU can be rented for 2.5$ per hour and can train around 63k tokens per second for a 1B model.
So you would need around 4,400 hours of GPU training costing only $11k
And costs will keep going down.
But, if the non-profit gives all its assets to the new legal entity, shouldn't the new legal entity be taxed heavily? The gift tax rate goes up to 40% in the US. And 40% of the value of openAI is huge.
Except that a plane has passengers. But this rocket had none. It did not even have cargo. And it crashed in a pre-evacuated zone. There is no need to have the same level of security for these two situations.
Counter-intuitively, larger models are cheaper to train. However, smaller models are cheaper to serve. At first, everyone was focusing on training, so the models were much larger. Now, so many people are using AI everyday, so companies spend more on training smaller models to save on serving.