Just keeping it up to date with competitors is much cheaper, by copying better ones like Qwen did with Claude. Also a bunch of research is trickling into open source / arxiv so catching up should continue becoming cheaper at least as a fraction of training from scratch
maybe you can preselect good ideas, build up guidelines describing most common pitfalls, extrapolate from ideas you already vetted etc and run on autopilot on a safe-ish subset
Nvidia's parakeet dropped recently with better performance and 0.6B params, so the rate of progress here looks good, probably next year (or mby the year after) they'll be running no probs
Not really answering your question, but:
One completely imo unnecessary category of sloppy software is electron apps.
It's totally ridiculous how little resources are put into alternatives like tauri given how most dekstop apps run on electron, and we know how bad it is.
I'm not disagreeing per se, but in my experience my tests are so disposable they're never worth investing that much engineering effort in past getting them to pass
The author makes an argument that at least looks like people choose YT over Nebula because YT is free. I, for example, already pay for Nebula, so I can watch it for free, but I still go to YT.
IMO it might be just a product problem. I opened nebula and:
* The same video had a better title on YT that was actually less clickbaity and more informative - assumedly because of YT algorithm for optimization
* Nebula auto set quality to 480p compared to 1080p in YT - if I wasn't tech-savvy I'd assume it's just worse quality.
* The loading times when you seek to part that's not loaded yet are 10x longer
* I missed comments
The recommendation algorithm is weaker too, I can't tell to what extent this is due to YouTube having simply more data and to what extent it's weaker engineering.
If they do some decarbonization real estate SaaS and need connections with both regulators or whoever issues certificates and REITs etc, then it makes sense. I don't think they need a lot of software for that.
To expand upon the other comment:
Indexing and multiplying with one-hot embeddings are equivalent.
IF N is vocab size and L is sequence length, you'd need to create a NxL matrix, and multiply it with the embedding matrix.
But since your NxL matrix will be sparse with only a single 1 per column, it'd make sense to represent it internally as just one number per column, representing the index at which 1 is. At which point if you defined new multiplication by this matrix, it would basically just index with this number.
And just like you write a special forward pass, you can write a special backward pass so that backpropagation would reach it.
In a 20-person office you'll pass by like 2 people on the way with the card, so it's not a problem. If the office was bigger, and it was a problem, you could just designate a spot for the card next to the pantry.
His argument is that it's effectively a legal moat now that protects monopoly. Like we shouldn't accept that you need to break the law to have a chance to compete with them.
I thought about it for a full day, and I have one idea for how to handle copyrighted data training.
It would need to be open / regulated and training till double descent would need to be disallowed, to make sure that the model is not memorizing the data.
I think this was overly self-critical - what would researching fully even mean? They can have however many sheep they want hidden within one-mile distance from Trafalgar square, no-one would expect the author to scour every possible location ensuring that there is no sheep hidden lol just to make a funny post 100% sure to be true