"How can we be sure that our radar sensor does not trigger an unintentional break due to a plastic bag blowing in the wind? We collect 10000h of real world driving evidence."
10,000 hours of real world driving data is probably several orders of magnitude too little? Would that really give you confidence that a system can handle a plastic bag blowing in the wind?
I would have thought that this specific scenario would also depend more on factors such as: likelihood of the object being a person or animal vs inanimate, estimated cost of impact to the object and car, and secondary consequences of a rapid deceleration of the car.
3. Read blogs like "The Morning Paper" (https://blog.acolyer.org) but skip fields that are outside your scope. You don't have time to follow more than one or two (or three) major fields.
4. Use Google Scholar to find the most cited papers, or to find papers that build on papers you think are good.
5. Keep an eye out for the conferences where these papers were presented. Then read the other papers that were also presented.
6. When you come across an amazing paper, read other papers by the same authors or supervisors.
7. If you're lucky you might also find good "survey" papers that cover and reference the state of the art.
8. Lecture notes from Stanford or MIT or another university can also be a great way to get a big picture of the evolution of techniques for a given data structure or problem. For example, these lecture notes are just brilliant for getting started with stuff around memory hierarchies: https://www.eidos.ic.i.u-tokyo.ac.jp/~tau/lecture/parallel_d...
These are a few tricks that I find useful. What else?
Here are some ways to learn deeply and be relevant:
1. Pick a data structure (such as a hash table or LSM-Tree) then read all the literature there is to read, every single paper that's great, following the best conferences year after year, and implement a 10x faster or more scalable version for the std lib of your favorite language.
2. Pick a fault model (such as storage faults, network faults, cryptography faults) then read all the literature there is to read, every single paper that's great, following the best conferences year after year, and write a fault injection or fuzzing harness to break some of the most respected storage/network/cryptography systems (for examples, see the work done by Remzi and Andrea Arpaci-Dusseau on storage faults, Kyle Kingsbury on Jepsen, and Guido Vranken on Cryptofuzz: https://github.com/guidovranken/cryptofuzz).
3. Pick a software field (such as web applications, mobile applications, native applications, file formats such as Office Open XML, or protocols such as SMTP, MIME, HTTP, QUIC) then read as many CVE reports and bug bounty reports as you can find, and then start participating in bug bounty programs within this field. Pick a target and give yourself a goal, e.g. DoS, RCE or read/write access, and do the work to make it happen. Chain as many steps as you can. Automate and enumerate. You'll find a way in if you keep at it. There's nothing like crafting an exploit to change the way you think about programming.
As you gain experience in data structures, storage/networking/cryptography, and security, you'll find this translates well to most software engineering work. You'll gain a speed/safety/security way of thinking, you'll have fun being curious and learning along the way (and hopefully you'll earn a bounty or two and get some CVEs under your name).
I think this comes down to realizing that everything in life is eventually hard. Provided you're doing something worthwhile [1], when the going gets hard, don't look for easier pastures. They're all difficult. Just keep plowing away. And don't doubt. Where there is no doubt, the wind is your companion.
[1] Ecclesiastes is a pretty great book to help figure out for yourself what exactly this is. It's not for nothing that the book uses the same word for "meaningless" or "empty" or "vanity" or "vapor" or "breath" as its refrain. It's also a beautiful book in itself as literature: https://en.wikipedia.org/wiki/Ecclesiastes#Influence_on_West...
"just cranking out variations of that formula as papers."
You'd be surprised how often variations on a formula are all it takes to destroy security. Security is enumeration at the end of the day. If it takes a paper mill to "count all the ways" then I'm grateful.
As someone who loves C, Zig has replaced C for me and I don't plan to go back.
There are just too many benefits, not least developer velocity and readability.
Zig is also very approachable to anyone who is familiar with TypeScript, in a way that C isn't. This means more of your team can get excited and get involved, at least in terms of reading the code.
And with Zig, where you have to spend a little extra maintenance time paying for the language being pre 1.0, that's still nothing compared to the upfront and long-lived costs associated with C. You can develop so much faster in Zig compared to C, that you're ahead even pre 1.0.
In the general sense, no, not yet, of course, but in one sense, Zig is probably more production ready than Rust, for example on embedded systems with obscure architectures where OOM-safety is essential.
Here is a talk on using Zig in exactly this kind of environment in production:
Raft is essentially Multi-Paxos under the hood, with some overstrict invariants that can impact availability and latency. If you need a consensus algorithm, then it's worth doing a survey of the literature beyond Raft.
Other Multi-Paxos variants are Viewstamped Replication Revisited, which has deterministic leader election for lower latency in the common case and thus no issue of split votes, and which I personally find easier to understand than Raft, also making less demands of the underlying hardware (i.e. VRR can do in-memory leader election, so you're safe in the event of disk failure, corruption, lost writes, and misdirected writes, which Raft does not deal with).
For all these Multi-Paxos variants, you can also look at Flexible Paxos, which is a simple technique for making leader election quorums slightly more expensive (since leader elections are infrequent) in order to make replication quorums much cheaper (since they are the common case critical path). You can use Flexible Paxos to save 20% in the number of hardware nodes for the same common case failure tolerance. Flexible Paxos is a game changer.
I would say quality is undervalued when it comes to technology. Technology suffers from a plague of bloat, and it's getting worse as high level languages pile on the dependencies and abstractions, all the while justifying the waste with "quality is overrated".
I find it hard to believe that you can ship a high quality product without caring about the quality of the components going into it, or that this won't cost you down the line. It only takes a few "good enoughs" in terms of probability theory to end up with a multitude of poor product tail results.
It's also not enough to get the job done, you need to maintain what you ship. You don't need to choose the absolute best tool, but choosing quality tools pays dividends over time.
I don't think it's a question of "boring" or "proven" vs "latest" or "newer". It's not even an 80/20 split between "use proven technology" and "explore new tools".
What it all comes down to at the end of the day is "quality".
You need to have an eye for quality, you need to be able to recognize good design decisions when you see them, appreciate their worth and weigh their potential. You need to be able to make up your mind for yourself.
If you can spot quality in a language, framework (assuming you even need one) or database, then "boring" or "proven" become meaningless phrases.
Quality transcends age and hype.
When you can spot quality far off, you can paddle out to the swell, be there as it breaks, and enjoy a long quality ride that's not cut short because of irrelevance. You won't need to see thousands of surfers on a wave to know it's a quality break.
With technology, the earlier you can spot quality, the sooner you can start riding the wave and benefitting from everything that comes with that (long term experience with that technology, deep understanding, the opportunity to shape or introduce key design decisions at a critical stage, learning from new ideas that can compound earlier into your own output, new connections and marketing opportunities). That's how it always works. There's no point catching a wave because everyone else is riding it. By then, you probably need to be paddling out for the next big swell.
To be clear, this is more than "skating to where you think the puck will be". This is looking for quality and having an eye for it, and being sure of it when you find it.
Is there any syscall to do the equivalent but from C?
e.g. If you were pre-allocating hash tables for a caching service, it would be convenient if the config could stipulate 80% of the system memory instead of the user having to specify the exact amount.
Cool! I had a similar empirical experience working on a Cauchy Reed-Solomon encoder awhile back, which is essentially measuring xor speed, but I just couldn't get it past 6 GiB/s per core either, until I guessed I was hitting memory bandwidth limits. Only a few weeks ago I stumbled on the actual formula to work it out!
It means if a system can only do X of something per second, then if you push the system past that, new arriving stuff has to wait on existing work in the queue, and things take longer than if the queue was empty. You can think of it like a traffic jam and it applies to most systems.
For example, our local radio station here in Cape Town loves to talk about "queuing traffic" when they do the 8am traffic report, and I always think of Little's law.
Bufferbloat is another example of queueing delay, e.g. where you fill the buffer of your network router say with a large Gmail attachment upload and spike the network ping times for everyone else sharing the same WiFi.
10,000 hours of real world driving data is probably several orders of magnitude too little? Would that really give you confidence that a system can handle a plastic bag blowing in the wind?
I would have thought that this specific scenario would also depend more on factors such as: likelihood of the object being a person or animal vs inanimate, estimated cost of impact to the object and car, and secondary consequences of a rapid deceleration of the car.