These companies fail because vertical integration, and even a monorepo, is needed to make these efforts successful. This is completely at odds with the existing OEM/Tier 1 business model and engineering process grown up around it. Also, neither OEM nor Tier 1 have software cultures up to the challenge.
This is why the Chinese OEMs, Tesla, and Rivian are able to move fast.
Under deadline scheduling, every pending job _eventually_ has highest priority as time elapses. (Assuming new jobs can’t arrive with deadlines in the past.) Every job is eventually serviced.
The “pain” experienced in an overload situations is spread among all late jobs. Contrast this with fixed-priority scheduling, where lowest priority jobs will be starved completely, until the overload is resolved.
In terms of processing raw bytes, inference on raw camera data, bypassing the ISP, is a more relevant application. This can reduce latency in camera inference.
SCHED_IDLE is problematic. There is no priority inheritance with SCHED_OTHER threads. SCHED_OTHER threads block for prolonged periods of time when they contend for a mutex in the kernel with a SCHED_IDLE thread. The SCHED_IDLE thread holds the mutex, but it is unable to complete its critical section because it keeps getting preempted.
That's not strictly true. There are dozens of real-time sleep-based locking protocols that support bounds on worst-case blocking time. Now, even if you have a suitable protocol available to you, your deadlines may be so tight that you can't afford the overhead of context switching. Then yes, you may need a lock-free or wait-free synchronization mechanism.
This is why the Chinese OEMs, Tesla, and Rivian are able to move fast.