It is not a reasonable assumption to compare your local cluster to the largest clusters within DOE or their equivalents in Europe/Japan. These machines regularly run at >90% utilization and you will not be given an allocation if you can’t prove that you’ll actually use the machine.
I do see the phenomenon you describe on smaller university clusters, but these are not power users who know how to leverage HPC to the highest capacity. People in DOE spend their careers working to use as much as these machines as efficiently as possible.
nvidia absolutely gives deals to national labs and universities. See Crossroads @ LANL, Isambard in the UK, Perlmutter @ LBL. While AMD is being deployed at LLNL and ORNL, Nvidia isn’t done with their HPC game. Maybe not at the leadership level, but we’ll see how Oak Ridge and LANL decide their next round of procurements
HPC, or scientific computing more generally, are slow when it comes to adopting modern software development practices. Most researchers used to only care about results above all, rather than reliability or reproducibility for other users. DOE initiatives [0] in the past couple years have done a lot to change this, though there are significant hurdles from a security and logistical standpoint [1]. For instance, to uncover issues in a PR, you might need to run the code at >64 nodes on a shared system.