This sort of thing has been building up for a long time. This is mainly due to the Greed of the SF tenants to who prevent developers from building more units with local politics NIMBY. By constricting the supply artificially it prevents new buildings from going up. Also, the mayor is to blame as well, for incentivizing more companies to come to SF making the situation even worse.
I could imagine a middle school dropout with an interest in programming, applying for that salary. That would be a great salary to make for a middle school dropout.
I've lived in the bay area for 10+ years as a Software Engineer and follow all the local news. But i don't get the sense that the things you mentioned are more popular than the other stuff you said you like. There's lots of people here who like talking about politics, money, science and drinking beer. It's not nearly as white and black here as you imagine.
Your suggesting Academic Researchers should be consultants for VCs. Sometimes they are! but, just not full time. Usually, they might be on a board of advisors, for a small amount of stock, or maybe as an unofficial advisor.
Now let's cover why not. Simply put: academic research often is only tangentially related, and thus of limited value. The people with the most direct expertise in the area in question, are the ones who are actually doing the technology they're developing, on a day to day basis.
Hiring, often times has biases. There could be any number of reasons: either technical or personal. Hiring decisions are often subjective, despite the fact that they try to be somewhat objective. And it's especially hard for people with less than 5 years of experience. Keep trying, you'll get it eventually.