GCP is pretty pleasant overall. The API, command line, and UI are all pretty good. Vast majority of functionality is supported in all three places too. Like all providers, they have a few services that suck but overall Google has done a great job at developer experience.
There are several use cases where ML can help even if it isn't perfect or even just better than random. Here is one example in NLP/search.
Let's say you have a product search engine and you analyzed the logged queries. What you find is a very long tail of queries that are only searched once or twice. In most cases, the queries are either misspellings, synonyms that aren't in the product text, or long queries that describe the product with generic keywords. And the queries either return zero results or junk.
If text classification for the product category is applied to these long tail queries, then the search results will improve and likely yield a boost in sales because users can find what they searched for. Even if the model is only 60% accurate, it will still help because more queries are returning useful results than before. However you don't apply ML with 60% accuracy to your top N queries because it could ruin the results and reduce sales.
Knowing when to use ML is just as important as improving its accuracy.
If you can write a SQL query or a set of SQL queries to do your transformation, then you can use DBT. DBT doesn't do transformation itself rather it helps you manage all the dependencies between your SQL models. Whether you can use SQL depends on your data and database/warehouse functionality. For example, JSON parsing support is pretty good now in many databases and warehouses. If your objects can be represented as JSON, then you could write SQL via DBT to parse the objects into columns and tables.
DBT has two main innovations. First, everything is a SELECT statement and DBT handles all the DDL for you. You can handle DDL yourself if you have a special case too. Second, the ref/source macros build a DAG of all your models so you don't have to think about build order. There are other innovations but those are the main ones.
You can give it truly pure SQL in both models and scripts, and mixing in Jinja if you need it for dynamic models. But I'd recommend at least using ref/source.
I think using a data warehouse as your data lake or lake house is optimal. Even for data that isn't relational. Storage is so cheap now and is decoupled from compute costs for several providers that I don't even give it a thought. You get a fast, scalable SQL interface which is still nice and useful for non-relational data. Then all, or most, of the transformations needed for analysis can be pure SQL using a tool like DBT. In my experience, it greatly simplifies the entire pipeline.
For my search use case, documents are mostly single topic and less than 10 pages. However I have found embeddings still work surprisingly well for longer documents with a few topics in them. But yes, multi-topic documents can certainly be an issue. Segmentation by sentence, paragraph, or page can help here. I believe there are ML-based topic segmentation algorithms too, but that certainly starts making it less simple.
I guess it depends on what you mean by "simple". The algorithms are complex but there are good tools that implement them. I would imagine smaller companies would use off the shelf tooling, and I would argue that is simpler. Vector embeddings are so unbelievably powerful and often yield better results than classical methods with one of the good tools + pretrained embeddings.
Specifically for search, I use them to completely replace stemming, synonyms, etc in ES. I match the query's embedding to the document embeddings, find the top 1000 or so. Then I ask ES for the BM25 score for that top 1000. I combine the embedding match score with BM25, recency, etc for final rank. The results are so much better than using stemming, etc and it's overall simpler because I can use off the shelf tooling and the data pipeline is simpler.
Yes I know for sure. Postgres search is essentially an easier to use regex engine. If you have a recall-only use case and/or a small dataset, then that works great. As soon as you need multiple languages, advanced autocomplete, misspelling detection, large documents, large datasets, custom scoring, etc you need Solr or ES.
I disagree based on the trends, especially in the US where few, if any, new coal plants are being built and most existing ones are scheduled to be converted to natural gas or shut down. But let's assume that a significant amount of coal will continue to be burned over the next 20 years. Do you think we should stop innovation in other sectors until we are off coal?
I grew up in Cincinnati. You are correct, however traffic is bad enough now that a subway or light rail would make a big difference. Unfortunately it is unlikely the voters would approve a tax increase to support that.
I personally try to make decisions using probability. I understand where you are coming from but there is one factor missing in your analysis: this situation is literally life and death. That changes the math a bit to something more akin to "better safe than sorry" in my opinion. We have found a guaranteed path out of this mess. There may be other faster paths that save more lives, but it could also end up killing millions more too. I'm all for experimenting but I take issue with making the experiment the policy when lives are on the line.
We have no data for mRNA vaccines except those trials. We don't know if the historical data on traditional vaccines are applicable here. As such, it is also possible that a single dose isn't good enough or a single dose is good enough but only for a couple months. We just don't know and it would be way worse if it turns out we have to re-vaccinate everyone because we were impatient.
Yeah I agree, I would like to see some percentage be lottery based. I think you should still need to write a proposal, but if the proposal isn't selected then it goes in the lottery pool. There are two reasons to still write the proposal: helps organize the researcher's thoughts and shows the researcher is serious.
I think a better analogy would be living near an airport like I do. Depending on the day and weather, even my small airport is quite loud and annoying. My neighbor's motorcycle is also louder than the planes, except the occasional military jet, but that noise is far less frequent than the planes. I think drone delivery will mean everyone in cities and suburbs will feel like they live near a small airport on a clear day. I too am not looking forward to the flying drone future. I think we should be working on reducing noise as much as possible, not create more in the name of slightly more convenient consumerism.