Yeah I agree with this. I will try to use it in earnest on my next project.
That metric is the key piece. I don't know the right way to build an automated metric for a lot of the systems I want to build that will stand the test of time.
I think the real problem with using DSPy is that many of the problems people are trying to solve with LLMs (agents, chat) don't have an obvious path to evaluate. You have to really think carefully on how to build up a training and evaluation dataset that you can throw to DSPy to get it to optimize.
This takes a ton of upfront work and careful thinking. As soon as you move the goalposts of what you're trying to achieve you also have to update the training and evaluation dataset to cover that new use case.
This can actually get in the way of moving fast. Often teams are not trying to optimize their prompts but even trying to figure out what the set of questions and right answers should be!
I've used browser rendering at work and it's quite nice. Most solutions in the crawling space are kind of scummy and designed for side-stepping robots.txt and not being a good citizen. A crawl endpoint is a very necessary addition!
Lots of companies have automations with Zapier etc. to upload things like invoices or other documents directly to notion. Or someone gets emailed a document with an exploit and they upload it.
I don't disagree it is unsustainable, I'm really trying to be more precise about whether anyone is getting value out of the tools. I'm just really skeptical that nobody is getting value.
The article does mention that OpenAI has huge revenue.
> While The Information reported that OpenAI's revenue is $3.5 to $4.5 billion a year in July, The New York Times reported last week that OpenAI's annual revenues have "now topped $2 billion," which would mean that the end-of-year numbers will likely trend toward the lower end of the estimate.
But then the author claims that the business value is questionable.
> And even if they did, it isn't clear whether generative AI actually provides much business value at all. The Information reported last week that customers of Microsoft's 365 suite [snip]
I would have appreciated a deeper discussion of why OpenAI's revenue isn't a data point toward generative AI having some business value. Presumably if nobody was using generative AI in a way that gives them value, OpenAI wouldn't be using all those GPU hours. That's what I was missing from the article personally.
I always find it really strange when articles like this claim nobody is paying for generative AI. I can't find reliable stats on this but there are at least a million ChatGPT Plus subscribers. Does that not count?
It takes time for new advancements to get proliferated through the economy.