Are the inputs and the responses in a non English language? LLM APIs can get costly for non English, sometimes as much as 10x due to more tokens being consumed. Not sure what's the solution here.
Also, maybe you can use sometime kind of caching combined with some mbeddongd search to serve the previous response, if the input is similar above a certain threshold.
"All employees below Principal Engineer, grades 7 to 11, will get a 5% cut, 10% cuts will be instituted for VPs, and the executive leadership team will take a 15% cut, with Pat Gelsinger taking a 25% cut. "
So the OP would rather have Intel fire thousands of employees?
I spent few weeks last year building a text to sql tool using codex model to do something like this but for all kinds of data sources. We pivoted away to something else for various reasons.
But your approach is much better. Pandas is used a lot. Build a tool on top of pandas. This is awesome.
AI chatbots(or equivalent AI answering machines) vs Search Engines
Search has 2 kinds of users.
1 trawls through multiple pages of results to find what they are looking for. This is you and me but we are in minority.
2 clicks on the first link(or sometimes 2nd) and that's the end of that particular search. The majority.
2nd will decide the winners
Academics, being the 1st kind are the reasons for arguments like those in the article. They are not wrong but to the majority, it won't matter.
And I am willing to guess the Code Red at Google is about the 2nd type of users. They are the bread and butter.