I shared the following message about the learnings of using ChatGPT in production under the premise of "this will replace mundane coding tasks" in our Discord. The result is disappointing. I thought it might be of interest to the HN crowd.
# Context
The release of ChatGPT lead to a discussion if ChatGPT could be used to auto generate a config file that takes the source codes structure into account. Manual prompting yielded amazing results, see the screenshots in [0]. But, turning the manual prompting into an automated system led to a massive effort that took >2 months (not full-time, but still a lot of effort!). The idea of "just fine-tune a prompt and done" is, at the moment, far from true. Besides, the result is most likely wasted efforts due to:
a) simplifying the config made the requirement for GPT style "write the config for me" obsolete.
b) hardcoded heuristics are easier to iterate on, more cost efficient (0 costs) and easier to deploy (no API calls needed).
# The message
FINALLY AUTO GENERATION OF THE CONFIG
https://www.loom.com/share/85d77004aa4c4bea9752d959c229e577
.
The irony is that this is not needed anymore. The plugin API simplified the config interface so much that initializing configs only differs by the used plugins and their respective configs.
The only useful thing that chatgpt provides is deriving what files in a filesystem are translation files. But a hardcoded version might be faster and more resilient.
Learning: The GPT hype is ... well hype. The amount of work that went into this feature is insane. The work boils down to software engineering. Sure, tools will emerge that make prompt engineering easier but so does the previous ML hypetrain led to endless nocode machine learning tools. Yet, there is no "disruptive everyone builds ML models" (for GPT "everyone will become a developer") nocode machine learning tool in sight. Maybe signaling that the premise of ML NoCode/GPT of "this works for everything" is not true. Deriving what files are translation files, yep. Deriving an entire config file, maybe not.
# Context
The release of ChatGPT lead to a discussion if ChatGPT could be used to auto generate a config file that takes the source codes structure into account. Manual prompting yielded amazing results, see the screenshots in [0]. But, turning the manual prompting into an automated system led to a massive effort that took >2 months (not full-time, but still a lot of effort!). The idea of "just fine-tune a prompt and done" is, at the moment, far from true. Besides, the result is most likely wasted efforts due to:
a) simplifying the config made the requirement for GPT style "write the config for me" obsolete.
b) hardcoded heuristics are easier to iterate on, more cost efficient (0 costs) and easier to deploy (no API calls needed).
# The message
FINALLY AUTO GENERATION OF THE CONFIG https://www.loom.com/share/85d77004aa4c4bea9752d959c229e577 .
The irony is that this is not needed anymore. The plugin API simplified the config interface so much that initializing configs only differs by the used plugins and their respective configs.
The only useful thing that chatgpt provides is deriving what files in a filesystem are translation files. But a hardcoded version might be faster and more resilient.
Learning: The GPT hype is ... well hype. The amount of work that went into this feature is insane. The work boils down to software engineering. Sure, tools will emerge that make prompt engineering easier but so does the previous ML hypetrain led to endless nocode machine learning tools. Yet, there is no "disruptive everyone builds ML models" (for GPT "everyone will become a developer") nocode machine learning tool in sight. Maybe signaling that the premise of ML NoCode/GPT of "this works for everything" is not true. Deriving what files are translation files, yep. Deriving an entire config file, maybe not.
## Footnotes
[0] https://github.com/inlang/inlang/discussions/408#discussion-4914727