We are looking for such a solution combined with data available via a Google datastudio connector. Could be an awesome feature to attract agencies as customers, who need to report to their clients.
I think that you are at least 20 years late for building a real domain business. I have been into in that game for more than 15 years and yes, it is fun, but it's not a (profitable) business. And if you want to make it profitable, it's not even close to a sidebusiness.
Thank you very much for sharing your experience with HN. From my own point of view (as the CEO of a performance marketing agency working with Google Ads & clients budgets daily) I have to add, that this are learnings your agency should have protected you from.
Showing the bot something different than the user could be seen as cloaking. We also do that in some cases. Just be really carefull with that strategy.
Depending on the product, i would suggest option 1 or 3. Maybe you can find a limitation for the free (lifetime) version, where the free users can help you to get some traction. (Like trello.com does)
I have already built several sideproject over the past 15 years. For me the hardest part as a marketer is to find a developer, mostly freelancers, who are willing to "dive into" the idea and work with the same passion as i do on the project, even if they are usually getting paid by hour.
The biggest issue for me was finding a suitable developer who is willing to grow the one ore another project together with me. So if any of you guys is interested in a marketing partner and willing to invest his time into a project and split the revenue, i would be happy to get in tough with you.
Now let´s come you back to your question - how to come up with an idea? I will discribe you my way:
note down every "problem" somebody is telling you. No matter if it means "find the cheapest flight price", "would be cool if tool X could do Z" or if somebody tells you a terrible workflow within his company. In addition to that, we are all surfing the web all day long, so simply also note down things you like somewhere, and make screenshots / screencasts, if something is really great.
when you do this some days / week, you will generate hundreds of cool ideas within a very short period of time. but the idea itself is worth nothing at all. the idea itself is just an idea.
you can definitely rank as a small business as well in the organic search results, without any adwords budgets, but SEO is not as easy as it was 6-8 years ago - that´s for sure. if you are interested, i can have a look at your company´s site and provide some ideas / feedback, if you want to.