Summarize content of website with provided link
2 comments
There are a few issues:
1. Extract just the text from the page. This means parsing the HTML and getting down to just text without stuff from the sides header or footer getting included. Also if the sites renders via JavaScript you’ll need a headless browser.
2. The amount of text can’t exceed 4000 or so words. 8000 with GPT4 but still some limit. You have to ask for multiple summaries and then summarize the summaries for long articles.
3. Making the api request to openai is not free. So at scale a website providing this service would have to charge for it. Or the other approach is to ask the user to enter their openai token and then it’s their account billed.
1. Extract just the text from the page. This means parsing the HTML and getting down to just text without stuff from the sides header or footer getting included. Also if the sites renders via JavaScript you’ll need a headless browser.
2. The amount of text can’t exceed 4000 or so words. 8000 with GPT4 but still some limit. You have to ask for multiple summaries and then summarize the summaries for long articles.
3. Making the api request to openai is not free. So at scale a website providing this service would have to charge for it. Or the other approach is to ask the user to enter their openai token and then it’s their account billed.
Hey I can help I just made a similar tool myself
What is the recommended approach?
Any GPT API we can use directly?
I have spent lots of time researching it, but cannot find any API we can use. OpenAI's AI seems not helping too.
We hope to find some API to summarize the content of website with specific website link. (I am willing to pay)