Ha, fair point — "every website" is the vision, 6 is the MVP :)
The adapter model is designed so adding a new site is a single YAML file (~30 min of work, or ~3 min with a coding agent). No Python code needed for most sites. PRs welcome if there's a site you'd want to see!
Hey HN, I built this because my AI agent was spending 8 seconds and 300MB of RAM just to search X. That felt wrong — the data is right there behind one HTTP request, but the "standard" approach is to launch a full browser, render the page, and scrape the DOM.
web2cli makes direct HTTP requests using your browser cookies. No Chromium, no Selenium, no headless anything. The tricky part was TLS fingerprinting - Cloudflare blocks Python's default TLS stack (JA3 fingerprint mismatch), so web2cli uses curl_cffi with BoringSSL to impersonate Chrome at the TLS level. X.com was even harder - their search requires a cryptographic nonce
generated by obfuscated browser JS, which the community reverse-engineered.
Six adapters today: HN, X, Discord, Slack, Stack Overflow, Reddit. Each adapter is a YAML file - writing a new one takes ~30 minutes (or ~3 minutes for your coding agent) and doesn't require Python code for most sites (although it's possible to add a custom python provider, like I did with X).
I'm working on web2cli Cloud - think "OAuth for sites that don't have OAuth." Your users log in via a sandboxed browser, your agent gets an opaque session token, cookies never touch your server.
Happy to go deep on the adapter architecture, anti-bot bypasses, or the economics of browser automation vs direct HTTP.
This is very interesting write up. Kudos for the author for completing it and showing the whole process. Nevertheless, coding skills were still necessary.
Right now I'm working on a tool to achieve such kind of automation of a website, but without code. Mostly without the glitch part from the post. Working on any kind of hosting, no matter what underlying technology - WordPress, GH pages, whatever. With interface similar to IFTTT where you connect 3rd party apps to a website to populate content - like in the example from the post - data from Instagram on the website. Or data from Slack bot on website (D&D example).
UX for this project is not an easy stuff. Also, we are thinking what services are most important to be on the platform at the very beginning. We would love to hear any kind of feedback. If you are interested find my email in my bio.
Replying to the child comment from whorleater - that's a good catch! Our mistake with the domain redirects and certs. Will fix it ASAP. Thanks for spotting this bug.
If you don't like to play with CLI and want to host files on Amazon's S3, go with the Sheetsu Pages [1] (I'm co-founder).
- It's uses Amazon's S3 CDN, actually your site is deployed to their all edge locations
- supports https
- you can add your custom domain
- you upload files to Google Drive, click sync button and it's done
The adapter model is designed so adding a new site is a single YAML file (~30 min of work, or ~3 min with a coding agent). No Python code needed for most sites. PRs welcome if there's a site you'd want to see!