rustdeveloper · HackerTrans

rustdeveloper

no profile record

Submissions

Bulk Google Index Checker

serpnames.com

1 pointsby rustdeveloper3 lata temu0 comments

comments

w zeszłym rokudiscuss

Happy to suggest another web scraping API alternative I rely on: https://scrapingfish.com

2 lata temudiscuss

Interesting data for sugar in products scraped from Walmart: https://scrapingfish.com/blog/scraping-walmart

2 lata temudiscuss

This is correct, my friends from Scraping Fish are hosting https://compareproxy.com to help people find proxy for web scraping. I'm happy to "push" for Scraping Fish as I'm also a satisfied user who received a lot of help from the founders for my web scraping projects.

2 lata temudiscuss

And many more: https://compareproxy.com/

2 lata temudiscuss

For web scraping at scale you want to get lost in the crowd. This usually means being (or pretending to be) chromium on windows. Unusual browsers are suspicious, detected or have very distinct fingerprint.

2 lata temudiscuss

This is a terrible news :( I know it was an option for web scraping and I used in once. I’m curious what is the real reason they took it down.

2 lata temudiscuss

Surprisingly, according to you tool, HN is neutral on “web scraping”. I noticed others also reported bias for neutral on other keywords.

2 lata temudiscuss

There are also SaaS products with usage based pricing. It depends on what SaaS or software or product it is. Different pricing model works for different things.

2 lata temudiscuss

Actually, they do allow this. I store my photos and iPhone backup on synology NAS. You just have to devote some time to set it up yourself.

2 lata temudiscuss

[flagged]

2 lata temudiscuss

I'm using Scraping Fish because of their pay-as-you-go style pricing as opposed to subscription with monthly scraping volume commitment. And they don't charge extra credits for JS rendering or residential proxies because the cost of each request is the same: https://scrapingfish.com

3 lata temudiscuss

This reminds me of https://scrapingfish.com/blog/are-most-rust-jobs-in-crypto :)

3 lata temudiscuss

Do we know how LLMs available in OpenLLM and other open source LLMs compare to different versions of GPT models? I know there’s a leaderboard on huggingface: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb... but it doesn’t contain GPT models.

3 lata temudiscuss

Does it give access to the internet? Could it be used with this setup: https://scrapingfish.com/blog/byo-mobile-proxy-for-web-scrap... ?

3 lata temudiscuss

For $20 you can get over 10k requests from scraping fish: https://scrapingfish.com/buy

3 lata temudiscuss

I don’t see how any LLM would help me with a high quality proxy, which is what I actually need in web scraping and I’m using https://scrapingfish.com/ for this.

4 lata temudiscuss

There are tutorials for building you own mobile proxy pool so it’s very accessible these days: https://scrapingfish.com/blog/byo-mobile-proxy-for-web-scrap...

4 lata temudiscuss

I'll be working on my own mobile proxy build: https://scrapingfish.com/blog/byo-mobile-proxy-for-web-scrap...

4 lata temudiscuss

I initially built a system for web scraping but was constantly running into issues of getting blocked, even when using good quality residential proxies. I had to constantly investigate why I'm getting blocked and update tools. Sometimes the effort was significant when I had to switch to a different framework which was giving me a better success rate.

Then, I switched to web scraping API (I'm using https://scrapingfish.com as they have convenient pricing for my use case, but there are other alternatives). Now I only have to maintain parsing logic in scrapers. It also actually reduced my costs of scraping since I no longer pay for proxies which are more expensive for my scale than a web scraping API.

4 lata temudiscuss

This looks really cool! There was a tutorial posted on HN about building mobile proxy pool with RPI that had obvious limitations: https://scrapingfish.com/blog/byo-mobile-proxy-for-web-scrap... It seems this could be a solution to scale capabilities of a single RPI.