The most comprehensive tool I've built so far was a JSONPath playground [0] when I was working on a game where I found myself writing a fair bit of JSONPath (well, JSONPath Plus specifically) expressions by hand and wanted to be able to test them out in the same way that I would on regex101 when writing regular expressions.
I realistically probably would have only saved myself less than an hour of crafting those expressions if the tool had already existed (with this level of detail, there are lots of many simpler ones already for it), but I would have spent a solid 40-50 hours of bouncing between manually crafting and writing detailed instructions to direct the agents to get this tool there.
I work at SerpApi [0], and we offer a free tier that may serve your needs if you're just looking to do programmatic searches periodically.
Much of the reason people go with a service like ours is because of the difficulty with rolling your own reliable solution. Happy to answer any questions you might have as well!
The most recent project I’ve been working through has been a tool for JSON query evaluation and debugging [0] inspired by how easy regex101 is to use.
I couldn’t find any that were as nice or as powerful to use for writing JSONPath queries, so instead of spending an hour crafting and testing them manually, I spent >40 hours building this tool to save myself half an hour.
I've got a few things on my plate that I bounce between building for fun:
1. A point-and-click adventure game: making it with incredibly heavy technical constraints "just because"
2. A coding puzzle game of rapidly escalating difficulty
3. Part of #2 had me needing to craft some JSON Path queries and I felt like there wasn't anything nice to build and test them with, so I built this tool for it (inspired by the amazing regex101): https://jsonpath101.com/
4. A website where I write about text-based browser games
I don't mean that users are following the links to `acme.com` and `demo.com` type domains in documentation; I mean that bots are likely finding and following many links to them because of their widespread use in documentation.
If you search for `site:github.com "acme.com"` in Google, you'll find numerous instances of the domain being used in contrived links in documentation as an example of how URLs might be structured on an arbitrary domain and also in issues to demonstrate a fully qualified URL without giving away the actual domain people were using.
This means that numerous links are pointing to non-existent paths on `acme.com` because of the nature of how people are using them in documentation and examples.
I suspect part of the issue is that people are still using things like `acme.com` and `demo.com` as an example domain in their documentation and tests instead of relying on `example.com` which is reserved exactly for this purpose [0]
SerpApi | https://serpapi.com | Junior to Senior Fullstack Engineer multiple positions | Customer Success Engineer | Hiring Coordinator | Python/Ruby/PHP/Js/Rust/Cotlin/C#/Crystal/Nim/Elixir Developer Advocate positions | Based in Austin, TX but remote-first structure | Full-time | ONSITE or FULLY REMOTE | $150K - 180K a year 1099 for US or local avg + 20% for outside the US
SerpApi is the leading API to scrape and parse search engine results. We deeply support Google, Google Maps, Google Images, Bing, Baidu, and a lot more.
Our current stack is Ruby, Rails, MongoDB, and React.JS. We are looking for more Junior and Senior FullStack Engineers. We have an awesome work environment: We are a remote first company (before Covid!). We do continuous integration, continuous deployments, code reviews, code pairings, profit sharing, and most of communication is async via GitHub.
We value super strongly transparency, do open books, have a public roadmap, and contribute to the EFF.
SerpApi | https://serpapi.com | Junior to Senior Fullstack Engineer multiple positions | Customer Success Engineer | Hiring Coordinator | Python/Ruby/PHP/Js/Rust/Cotlin/C#/Crystal/Nim/Elixir Developer Advocate positions | Based in Austin, TX but remote-first structure | Full-time | ONSITE or FULLY REMOTE | $150K - 180K a year 1099 for US or local avg + 20% for outside the US
SerpApi is the leading API to scrape and parse search engine results. We deeply support Google, Google Maps, Google Images, Bing, Baidu, and a lot more.
Our current stack is Ruby, Rails, MongoDB, and React.JS. We are looking for more Junior and Senior FullStack Engineers. We have an awesome work environment: We are a remote first company (before Covid!). We do continuous integration, continuous deployments, code reviews, code pairings, profit sharing, and most of communication is async via GitHub.
We value super strongly transparency, do open books, have a public roadmap, and contribute to the EFF.
I like the idea of it; though I think the latest update (around puzzle solveability?) may have broken the game as I'm unable to begin in any browser I've tried:
Uncaught TypeError: can't access property "puzzleLog", G is null
If you're interested in giving the tool a shot, feel free to shoot me an email (take my username and insert an @ after david and a .com at the end) and I'll happily give you access after I get it up somewhere publicly accessible -- possibly in the next couple of weeks.
We have ASIC (Australian Securities & Investments Commission) that handles company registrations, notices, etc. and they maintain a register of insolvency notices and liquidations.
This can be a reasonable place to go to look for distressed businesses/assets too and I've considered using them as a source with my aggregation/search engine, though they don't really have the same type of information as a business for sale listing so they fall somewhat outside of the main type of results that I otherwise display.
Other reasonable places I've seen too, though in incredibly low volumes (think 0-3 listings a month), are commercial auction houses/sites where they'll list a business for sale or the full assets of a business. The main issue with that it is that they're so low volume that I'm not sure it's worth spending the time ingesting them this early on while there's still many other larger listing sources.
In Australia, brokers need to be licenced real estate agents (even as a buyers agent in many states), so I suspect there's a relatively decent culture of people who are serious about selling their business going through brokers and resulting in listings on at least one of the major platforms.
There's just shy of 90,000 unique listings I'm tracking (i.e. after de-duplication) on these platforms.
On the traditional classifieds sites and things like Facebook groups focusing on these, there's a significantly smaller number of listings/ads for business sales (e.g. a couple of thousand).
I think where there are definitely hidden gems is where there are many small business owners at or close to retirement age where they haven't planned for a sale at all. For example, a family member nearing retirement age has a small business they're just intending to shut down because they "couldn't be bothered" selling it. I've heard people have had reasonable success just approaching local businesses like this that have older owners OR asking accountants if they have any clients that are thinking of selling.
I've been working on a "businesses for sale" aggregation/search engine that sources data from all of the major "business for sale" type platforms in Australia and de-duplicates listings, extracts data like revenue/profit/etc, and normalises it all for quick browsing.
I have a couple of family members and friends who are looking to buy businesses (separately), and it's been much more time-consuming than you'd expect just to browse through listings to determine if they're relevant to you or not.
The platforms seem to mostly follow the same format as real estate listings (as the brokers seemingly rely on the same software/data formats), with one big blob of freeform text that contains the various information that you'd ideally just be reading at a glance.
Add to the fact that there are over 15 "business for sale" type platforms in Australia where they have a minimum of 1,000 listings and at least 10 platforms with between 100-1,000 listings, you can easily burn hours looking through them individually.
I'm currently covering 12 of the top 15 (ranked by number of listings they contain) platforms and I just tinker away once or twice a month, adding support for new platforms.
I should probably release it and get some feedback at some point, but I suffer a bit from "it needs more polish before I let people other than my family and friends use it"
I've been working on and off on a client-side* SERP rank tracker: https://serpowl.com/
I wanted a simpler alternative to the self-hosted SerpBear tool that I could use and share, so this is the result.
It uses SerpApi (where I work) as the data source for what actually executes the SERP scraping because it's much too complex to have purely client-side, but 100% of the rank tracking portion is client-side.
It's not fully complete and there's definitely rough edges with it, but because of the data source, it supports a large number of search engines right off the bat.
SerpApi | https://serpapi.com | Junior to Senior Fullstack Engineer positions | Customer Success Engineer | and more... | Based in Austin, TX but remote-first structure | Full-time | FULLY REMOTE or ONSITE | $150K - 180K a year 1099 for US or local avg + 20% for outside the US
SerpApi is the leading API to scrape and parse search engine results. We deeply support Google, Google Maps, Google Images, Bing, Baidu, and a lot more.
We are still hiring - we have so much work and we're after great people to help out! I only started at SerpApi late last year myself and can't recommend it enough.
We do continuous integration, continuous deployments, code reviews, code pairings, profit sharing, and most of communication is async via GitHub.
Our current stack is Ruby, Rails, MongoDB, and React.JS. We are looking for more Junior and Senior FullStack Engineers. We have an awesome work environment: We are a remote first company (before Covid!).
We very strongly value transparency, do open books, have a public roadmap, and contribute to the EFF.
When you apply, please mention that you saw David's post on HackerNews.
If you've previously applied and didn't make it through, please feel free to reapply if it's been a while and you think you would make a better fit than previously.
You can contact me at david at serpapi.com if it's been a few days and you think your application may have fallen through the cracks.
Making it harder for bots usually means that it drives up the cost for the bots to operate; so if they need to run in a headless browser to get around the anti-bot measures it might mean that it takes, for example, 1.5 seconds to execute a request as compared to the 0.1 seconds it would without them in place.
On top of that 1.5 seconds is also that there is a much larger CPU and memory cost from having to run that browser compared to a simple direct HTTP request which is near negligible.
So while you'll never truly defeat a sufficiently motivated actor, you may be able to drive their costs up high enough that it makes it difficult to enter the space or difficult to turn a profit if they're so inclined.
Very impressive work! I always enjoy a good write up about reverse engineering efforts and yours was really simple to follow.
Many popular/large websites and bot protection services usually have environment checking as a baseline and mouse-movement tracking in some of the more aggressive anti-bot checks.
It's always interesting to see how long it takes from when the measures have been defeated/publicised until the service ends up making changes to their mechanism to make you start over (hopefully not from scratch).
It's currently just where I'm posting miscellaneous experiments I've tinkered with (mainly just about trying to abuse CSS at this stage)