Meanpath - Search the page source, server headers and visible text on 150m sites(meanpath.com)
meanpath.com
Meanpath - Search the page source, server headers and visible text on 150m sites
https://meanpath.com
5 comments
This is true :) been a great tool to write a spider in, fast native code, a strict type system and very cheap threads has made it much faster than it otherwise would have been. ZeroMQ has also been a great help in modularising it. I'll try to get a piece out on the technical architecture later. (I'm the CTO, btw.)
My favourite bits of the project;
Elasticsearch
ZMQ
Golang & Haskell
Unbound DNS
And a bit of old Python glue to hold everything together.
It's funny how long things can last when you think you would have outgrown them pretty quickly. A tiny round robin domain server that maxes out at around 60k domains per second on just a single core to distributed spider bots though is just about easy with Python + ZMQ. Plenty to keep a swarm of far more complex and well engineered Haskell crawlers humming away and throwing the results to Golang indexers to push into the Elasticsearch core for ...
Well, I'll leave the rest of it to mwotton's blog post, suffice to say it's been one of those life changingly awesome projects of doom. I'm sold on the benefits of embracing polyglot.
Elasticsearch
ZMQ
Golang & Haskell
Unbound DNS
And a bit of old Python glue to hold everything together.
It's funny how long things can last when you think you would have outgrown them pretty quickly. A tiny round robin domain server that maxes out at around 60k domains per second on just a single core to distributed spider bots though is just about easy with Python + ZMQ. Plenty to keep a swarm of far more complex and well engineered Haskell crawlers humming away and throwing the results to Golang indexers to push into the Elasticsearch core for ...
Well, I'll leave the rest of it to mwotton's blog post, suffice to say it's been one of those life changingly awesome projects of doom. I'm sold on the benefits of embracing polyglot.
yes, I probably should have mentioned the other 90% of the project, hey :)
This is an exact copy of my site NerdyData.com. Are you serious right now, I was using that layout just a week ago for my homepage!
You stole our idea and our theme verbatim? How can you live with yourself?
You stole our idea and our theme verbatim? How can you live with yourself?
The layouts don't really look anything alike and I think they may be using an off-the-shelf theme for their landing page. Also their site works... have you launched yet?
I apologize for my previous comment, I was reacting out of anger.
This is the page they copied http://nerdydata.com/home2.php. It was our wait list for the last 2 months while we developed our site, which now has a new homepage we put up last week. http://nerdydata.com
But no hard feelings, maybe you had the idea independently and happened to use the same exact theme and messaging.
This is the page they copied http://nerdydata.com/home2.php. It was our wait list for the last 2 months while we developed our site, which now has a new homepage we put up last week. http://nerdydata.com
But no hard feelings, maybe you had the idea independently and happened to use the same exact theme and messaging.
Well I guess that's the risk you take by using an off-the-shelf theme... your copy differs significantly if that's any consolation...
edit: (this is the theme https://wrapbootstrap.com/theme/beaker-responsive-working-la...)
edit: (this is the theme https://wrapbootstrap.com/theme/beaker-responsive-working-la...)
Wow very impressive - I can see so many diverse uses of this (the least commercial but fun one would be up to the minute javascript framework popularity contests!).
All the best with the continuing launch.
All the best with the continuing launch.
repost from deleted:
Wicked cool concept, I can definitely see this being useful on several fronts. Just out of curiosity, how are you determining the "links to your competitors but not yours"?
The example query is """ (www.nike.com OR nike.com) NOT (www.adidas.com OR adidas.com OR redirect:nike OR domain:nike) """
I assume this is assuming that "you" are Adidas and "your competitor" is Nike?
Correct. Adidas would be able to see sites talking about their competitor Nike that are not linking to Adidas and also do not contain Nike as part of the domain. This makes sure you screen out all the Nike sub-sites like runwithnike.com which obviously don't link to Adidas.
Cool idea, I still wonder how you'll cope keeping that entire index in memory :)
Considering I'm doing my own Haskell based tech in my business (and I'm somewhat actively in the core community), It pleases me greatly to see Haskell/ghc used in a rich array of interesting businesses!