Ask HN: Who is building a new kind of search engine?
In PG's Frighteningly Ambitious Startup Ideas essay he talks at length about building a new kind of search engine. I was curious, who is working on this angle and makes your search engine innovative?
47 comments
I have an idea( prototype? ) for a search engine that could be disruptive, with a bit of luck. It has nothing to do with bangs and slashtags.
I've hired a freelancing coder to do some "wrapping up" so I could show the prototype but it is over a month now and still I haven't heard from him.
I do not want to engage too much time and resources into this (hiring stationary coders) but this really is the search engine that is closest to disrupting Google and Bing.
I believe that slashtags and bangs and non-spam results are not the way to go, my search engine tries a different route altogether which those two big engines ( Google and Bing ) couldn't probably adopt because they are simply too big.
I can produce results semi-manually and all I need is automation for that method but the coder I've hired, stopped responding over a month ago.
I really see big future in it but I do not want to invest too much effort and money into something so risky.
I wish I could show you what it is all about but it would reveal too much. And without the automated prototype, some good coder could simply copy it in like 2 weeks.
I believe that for $10000 I could present a really basic prototype of something that is disruptive to Google and Bing and if it isn't I think nothing will ever be.
I've hired a freelancing coder to do some "wrapping up" so I could show the prototype but it is over a month now and still I haven't heard from him.
I do not want to engage too much time and resources into this (hiring stationary coders) but this really is the search engine that is closest to disrupting Google and Bing.
I believe that slashtags and bangs and non-spam results are not the way to go, my search engine tries a different route altogether which those two big engines ( Google and Bing ) couldn't probably adopt because they are simply too big.
I can produce results semi-manually and all I need is automation for that method but the coder I've hired, stopped responding over a month ago.
I really see big future in it but I do not want to invest too much effort and money into something so risky.
I wish I could show you what it is all about but it would reveal too much. And without the automated prototype, some good coder could simply copy it in like 2 weeks.
I believe that for $10000 I could present a really basic prototype of something that is disruptive to Google and Bing and if it isn't I think nothing will ever be.
"those two big engines ( Google and Bing ) couldn't probably adopt because they are simply too big." and "some good coder could simply copy it in like 2 weeks." seem very contradictory.
The idea is really simple but the results are nothing like normal search engine (it works differently than blekko, DDG, Google, Bing etc. ).
People are used to the way Google works. If they(Google) tried to adopt this new way, I am sure many people would resign / change their provider. But still for many people, it would be a better experience.
People are used to the way Google works. If they(Google) tried to adopt this new way, I am sure many people would resign / change their provider. But still for many people, it would be a better experience.
If its just about ux then you should consider that google will put that option as an extra style of search and starve off your users. Either way if it's innovative get it done, if you don't make money you've at least improved the tech that is available out there (from you or google).
Sounds like an interesting idea from a UX perspective. If you're looking to get this developed quickly, go ahead and contact me here: http://prontotype.us/ (I would have contacted you if I could find your email)
I'm curious. Email me (my contact info is in my profile), I'll sign an NDA and everything.
Can you share one or more sample queries with sample results?
[deleted]
Do you think Twitter has the potential to grow into a new kind of search engine (http://idealab.talkingpointsmemo.com/2012/07/what-twitters-n...)?
Unlikely I know, but their access to true real-time data presents interesting possibilities I think.
Unlikely I know, but their access to true real-time data presents interesting possibilities I think.
I would assume they're heading in that direction given that they decided not to renew their contract with Google which used to let Google display real-time search results from Twitter.
Especially since it seems they have the mobile space better figured out that practically anyone (Facebook, Google, etc.).
I dont use it, but Blekko had some interesting ideas with slashtags
http://blekko.com/
http://blekko.com/
I think wolframalpha has a great potential to disrupt current search engine market in future.
I didn't write a search engine of any kind, I wrote a search front-end of sorts which redirects you to actual search engines (or Wikipedia if your query is an exact article title match). It doesn't handle not-ASCII text very well and there are some other bugs I should iron out and sometimes Heroku is down (hosted for free so I shouldn't complain) but it works well enough for me 99% of the time so I'm not actively working on it.
http://s.xqz.ca/
http://s.xqz.ca/
There is a Firefox extension similar to this.
https://addons.mozilla.org/en-us/firefox/addon/instantfox/
I would love to see if Facebook could create a better search engine by using an individual's social network to weight links instead of backlinks.
It might be too much work just to see if it is useful but I'm thinking they have a ton of data about where everyone goes on the web because the Like button is so pervasive. Why not use that data to make search extremely personalized?
Reasons why I like this:
1) It's not like anyone currently uses facebook for search, they literally have nothing to lose.
2) It would be interesting to see if the aphorism, "Show me your friends and I'll show you your future" holds up.
3) It's data that google doesn't have access to, unless G+ takes off.
4) It would be cheaper to crawl the web, essentially your users are your web crawlers.
Take the same graph theory algorithms used for backlinks and apply it to views and instead of page rank use time that their friends spend on the site.
Then you can do some interesting stuff like weight the page rank more heavily for individuals you interact with more.
Or super creepy (but statistically interesting) stuff could be done like if you want someone to reconnect with a friend they haven't spoken to in a while, weight the "long lost friend's" pages more heavily. Essentially, influence people to talk to each other by converging their data about the world.
EDIT: Another thought - if websites know that they're only admission into the search engine is putting a Like button on their page it gives them an incentive to do so.
It might be too much work just to see if it is useful but I'm thinking they have a ton of data about where everyone goes on the web because the Like button is so pervasive. Why not use that data to make search extremely personalized?
Reasons why I like this:
1) It's not like anyone currently uses facebook for search, they literally have nothing to lose.
2) It would be interesting to see if the aphorism, "Show me your friends and I'll show you your future" holds up.
3) It's data that google doesn't have access to, unless G+ takes off.
4) It would be cheaper to crawl the web, essentially your users are your web crawlers.
Take the same graph theory algorithms used for backlinks and apply it to views and instead of page rank use time that their friends spend on the site.
Then you can do some interesting stuff like weight the page rank more heavily for individuals you interact with more.
Or super creepy (but statistically interesting) stuff could be done like if you want someone to reconnect with a friend they haven't spoken to in a while, weight the "long lost friend's" pages more heavily. Essentially, influence people to talk to each other by converging their data about the world.
EDIT: Another thought - if websites know that they're only admission into the search engine is putting a Like button on their page it gives them an incentive to do so.
I am working on a very specialized search engine for software libraries. You can find it here: http://www.versioneye.com.
I don't think that we need better search algorithms for search engines. I think we just need more filters in the internet. With filters I mean more specialized search engines who are not crawling everything. If I am typing in "Spring" into the search field, I am not interested in "Spring", I am interested in the spring framework. But because most of the people on this planet are not software developers and Google don't know that I am a developer, the spring framework will not be the first result on google.
Of course we need general search engines like google. But I would love to use more specialized search engines who are just showing results to a special topic.
I would like to hear your thoughts about this.
I don't think that we need better search algorithms for search engines. I think we just need more filters in the internet. With filters I mean more specialized search engines who are not crawling everything. If I am typing in "Spring" into the search field, I am not interested in "Spring", I am interested in the spring framework. But because most of the people on this planet are not software developers and Google don't know that I am a developer, the spring framework will not be the first result on google.
Of course we need general search engines like google. But I would love to use more specialized search engines who are just showing results to a special topic.
I would like to hear your thoughts about this.
I consider writing Digest for a 3rd time. The first time was a proof of concept in Perl, the 2nd time was product in Java for [name of customer deleted]. The 3rd version could be in PHP for the masses.
Digest is not a normal search engine, but an XML description language to tell a spider where to find what on a web layout. Its aimed on the dark side of the internet, able to login on website, use cookies, and even delegate captcha solving.
The main problem of Digest is that its maintenance intensive because websites are changing layouts regularly. The idea of a Digest for the masses would be an open source distributed search engine, that runs on normal peoples web servers. So every (power) user has access to the search engine itself.
My problem, and question to this audience is: How to make money with such an project? The only idea I have currently is kickstarter.
Digest is not a normal search engine, but an XML description language to tell a spider where to find what on a web layout. Its aimed on the dark side of the internet, able to login on website, use cookies, and even delegate captcha solving.
The main problem of Digest is that its maintenance intensive because websites are changing layouts regularly. The idea of a Digest for the masses would be an open source distributed search engine, that runs on normal peoples web servers. So every (power) user has access to the search engine itself.
My problem, and question to this audience is: How to make money with such an project? The only idea I have currently is kickstarter.
I only see Gabriel Weinberg's DuckDuckGo : http://ddg.gg
That's not a new kind of search engine, it is a frontend for bing that can be used without the shame of having to say you use bing.
As far as I know, nobody is building a new kind of search engines, although I'd guess that a new kind of search engine won't really look like a search engine at first.
As far as I know, nobody is building a new kind of search engines, although I'd guess that a new kind of search engine won't really look like a search engine at first.
ddg doesn't just use bing, it uses a multitude of sources. http://help.duckduckgo.com/customer/portal/articles/216399-s...
That is an interesting claim about DDG being a wrapper for bing. Could you back it up with citations please?
I thought it was common knowledge that DDG uses Bing and a few other sources, except Google. The difference is that they don't track the users in any way and they don't put ads on the page. The fact that it uses Bing is why relevancy is not very good on DDG, though.
https://www.ixquick.com is also another alternative to DDG, doing much of the same thing.
http://en.wikipedia.org/wiki/Ixquick
https://www.ixquick.com is also another alternative to DDG, doing much of the same thing.
http://en.wikipedia.org/wiki/Ixquick
Thank you! I did not know the full details of this.
I think DDG is best considered a search broker, it federates the queries out to other engines, like bing, or others under query syntax control, and I believe they are also crawling some or their own content, although that's mostly beyond any individuals capability at this point because the internet has become quite large. The broker also merges federation sources, however it isn't a search engine in the traditional sense because the indexes (posting lists) are in google, bing, lucene etc.
* edited for autocorrect errors
* edited for autocorrect errors
Shameless plug: we're working on a product focused solely on opinions called Cheerboo (http://www.cheerboo.com/).
It's part search/discovery part social, and is trying to solve the problem that all the opinions shared in places like Twitter/Facebook aren't preserved, aggregated or made useful outside of the social graph.
We organize opinions around topics, make people weigh in on topics with a binary rating system (cheer or boo) and use the cheers/boos to give each topic a "RottenTomatoes-style" score. Let us know your thoughts - we love feedback.
It's part search/discovery part social, and is trying to solve the problem that all the opinions shared in places like Twitter/Facebook aren't preserved, aggregated or made useful outside of the social graph.
We organize opinions around topics, make people weigh in on topics with a binary rating system (cheer or boo) and use the cheers/boos to give each topic a "RottenTomatoes-style" score. Let us know your thoughts - we love feedback.
I love the concept and the design, but I have a few suggestions:
1. Give immediate visual after clicking the cheer/boo buttons. It can take a few seconds for the loading icon and comments to appear, so having the button lighten or change at all would let me know that it is in fact working.
2. Show the number of comments on each subject so that users have an idea of how popular a subject is at a glance without loading the comments.
3. Add a setting to not automatically load comments after cheer/boo -ing.
1. Give immediate visual after clicking the cheer/boo buttons. It can take a few seconds for the loading icon and comments to appear, so having the button lighten or change at all would let me know that it is in fact working.
2. Show the number of comments on each subject so that users have an idea of how popular a subject is at a glance without loading the comments.
3. Add a setting to not automatically load comments after cheer/boo -ing.
Thanks so much for the feedback! Agree with your points and we'll work on implementing.
Shameless plug: My startup, Semantics3 (http://www.semantics3.com/), is building an vertical based search engine for products data - track product pricing changes, get detailed metadata etc. Our current strategy is to monetize on selling the indexed data as part of a pay-as-you-go api.
Later on we plan to build a consumer facing website/search engine, essentially a front-end for our api, once we have moved into building large indexes around people and places (the two other verticals we are targeting)
Later on we plan to build a consumer facing website/search engine, essentially a front-end for our api, once we have moved into building large indexes around people and places (the two other verticals we are targeting)
Does there exist any non-commercial search engine? When I say "non-commercial", I don't just mean a search engine that doesn't make money, I mean a search engine that excludes all commercial websites. You accept credit cards? Sorry, you're out. You have an ad on your page? Out. Simple as that. I suspect this view of the web could bring back some of the old magic feeling. It would include Terry Tao's blog, and all of Wikipedia, but exclude the SEO'd crap that Google has incentivized.
Now, that's a really good idea. But... on the flipside I just wonder where the incentive would be for people to eventually remove the ads.
Privacy, access to different engines (when there is an API), reactive search (refine your search within the results).
http://kligl.com
Speaking of which, there are 'open' indexes of the web that anyone can download and use to build their own search product. But I can't remember the name of them. Can anyone help? Googling it results in dismal failure.
I think there is still huge room for innovation in search. Google has taken a different track, and they're focusing on Machine Intelligence. (in my opinion) search is just a nice side effect, but I really don't think it's their focus any longer.
I think there is still huge room for innovation in search. Google has taken a different track, and they're focusing on Machine Intelligence. (in my opinion) search is just a nice side effect, but I really don't think it's their focus any longer.
Maybe it's just me, but search engines suffer from a vicious circle. Noone uses your search engine unless you have enough indexed pages, and you can't index the entire web (or even a small portion of it) unless you have enough generated $$$ for high-performance servers.
That probably explains why search engine space is dominated by a few power players, with newer players like DuckDuckGo being used by only a dedicated bunch of fans.
That probably explains why search engine space is dominated by a few power players, with newer players like DuckDuckGo being used by only a dedicated bunch of fans.
My side project isn't exactly a search engine, more of a front end to get recent information from blekko, twitter, Facebook and Google plus.
www.unscatter.com
I use it to keep up to date with topics on a regular basis and also when searching for recent info on topics
I haven't heard about common crawl mentioned in another comment, going to go check that out now.
www.unscatter.com
I use it to keep up to date with topics on a regular basis and also when searching for recent info on topics
I haven't heard about common crawl mentioned in another comment, going to go check that out now.
DRIFT is a new image "discovery engine" I discovered here recently.
http://drift.io
http://drift.io
Pretty product - how long has it been around? I didn't see anything on crunchbase about it.
Not exactly a new kind of search engine, but http://www.millionshort.com
I think it strips out the top however many results you get from google. It can possibly (depending on the search), get rid of SEO heavy results that don't have much useful content.
I think it strips out the top however many results you get from google. It can possibly (depending on the search), get rid of SEO heavy results that don't have much useful content.
Try http://www.verticalset.com/
Specifically http://www.verticalset.com/extensions
And build your own here
http://www.verticalset.com/extensions/new
Specifically http://www.verticalset.com/extensions
And build your own here
http://www.verticalset.com/extensions/new
Not mine, but there's http://discoveryengine.com
for finding stuff again (~one third of all your searches)
http://archify.com - What You See Is What You Search is your private search engine for web/FB/TW/email/..
Disclaimer: I'm archify team member
http://archify.com - What You See Is What You Search is your private search engine for web/FB/TW/email/..
Disclaimer: I'm archify team member