One company's plan to build a search engine Google can't beat(protocol.com)
protocol.com
One company's plan to build a search engine Google can't beat
https://www.protocol.com/neeva-search
321 コメント
it subscribe model is OK for a starter but what are the search algorithms employed should produce open source technology if its want vast adoption
> putting search results back at the top of search results
Watching Google slowly fill more and more of the search results with ads, this is an obvious and very welcome idea.
Watching Google slowly fill more and more of the search results with ads, this is an obvious and very welcome idea.
Related: https://hammer.lol/
Even if they make a superior product, how will they get users to use it anywhere near the scale they use Google?
> Rather than try to build a search infrastructure from scratch, Neeva instead opted to use Bing's search API for its basic results.
So it's another UI over bing like duckduckgo? I'm not too optimistic, at the moment there are fundimental issues with how search engines interpret text and rank results.
So it's another UI over bing like duckduckgo? I'm not too optimistic, at the moment there are fundimental issues with how search engines interpret text and rank results.
Search engines are such a big thing they should be open sourced and distributed over the community. It's like the most basic infrastructure the internet needs to work and we are outsourcing it.
If you could return search results that actually contain the keywords I entered and not waste 5 minutes of my time returning irrelevant results because you only found 3 relevant results (you know, the ones I was actually looking for) to pad out the results, that would be a massive improvement. Search has gotten so bad that I feel like the only reason no one has toppled Google at it is because search engines are almost passe. No one cares about them much any longer because people go to a handful of site for specific types of content rather than find data scattered and unorganized across the web.
I hope this works out. I'm a big believer that companies eventually take the shape of their business models, and if you're free but serve ads, you're an ads company, not a search company. So it'll be interesting to see how a company without the ads tension ends up evolving.
That said, search is a really hard problem to solve (even if you can take shortcuts like using Bing's API).
That said, search is a really hard problem to solve (even if you can take shortcuts like using Bing's API).
Using Bing's API isn't really going to help with some of the problems.
One thing that would be really helpful, stop counting the words appearing on links on the sidebar or other non-content part of the page as important for my search. It's amazing how many searches go astray because someone has some words on a sidebar that don't have anything to do with the content of the page. You would think with all this ML someone would teach a search algorithm to ignore it.
One thing that would be really helpful, stop counting the words appearing on links on the sidebar or other non-content part of the page as important for my search. It's amazing how many searches go astray because someone has some words on a sidebar that don't have anything to do with the content of the page. You would think with all this ML someone would teach a search algorithm to ignore it.
How would somebody write their own search engine?
I'm imagining some access patterns like this:
If I know what site I want to get an answer from (youtube), i can just download lots of text from youtube and search those text files for the string im looking for, making a map of files that match that string (or youtube could expose an API for me to use of course)
if i dont know what site i want an answer from, well this becomes harder, and presumably if i dont have the space to store a text only copy of the internet for me to grep through, then a 3rd party web copy (such as wikipedia) is probably a good starting point???
I'm imagining some access patterns like this:
If I know what site I want to get an answer from (youtube), i can just download lots of text from youtube and search those text files for the string im looking for, making a map of files that match that string (or youtube could expose an API for me to use of course)
if i dont know what site i want an answer from, well this becomes harder, and presumably if i dont have the space to store a text only copy of the internet for me to grep through, then a 3rd party web copy (such as wikipedia) is probably a good starting point???
Hey, this is great. Sridhar was my VP for 3 years of my tenure at Google. Hell of a nice man. Great that he got out of the ad business and I absolutely think this a great thing to be working on!
Having actually built a different kind of search engine (see https://hnprofile.com)
I’m kinda miffed that all these posts use bing. I’m sorry, that’s not a new search engine, it’s a new UI.
I’m kinda miffed that all these posts use bing. I’m sorry, that’s not a new search engine, it’s a new UI.
How to build a good search engine:
1. Actually return the results matching the words that people typed in your search box. The more they match, the more they go up.
More and more, it has become extremely disappointing what you get back from Google (and others as well). Verbatim search (where you surround exact terms in quotes), seems to have vanished in the last year or so. More often than not I have clicked on a result, only to find out that my particular query is nowhere to be found on the site, but some "related stuff" does.
1. Actually return the results matching the words that people typed in your search box. The more they match, the more they go up.
More and more, it has become extremely disappointing what you get back from Google (and others as well). Verbatim search (where you surround exact terms in quotes), seems to have vanished in the last year or so. More often than not I have clicked on a result, only to find out that my particular query is nowhere to be found on the site, but some "related stuff" does.
These paid services never seem to get off the ground. It's asking too much for someone to pay for something right off the bat.
Not saying paid services will never work, but I feel something like this could only work if they make it free from the beginning, actually offer better search, build a rabid fanbase, then start charging for it.
Not saying paid services will never work, but I feel something like this could only work if they make it free from the beginning, actually offer better search, build a rabid fanbase, then start charging for it.
These two guys are super smart. But, how are they planning to make money?!
Asking users to pay for something they've been getting free for 22 years is a poor business model. Look at news websites.
I have to admit this has a great title.
I am familiar with Neeva and listened to the podcast with its founder which I recommend to people interested in the field.
https://podcasts.apple.com/us/podcast/sridhar-ramaswamy-your...
I am a bit skeptical about the whole thing for two main reasons:
- Sridhar spent 15 years building the behemoth that is Google's ad business. It has to be hard to shrug off 'don't be evil until you do' culture and mindset that easily. - I think they can do a good job at specialized searches (like the one illustrated in the article). But the problem is that the web search is general, not specialized. They can not expect the users to pay $30/mo and still go to Google for general searches. Bing API results are subpar to Google results.
I do welcome a paid model for a well executed search engine. I would personally pay $100/mo for one that meets my needs.
I am familiar with Neeva and listened to the podcast with its founder which I recommend to people interested in the field.
https://podcasts.apple.com/us/podcast/sridhar-ramaswamy-your...
I am a bit skeptical about the whole thing for two main reasons:
- Sridhar spent 15 years building the behemoth that is Google's ad business. It has to be hard to shrug off 'don't be evil until you do' culture and mindset that easily. - I think they can do a good job at specialized searches (like the one illustrated in the article). But the problem is that the web search is general, not specialized. They can not expect the users to pay $30/mo and still go to Google for general searches. Bing API results are subpar to Google results.
I do welcome a paid model for a well executed search engine. I would personally pay $100/mo for one that meets my needs.
Yet another startup with product that doesn't fit the market.
Want to build a superior search engine?
OBEY YOUR FREAKING BOOLEAN OPERATORS (search syntax, whatever) - something Google, DDG, Bing, et al, no longer have any interest in doing.
OBEY YOUR FREAKING BOOLEAN OPERATORS (search syntax, whatever) - something Google, DDG, Bing, et al, no longer have any interest in doing.
Google can beat them by also offering a paid-for search engine. People trust Google with their mail and phones, they will also trust google with privacy-respecting search.
On the other hand, how come Cliqz has been shut down instead of being sold?[1] Are there no companies with deep pockets who are interested in containing Google's revenue besides MS? E.g. since Cliqz was so privacy focussed, wouldn't that have been a great start for Apple to have a privacy respecting search engine?
[1] https://news.ycombinator.com/item?id=23909484
On the other hand, how come Cliqz has been shut down instead of being sold?[1] Are there no companies with deep pockets who are interested in containing Google's revenue besides MS? E.g. since Cliqz was so privacy focussed, wouldn't that have been a great start for Apple to have a privacy respecting search engine?
[1] https://news.ycombinator.com/item?id=23909484
Lately I find myself inserting "site:reddit.com" at the end of my Google searches. Most of the stuff that shows up by default is pure garbage that is riddled with ads and doesn't answer my question quickly.
Not mentioned in the article or in the comments so far, but Neeva has raised $37.5m in funding[0]. I'm curious how that money will be spent, if they're not actually spending it on building a new search engine. Is it going to be mostly spent on buying the results in from Bing, and/or on advertising their new ad-free search, and/or something else?
[0] https://tech.economictimes.indiatimes.com/news/startups/7649...
[0] https://tech.economictimes.indiatimes.com/news/startups/7649...
> See, Google's business model won't allow it to compete with Neeva. It can't get rid of ads because ads are its whole business.
That may sound great, but it's just not true. Everybody working at a company that pays for gsuite knows this. Gmail in gsuite is not part of the ad-financed services. Instead, you pay. Google could any day offer the same thing to consumers with search. Pay some fee every month, opt out of ads. Facebook could do the same thing, btw. Boom, there goes the "novelty" of Neeva's business model.
Now comes the question why don't Google, FB etc have such offerings yet. Many people would choose them. My guess is that they would need to have quite steep pricing, if they would want to 100% compensate for lack of ad income for each user who pays instead. Nobody would want to pay FB 50 USD every month just to see their uncle's Trump propaganda. That would be quite the eye opener about how much each user is really worth to FB or Google in money terms.
Maybe services like Neeva create a market though and push FB and Google to at least consider it.
That may sound great, but it's just not true. Everybody working at a company that pays for gsuite knows this. Gmail in gsuite is not part of the ad-financed services. Instead, you pay. Google could any day offer the same thing to consumers with search. Pay some fee every month, opt out of ads. Facebook could do the same thing, btw. Boom, there goes the "novelty" of Neeva's business model.
Now comes the question why don't Google, FB etc have such offerings yet. Many people would choose them. My guess is that they would need to have quite steep pricing, if they would want to 100% compensate for lack of ad income for each user who pays instead. Nobody would want to pay FB 50 USD every month just to see their uncle's Trump propaganda. That would be quite the eye opener about how much each user is really worth to FB or Google in money terms.
Maybe services like Neeva create a market though and push FB and Google to at least consider it.
Why another general purpose search engine? How about a niche, for example a search engine that can be extensively controlled by using Boolean algebra, Regex, first-order logic?
Googles results got less and less reasonable over the last years.
Googles results got less and less reasonable over the last years.
Maybe the pendulum could swing back the other way slightly and we could have room for an older Yahoo-style indexed search engine. This wouldn't be something that would be browsed instead of search, but if you had a general categories list indexed, say 'finance' or 'graphic design', users could enter those sections and search sites categorised accordingly. Perhaps some ranking could be done based on search terms used within a category and what site a user ends up visiting (e.g. many users end up visiting a certain stackexchange link when searching for 'parsing json in python' within the 'computer programming' category, and so its rank increases). Heck instead of ads, maybe each category page could have a list of 'popular / trending sites on this topic' section at the topic that pages could pay to be placed in.
Not sure if this solves any user problem to be honest, and the idea is only appealing to me because I think some amount of domain expertise and human curation could go into categorising pages. While this sounds (and no doubt would be) labour intensive, if we consider the number of domains that users actually visit when conducting a search (i.e. ignoring anything past page 1 of google) then perhaps it's not so extreme.
Not sure if this solves any user problem to be honest, and the idea is only appealing to me because I think some amount of domain expertise and human curation could go into categorising pages. While this sounds (and no doubt would be) labour intensive, if we consider the number of domains that users actually visit when conducting a search (i.e. ignoring anything past page 1 of google) then perhaps it's not so extreme.
So, finally a paid search engine that does not have to rely on ads for revenue.
So far it's based on Bing, which does. This makes it a bit a hard sell, compared to an intelligent ad blocker.
The most important problem of search engines is SEO spam. Google themselves sort of have a moral hazard to not be too stringent on SEO spam, because it shows ads by Google, increasing Google's revenue.
OTOH I wonder if the subscription revenue is going to be sufficient to have access to a reasonably good search index and enough processing power to efficiently combat SEO spam while returning relevant results. This takes your own data centers run frugally, because fees of something like AWS or Azure will just be exorbitant for a global search engine and a global search index.
I wonder if companies aiming to provide alternative search engines will cooperate on maintaining a common index, to distribute the massive costs of doing that. They could even publicly sell access to it, at a point where running a competing search engine won't be practical; e.g. researchers would buy it.
So far it's based on Bing, which does. This makes it a bit a hard sell, compared to an intelligent ad blocker.
The most important problem of search engines is SEO spam. Google themselves sort of have a moral hazard to not be too stringent on SEO spam, because it shows ads by Google, increasing Google's revenue.
OTOH I wonder if the subscription revenue is going to be sufficient to have access to a reasonably good search index and enough processing power to efficiently combat SEO spam while returning relevant results. This takes your own data centers run frugally, because fees of something like AWS or Azure will just be exorbitant for a global search engine and a global search index.
I wonder if companies aiming to provide alternative search engines will cooperate on maintaining a common index, to distribute the massive costs of doing that. They could even publicly sell access to it, at a point where running a competing search engine won't be practical; e.g. researchers would buy it.
There's already a paid, curated search engine Google can't beat in certain verticals: Westlaw and LexisNexus, but they are based on vast curated and annotated data sets of legal case law that deliver enormous value to end users of those services.
The more I think about it, the more I think the search engine problem has no (currently known) solution. The options are:
1) Word-based crawling/indexing: quickly abused by spammers.
2) IA/ML-based: I think this is the current model (?), but after a while the machine got "clever" and it makes Google to think it knows better than me about what I am looking for, and returns result for "most people" tastes. The problem is when you are not "most people", and/or you are looking for some niche topic/work related/tech stuff/etc. Simply trying to discover new things like an interesting blog or a small shop it's impossible.
3) Paid-based: as in the article, and might be a good idea. But I think it has to run on a custom indexer. Why would I pay for Bing results?
4) Aggregators: a search engine that returns results from a bunch of other search engines, like DDG and others.
5) A mix of the above?
So unless new ideas come to the rescue, I think it's always going to get worse.
1) Word-based crawling/indexing: quickly abused by spammers.
2) IA/ML-based: I think this is the current model (?), but after a while the machine got "clever" and it makes Google to think it knows better than me about what I am looking for, and returns result for "most people" tastes. The problem is when you are not "most people", and/or you are looking for some niche topic/work related/tech stuff/etc. Simply trying to discover new things like an interesting blog or a small shop it's impossible.
3) Paid-based: as in the article, and might be a good idea. But I think it has to run on a custom indexer. Why would I pay for Bing results?
4) Aggregators: a search engine that returns results from a bunch of other search engines, like DDG and others.
5) A mix of the above?
So unless new ideas come to the rescue, I think it's always going to get worse.
I want to share a recent search I did through Google -- at work I wanted to look up how to implement conditionals in Microsoft Forms (i.e. the next question in the form is based on the previous answer). I searched for "microsoft forms conditional" (without quotes) and the first search result on Google was this page on how to "use branching in Microsoft Forms" - https://support.microsoft.com/en-us/office/use-branching-in-...
That page doesn't contain the word "conditional" at all - the word that Microsoft uses is "branching" but Google deduced that it was the best result, which was perfect. The same search in either Bing or DDG produces results that all have the words "microsoft", "forms", and "conditional" in them and none of them link to the page I mentioned above, which I consider to be the best result.
Moral of the story? Search is hard, but Google does it better than everyone else.
Also, I learned via this thread that DDG is just a wrapper around Bing, which explains why the search results between DDG and Bing were near identical - they even have the matching video suggestions.
That page doesn't contain the word "conditional" at all - the word that Microsoft uses is "branching" but Google deduced that it was the best result, which was perfect. The same search in either Bing or DDG produces results that all have the words "microsoft", "forms", and "conditional" in them and none of them link to the page I mentioned above, which I consider to be the best result.
Moral of the story? Search is hard, but Google does it better than everyone else.
Also, I learned via this thread that DDG is just a wrapper around Bing, which explains why the search results between DDG and Bing were near identical - they even have the matching video suggestions.
I'm semi-seriously considering writing my own.