We strive to earn our users trust every day. We read every piece of feedback that comes in and try to make the product better. At the end of the day, we only make money if our users use our product and opt to pay for it, so our incentives are as aligned with our users as they can be.
(co-founder) Actually,
1. you can just search on neeva.com w/o signing up for an account
2. If/when you sign up for an account, you can choose between our "free basic" (free) product and our "premium" (paid) product. The premium product includes access to a VPN and a password manager.
All these options are ads-free and private. (To experience Neeva AI, you need an account for now.)
(co-founder here) We are working on a bunch of freshness improvements across the board (ranking, indexing, crawl) that should help with this. Thanks for the feedback and keep it coming!
tldr; there are two big challenges in crawling the web:
* quantity -- how do you crawl the web at O(B) pages per day
* quality -- how do you make sure you are crawling the high quality parts
On the quantity side, the web is a much trickier place to crawl than 10-15 years ago. Even if you build a system that is well behaved, respects rate limits and work w/ webmasters and CDNs to make sure you are treated as a good bot, the following things will bite you:
* some sites only allow googlebot/bingbot via robots
* even when sites allow all search crawlers, you'll see 429s, 503s, crawl delay directives that limit you to very low qps, and other mysterious errors. (most retailers and aggregators)
* working w/ webmasters works only if you manage to get a response from them (good luck)
* many sites require JS rendering (which is 100x more expensive in terms of the number of assets you are crawling)
On the quality side, crawl prioritization works best when you have click data, which most small search engines don't have enough of. In the absence of that, good seed sets and high quality inlinks are your friend.
In other news, eng blogs are being written up as we speak. Will post here on HN as they roll out. Also, like marginalia_nu pointed out, we share whatever we've been working on on a weekly basis on our Twitter account.
(Founder of Neeva here)
Query-independent page and site signals like PageRank (or its site variants) have limited utility in a search ranker. Mostly, they are useful in weeding out bad pages and sites from your index, tie-breaking when you hit shard limits in retrieval and a few other edge cases.
The signals that matter the most:
1. Anchor text (and all variants of smearing and distinguishing between high quality and low quality anchors)
2. In aggregate, which pages got clicked on on any given query (and all smearing variants -- using ngrams, embeddings, ...). At Neeva, we use it for retrieval and scoring.
3. Query understanding signals mined from the query-click bi-partite graph and the query-query session refinement graph.
4. Page summarization signals built on top of 1 and 2 and body text.
5. To a lesser extent, query-independent page quality signals
Whether you use term-based retrieval or nearest neighbor (embedding) retrieval, a heuristic combination of signals or LambdaMart, whether you calibrate to human eval or clicks, whether your topical relevance function is hand crafted or uses a combination of deep learning and IR signals are all details past that.
tldr; there's a lot of craft in a search ranker, and no one silver bullet. Definitely not just PageRank.
(Founder of Neeva here) On OP's question -- don't underestimate the power of defaults. Google owns Chrome and Android. Google pays Apple to be the default on Safari. Google pays Firefox to be the default. Bing is default on Edge. And so on. All the platforms make it very hard to change the defaults. And on platforms like Safari, as a user, you can't even change the default search provider if you want to.
(Founder of Neeva here) Appreciate the input and hear your frustration. This is a carryover from changes we made to introduce the free tier earlier in the quarter. We are making imminent improvements on modal frequency and getting them out of your way. Stay posted.
(Founder of Neeva) Eval for search engines is about as hard as ranking for search engines. You need rating templates, raters, querysets, tooling and lots of time and patience. There are a number of vendors who can help you with the raters part, but the rest is still painstaking work you have to do yourself. Email me if you need help; we are happy to share findings.
(Founder of Neeva here) -- Chris -- I love the idea. Neeva does this in a contextually relevant manner to the intent of the query. For example, on health queries, we label all health sites as "trusted", "ad-supported" etc. and allow you to filter down to the appropriate subset of results. For programming queries, we label sites as "official sites", "forums", "blogs", "code repos", "programming websites" (the SEO-ed ones). We work with human raters to do this. Would love to hear if you find it useful and what other labels would be of use.
(Founder of Neeva here) Colin -- huge congratulations on the 5B milestone. Crawling the web, building and serving a large index, and powering your own ranking is not for the faint of heart. Great job!
Founder of @Neeva here. In my experience, this is mostly PR that will go nowhere -- these are efforts that are not structurally set up to succeed given the internal constraints at Google. The Chrome sandbox efforts have been going on for 3-4 years with no progress.
The mobile app ads businesses (UAC campaigns) at Google are just too profitable, and depend too much on IDFA/ADID tracking for the Android team to be able to do anything meaningful here. So we all know the stakes here, and to put this abstract "tracking" into "real world" perspective, a big use case of ADID/IDFA tracking is gaming advertisers identifying "whales" (kids/adults who spend a shit ton of money on their games) and accurately targeting them across all their usage on the phone. If your kids are being tricked into spending hundreds of dollars by these ads, this is why.