This is probably why everyone still relies on Google(robotmay.com)
robotmay.com
This is probably why everyone still relies on Google
http://robotmay.com/post/23096164275/these-are-the-crawl-stats-for-the-last-month-for
11 comments
Yep, I tried to use DDG for some months, but often find me repeating the same search in Goog. However, usually I open a incognito browser window in Chromiums when I search in Goog, so it doesn't get my normal user tracking cookie.
You might want to look into your specific useage situation, but generally incognito shares the usual information with the server, it only switches off logging to the client.
For instance,in most cases, your wife will not know what you've searched for, but Google will.
Tor is an example of a technology that does what it seems like you're looking for.
There also may be options in your Google settings to opt out of some of their info collection.
For instance,in most cases, your wife will not know what you've searched for, but Google will.
Tor is an example of a technology that does what it seems like you're looking for.
There also may be options in your Google settings to opt out of some of their info collection.
Incognito mode has its own ephemeral cookie store, so it serves exactly the purpose he suggested. Yes, that doesn't address the fact that web protocols can still leak quite a bit of identifying information.
Torbutton tries to address information leakage, but it incurs its own costs (e.g. blocking things like HSTS and sharing the entire connection with whoever runs the tor nodes).
Torbutton tries to address information leakage, but it incurs its own costs (e.g. blocking things like HSTS and sharing the entire connection with whoever runs the tor nodes).
Wait, DDG is using Bing? I thought it was an independent and written from scratch enterprise. Does it depend on Bing for everything?
"DuckDuckGo gets its results from over 50 sources, including DuckDuckBot (our own crawler), crowd-sourced sites (in our own index), Yahoo! BOSS, embed.ly, WolframAlpha, EntireWeb, Bing & Blekko. For any given search, there is usually a vertical search engine out there that does a better job at answering it than a general search engine. Our long-term goal is to get you information from that best source, ideally in instant answer form."
source: http://help.duckduckgo.com/customer/portal/articles/216399-s...
source: http://help.duckduckgo.com/customer/portal/articles/216399-s...
Yeah, DDG uses Bing as one of it's sources. It's similar to the "meta search engines" of ye olden days.
DDG's big schtick (other than privacy) is to use lots of sources and then heavily filter the results/supplement them with additional sources (wiki, stack overflow, etc).
The founder of DDG has said he specifically wants to avoid writing a general-purpose spidering engine because you can sink millions of dollars into it and still be miles behind the competition.
DDG's big schtick (other than privacy) is to use lots of sources and then heavily filter the results/supplement them with additional sources (wiki, stack overflow, etc).
The founder of DDG has said he specifically wants to avoid writing a general-purpose spidering engine because you can sink millions of dollars into it and still be miles behind the competition.
No, there are over 50 sources: http://help.duckduckgo.com/customer/portal/articles/216399-s...
And they aren't just copied verbatim (with the same ranking, etc), but are re-ranked and mixed.
Aye, I think Cloudflare had Yahoo! in there as a bit of a legacy thing. I find myself appending !g to a lot of my queries on DDG; but I'm fine with that, as it offers me some interesting features that I don't get on Google.
Have you tried Blekko?
Those stats look a little off, and I wonder if it's because the site in question is pretty new. For comparison here's a screenshot of the crawl information from my CloudFlare analytics panel (the same information as gathered by the OP): http://i.imgur.com/vrKlz.png
jgc.org has been around since 1997, daysoutnearme.com since April, 2012. I'm guessing that the other engines don't yet know about the site.
jgc.org has been around since 1997, daysoutnearme.com since April, 2012. I'm guessing that the other engines don't yet know about the site.
Ah ha, that's really interesting. My site hasn't been around very long, but it has been submitted to both Google & Bing for the same amount of time. Nice to know that it balances out over time, but it looks like Google pays more attention to new sites than Bing does. Or Google has more resources with which to crawl sites, of which I have no doubt.
My impression, based on eyeballing logs for my own low traffic, unimportant sites: Google collects and uses stats on how often pages change, how often they come up in searches, etc., and for static and/or low importance pages they scale crawling back. This frees up crawling resources to spend elsewhere, like on important, frequently updated pages, new sites, et al. I haven't looked closely in a couple of years, but the pattern used to be that new content would be crawled fairly agressively and after a while it would back off to an "appropriate" level.
Back then, other search engines took a good while to begin crawling my new stuff in earnest, and once they did their crawling pattern did not seem to change much over time between a (relatively) popular changing page vs. a lonely static page.
Back then, other search engines took a good while to begin crawling my new stuff in earnest, and once they did their crawling pattern did not seem to change much over time between a (relatively) popular changing page vs. a lonely static page.
> I'm guessing that the other engines don't yet know about the site.
Didn't he say he submitted sitemaps?
Didn't he say he submitted sitemaps?
I tried DDG, but Goog's results are just better. And when I do tail -f log/access_log on my box, GBot is always there, hitting my sites every few seconds 24/7. Sometimes Bing is around, but less frequent and much slower.
I always knew Google hit more (especially if you're watching the access logs) but I didn't realise quite how much more until I saw the graphs on Cloudflare.
Google search is overwhelmed by new "hot" topics rather than solid valuable enduring sites. It's almost as if their news crawl has invaded their web crawl. I think the rush to have "fresh" content in the first 10 links is actually making Google significantly less useful.
Ever tried searching a business phone number in Google?
No matter if said business has a page for 10yrs. The first 2 pages well be phone directory seo spam sites that cropped up last couple years.
No matter if said business has a page for 10yrs. The first 2 pages well be phone directory seo spam sites that cropped up last couple years.
OT, does anybody actually know how to get Bing to index your site? I looked for maybe an hour with no luck.
http://bing.com/webmaster. Good luck, it took about 2 weeks before it even hit it once :\
> I looked for maybe an hour with no luck.
1. On the Bing homepage, click "Help"
2. Click "Find Help for site owners"
Arrive at the answer to your question.
1. On the Bing homepage, click "Help"
2. Click "Find Help for site owners"
Arrive at the answer to your question.
That's one of the paths I went down before, got to a place where I had to sign into windows live, signed in, and arrived at a dead end. Robotmay's link above looks good though!
Reasons people still uses Google:
(Perception of) better results: until proven otherwise - why would you use an alternative/inferior SE - unless you value other things like privacy more than the content of the results - this doesn't apply to most people
Fast/it just works: give me a compelling reason to switch. "Privacy" really isn't enough for most people.
It's default: the default FF SE? The default homepage for a bunch of people? Default search in Chrome? Check. Check. Check
Network effect: Remember the expression "Nobody ever got fired for choosing IBM" (that might be too old for a lot of the readers here) - but the same thing applies - "nobody ever got in trouble for choosing Google"
(Perception of) better results: until proven otherwise - why would you use an alternative/inferior SE - unless you value other things like privacy more than the content of the results - this doesn't apply to most people
Fast/it just works: give me a compelling reason to switch. "Privacy" really isn't enough for most people.
It's default: the default FF SE? The default homepage for a bunch of people? Default search in Chrome? Check. Check. Check
Network effect: Remember the expression "Nobody ever got fired for choosing IBM" (that might be too old for a lot of the readers here) - but the same thing applies - "nobody ever got in trouble for choosing Google"
> Default search in Chrome?
It's not default in Chrome, it asks when you first use it what you wanna use as the default search. You have to opt in to google and bing is just one click away. Most still choose google anyway.
In contrast, to change bing from being default search, on the default browser, on the most used OS in the planet. You still have to go inside preferences and opt out, which takes several clicks. And then, Microsoft still tries to sneak bing to you if you install MSN (windows live or w/e it's called now), which will change your default search engine on FF and Chrome to Bing. Unless you opt out during instalation.
And still people choose google.
It's not default in Chrome, it asks when you first use it what you wanna use as the default search. You have to opt in to google and bing is just one click away. Most still choose google anyway.
In contrast, to change bing from being default search, on the default browser, on the most used OS in the planet. You still have to go inside preferences and opt out, which takes several clicks. And then, Microsoft still tries to sneak bing to you if you install MSN (windows live or w/e it's called now), which will change your default search engine on FF and Chrome to Bing. Unless you opt out during instalation.
And still people choose google.
In Europe IE will also asks you to choose a default search engine on first start.
For me (India) it actually had chosen Bing already, I was asked to confirm the choice and I could select others if I wanted to.
I never had a problem with Google results. The only problem i always have is when there is not enough content about a topic. These are the times when i try another search engine, but the results are often worse then. I make around 20k searches a year with Google and i know how to find stuff with it, if there is something to find.
Google is free and Bing/DDG/Yandex make nothing better that would make me switch. There is not really a problem to solve when it comes to find available stuff. Quora and Stack Overflow try to solve a problem Google isn't good at. Getting answers to stuff when there is nothing to find out there.
Google is free and Bing/DDG/Yandex make nothing better that would make me switch. There is not really a problem to solve when it comes to find available stuff. Quora and Stack Overflow try to solve a problem Google isn't good at. Getting answers to stuff when there is nothing to find out there.
http://millionshort.com popped up a while back and I've actually found it incredibly useful, especially for any topics more likely to be found on little personal sites.
All equally valid points. For me I find that the quality of results is what still draws me back. I use DDG for most things these days (as it's especially good for quickly finding repos on GitHub etc), but if I have anything less certain (like a caching question) then I often find myself using Google.
Mildly off-topic, but even if I preferred google's results to duckduckgo's, I would still use duckduckgo's "g! my search phrase" over google - just for the better interface/styling that ddg offers.
I must admit that a significant portion of my love for DDG is because of the duck. It just looks so enthusiastic.
It's funny... just runned the query "days out near me" on google and bing. Bing shows the site at number one while google only shows it at number 3, 4 and 5 and never the top domain.
Hah, that's quite entertaining. Doing a non-bubbled search on Google has it at 2/3/4/5 for me. The site at #1 is, I'm assuming, much older.
I dunno... I have a site that would get creamed by Baidu's web crawler if I didn't block it, but never got noticeable traffic from it.
From what I can tell, Bing is very picky in indexing pages, whereas Google will index and ignore them if not up to their standards.
I use Bing and DDG and maybe once or twice a week I need to use Google to double check. Bing is more than fine for "most users," but Google has a powerful brand, they have a lot of goodwill and they open their checkbook to buy traffic (Mozilla, Opera, iOS, portals etc.)
Edit: All the above (even extra Googlebot servers to index new sites in seconds) are because Google can "monetize better," or has a lot of traffic which in turn attracts advertisers that bid up prices. Google is also shameless in adding dozens of ads in all places so they have lots of money to maintain their status quo. Put Bing or DDG results on Google.com, under the Google logo and very few "regular" people will complain.
I use Bing and DDG and maybe once or twice a week I need to use Google to double check. Bing is more than fine for "most users," but Google has a powerful brand, they have a lot of goodwill and they open their checkbook to buy traffic (Mozilla, Opera, iOS, portals etc.)
Edit: All the above (even extra Googlebot servers to index new sites in seconds) are because Google can "monetize better," or has a lot of traffic which in turn attracts advertisers that bid up prices. Google is also shameless in adding dozens of ads in all places so they have lots of money to maintain their status quo. Put Bing or DDG results on Google.com, under the Google logo and very few "regular" people will complain.
A random guess:
I reckon that people still use google because they have learned to "trust" google's results and they are afraid that if they use another search engine they will miss out on a useful result.
So, I wonder if new sites should show results comparisons to give new users confidence.
I reckon that people still use google because they have learned to "trust" google's results and they are afraid that if they use another search engine they will miss out on a useful result.
So, I wonder if new sites should show results comparisons to give new users confidence.
I do value my privacy, for instance, I switched from gmail to fastmail even if fastmail has less functionality. But for search there is not a real alternative for me.