Duck Duck Go Architecture(gabrielweinberg.com)
gabrielweinberg.com
Duck Duck Go Architecture
http://www.gabrielweinberg.com/blog/2009/03/duck-duck-go-architecture.html
22 comments
Thanks! Yeah, we've gotten a lot better since then. On the zero-click info (what is x?) we use lots of sources now beyond Wikipedia (e.g. Crunchbase) as well as have a bunch of new features (http://duckduckgo.com/about.html). We also have both an iPhone app and an API :)
iPhone App: http://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftwa...
API info: http://duckduckgo.com/api.html
iPhone App: http://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftwa...
API info: http://duckduckgo.com/api.html
Whoops, should have spotted both of those :p
Both look good though!
Both look good though!
I'm not exactly a search connoisseur or even pay much attention to startups but wow. This is one of the first search innovations I've found that was any good, in fact it's great.
I've just replaced my google FF keyword, 'g', with duckduckgo link. I'll try it out for a week or so and see if I still think it's as awesome as I do now.
I've just replaced my google FF keyword, 'g', with duckduckgo link. I'll try it out for a week or so and see if I still think it's as awesome as I do now.
Thanks! Please email me your thoughts after the week--always looking for real feedback from real users. Also note we have a FF extension: https://addons.mozilla.org/en-US/firefox/addon/10536
Cool article. Couple of queries :-)
It mentions that static content is served from nginx, but then mentions S3 later on as well. Why not just Amazon for all the static content?
Mentions nginx being more stable than Apache - you mean no-crash stable, or more consistent under load?
It mentions that static content is served from nginx, but then mentions S3 later on as well. Why not just Amazon for all the static content?
Mentions nginx being more stable than Apache - you mean no-crash stable, or more consistent under load?
Thanks! I use S3 to serve images that are loaded after the initial page load (incidentally, using YUI's ImageLoader: http://developer.yahoo.com/yui/imageloader/). I use nginx to serve static files, e.g. HTML, as well as images that get loaded on page load, e.g. logo. Why not S3 for the latter? Too slow. Amazon's (relatively) new content network may be faster, though I haven't tried it yet.
Wrt nginx being more stable, both. I've crashed Apache as well as pseudo-crashed it (process still running, but not serving right). nginx hasn't crashed yet. It's also more consistent (and faster) under load. However, in Apache's defense, it does a lot more, e.g. mod_perl vs nginx proxy to FastCGI.
Wrt nginx being more stable, both. I've crashed Apache as well as pseudo-crashed it (process still running, but not serving right). nginx hasn't crashed yet. It's also more consistent (and faster) under load. However, in Apache's defense, it does a lot more, e.g. mod_perl vs nginx proxy to FastCGI.
Cool - well I'm on Apache now, on a very undersized server - so I guess I'll have to give nginx a shot.
Not sure how you'd extract more speed from your current setup. The natural next step would be doing work in parallel - this something you do/have considered?
If you do try the Amazon CDN, let us know how you go :)
Not sure how you'd extract more speed from your current setup. The natural next step would be doing work in parallel - this something you do/have considered?
If you do try the Amazon CDN, let us know how you go :)
I use multiple worker processes in nginx (2 x # of cores), which work in parallel. Then FCGI::Engine is configured to run multiple processes as well. Finally, I use the external JS and image files (in part) to speed up the client-side via parallel processing of requests. (Note that IE7 and below limit concurrent connections per host by default often to 2: http://msdn.microsoft.com/en-us/library/cc304129(VS.85).aspx).
This passed my "does this not suck?" test. Good work. A search engine that does not suck and is not named Google.
gigablast finds stuff for me that others miss.
Question: why are you going with FastCGI over mod_perl? Issues with mod_perl debugging/memory leaks? Desire to use nginx over Apache for higher throughput? The big advantage -- to me -- of mod_perl is the power of having your code live within Apache's process space.
I wonder if mod_perl works with event_mpm, yet.
I wonder if mod_perl works with event_mpm, yet.
d) All of the above.
Initially I didn't think anything could beat mod_perl speed, but after benchmarking, this setup is just as fast. It's also more stable, and uses less memory. And yes, also it is easier to debug and I was getting memory leaks on mod_perl.
Don't get me wrong, I love mod_perl. I used it at my last company and on many other projects before this one. And I still use it for other stuff.
Last I looked I think you can run mod_perl with event_mpm but I didn't like this: "The current Apache::DBI should be usable under threaded mpm, though it doesn't share connections across threads. Each Perl interpreter has its own cache, just like in the prefork mpm." (http://perl.apache.org/docs/2.0/user/performance/mpm.html)
Initially I didn't think anything could beat mod_perl speed, but after benchmarking, this setup is just as fast. It's also more stable, and uses less memory. And yes, also it is easier to debug and I was getting memory leaks on mod_perl.
Don't get me wrong, I love mod_perl. I used it at my last company and on many other projects before this one. And I still use it for other stuff.
Last I looked I think you can run mod_perl with event_mpm but I didn't like this: "The current Apache::DBI should be usable under threaded mpm, though it doesn't share connections across threads. Each Perl interpreter has its own cache, just like in the prefork mpm." (http://perl.apache.org/docs/2.0/user/performance/mpm.html)
complete side note - but your favicon has a lot of aliasing when it's on a dark background (i.e. as it is when you have ddg in an inactive chrome tab). the circle looks nice on white, but you may want to adjust it so it reads a little better against other mattes.
aside from that tiny-minor issue, ddg seems cool & the overview of the architecture is very clear. i'm going to give it a try as my default chrome search engine & figure out just how behaviorally indoctrinated google has me and how your architecture holds up as more people use the service.
aside from that tiny-minor issue, ddg seems cool & the overview of the architecture is very clear. i'm going to give it a try as my default chrome search engine & figure out just how behaviorally indoctrinated google has me and how your architecture holds up as more people use the service.
Thanks! I will look into the favicon issue. Please send me an email after you try it for a while with your thoughts.
[deleted](2)
Just had a look again and I now think it's pretty awesome. Just for a really speedy "what the hell is this?" type of query it looks absolutely amazing. I love the AJAX extra search links, makes it all feel really snappy.
Works great on the iPhone as well, I've added an icon to my home screen. You might want to consider making an app for the app store (you could pretty much just make an app that loads the site as it is).
Have you considered adding an API? I currently have my Emacs set up to do a dictionary lookup (using the dictd protocol) on a word when I hit C-cl. I'd love to add Duck Duck Go results to the buffer this displays!