How We Built a Cutting-Edge Color Search App(bits.shutterstock.com)

43 pointsby mattangriffelvor 12 Jahren16 comments

bits.shutterstock.com

How We Built a Cutting-Edge Color Search App

http://bits.shutterstock.com/2014/01/30/how-we-built-a-cutting-edge-color-search-app/

11 comments

This is really awesome. For anyone who wants a naive, poor man's implementation of this, I have an unfinished Ruby gem that's a good starting point for you: https://github.com/JoshSmith/kaleidoscope

Here's how I got it to work (cribbed from my own README):

TL;DR: I used k-means clustering to segment a database of images into color bins for quick searching.

Using imagemagick, I ran histograms on images and converted their top n most frequent colors into Lab* color space for an approximate representation of human vision.

Colors were then matched to a user-defined set of colors using Euclidean distance, i.e. a "bin". I could choose any array of RGB values of arbitrary length.

I then stored hexadecimal values of the image's original color and the matched color, along with the frequency of that color within the image (for sorting based on frequency) and the Euclidean distance (for sorting by tolerance).

Then finding images close to a certain color was as simple as Photo.all.with_color('#993399') and order by frequency and Euclidean distance. Here's a photo of the results: https://github-camo.global.ssl.fastly.net/89cc87ac84cd3a1d12...

I might spend some time reverse-engineering Shutterstock's implementation, since it sounds way better than mine and clearly works at scale. But for my purposes, my own implementation worked just fine.

If you want help implementing it, feel free to reach out to me!

clbeckervor 12 Jahren

That looks pretty cool. As far as scaling issues went, the bulk of them were addressed just by using Solr on pretty beefy hardware (for our full library of 30+ million images, we're running on Dell r510s w/ 24 cores and 128GB Ram). Of course, depending on your hardware limitations there's compromises you can make to increase speed / reduce memory usage - i.e. the fewer fields you need to index and sort on the better & the lower the cardinality of each field, the better too. Also, since our input only consisted of one slider, we were able to run all the distance calculations beforehand, and just store a score that represented the image's distance from the given slider position - something like that might also work for you, since you're using a limited palette of input colors too.

kevinhvor 12 Jahren

clbecker, your account has been shadowbanned so your replies won't show up on any posts. This is very unfortunate because it seems like you're the origin of this post. So, make a new account or something.

Everyone else: If you want to see his comments, turn on the showdead option in your profile.

[deleted]vor 12 Jahren

darshamvor 12 Jahren

Reminds me of TinEye's multicolr demo [0] that searches through CC-licensed images on Flickr. Their multiple color feature was really nice (however they don't have shuttershock's keyword filtering.)

I wonder if anyone ever bought TinEye's color-search-engine-as-a-service [1]. The as-a-service model seems really awkward for something that requires so much integration, and this new shuttershock feature (developed from the ground up) seems to confirm this.

[0] http://labs.tineye.com/multicolr

[1] http://services.tineye.com/MulticolorEngine

streblervor 12 Jahren

I'm really interested in that question too, did anyone buy TinEye's color search service? I hope so.

Personally, I think the TinEye color results are better than Shutterstock's approach....although having meta-data alongside is definitely a must.

basseqvor 12 Jahren

I think the next step for them is to cluster by similar images. A green hue, for instance, shows a lot of similar-looking pictures of leaves—better to show one picture and have a "show similar images" feature to dig into a finer level of variance.

V. cool, though.

streblervor 12 Jahren

That'd be a good next post, color retrieval using solr.

frikvor 12 Jahren

Airliners.net had something similar since 2005.

http://www.airliners.net/similarity/

It was on Slashdot back then: http://tech.slashdot.org/story/05/05/04/2239224/searching-by...

clbeckervor 12 Jahren

Thanks to the Hacker News Gods, my account is no longer blocked on here. I'm happy to answer any questions about the original blog post.

kfkvor 12 Jahren

Is somebody thinking on applying this to a different stock photo model? I mean, I have heard shutterstock takes a big cut and pays very little to photographers. Nobody out there with these innovative ideas but with a business plan more friendly for photographers?

carlobvor 12 Jahren

The three sliders here correspond to averages of the three LCH channels. Has anyone thought of looking into the second moments of those? As in variances and covariances in order to get high contrast in luminance or high contrast in hue…

frikvor 12 Jahren

"the prototype had over 20 sliders to control all the visual attributes"

I cannot read the slider labels, the screenshot is very low-res :(

clbeckervor 12 Jahren

Sorry about that! The labels on the actual prototype were a bit vague and undescriptive anyway - basically most of the sliders represented various statistics taken over the histograms for each dimension in the LCH colorspace (mean, median, stddev, etc), and then there were a few magical sliders that, to this day, I think only one engineer around here knew how they worked...

[deleted]vor 12 Jahren

(2)

abvdaskervor 12 Jahren

This is incredibly cool. Shutterstock has done a great job in creating a beautiful product to address a fun problem. Kudos.

bpphillipsvor 12 Jahren

Neat! It'd be nice to see some more discernible screenshots of the prototypes to compare with the final version.

iterablevor 12 Jahren

this is dope. Shutterstock is bomb