Thanks :) Hope it will be of use to people.
We don't use solr, we use elasticsearch and we also don't hold the unzipped fingerprints in the same format the default echoprint does. But for this dump we exported everything in the official echoprint format so people can use it with the default cluster.
So far we've been hesitant to go that route as the current 7m fingerprints come down to 42m+ documents in Lucene.
I.e. If we were to save 20 versions of each fingerprint in different bpms we'd quickly have about a billion+ documents on our hands to be searched on every query (with ~1000-2000 hashes per document)
One-click hosters and cloud hosters with sharing options(such as docs.google.com), several music streaming sites, usenet.
Also the amount of mp3s hosted on regular webservers which are indexed by and easily found using the usual search engines is mindblowing
The reason for it being far from perfect is likely a combination of both. If the correct song is indexed there is a high probabiliy for us to find the right match. However if its not, with a bit of bad luck a false positive can happen easily with the default solution (and ours too).
Also when analysing a youtube video it can happen that in a 30sec snippet only 10 secs are a matching song and 20 are unrealated or 15 are one matching song the other 15 match a different one in which case 2 tracks or multiple versions of 2 different tracks will have relatively OK scores. Deciding what to consider a match (or whether to try different queries for the same or slightly altered timespan prior to deciding) is not trivial in these cases and our changes are mostly concerning when a match will be considered a match by altering thresholds and how matching truescores will be looked at in relation to other fingerprints true scores. Due to issues like these, specifying a timeframe for analysis will often produce better results.
The user can paste a youtube url which will then be analysed, fingerprinted and matched against a database of 7+ million audio fingerprints. It does not only identify a single song but is able to identify multiple songs contained in a single file or video and generates a timeline listing which tracks it contains at which time.
Our matching algorithm is based on the open source echoprint-codegen fingerprinting method, which we have built our own stack around:
- Replaced Solr/Tokyo Tyrant with Elasticsearch
- Reimplemented matching-logic
- Crawlers search multiple sources for audio files to be indexed (mp3s arent stored long term, only fingerprinted then deleted)
- Indexing about 1 new track per second
- Found method to verify unrealiable ID3 tags (in progress, current database also includes unferified)
I try to code as much as possible as early as possible. I throw away lots of stuff and recode it. Besides that I have an eye for stuff that is "similar" and can be abstracted. If someone wants an estimate, I guess as good as possible.
Big code is idealy split into one-person-chunks each with a documented API, but sometimes many people have to work on the same "files". Then big code is split between multiple people that sit nearby and communicate personally while discussing implementations based on technical arguments.
How to make a product of software is a different story. But I guess it works when you design your product in estimateable pieces and adapt fast to changing requirements.
Also I am pretty sure I forgot one or two things...
The state of JavaScript in 2015: it still sucks.
I just wish the webbrowser creators would implement a way of other languages to be executed as a replacement of JavaScript. Not gonna happen, I know...