Custom python crawler staging data in S3, extractors pick it up from there enriching it using DeepMoji & Fasttext pretrained models for sentiment along with some custom trained SVMs to measure influence, data gets staged in Mongo, then sent to Elastic.
We're building a suite of tools that let crypto enthusiasts measure the aggregate social sentiment for any coin to provide clues on where the price will go.
We're crawling and indexing over 24M+ posts on the 2 most popular crypto boards: Bitcointalk and Reddit (the crypto related subreddits), and working on adding Telegram, YouTube, and Twitter. We're performing NER to extract the names of the coin, doing sentiment analysis (combination of pre-trained DeepMoji and fastText models), and overlaying historical coin price.
We've also implemented our own search engine on top of this data (https://amuletplatform.com/data/) that ranks results based on the quality of the author (we detect users who shill or post low quality content, and rank them lower).
Still got a lot of work todo, but wanted to get some early feedback!