Hey HN, we recently released a new search implementation at Rollbar. It combines different kinds and scales of data across Clickhouse, MySQL and Elasticsearch. The results are that Rollbar's item search is now much faster and also supports many many more searches. Hope you find the technical details interesting - questions welcome.
* We're a ~30-person team (SF, Budapest, remote US and Europe) with a mission to help developers build software quickly and painlessly
* We help tens of thousands of developers find and fix errors faster.
* Our backend handles billions of errors with low latency and high reliability
* Our front-end allows developers to discover and drill down across millions of errors in real-time
* Our customers are some of the best engineering teams in the world, including Twilio and Shipt
* Our values are honesty, transparency, pragmatism, and dependability
* Our tech stack includes Python, Node.js, TypeScript, and React; MySQL, Kafka, ClickHouse, Elasticsearch, and Redis; Kubernetes, Terraform, and Google Cloud Platform.
* We're a ~30-person team (SF, Budapest, remote US and Europe) with a mission to help developers build software quickly and painlessly
* We help tens of thousands of developers find and fix errors faster.
* Our backend handles billions of errors with low latency and high reliability
* Our front-end allows developers to discover and drill down across millions of errors in real-time
* Our customers are some of the best engineering teams in the world, including Twilio and Shipt
* Our values are honesty, transparency, pragmatism, and dependability
* Our tech stack includes Python, Node.js, TypeScript, and React; MySQL, Kafka, ClickHouse, Elasticsearch, and Redis; Kubernetes, Terraform, and Google Cloud Platform.
* We're a ~30-person team (SF, Budapest, remote US and Europe) with a mission to help developers build software quickly and painlessly
* We help tens of thousands of developers find and fix errors faster.
* Our backend handles billions of errors with low latency and high reliability
* Our front-end allows developers to discover and drill down across millions of errors in real-time
* Our customers are some of the best engineering teams in the world, including Twilio and Shipt
* Our values are honesty, transparency, pragmatism, and dependability
* Our tech stack includes Python, Node.js, TypeScript, and React; MySQL, Kafka, ClickHouse, Elasticsearch, and Redis; Kubernetes, Terraform, and Google Cloud Platform.
* We're a ~30-person team (SF, Budapest, remote US and Europe) with a mission to help developers build software quickly and painlessly
* We help tens of thousands of developers find and fix errors faster.
* Our backend handles billions of errors with low latency and high reliability
* Our front-end allows developers to discover and drill down across millions of errors in real-time
* Our customers are some of the best engineering teams in the world, including Twilio and Shipt
* Our values are honesty, transparency, pragmatism, and dependability
* Our Tech stack includes Python, Node.js, TypeScript, and React; MySQL, Kafka, Clickhouse, Elasticsearch, and Redis; Kubernetes, Terraform, and Google Cloud Platform.
Tech stack includes Python, Node.js, TypeScript, and React; MySQL, Kafka, Clickhouse, Elasticsearch, and Redis; Kubernetes, Terraform, and Google Cloud Platform.
Hi, Brian from Rollbar here. We believe that the items listed comprise the entirety of the scope. We will be able to state definitively once forensic analysis is complete.
GitHub tokens are not exposed. More specifically: customer credentials stored for third party integrations (i.e. GitHub, Slack, JIRA) are stored encrypted using a key that is not stored in the database, so those are not exposed.
Reading every error is a great idea, but you don't need to read them all yourself - let the computer read them for you.
This is what error aggregators (e.g. Rollbar) do. Now you've got 10-1000x leverage on your time because you're reading unique errors instead of every occurrence, and you've got more data to debug with because you've got data about the aggregates ("when did it start? what request params are common? what servers is it on?") in addition to data about the individual occurrences. You can still commit to solving them all ("Rollbar Zero"). It's just a whole lot easier.
Agree about synthetic data. My point is that AI-powered applications that are deployed in production generate more _real_ data which can be used for training. For example, self-driving cars generate tons of data about how their models perform, as a result of the cars driving around. Similarly, code-writing AI applications will generate feedback in the form of errors, logs, etc. which is can be fed back into the models as training data.
This analysis misses the impact of AI models being deployed, like is happening rapidly right now. Production applications built on AI will provide ample (infinite?) additional training data to feed back into the underlying models.
I like this idea a lot. In a sense it's thinking of your application as a spreadsheet, where the database is a data tab and the frontend is the summary tabs. If there's a change to the data tab, the "spreadsheet engine" (or "materialized view engine" in your case) walks the dependency graph and updates all the relevant parts of the summary tabs.
In the Rollbar UI, we implemented a lot of this type of logic by hand to make our real-time aggregated Items list work (i.e. each time an Item has a new occurrence, we update its stats in the database and push the diff to connected clients). It would save an immense amount of code to have had a solution that did this out of the box.
The closest thing I'm aware of to this is BigQuery's materialized views ( https://cloud.google.com/bigquery/docs/materialized-views-in... ), which take care of making otherwise expensive queries cheap fast, but they are rather limited (i.e. no subqueries or joins), and don't have the "streaming output of changes" you describe.