Wade gave a memorable talk during our YC batch. When asked how he ran Zapier remotely, which was unusual at the time, he said something along the lines of "When I want to talk with the team, I open Slack. When I want to file a support ticket, I open Zendesk."
Ironically, we were the last in-person batch that was cut short by COVID, so his comments turned out to be quite helpful.
+1 on existing log viewers being particularly well suited for text over non-textual assets. My experience here is limited but I believe Grafana has a dynamic image plugin if you store a link to an asset in blob storage or Base64 encode it.
I've also heard of people storing those links in a database like Snowflake then creating displays on top using Tableau or Looker, to avoid having to build a web app from scratch.
Thanks for reading! Including examples is a great point, because otherwise the article can be kind of abstract, especially because each person has a different mental model of data. I'll add some later on.
Maybe thermodynamics is a hammer that makes all things seem like nails, but the connections pop up all over the place. Entropy is another highly applicable concept to data systems.
OP here, I only try to write and share things that I find personally interesting, so if it came across as marketing fluff that was the opposite of what I was aiming for :/. But I do appreciate you reading the whole thing. FWIW I also thought including PhD might be pretentious.
OP here, I posted this a few days ago and was surprised to see it on the front page this morning. Not sure why it says I submitted 4 hours ago when I wasn’t awake, maybe the second-chance pool (https://news.ycombinator.com/item?id=26998308)?
But I’m also generally skeptical of high upvote/comment ratios, because as a long-time HNer too I also want to read things that are genuinely interesting. In this case, I can promise you neither I nor anyone on the team is soliciting upvotes for this post.
On that note, if anyone has any comments about the content itself, happy to discuss further.
Not yet, but eventually! We’re focused on data in warehouses and transactional DBs right now, just to limit the amount of integrations we need to build to start. We definitely plan to integrate with application sources like Google Analytics down the line though. Upstream applications are ultimately the sources of data truth, after all.
I wanted to +1 what you said about “organizations that don’t have a proper data warehouse and dedicated BI staff.” At the end of the day, a huge number of companies (maybe even most?) don’t have dedicated data teams but still want to know be alerted about data anomalies. Heck, we at Metaplane even fall into that camp.
Our customers think of BigEye, Anomalo, and Monte Carlo very similarly (needing to go through a sales process, spending quite a bit of money), so this answer to a previous question about Monte Carlo might be useful: https://news.ycombinator.com/item?id=29228070 (linking to avoid redundancy)
Integrating with Microsoft SQL Server is definitely on our roadmap in the coming months. If you're up to discuss your use case, please reach out to [email protected] because we’d love to explore building this integration for you!
Amazing how many companies use dbt + Snowflake right? Such a different world from 2014…
Good idea, we actually do have a dbt integration that pulls in lineage and job metadata from your dbt manifests: https://docs.metaplane.dev/docs/dbt. Eventually we want to let you configure Metaplane tests from your dbt YAML.
Pricing is still in flux to be honest. We wanted to start with a price that was approachable for small teams, comparable to other tools in your stack, and could be paid for without going through a whole procurement process. But we’re trying to stay as flexible on pricing as possible!
Thanks, and definitely! We're making a big push in the coming months to keep building out our downstream BI integrations. The Superset API is quite nice so we're looking forward to working with it.
Good question! Both Datafold and Atlan support data monitoring as a secondary feature, but have different main focuses:
Datafold is primarily known for their Data Diff regression testing that simulates the result of a PR on your data within a CI/CD workflow. There’s definitely a need for proactively preventing data issues from occurring in the first place, but issues introduced via code are only one subset of potential data quality issues.
Metaplane is focused on catching the symptoms first via continuous monitoring. Regression tests don’t replace the need for observability, and vice-versa.
Atlan is primarily known for their data workspace features that make collaboration easier, like a data dictionary, SQL editor, and governance.
Data collaboration is a huge unsolved problem and data monitoring does play a role there. But Metaplane is focused squarely on the problem of detecting data issues and giving you relevant metadata to prioritize and debug.
Nice work! Love the support for merging profiles together and also profiling unstructured data. I could’ve used this all of the time back in research days, instead of having to do the profiling by hand and from scratch every time.
We definitely want to explore suggesting data tests based on profiling. Don’t be surprised if you see a fork!
Haha you're definitely not alone — this happened so much we ended up getting http://meatplane.com. Maybe we'll consider a rebrand in the near future :)
The main difference is how we get into the hands of data teams. To use Monte Carlo you’ll need to book a demo with their sales team to see the product, go through an implementation process, and potentially pay as much for a data monitoring tool as you do for your database.
We want every data team to have have observability as soon as they have data in a warehouse, and that means: 1) letting you implement the tool within 10 minutes, without talking to us, 2) providing a free plan and paid plans that make sense for modern data teams, and 3) focusing on being as helpful as we can with as little configuration as possible.
Monte Carlo has built a strong team and has done a great job telling the story of data observability for larger companies. Overall we want to support even the smallest data teams, who we feel are being underserved by other companies in this space.
Great question! While right now we are focused on our cloud native application, we do plan on supporting an on-prem version for companies that require hosting themselves.
The reason our customers haven’t required this is because we’ve tried to take security seriously from day one and Metaplane doesn’t store any customer data (just the metadata). We received our SOC2 Type II report, support IP whitelisting, SSH/Reverse SSH tunnels and are always exploring other integration options like AWS’ PrivateLink.
That said, we definitely understand the need to keep even metadata on-prem, so we plan on tackling that later next year.
We wanted to make Metaplane free for individuals and approachable for teams, and found that $400/mo is justified by the cost of engineering time saved and is comparable to other paid tools that a smaller data team might use (like dbt, Fivetran, Hightouch).
That said, we’re still experimenting with pricing though, and I can see the argument for a tier that’s more suited for individual paid plans. We also want the free plan to be pretty generous — is there a specific constraint that feels too limiting? Thanks for your feedback!
Thanks for sharing your use case :) We don't support Clickhouse yet, but this is exactly the kind of problem that we want to solve for data teams. You shouldn’t need to write and deploy code to monitor every metric or data that needs to be accurate and timely.
Looking forward to hearing what you think, and please reach out to [email protected] because we’d love to explore building this integration for you!
Having done one part of undergrad at a "liberal-artsy school" and another part at a larger tech school, I couldn't agree more.
If you were a lecturer at a research-focused institution, in which the quality of your teaching is a minor factor in your "success", and the social norm is mediocre teaching, would you teach at a high level?
Ironically, we were the last in-person batch that was cut short by COVID, so his comments turned out to be quite helpful.