Infrastructure has become a key driver of observability cost. More than data volumes, this is because of approaches like replication, cross AZ data transfer and almost no compression.
In this post we explain the infra cost of running Parseable for 100TB telemetry data on a daily basis. This is based on extrapolating ingestion volumes for an hour.
Looking forward to your comments / feedback on this!
Traces are one of most difficult observability signals to work with. You need a lot of work on the application and infrastructure instrumentation to get it right.
Once the hard work is done on the client side, you'd expect to extract max value from the this trace data (and not set small retention to save cost).
We worked hard to understand (and explain) how OTEL traces can be efficiently stored, queried and analysed on Parseable using the ubiquitous Parquet file format. Read on!
Columnar formats allow side stepping of some of the most critical issues that plague time series databases. Each label essentially becomes a column which is then efficient to parse through.
Minio is deliberately designed in this manner for scale. It is in our intention to make storage deployment to be simpler and scaling is achieved by deploying many smaller instances.
So each minio cluster is a deployment unit which scales with your orchestration software like kubernetes. In our view creating a gigantic cluster with single namespace is scary, where single server can take down the the entire data center.
With Minio's design the intention is to isolate such occurrences and safely operate/upgrade clusters without disturbing a single namespace.
Servers crash all the time. But it is important to make sure applications, and hence the business, doesn’t suffer. This is why service availability is one of biggest concerns for operational engineers deploying applications in the cloud.