Vector is designed to work well with systems like Kafka, not to replace them. While it does have a _very_ simple durable queue in the optional disk buffer feature, it is nowhere near the durability, fault tolerance, performance, etc of a full Kafka cluster and we would not recommend thinking of them as the same type of system.
That being said, we do know of a few cases where Vector's in-flight processing and routing capabilities were enough that a full Kafka-based pipeline was no longer needed. This ability to push computation out to the edge and reduce the need for centralized infrastructure is one of the aspects of Vector that we're most excited about.
Yes, streaming k8s pod logs to Kafka and/or S3 is a great example of when you could use Vector.
The "collect, transform, and route all your logs, metrics, and traces" bit is our most succinct explanation of what Vector does, but I'll admit it's still not as clear as we'd like. To expand it slightly, Vector is a tool to collect observability data (logs, metrics, etc) from wherever it's generated, optionally process that data in-flight, and then forward it to whatever upstream system you'd like to consume it. It does this by providing a variety of different components that you configure into whatever pipeline you need. In your example, you could use our new k8s source and plug it into our Kafka sink, our S3 sink, or both.
Thanks for pointing this out! That limitation is largely a holdover from when the Kafka sink was written and our support for accepting multiple data types was not as good as it is now. As things stand today, it should be a pretty simple change to enable this.
I'll go ahead and open an issue to get that addressed, but in the future please feel free to do so yourself for anything that's tripping you up! We really value this kind of feedback and try to address it as promptly as possible.
I've looked into OpenCensus/OpenTracing/OpenTelemetry (and the apparently unaffiliated OpenMetrics?) a bit, but I'm not as familiar as I'd like to be. It does seem like they're focused primarily on application-level instrumentation and the ability to ship that metrics and tracing data to different backends.
Vector's perspective is that your applications and infrastructure are already emitting all kinds of interesting data via logs, metrics, etc, and the primary challenge is to collect, enrich, and manage the storage of that data. We have no plans to integrate Vector into your application or introduce some kind of Vector-specific method of exporting data.
We'll definitely be watching OpenTelemetry as it moves forward and would very much like to be a compatible part of that ecosystem. To the degree that they use common open standards for their communication protocols, that could just fall out naturally.
Thanks for your interest! And please feel free to get in touch if you have any questions or feel there are things we could do to better support your use case: https://vector.dev/community/
Absolutely, Veneur is something we looked at quite a bit when it popped up. It's clear Stripe was feeling a lot of the same pain points we were when we started building Vector and they've come up with something really impressive.
As you mentioned, it seems they've focused more on metrics out of the gate, while we've spent more of our time on the logging side of things (for now). We're working to catch up on metrics functionality, but interoperability via SSF is an interesting idea!
Vector can act as a Splunk forwarder, but is designed to be much more flexible.
In addition to forwarding to more storage systems (S3, Elasticsearch, syslog, etc), Vector can do things like sampling logs, parsing them, and aggregating them into metrics. Depending on your needs, this makes it easier to reduce your Splunk volume and reduce costs, transition to something like an ELK stack, etc.
We're also working to build up the metrics side of Vector's capabilities. In a way, you can think of Vector as a stream processing system for observability data, capable of feeding into a variety of storage backends.
We really love the idea of mtail and think it's criminally underused.
A big part of the reason we started building Vector was to integrate that kind of functionality into a larger project, so people wouldn't have to get over the hump of discovering, setting up, and rolling out a whole separate tool.
We're definitely not as mature as mtail yet, but we're working really hard to get there.
It's still slightly rough around the edges, but Vector can actually ingest metrics today in addition to deriving metrics from log events. We have a source component that speaks the statsd protocol which can then feed into our prometheus sink. We're planning to add more metrics-focused sources and sinks in the future (e.g. graphite, datadog, etc), so check back soon!
That's a good way to think about it! Heka was a big inspiration. The design isn't exactly the same, but we're aiming to solve a lot of the same problems.
Hi! I work on Vector. For a motivating example, let's say you have an application fronted by nginx. Using Vector would allow you to ingest your nginx logs off disk, parse them, expose status code and response time distributions to prometheus, and store the parsed logs as JSON on S3.
There are obviously plenty of ways to accomplish that same thing today, but we believe Vector is somewhat unique in allowing you to do it with one tool, without touching your application code or nginx config, and with enough performance to handle serious workloads. And Vector is far from done! There's a ton more we're working to add moving forward (thinking about observability data from an ETL and stream processing perspective should give you a rough idea).
Yep, you're right, a plain FutureResult works fine in this case. I was aware of the general rule about boxing futures, but didn't realize it was unnecessary in this case. Thanks for pointing it out! I'll update the post.
Interesting post, but that much scaffolding around the tests is a bit of a red flag to me.
I personally try to move as much logic as possible into easily testable, independent methods. This leaves things like Rake tasks as very small, "obviously correct" shells around the actual logic.
I highly recommend checking out the talk Boundaries by Gary Bernhardt[0], which explains this idea far better than I can.
I've always wondered how difficult it would be to extract the vim "engine" into a library that could easily be reused by other programs. I can only assume the answer is "very difficult".
Another idea would be some kind of VimSpec that emulators could be run against to see how well they cover the whole range of vim actions. Again, probably not very feasible.
Yes, that's true. The approach has it's drawbacks though (validates_uniqueness_of being a prime example).
As much as I like Rails, I sometimes wish it were designed with the database playing more of a role than just dumb storage. Defining things like validations in Ruby is certainly nice and easy, but the database is much better suited to actually enforcing such constraints.
Even easier would be to setup cloudflare in front of the site.
All you have to do is set the cache control headers for your landing page to something like "max-age=300, public". That way cloudflare's servers will eat the vast majority of your requests, but no one will see content older than a few minutes.
We got HN'd the other day and this kept our median response time around 25ms, even for our slow rails app.
Of course you have to make sure your landing page is publicly cacheable for this to work (i.e. no user specific data), but that's a small price to pay for a drastic reduction in the amount of traffic you need to serve.
That being said, we do know of a few cases where Vector's in-flight processing and routing capabilities were enough that a full Kafka-based pipeline was no longer needed. This ability to push computation out to the edge and reduce the need for centralized infrastructure is one of the aspects of Vector that we're most excited about.