BuffDB is a Rust library to simplify multi-plexing on edge devices(github.com)
github.com
BuffDB is a Rust library to simplify multi-plexing on edge devices
https://github.com/buffdb/buffdb
5 comments
What does "Simplify multiplexing on edge devices" even mean?
I think it's a proxy between your software and SQLite with a new database API. I guess "multiplexing" is a big word for saying you can someday swap SQLite for something else.
The CLI shows key-value store features.
I don't know if this software has real world savings in performance. I don't think I would ever use this software.
The CLI shows key-value store features.
I don't know if this software has real world savings in performance. I don't think I would ever use this software.
Like if you want your frontend to write to both Sqlite and RocksDB?
I guess they mean number of frontends times number of backends? https://github.com/buffdb/buffdb/issues/19
Is it a protobuf based database store, or is it a database that uses grpc as its connection communication? Could be a bit clearer from the frontpage
> protobuf
such an awful format, I wish people would stop using it
such an awful format, I wish people would stop using it
I think it has many good parts. There are waaaaay worse formats I have worked with over the years. Like COM, java RMI, many variants of SOAP, handcrafted json in countless variants…
Why?
The format is not that bad.
The binding/libraries OTOH are often awful and they often require unnecessary full-message deserialization
The format is not that bad.
The binding/libraries OTOH are often awful and they often require unnecessary full-message deserialization
not self describing. if it was just the field names I could deal with that, but even the values are ambiguous, since the same type is used for Bytes and embedded Messages. the worst part is the wire type integer has two unused values, so they easily could have added a wire type for embedded messages
Self-describing is point-less for serializations. There is a great deal of history here. ASN.1 has self-describing encoding rules such as BER/DER/CER, XER (XML), JER (JSON), and GSER (never mind), and it has non-self-describing serializations like PER (packed encoring rules) and OER (octet encoding rules). XML and JSON are self-describing, naturally. FastInfoSet is a PER-based fast encoding for XML, because it turns out that XML is slow (imagine that). XDR is a non-self-describing serialization format that resembles OER but with 4-octet alignment. Flat buffers is essentially an OER-ish encoding for the same IDL as protobufs, and is much better than protobufs.
It would be nice if the next serialization format either is truly original or just solves problems that somehow none of the many existing schemes do.
How many serialization formats are there? See: https://en.wikipedia.org/wiki/Comparison_of_data-serializati... (which is NOT a complete list).
It would be nice if the next serialization format either is truly original or just solves problems that somehow none of the many existing schemes do.
How many serialization formats are there? See: https://en.wikipedia.org/wiki/Comparison_of_data-serializati... (which is NOT a complete list).
> Self-describing is point-less for serializations
you couldn't be more wrong. what happens when you lose the schema, or never had access to it in the first place? think from the point of view of reverse engineering
you couldn't be more wrong. what happens when you lose the schema, or never had access to it in the first place? think from the point of view of reverse engineering
I imagine when someone is choosing a serialization solution they probably don't really care about people trying to reverse engineer it... And if they did, they would just make schemas available instead.
And if you lose your own schema, then you probably have more serious underlying problems.
And if you lose your own schema, then you probably have more serious underlying problems.
just imagine a world where protobuf replaces JSON, would you really want that? you're not thinking big picture
All the binary JSON schemes, like CBOR for example, are all about making encoding and decoding faster. So the world kinda wants this even if you don't.
Do I want protobufs replacing JSON? No. I want flatbufs augmenting JSON (and XML, and ...).
I maintain an ASN.1 compiler and run-time that supports transliteration of DER to JSON. This is possible because the syntax/schema language/IDL and the encoding rules are separable. This is how you get the best of both worlds. You can use optimized binary encoding rules for interchange but convert to/from JSON/XML/whatever as needed for inspection.
Do I want protobufs replacing JSON? No. I want flatbufs augmenting JSON (and XML, and ...).
I maintain an ASN.1 compiler and run-time that supports transliteration of DER to JSON. This is possible because the syntax/schema language/IDL and the encoding rules are separable. This is how you get the best of both worlds. You can use optimized binary encoding rules for interchange but convert to/from JSON/XML/whatever as needed for inspection.
I actually don't think it'd be so bad.
Ideally it is capnproto instead of protobuf, but in general the same applies.
It is a _transport_ protocol. You can still write your config or data in any format you like.
Ideally it is capnproto instead of protobuf, but in general the same applies.
It is a _transport_ protocol. You can still write your config or data in any format you like.
you're not getting it. imagine if every single public website today using JSON switches to protobuf, you instantly lost most if not all ability to reverse those sites.
All these sites go through an evolution where they start out having open APIs, attracting third party developers, then eating their lunch and closing the APIs. No one in this sub-thread is saying that's desirable. But also nothing you can do can stop them from using something other than JSON.
You are assuming websites offer an end user api via json but that's just a side effect, an implementation detail.
Why would you lose it? NFS never lost its XDR schema, for example. Do you have any examples where the schema got lost?
Protobuf is a tag-length-value (TLV) encoding. It's bad. TLV is the thing that everyone loves to hate about ASN.1's DER.
> It's bad
I'm not sure that's a helpful way to look at things.
There are tradeoffs. Can you elaborate more about what aspects of a TLV encoding you find problematic? Is it decoding speed? The need to copy the encoded value into a native value in order to make use of it? Something else?
I'm not sure that's a helpful way to look at things.
There are tradeoffs. Can you elaborate more about what aspects of a TLV encoding you find problematic? Is it decoding speed? The need to copy the encoded value into a native value in order to make use of it? Something else?
TLV encodings are always redundant and take more space than non-TLV encodings. Therefore they are a pessimization. As well, definite-length TLV encodings require two passes to encode the data (one to compute the length of the data to be encoded, and one to encode it), thus they are a) slower than non-TLV binary encodings, b) not on-line for encoding.
Yes all those are indeed pessimizations during encoding but are features when decoding: the decoder can skip decoding fields and tolerate unknown fields.
Now, you may disagree that tolerating unknown fields is a features (as many people do), but one must understand the context where protobuf has been designed, namely the situation where it takes time to roll out new versions of binaries that process the data (either in API calls or on stored files) and thus the ability to design a schema evolution with backward and forward compatibility is worth a few more cycles during encoding.
Not all users have that need and hence there exist other formats, but I wouldn't dismiss the protobuf encoding as flatly "wrong" just because you don't have the requirements it has been designed for.
Now, you may disagree that tolerating unknown fields is a features (as many people do), but one must understand the context where protobuf has been designed, namely the situation where it takes time to roll out new versions of binaries that process the data (either in API calls or on stored files) and thus the ability to design a schema evolution with backward and forward compatibility is worth a few more cycles during encoding.
Not all users have that need and hence there exist other formats, but I wouldn't dismiss the protobuf encoding as flatly "wrong" just because you don't have the requirements it has been designed for.
> the decoder can [...] tolerate unknown fields.
See ASN.1's extensibility rules. If you mark a record type (SEQUENCE, in ASN.1 parlance) as extensible, then when you later add fields (members, in ASN.1 parlance) the encoding rules have to make it possible to skip/ignore those fields when decoded by software implementing the pre-extension type. PER/OER will include a length for all the extension fields for this purpose, but it can be one length for all the extension fields in each round of extensions rather than one per (which would only save on the type "tag" in TLV).
> the decoder can skip decoding fields
This is mainly true for on-line decoders that produce `{path, leaf value}` tuples _and_ which take paths or path filters as arguments.
> Now, you may disagree that tolerating unknown fields is a features (as many people do), but one must understand the context where protobuf has been designed, namely the situation where it takes time to roll out new versions of binaries that process the data (either in API calls or on stored files) and thus the ability to design a schema evolution with backward and forward compatibility is worth a few more cycles during encoding.
This is the sort of thing I mean when I complain about the ignorant reinvention of the wheel that we all seem to engage in. It's natural and easy to do that, but it's not really a good thing.
Extensibility, versioning, and many many other issues in serialization are largely not new, and have been well-known and addressed for decades. ASN.1, for example, had no extensibility functionality in the early 1980s, but the only encoding rules at the time (BER/DER/CER), being TLV encodings, naturally supported extensibility ("skipping over unknown fields"). Later formal support for extensibility was added to ASN.1 so as to support non-TLV encodings.
ASN.1 also has elaborate support for "typed holes", which is what is referred to as "references" in [0].
ASN.1 gets a lot of hate, mainly for
And every time someone re-invents ASN.1 they miss important features that they were unaware of.
Meanwhile ASN.1 is pluggable as to encoding rules, and it's easy enough to extend the syntax too. So ASN.1 covers XML and JSON even. There's no other syntax/standard that one can say that for!
Next time anyone invents a new syntax and/or encoding rules, do please carefully look at what's come before.
See ASN.1's extensibility rules. If you mark a record type (SEQUENCE, in ASN.1 parlance) as extensible, then when you later add fields (members, in ASN.1 parlance) the encoding rules have to make it possible to skip/ignore those fields when decoded by software implementing the pre-extension type. PER/OER will include a length for all the extension fields for this purpose, but it can be one length for all the extension fields in each round of extensions rather than one per (which would only save on the type "tag" in TLV).
> the decoder can skip decoding fields
This is mainly true for on-line decoders that produce `{path, leaf value}` tuples _and_ which take paths or path filters as arguments.
> Now, you may disagree that tolerating unknown fields is a features (as many people do), but one must understand the context where protobuf has been designed, namely the situation where it takes time to roll out new versions of binaries that process the data (either in API calls or on stored files) and thus the ability to design a schema evolution with backward and forward compatibility is worth a few more cycles during encoding.
This is the sort of thing I mean when I complain about the ignorant reinvention of the wheel that we all seem to engage in. It's natural and easy to do that, but it's not really a good thing.
Extensibility, versioning, and many many other issues in serialization are largely not new, and have been well-known and addressed for decades. ASN.1, for example, had no extensibility functionality in the early 1980s, but the only encoding rules at the time (BER/DER/CER), being TLV encodings, naturally supported extensibility ("skipping over unknown fields"). Later formal support for extensibility was added to ASN.1 so as to support non-TLV encodings.
ASN.1 also has elaborate support for "typed holes", which is what is referred to as "references" in [0].
ASN.1 gets a lot of hate, mainly for
a) its syntax being ugly (true) and hard to parse (true-ish)
b) the standard being non-free (it used to be non-free, but it's free now, in PDF form anyways)
c) lack of tooling (which is not true anymore).
And every time someone re-invents ASN.1 they miss important features that they were unaware of.
Meanwhile ASN.1 is pluggable as to encoding rules, and it's easy enough to extend the syntax too. So ASN.1 covers XML and JSON even. There's no other syntax/standard that one can say that for!
Next time anyone invents a new syntax and/or encoding rules, do please carefully look at what's come before.
[0] https://en.wikipedia.org/wiki/Comparison_of_data-serialization_formatsYou make it sound like protobuf was invented yesterday. Sure it's not so old like ASN.1 but protobuf now about a quarter century old and battle tested as the main interchange format for at least one giant company with gazillions of projects that needed to interoperate.
One of the design requirements was simplicity and ease of implementation and for all the love in the world I can muster for ASN.1 I must admit it's far from simple.
IIRC complete and open implementations of ASN.1 were/are rare and the matrix of covered features didn't quite overlap between languages.
One of the design requirements was simplicity and ease of implementation and for all the love in the world I can muster for ASN.1 I must admit it's far from simple.
IIRC complete and open implementations of ASN.1 were/are rare and the matrix of covered features didn't quite overlap between languages.
A subset of ASN.1 can be simple enough. There was no need to re-invent the wheel, and to do it badly.
> IIRC complete and open implementations of ASN.1 were/are rare and the matrix of covered features didn't quite overlap between languages.
When protobufs was designed there were zero protobufs implementations.
> IIRC complete and open implementations of ASN.1 were/are rare and the matrix of covered features didn't quite overlap between languages.
When protobufs was designed there were zero protobufs implementations.
[deleted]
Another local first library. The movement is taking off.
Yet another title to drive embedded device designers and control engineers up the wall.
Why don't you make up your own bullshit words instead of randomly picking stuff from other places? That's not even multiplexing.
Why don't you make up your own bullshit words instead of randomly picking stuff from other places? That's not even multiplexing.