Embed is in C23(thephd.dev)
thephd.dev
Embed is in C23
https://thephd.dev/finally-embed-in-c23
338 comments
Ha I suggested this on the C++ proposals mailing list 7 years ago:
https://groups.google.com/a/isocpp.org/g/std-proposals/c/b6n...
Enjoy the naysayers if you like! I'm glad someone spent the time and effort to push past them. Bit too late for me - I have moved on to Rust which had support for this from version 1.0.0.
> There's also the standard *nix/BSD utility "xxd".
> Seems like the niche is filled. Or, at least, if you want to claim that
> (A) XPM
> (B) incbin
> (C) "xxd -i"
> (D) various ad-hoc scripts given in http://stackoverflow.com/questions/8707183/script-tool-to-co...
>...do NOT completely fill this evolutionary niche
> This ultimately would encourage a weird sort of resource management philosophy that I think might be damaging in the long run.
> Speaking from experience, it is a tremendously bad idea to bake any resource into a binary.
> I'll point out that this is a non-issue for Qt applications that can simply use Qt's resources for this sort of business.
(Though credit to Matthew Woehlke, he did point out a solution which is basically identical to #embed)
> I find this useless specially in embedded environments since there should be some processing of the binary data anyway, either before building the application
In fairness there was a decent amount of support. But given the insane amount of negativity around an obviously useful feature I gave up.
I wonder if there was a similar response to the proposal to include `string::starts_with()`...
https://groups.google.com/a/isocpp.org/g/std-proposals/c/b6n...
Enjoy the naysayers if you like! I'm glad someone spent the time and effort to push past them. Bit too late for me - I have moved on to Rust which had support for this from version 1.0.0.
> There's also the standard *nix/BSD utility "xxd".
> Seems like the niche is filled. Or, at least, if you want to claim that
> (A) XPM
> (B) incbin
> (C) "xxd -i"
> (D) various ad-hoc scripts given in http://stackoverflow.com/questions/8707183/script-tool-to-co...
>...do NOT completely fill this evolutionary niche
> This ultimately would encourage a weird sort of resource management philosophy that I think might be damaging in the long run.
> Speaking from experience, it is a tremendously bad idea to bake any resource into a binary.
> I'll point out that this is a non-issue for Qt applications that can simply use Qt's resources for this sort of business.
(Though credit to Matthew Woehlke, he did point out a solution which is basically identical to #embed)
> I find this useless specially in embedded environments since there should be some processing of the binary data anyway, either before building the application
In fairness there was a decent amount of support. But given the insane amount of negativity around an obviously useful feature I gave up.
I wonder if there was a similar response to the proposal to include `string::starts_with()`...
> told me this form was non-ideal and it was worth voting against (and that they’d want the pure, beautiful C++ version only[1])
I heard about #embed, but I didn't hear about std::embed before. After looking at the proposal, to me it does look a lot better than #embed, because reading binary data and converting it to text, only to then convert it to binary again seems needlessly complex and wasteful. I also don't like that it extends the preprocessor, when IMHO the preprocessor should at worst be left as is, and at best be slowly deprecated in favour of features which compose well with C proper.
Going beyond the gut reaction and moving on to hard data, as you can expect from this design, std::embed of course is faster during compilation than #embed for bigger files (comparable for moderately-sized files, and a bit slower for tiny files).
I'm not a huge fan of C++, but the fact that C++ removed trigraphs in C++17 and that it's generally adding features replacing the preprocessor scores a point with me.
[1]: <https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2020/p10...>
I heard about #embed, but I didn't hear about std::embed before. After looking at the proposal, to me it does look a lot better than #embed, because reading binary data and converting it to text, only to then convert it to binary again seems needlessly complex and wasteful. I also don't like that it extends the preprocessor, when IMHO the preprocessor should at worst be left as is, and at best be slowly deprecated in favour of features which compose well with C proper.
Going beyond the gut reaction and moving on to hard data, as you can expect from this design, std::embed of course is faster during compilation than #embed for bigger files (comparable for moderately-sized files, and a bit slower for tiny files).
I'm not a huge fan of C++, but the fact that C++ removed trigraphs in C++17 and that it's generally adding features replacing the preprocessor scores a point with me.
[1]: <https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2020/p10...>
This serves the same use as Rust's `include_bytes!` macro, right? Presumably most people just use this feature as a way to avoid having to stuff binary data into a massive array literal, but in our case it's essential because we're actually using it to stuff binaries from earlier in our build step into a binary built later in the build step. Not something you often need, but very handy when you do.
> Even among people who control all the cards, they are in many respects fundamentally incapable of imagining a better world or seizing on that opportunity to try and create one, let alone doing so in a timely fashion.
That does sound soul-crushing. Congrats on this achievement!
That does sound soul-crushing. Congrats on this achievement!
This reminds me, I'd argue that the explosion of JS frameworks can be mainly blamed on one thing: the lack of an <include src="somemodule.html"> tag. If you have that you basically have 80% of vue.js already natively supported. No clue why this was never added in any fashion. Change my mind.
Honestly I'm usually very wary of additions to C, as one of its greatest strengths (to me) is how rather straightforward it is as a language in terms of conceptual simplicity. There just aren't that many big concepts to understand in the language. (On the other hand there's _many_ footguns but that's another issue.)
That said, to me this seems like a great addition to the language. It's very single-purpose in its usage (so it doesn't seem to add much conceptual complexity to the language) and it replaces something genuinely painful (arcane linker hacks). I'm very much looking forward to using this as I often make single-executable programs in C. The only thing that's unfortunate is I'm sure it'll take decades before proprietary embedded toolchains add support for this.
That said, to me this seems like a great addition to the language. It's very single-purpose in its usage (so it doesn't seem to add much conceptual complexity to the language) and it replaces something genuinely painful (arcane linker hacks). I'm very much looking forward to using this as I often make single-executable programs in C. The only thing that's unfortunate is I'm sure it'll take decades before proprietary embedded toolchains add support for this.
There is way too much in C already.
The first commandment of C is: 'writing a naive C compiler should be "reasonable" for a small team or even one individual'. That's getting harder and harder, longer and longer.
I did move from C being "the best compromise" to "the less worse compromise".
I wish we had a "C-like" language, which would kind of be a high-level assembler which: has no integer promotion or implicit casts, has compile-time/runtime casts (without the horrible c++ syntax), has sized primitive types (u64/s64,f32/f64,etc) at its core, has sized literals (42b,12w,123dw,2qw,etc), has no typedef/generic/volatile/restrict/etc well that sort of horrible things, has compile-time and runtime "const"s, and I am forgetting a lot.
From the main issues: the kernel gcc C dialect (roughly speaking, each linux release uses more gcc extensions). Aggressive optimizations can break some code (while programing some hardware for instance).
Maybe I should write assembly, expect RISC-V to be a success, and forget about all of this.
The first commandment of C is: 'writing a naive C compiler should be "reasonable" for a small team or even one individual'. That's getting harder and harder, longer and longer.
I did move from C being "the best compromise" to "the less worse compromise".
I wish we had a "C-like" language, which would kind of be a high-level assembler which: has no integer promotion or implicit casts, has compile-time/runtime casts (without the horrible c++ syntax), has sized primitive types (u64/s64,f32/f64,etc) at its core, has sized literals (42b,12w,123dw,2qw,etc), has no typedef/generic/volatile/restrict/etc well that sort of horrible things, has compile-time and runtime "const"s, and I am forgetting a lot.
From the main issues: the kernel gcc C dialect (roughly speaking, each linux release uses more gcc extensions). Aggressive optimizations can break some code (while programing some hardware for instance).
Maybe I should write assembly, expect RISC-V to be a success, and forget about all of this.
I’m really amazed at how divisive this one is, and the number of comments here questioning what seems to me a really useful and well-thought-out feature, something I’d have loved to have used many many times over the years.
I guess the heated arguments here help me understand how it could have taken so long to get this standardised, though, so that’s something!
Congratulations and thank you to the OP for doing this, and thanks also for this really interesting (if depressing) view of the process.
I guess the heated arguments here help me understand how it could have taken so long to get this standardised, though, so that’s something!
Congratulations and thank you to the OP for doing this, and thanks also for this really interesting (if depressing) view of the process.
This is a really, really good feature and I am so glad it is finally getting standardized. C23 is shaping up to be a very good revision to the C standard. I’m hoping the proposal to allow redeclaration of identical structs gets in as well as you would finally be able to write code using common types without having to coordinate which would allow interoperability between independently written libraries.
Congratulations to the author. Things like this are why I hope Carbon exists. Evolving c++ seems like a dumpster fire, despite whatever compelling arguments about comparability you are going to drop on me.
Re: #embed </dev/urandom>
Just a random thought, but I'd expect a compiler to do exactly what's described if I tell it:
Just a random thought, but I'd expect a compiler to do exactly what's described if I tell it:
static char foo[123] = {
#embed </dev/urandom>
};
Interesting. I look forward to this. What I've been doing now to embed a source.png file is something like this, where I generate source code from a file's data:
in embed_dump.cpp:
in embed_dump.cpp:
#include <fstream>
#include <iostream>
int main(){
std::ifstream f;
f.open("./source.png");
std::cout
<< "//automatically generated by embed_dump from project files:" << std::endl
<< "const char embedded_tex[] = {";
char a;
while(f.good()){
f.read(&a,1);
std::cout << int(a) << ",";
}
std::cout << "};" << std::endl;
f.close();
}
main_stuff: main_stuff.cpp embedded_files.h
c++ main_stuff.cpp
embeded_files.h: embed_dump source.png
./embed_dump > embeded_files.h
embed_dump: embed_dump.cpp
c++ embed_dump.cpp -o embed_dumpWhat about creating object files from raw binary files and then linking against them? That's what I (and of course many others) do for linking textures and shaders into the program. It's a bit ugly though that with this approach you can't generate custom symbol names, at least with the GNU linker.
This #embed feature might be a nice alternative for small files. Well for large files you usually don't even want to store them inside the binary, so the compilation overhead might be miniscule, since the files are, by intention, small.
When I read the introduction of the article - about allowing us to cram anything we want into the binary - I was hoping to see a standard way to disable optimizations (When the compiler deletes your code and you don't even notice).
This #embed feature might be a nice alternative for small files. Well for large files you usually don't even want to store them inside the binary, so the compilation overhead might be miniscule, since the files are, by intention, small.
When I read the introduction of the article - about allowing us to cram anything we want into the binary - I was hoping to see a standard way to disable optimizations (When the compiler deletes your code and you don't even notice).
Off the top of my head, I think there's some niche use in embedding shaders so that they don't need to be stored as strings (no IDE support) or read at runtime (slower performance).
Other stuff:
https://twitter.com/rcs/status/1550526425211584512
nullptr! auto! constexpr!
https://twitter.com/rcs/status/1550526425211584512
nullptr! auto! constexpr!
If you want to start playing with this now, my C preprocessor Cedro (https://news.ycombinator.com/item?id=28166125, https://news.ycombinator.com/item?id=31193138) will insert the byte literals for the compiler:
https://sentido-labs.com/en/library/cedro/202106171400/#bina...
That’s the same that `xxd` does, for instance, and it works with C89/C99/C11.
The advantage is that since it uses the same syntax as C23, it is easier to switch to using the compiler: just remove the Cedro pragma `#pragma Cedro 1.0 #embed` and it will compile as C23.
The source code (Apache 2.0) and GitHub link are at: https://sentido-labs.com/en/library/
https://sentido-labs.com/en/library/cedro/202106171400/#bina...
That’s the same that `xxd` does, for instance, and it works with C89/C99/C11.
The advantage is that since it uses the same syntax as C23, it is easier to switch to using the compiler: just remove the Cedro pragma `#pragma Cedro 1.0 #embed` and it will compile as C23.
The source code (Apache 2.0) and GitHub link are at: https://sentido-labs.com/en/library/
Looks great. I've been writing cmake hacks to include assets in executables for too long.
Can’t you just add binary data into a custom section of your ELF executable?
5 years ago I wrote a small python script [1] to help me solve "the same problem".
It reads files in a folder and generates an header file containing the files' data and filenames.
Is very simple and was to helping me on a job. It has limitations, don't be too hard on me :)
[1] https://github.com/daxliar/pyker
[1] https://github.com/daxliar/pyker
This will simplify a lot of build pipelines for sure.
One thing that isn't clear from skimming the article, how do you refer to the embedded data again?
One thing that isn't clear from skimming the article, how do you refer to the embedded data again?
Everlasting glory to JeanHeyd Meneide, @thephantomderp, for getting this feature into C.
I am wondering, though - where does this stand in C++?
I am wondering, though - where does this stand in C++?
[deleted]
This is a cool feature and I'll likely be using it in the years to come. However, the posix standard command xxd and its -i option can achieve this capability portably today.
It will be useful to achieve it directly in the preprocessor however. I wonder how quickly can it be added to cpp?
It will be useful to achieve it directly in the preprocessor however. I wonder how quickly can it be added to cpp?
I've always used xxd -i for embedding, doesn't have the mentioned problems and works everywhere, as it simply outputs a header file with byte array.
It is something that I had wanted in C too, for a while, so I am glad that they added this #embed command.
How to read unsigned data? Is there a stadardized parameter, or does this require a vendor extension?
What situations is this useful for?
> “Touch grass”, some people liked to tell me. “Go outside”, they said (like I would in the middle of Yet Another Gotdang COVID-19 Spike). My dude, I’ve literally gotten company snail mail, and it wasn’t a legal notice or some shenanigans like that! Holy cow, real paper in a real envelope, shipped through German Speed Mail!! This letter alone probably increases my Boomer Cred™ by at least 50; who needs Outside anymore after something like this?
Touch grass indeed. Sure, #embed is a nice feature, but this self-indulgent writing style I can’t stand.
Touch grass indeed. Sure, #embed is a nice feature, but this self-indulgent writing style I can’t stand.
This is wrong.
It belongs in the linker and you can pull the symbol it creates in with extern. I’ve been doing this for about 25 years.
It belongs in the linker and you can pull the symbol it creates in with extern. I’ve been doing this for about 25 years.
Cs main strengthen is its portability and simplicity. Therefore we should be very conservative, and not add anything quickly. There are plenty of languages to choose form if you want a "modern" language with lots of conveniences. If you want a truly portable language there is really only C. And when I say truly, I mean for platforms without file systems, or operating systems or where bytes aren't 8 bits, that doesn't use ASCI or Unicode, where NULL isn't on address 0 and so on.
We are the stewards of this, and the work we put in, while large, is tiny compared to the impact we have. Any change we makes, needs to be addressed by every compiler maintainer. There are millions of lines of code that depend on every part of the standard. A 1% performance loss is millions of tons of CO2 released, and billions in added hardware and energy costs.
In this privileged position, we have to be very mindful of the concerns of our users, and take the time too look at every corner case in detail before adding any new features. If we add something, then people will depend on its behavior, no matter how bad, and we therefor will have great difficulty in fixing it in the future without breaking our users work, so we have to get it right the first time.