Making Win32 APIs More Accessible to More Languages(blogs.windows.com)
blogs.windows.com
Making Win32 APIs More Accessible to More Languages
https://blogs.windows.com/windowsdeveloper/2021/01/21/making-win32-apis-more-accessible-to-more-languages/
14 comments
There has to be a bottom to the stack. What I mean is, at some point types become bytes and instructions. Not every program or interaction is capable of being expressed at every levels of that stack, the invariants must be broken in code and maintained explicitly by programmers on either side of APIs. That simple fact is why FFI boundaries are inherently unsafe.
This doesn't have to do with C, it has to do with the nature of systems programming - not everything can be baked into the one type system to rule them all that is checked by all programs in all languages at compile time. Metadata is insufficient, you need runtime type checking, which is not free.
And to be honest I don't think this is an attractive thing to strive for. OS APIs are not that hard to program against (well, maybe Windows APIs are). Differences between OS's and versions are, however. We don't really need safer systems interfaces, what we need are standard ones that actually work to solve hard systems problems like asynchronous i/o, real time scheduling on non-realtime kernels, and arbitrary control flow. These are very unsafe things that do not have good solutions in the OS APIs, adding strict types where they are not needed creates little value for the systems programmers that want to make cool new systems.
This doesn't have to do with C, it has to do with the nature of systems programming - not everything can be baked into the one type system to rule them all that is checked by all programs in all languages at compile time. Metadata is insufficient, you need runtime type checking, which is not free.
And to be honest I don't think this is an attractive thing to strive for. OS APIs are not that hard to program against (well, maybe Windows APIs are). Differences between OS's and versions are, however. We don't really need safer systems interfaces, what we need are standard ones that actually work to solve hard systems problems like asynchronous i/o, real time scheduling on non-realtime kernels, and arbitrary control flow. These are very unsafe things that do not have good solutions in the OS APIs, adding strict types where they are not needed creates little value for the systems programmers that want to make cool new systems.
So much this. People (and languages) seem to forget that an HTTP request is a string of bytes (another “bottom of the stack”). A JSON object isn’t a type, it’s a string of bytes.
I would say it's more accurate to call an HTTP request or a JSON object a subset of the set of all strings.
If step out the realm of sets and into programing languages types are very similar to sets. Heck there is type theory which also is very similar to set theory, but has a few difference.
If step out the realm of sets and into programing languages types are very similar to sets. Heck there is type theory which also is very similar to set theory, but has a few difference.
[deleted]
A Rust `bool` is not a WinAPI `BOOL`, for one thing they have different sizes. A wrapper function could translate between the two... but even that's not universally possible. Depending on the API, `BOOL` can sometimes have three or more values (e.g. -1, 0 or 1). So whatever happens you need a safe wrapper around the function call that understands what `BOOL` means in this specific context.
Incidentally, there is no reason why the metadata can't be improved with more information that can be used to create safer wrappers. The winmd format isn't restricted to only that which can be represented in C's type system.
Incidentally, there is no reason why the metadata can't be improved with more information that can be used to create safer wrappers. The winmd format isn't restricted to only that which can be represented in C's type system.
winmd is limited to the .NET type system - an improvement over C, but still insufficiently expressive enough imo.
I suppose you could hack it by adding new Attributes...
I suppose you could hack it by adding new Attributes...
See: https://github.com/microsoft/win32metadata#winmd-emitter-ove...
It seems at least one of their hacks is not valid in the CLR, so that bridge has been crossed. Plus they do make use of attributes.
It seems at least one of their hacks is not valid in the CLR, so that bridge has been crossed. Plus they do make use of attributes.
I think this C# intermediate is just a tooling thing. The winmd format is already extended beyond .NET IMHO.
And I think, they have thought about that before they startet :)
And I think, they have thought about that before they startet :)
Some things aren’t expressable in winmd though - like Code Contracts’ preconditions, invariants, and postconditions: those require directly executable (but pure, stateless, and side-effect-free) methods to represent and encode those contracts. As Winmd binaries can’t contain executable code it means you’re limited to only using “common”/“known” contract functions and/or resorting to re-encoding the contract to a string containing the expression that’s stored in an attribute. Good luck with that.
Who does still use Code Contracts besides legacy code? They have been long dead.
I was only using it as an example of a concept that isn't well-representable in winmd.
As for CC specifically, I'd love to use it with .NET Core - I gather it doesn't need that much work to get going again. It's a shame Microsoft hasn't invested more into it, especially as there was talk of making CC an integral part of the C# syntax (as in, first-class support for `pure` functions in C# and a succinct syntax for attaching them as invariants/etc to C# types and methods). If this was done it would solve so many problems (e.g. compiler-enforced guarantees about which navigation-properties on Entity Framework entities are loaded) and maybe even elevate C# to compete with Ada and Eiffel for safety-critical systems!
As for CC specifically, I'd love to use it with .NET Core - I gather it doesn't need that much work to get going again. It's a shame Microsoft hasn't invested more into it, especially as there was talk of making CC an integral part of the C# syntax (as in, first-class support for `pure` functions in C# and a succinct syntax for attaching them as invariants/etc to C# types and methods). If this was done it would solve so many problems (e.g. compiler-enforced guarantees about which navigation-properties on Entity Framework entities are loaded) and maybe even elevate C# to compete with Ada and Eiffel for safety-critical systems!
Requiring VS Enterprise to actually use them properly, doomed their adoption.
WinRT/UAP/UWP (got to love marketing) is just COM as usual, with winmd instead of type libraries, IInspectable as base interface alongside IUnknown and extended ABI for generics, enums, arrays, structs.
Here is a nice overview of the whole story and how interrelated COM and .NET are.
https://arstechnica.com/features/2012/10/windows-8-and-winrt...
Here is a nice overview of the whole story and how interrelated COM and .NET are.
https://arstechnica.com/features/2012/10/windows-8-and-winrt...
> One issue I've had with any kind of auto-generated bindings based on metadata is that you get low-level types such as "Win32 BOOL" and "Win32 GUID" polluting the output. Mainstream languages already either have native bool and guid types, or have defacto standard libraries.
This shouldn't actually be a huge problem in Rust, because of the traits system. It should be possible to implement all the expected behaviors (Into/From, Add/Sub/Mul/Div, Ord/Eq, etc) for these special types and if library code is properly generic, it shouldn't care whether or not you're using the native type. If it does require a specific concrete type, From/Into should still make things reasonably ergonomic.
Re: safety, the typical approach has just been to keep one library that's a direct mapping to the C API, which is publicly unsafe, and then create a safe library abstracted on top of that which codifies the documented behaviors. That layer relies on documentation, and on humans to follow it, but then so does all C code ever written :)
This shouldn't actually be a huge problem in Rust, because of the traits system. It should be possible to implement all the expected behaviors (Into/From, Add/Sub/Mul/Div, Ord/Eq, etc) for these special types and if library code is properly generic, it shouldn't care whether or not you're using the native type. If it does require a specific concrete type, From/Into should still make things reasonably ergonomic.
Re: safety, the typical approach has just been to keep one library that's a direct mapping to the C API, which is publicly unsafe, and then create a safe library abstracted on top of that which codifies the documented behaviors. That layer relies on documentation, and on humans to follow it, but then so does all C code ever written :)
In my experience with both classic Win32 programming in the 1990s in C++ and then later in C# and now in Rust is that it is obscenely verbose. It takes a solid page of code to call a single method, and most of that is just noise.
A typical method call is:
PS: Win32 strings and Rust strings aren't compatible. I've already noticed that the current version of the Microsoft code-generator seems to use an awful lot of [u16] arrays, which are not the system OsString struct.
For comparison, the C# "String" type isn't directly compatible with Win32 strings either, but the FFI engine handles this conversion back-and-forth for you.
Similarly, few (any?) languages have good built-in support for the various kinds of "struct-followed-by-dynamic-data" typically returned by system APIs. Notably, C does not have any way to encode this in the type system, so API metadata tends to just leave this out or describe it in the documentation. If the metadata was extented to cover this kind of thing, then the generators could do all sorts of clever things with this scenario...
[1] A lot of people don't do this, they assume that the second call will always succeed. Programming in this style leads to rare random errors throughout the entire application. A loop is required for correctness for most (but not all) Win32 calls that return dynamic data...
A typical method call is:
Allocate some struct.
Zero it out.
Set a "length" field to the length of the struct.
Populate the "parameter" fields.
Call the API with the struct and 0 for the return buffer size.
You get back an expected error code for "buffer too small".
You allocate the required space in your buffer.
Call the method again. The amount of data changed! Loop and try again.[1]
You get your data! Now to check the error codes.
Look up error code using a Win32 method.
Free the buffer.
PS: Win32 strings and Rust strings aren't compatible. I've already noticed that the current version of the Microsoft code-generator seems to use an awful lot of [u16] arrays, which are not the system OsString struct.
For comparison, the C# "String" type isn't directly compatible with Win32 strings either, but the FFI engine handles this conversion back-and-forth for you.
Similarly, few (any?) languages have good built-in support for the various kinds of "struct-followed-by-dynamic-data" typically returned by system APIs. Notably, C does not have any way to encode this in the type system, so API metadata tends to just leave this out or describe it in the documentation. If the metadata was extented to cover this kind of thing, then the generators could do all sorts of clever things with this scenario...
[1] A lot of people don't do this, they assume that the second call will always succeed. Programming in this style leads to rare random errors throughout the entire application. A loop is required for correctness for most (but not all) Win32 calls that return dynamic data...
C99 has made all that a lot easier, you can setup, initialize a struct and call the function taking a pointer to that struct all in one call:
https://github.com/floooh/sokol-samples/blob/89f5825ab5d3690...
The Win32 window system functions and D3D11 are actually very convenient to work with in C99, better than C++ actually.
https://github.com/floooh/sokol-samples/blob/89f5825ab5d3690...
The Win32 window system functions and D3D11 are actually very convenient to work with in C99, better than C++ actually.
You're not wrong that win32 is very verbose.
> A loop is required for correctness for most (but not all) Win32 calls that return dynamic data...
A very good point, one very rarely mentioned.
Of course, the win32 windowing and common controls and single threaded, so if it's the same application it won't have changed underneath you.
> A loop is required for correctness for most (but not all) Win32 calls that return dynamic data...
A very good point, one very rarely mentioned.
Of course, the win32 windowing and common controls and single threaded, so if it's the same application it won't have changed underneath you.
Yes as a simple example of a wrapper:
std::string nicerGetComputerName() {
char buf[1024];
DWORD len = sizeof(buf);
memset(buf, 0, sizeof(buf));
GetComputerNameA(buf, &len);
return buf;
}#1) GetComputerNameA can fail. You didn't check the error message or convert it into a C++ exception.
#2) You probably want GetComputerNameW. GetComputerNameA uses an unspecified encoding that you'd have to handle.
#3) A 1KB buffer is excessive! Over-sized buffers can blow through the L1 cache all too easily, slowing down applications. It's also stack allocated, and this can be a problem in some scenarios.
#4) You're forcing the computer name to be heap allocated. This can also be problematic in some scenarios, and is slower. GetComputerName might be used in logging scenarios that are already trying to handle cases such as "out of memory"! Allocating more memory may be unacceptable.
You get the idea.
#2) You probably want GetComputerNameW. GetComputerNameA uses an unspecified encoding that you'd have to handle.
#3) A 1KB buffer is excessive! Over-sized buffers can blow through the L1 cache all too easily, slowing down applications. It's also stack allocated, and this can be a problem in some scenarios.
#4) You're forcing the computer name to be heap allocated. This can also be problematic in some scenarios, and is slower. GetComputerName might be used in logging scenarios that are already trying to handle cases such as "out of memory"! Allocating more memory may be unacceptable.
You get the idea.
What encoding will the string returned from GetComputerNameA be in? Is it the code page for the current regional setting?
Wouldn't it be safer to use GetComputerNameW to get a UTF-16 string?
Wouldn't it be safer to use GetComputerNameW to get a UTF-16 string?
FYI:
char buf[1024] = {0};To be fair everything starting from "Call the API with the struct and 0 for the return buffer size." Is fairly uncommon, at least in my experience.
I'm tackling this issue from two sides:
(1) Change the C-API to make it more "binding-generator-friendly", for instance by adding a range/slice-struct to th C-API which bundles a pointer and associated size, or specially named typedefs that only exist to give the binding generator hints for special case handling. By adding a C-type which represents a range/slice, it's easier to connect high level languages to the C API, and as a nice side effect, the C-API also becomes safer and more "correct", because pointers and associated sizes can be passed around as one item.
(2) Make the bindings-generator configurable on a per-language and per-API basis, this can be as simple as a map which overrides type- and function-names, or injects manually written code into the generated bindings.
The goal is to make the generated bindings more idiomatic to the target language.
This mostly works if you have control over the underlying C-API of course, e.g. the language bindings are created by the original C-library project, not as an external project to convert a fixed C-API.
I wrote a blog post about this whole topic:
https://floooh.github.io/2020/08/23/sokol-bindgen.html
...and here's an example of such semi-auto-generated Zig bindings:
https://github.com/floooh/sokol-zig/tree/master/src/sokol
...for instance note the "injected" helper functions here:
https://github.com/floooh/sokol-zig/blob/1c93f60ad178869b84d...
(1) Change the C-API to make it more "binding-generator-friendly", for instance by adding a range/slice-struct to th C-API which bundles a pointer and associated size, or specially named typedefs that only exist to give the binding generator hints for special case handling. By adding a C-type which represents a range/slice, it's easier to connect high level languages to the C API, and as a nice side effect, the C-API also becomes safer and more "correct", because pointers and associated sizes can be passed around as one item.
(2) Make the bindings-generator configurable on a per-language and per-API basis, this can be as simple as a map which overrides type- and function-names, or injects manually written code into the generated bindings.
The goal is to make the generated bindings more idiomatic to the target language.
This mostly works if you have control over the underlying C-API of course, e.g. the language bindings are created by the original C-library project, not as an external project to convert a fixed C-API.
I wrote a blog post about this whole topic:
https://floooh.github.io/2020/08/23/sokol-bindgen.html
...and here's an example of such semi-auto-generated Zig bindings:
https://github.com/floooh/sokol-zig/tree/master/src/sokol
...for instance note the "injected" helper functions here:
https://github.com/floooh/sokol-zig/blob/1c93f60ad178869b84d...
I believe that it is best for this kind of automatic binding to expose the raw native types, and not use the language idiomatic ones.
I mean, if you need to use the raw win32 api, there must be a reason you need raw access, and in such a case, conversion to language types is more likely to be a hindrance than help.
Also, if you need to call multiple native apis, it is easier to pass the native types around if the is no conversion to language types.
I mean, if you need to use the raw win32 api, there must be a reason you need raw access, and in such a case, conversion to language types is more likely to be a hindrance than help.
Also, if you need to call multiple native apis, it is easier to pass the native types around if the is no conversion to language types.
The win32metadata repository has documentation on type renaming, and mapping integers into .NET enumerations.
It is up to the generation tool to provide idiomatic bindings based on the information provided by win32metadata.
This is nothing new, WinRT works exactly like this across .NET, C++, JS, Rust and whatever language projection one feels like adding.
It is up to the generation tool to provide idiomatic bindings based on the information provided by win32metadata.
This is nothing new, WinRT works exactly like this across .NET, C++, JS, Rust and whatever language projection one feels like adding.
What is new is the definitions for more than just WinRT.
Yes, this is a welcome milestone from Project Reunion.
I was the de facto maintainer of JNA (https://github.com/java-native-access/jna) win32 bindings for years. The API metadata is welcome - we have always based all work on header files in the visual studio / windows SDK. Now there’s a change to generate that code - someone might want to try to replace hand crafted mappings in JNA with generated ones - there’s a huge test suite to work with.
First of all, thanks for the hard work.
I guess there is also the possibility to use the JNI replacement when it comes out of preview, although that would be constrained to Java 17 or later.
I guess there is also the possibility to use the JNI replacement when it comes out of preview, although that would be constrained to Java 17 or later.
Golang is also considering consuming this new metadata: https://github.com/golang/go/issues/43838
A good thing that is late for 25 years. Still better than never, WinAPIs never were an easiest thing to wrap.
It's a great move but nearly every lang on Earth has a solid 3rd party Win32 library built over years and battle hardened.
Waiting patiently for Microsoft to actually put some effort into the obscure TPM documentation along with making it easier for developers to implement.
Waiting patiently for Microsoft to actually put some effort into the obscure TPM documentation along with making it easier for developers to implement.
The TPM infrastructure is full of leaks that can’t be fixed, they’re not going to expose that with documentation. It’s like asking them to document a vendor to open-source their DRM.
> full of leaks that can’t be fixed
Not claiming it's perfect, saying it exists on nearly every cpu running Windows today. There's serious value for plenty of people out there to have access to these interfaces that are only internally documented and available to a small group of people from one company alone.
Only have to look at the history of adding randomness to CryptographyAPI-NextGen to understand where I'm coming from.
No one has been able to add an additional RNG source to the Windows entropy pool for ~15 years now, highly venerable Microsoft developers say this is a good thing in public for all their enterprise clients when basic maths proves it wrong in about 2 minutes.
They do not have your best interests at heart, no matter how big you are, in fact it's less likely the bigger you are.
Not claiming it's perfect, saying it exists on nearly every cpu running Windows today. There's serious value for plenty of people out there to have access to these interfaces that are only internally documented and available to a small group of people from one company alone.
Only have to look at the history of adding randomness to CryptographyAPI-NextGen to understand where I'm coming from.
No one has been able to add an additional RNG source to the Windows entropy pool for ~15 years now, highly venerable Microsoft developers say this is a good thing in public for all their enterprise clients when basic maths proves it wrong in about 2 minutes.
They do not have your best interests at heart, no matter how big you are, in fact it's less likely the bigger you are.
Rust for Windows uses this.
https://news.ycombinator.com/item?id=25862291
Now if MacOS followed suit, all would be right in the universe.
Interesting. The real thing here is the Win32 Metadata project: https://github.com/microsoft/win32metadata
It contains definitions that can be used to automatically generate bindings to any language. So, we have C# and Rust as examples, but the very same definitions could be used with languages like Zig.
Very cool initiative.
It contains definitions that can be used to automatically generate bindings to any language. So, we have C# and Rust as examples, but the very same definitions could be used with languages like Zig.
Very cool initiative.
I mainly use C#, and this is late, but really welcome!
I usually use Vanara for Win32 PInvoke wrappera, or hand-code them if they are undocumented or I only need a couple. Rarely I've come across memory issues with the wrappers provided by Vanara and PInvoke.net, where they've forgotten to return a buffer - these kind of problems are absolute bastards to track down, and the root cause is of course manually building wrapper code. Having an automated means to do this will be a huge boon.
I wonder though - quite often if I'm digging into Win32 stuff, it means I'm working on performance sensitive code, where I want to keep allocations low - I haven't looked I to it yet, but I hope C#/Win32 has an option to generate wrappers that take pointers instead of only using managed types!
I usually use Vanara for Win32 PInvoke wrappera, or hand-code them if they are undocumented or I only need a couple. Rarely I've come across memory issues with the wrappers provided by Vanara and PInvoke.net, where they've forgotten to return a buffer - these kind of problems are absolute bastards to track down, and the root cause is of course manually building wrapper code. Having an automated means to do this will be a huge boon.
I wonder though - quite often if I'm digging into Win32 stuff, it means I'm working on performance sensitive code, where I want to keep allocations low - I haven't looked I to it yet, but I hope C#/Win32 has an option to generate wrappers that take pointers instead of only using managed types!
There was some discussion on this earlier today: https://news.ycombinator.com/item?id=25885196
I gave their Rust bindings a try, but they still have work to do to even match the manual work that has been put into the winapi crate. I can say from experience is long, tedious, and error prone so it will definitely be better to be auto-generating the bindings.
Reminds me of Dan Appleman’s VB Win32 API book from ~20 years ago.
Oh so it arrived late, but better come now than never. Also it's super nice that I can use RUST for it.
The best way to access Win32 APIs has always been Delphi's VCL
E.g.: Rust has the UUID crate: https://crates.io/crates/uuid
Similarly, Java has had a UUID type since v1.5: https://docs.oracle.com/javase/8/docs/api/java/util/UUID.htm...
It's not just concrete types, but core language concepts often "don't translate". E.g.: Win32 is essentially a C-like set of bindings. In many languages such as C++, D, Zig, etc... this is not an issue. However the type system of C is much less strict that Rust, so the metadata has no concept of ownership and lifetimes. This means that if you want to consume Win32 APIs from Rust, you're still forced to wrap everything manually at least once to escape the "unsafe" sandpit.
A thought I had is that C is used for API definitions because "it is lowest common denominator". But that's the wrong thing to do, the metadata doesn't have to "sink to the lowest level"! An ideal metadata definition would be something akin to Rust's type system, or even stronger than that! "Degrading" a binding from a strong type system to a weak one is trivial and can be performed completely automatically. The converse is difficult and error-prone even for humans, and impossible for code generators. You can't synthesise this missing data!
Even the read-only guarantee is too weak. E.g.:
The two methods have identical metadata signatures, but the information required by a Rust programmer or a Rust code generator is missing...
There are many such abstract API concepts missing from C, such as thread safety, process integrity safety, non-returning calls, blocking vs async, etc...