>In many ways NT was a new, ground up implementation of “VMS NT”.
Most definitely. There was a lot of design cleanup from VMS (e.g. fork processes -> DPCs, removing global PTEs and balance slots, etc), optimizations (converging VMS's parallel array structure of the PFN database into one), and simplification (NT's Io subsystem with the "bring your own thread" model, removing P1 space, and much more). SMP was also designed into NT from the beginning. You can start seeing the start of these ideas in the MICA design documents but their implementation in C instead of Pillar (variant of Pascal designed for Mica) in NT was definitely the right thing at the time.
Before VMS there was the family of RSX-11 operating systems which also had ASTs (now called APCs in NT parlance), IRPs, etc. Dave Cutler led the RSX-11M variant which significantly influenced VMS. The various concepts and design styles of the DEC family of operating systems that culminated in NT goes back to the 1960s.
It's sad that the article didn't mention VMS or MICA since NT didn't magically appear out of the void two years after Microsoft hired the NT team. MICA was being designed for years at DEC West as part of the PRISM project.
>I don’t think anyone has ever bothered to make FS code async, it’s already quite horrible as it is. Not in a mainstream kernel anyway. [...] Same for Windows disk IO with IOCPs.
IOCP merely allows IO completion to happen on worker threads in a threadpool rather than the particular thread that initiated the IO request. Many concepts in the NT kernel come from a long line of production kernels in mainstream DEC operating systems going back to RSX-11M and VMS. That entire lineage of operating systems (culminating in NT) all have true asynchronous IO using IO Request Packets (IRPs) within the kernel to initiate/queue IO requests and immediately return. There are individual cases within NTFS where blocking will occur but those are special cases rather than the rule as the kernel IO system in general is entirely async.
"Embedded software" and "embedded system" also have similar issues with being nebulously defined in various situations. Some things will never perfect and at least firmware is kind of cute.
Usually this is still host firmware and not a secondary controller, at least on x86 platforms. To detect/use the USB controller you still need to configure the chipset/root complex and do an initial PCIe bus scan to set up PCIe BAR apertures. After that occurs you need a (primitive) USB stack that is able to talk to the USB controller to enumerate the USB devices as well as block storage and filesystem layers. All of this code is implemented as a collection of DXE drivers that implement various UEFI protocols which are initialized in the UEFI DXE phase that runs after SEC and PEI phases. On Intel platforms PEI does things like training the DRAM and PCIe links so memory is always available to DXE. Unfortunately, there's still a lot of code that needs to run to get this to work.
After FW update binaries are located it's not uncommon to write them to a scratch flash and then reset the system. On reset somewhere in the flow the scratch flash is checked for an update and then the hardware sequenced flashing registers in the chipset are utilized to actually flash the firmware. Another reset is performed to boot from the freshly flashed firmware.
There are variations on this flow depending on the firmware implementation and platform/vendor which can simplify it but that is usually the basic idea. Various microcontrollers are definitely employed for other platforms (even on x86, such as the embedded controller though these perform auxiliary tasks to the host firmware rather than the whole thing).
Updating without a CPU installed usually is on the embedded controller itself but that's not a normal update flow IIRC.
Unfortunately, I don't believe there is any up to date and detailed public documentation on the modern DRTM flows that exist on both Intel and AMD platforms. Maybe documentation has been recently updated but I’m not sure if I’m able to share more beyond what I already have.
DRTM does not remove any malicious firmware provided code or data. Traditionally it is merely a measurement mechanism that happens after ExitBootServices which measures platform state in an unforgable manner. Practically the DRTM event can also cause certain chipset registers to get locked or SMM supervisors to get launched depending on the platform. SMM and ACPI tables (on some x86 platforms certain tables are rebuilt) are measured into a PCR by the secure loader or security processor during the DRTM event. The idea is that if malicious code or data was present then the PCR values wouldn't match the previous boot session and TPM secrets wouldn't unseal.
While what you said is technically correct, it is by design and any compromised firmware can do as it pleases before the DRTM event at the cost of getting caught and having the device fail attestation or not be able to access encrypted data (depending on what policy is layered on top of DRTM itself as it is just a security primitive). By having PCRs get reset during the DRTM event secrets are much more reliably able to be sealed to specific PCR values.
I'm not sure why you have legacy in quotes, that is what it is. Modern platforms do not act like a PC AT and there is a ton of cruft to keep those interfaces working which is set up and facilitated by firmware (not so open, at least TianoCore exists regardless of your thoughts on UEFI). Modern x86 systems programming is so much more than these interfaces. I think that it would be beneficial for others to understand how things are actually working on modern platforms.
Regarding your linked post:
For corporate users legacy methods of booting /are/ a security threat. Firmware attacks are a real threat for enterprises. Having IBVs all hand roll things like the S3 boot script or communication with SMM (not using standardized com buffers) is not wise. Secure boot combined with modern security features like using DRTM to remove firmware from the trusted computing base are important and should not be discounted because it's not an 40 year old interface. The PC platform is historically extremely insecure and there is a ton of work still to be done to effectively provide the primitives needed to secure the platform against determined attackers.
If you want to have a USB stack in SMM owning your USB controller to emulate port 60h with IO SMI traps that's fine but it's not a reason to hold the platform back.
These are good examples and this might be slightly off topic but I wish that there were more resources on how modern x86 platforms actually work instead of using these legacy interfaces that are either emulated in SMM with IO port trapping or relegated to requiring chipset hacks to keep them working (such as VGA device hacks in the PCI spec or well known chipset legacy IO ports decoding to the LPC bridge).
While it is essentially impossible to truly understand every aspect of todays platforms I think that one can build a lot of intuition about how things work by studying how transaction routing is handled in the platform, e.g. how the uncore and chipset routes TLPs (whether it's raw PCIe or packets essentially wrapped by other interconnect protocols like IF/HyperTransport/DMI/QPI/etc) through the system with various base address registers and decoding windows. This gives way to understanding how modern MSI interrupts and DMA works as well. While I understand that truly going into things like PCI configuration, ACPI, UEFI, chipset interfaces on bus 0, modern booting, DMA and interrupt remapping, DRAM, register locking and security, security processors, power management, various types of runtime firmware, etc is a a many-years-long exercise I think that the core ideas don't need to be spread around all over the place for one to get familiarity with the ideas and patterns that underpin our PCs.
I think that a general conceptual overview on which one can hang more detailed knowledge acquired by reading specs is needed with the amount of complexity that exists today. There are some really good resources that I've collected over the years but there is still something missing. Maybe I or someone else will get around to it some day.
A lot of critical system configuration is stored in the SYSTEM hive which isn't explicitly backed by a file mapping and is loaded at boot via firmware services so this will be fast as it is non-paged and mapped in kernel space for the entire boot session. On newer builds of Windows 10 other hives are memory mapped into the usermode address space of the minimal Registry process. Whenever you do a registry read the kernel will temporarily attach your thread to the Registry process' address space and the read to the UM mapped section will occur which will naturally fault in the data from disk. The requesting process' thread will then be unattached and the information will be returned. Since non-SYSTEM/ELAM hives are memory mapped the kernel's cache manager and memory manager subsystems are the ones that "own" and control the mapped memory. The file cache is tuned based on the particular system's hardware characteristics to be as performant as possible. There are registry-specific caches in between to reduce the need to attach to the Registry process but this isn't going to be a disk IO speed bottleneck.
The program you're thinking of is procmon which is a part of the Sysinternals suite of tools.
This is correct. The functionality of LOADALL now largely happens in the RSM instruction which returns back to normal execution context from an SMI. In modern firmware, before the SMI handler is invoked by the SMM dispatcher the firmware will switch the mode away from real mode and have its own GDT/descriptors. All this needs to be restored on SMI return thus RSM will parse a specially formatted block of processor context to dump everything back.
Most legacy code is this way unfortunately. The actual techniques employed by the "called out" code are not uncommon. I don't think that people are defending the lack of clarity but rather what the code is doing itself. It could be written more clearly, though one could argue that the lengthy series of articles does just that. Maybe it's an exercise for you to write it how you want.
Have you taken a look at some major C codebases? It's not justification but an appeal to authority seems pretty unnecessary :)
You see similar code to his unconditional executeNoMatch at the end of the table in table based code all the time. Additionally forEachObject is a macro that immediately initializes a variable. For projects like this the code is completely fine. I'm not sure what is particularly dangerous about this code for a small to medium size project.
Most definitely. There was a lot of design cleanup from VMS (e.g. fork processes -> DPCs, removing global PTEs and balance slots, etc), optimizations (converging VMS's parallel array structure of the PFN database into one), and simplification (NT's Io subsystem with the "bring your own thread" model, removing P1 space, and much more). SMP was also designed into NT from the beginning. You can start seeing the start of these ideas in the MICA design documents but their implementation in C instead of Pillar (variant of Pascal designed for Mica) in NT was definitely the right thing at the time.