Interesting. Both timely, given the recent discovery of the massive failure to disclose the asset hiding in his family trust by Carl Bates, yet also rendered less pertinent because of it - in reality a wide swathe, and possibly a majority of government ministers now are routinely not disclosing substantial gifts at all, let alone other forms of what are indisputably corrupt transactions.
The use of trust structures for tax efficiency is one thing, but they are so opaque they largely make a mockery of disclosure both because the size of the trust's asset base is opaque and often the trustee structures are pure paper fictions. It's still good to have this though, well done you.
> Are these people you know actively applying for better jobs and not getting them?
No, because the decades have worn them down. It's been a long time since there have ever been jobs in NZ advertised requiring hard development skills, and the tiny handful that do come up in public tend to be for very specialized verticals where they made hard demands on past experience in that niche area first and foremost over everything else.
> The other is happy with the chill job they have
I mean, I can't blame them for that - there are lots of toxic employers, ageism, credentialism, etc etc. If you're just going to be underemployed doing kiddie-level work anyway, better the devil you know, particularly if you have a family to take care of.
It's all just a big old mess of market failure, though. The problem isn't that the talent pool isn't there, it's that since there's no VC money around, the firms that _really_ need that talent can't pay what the hungrier younger people (who want more than anything else to get on the FAANG gravy train) want and the fantastically talented folks mostly can't earn any kind of premium remotely matching their business value and so make lifestyle choices it's hard to pry them out of.
> But paradoxically there is simultaneously a lack of top tier within New Zealand
Really? I'm aware of some extremely top-tier and wildly underemployed talent - the problem is that the NZ market has almost no companies that need or are interested in good people with hard skills - it's almost all very basic web dev. Pretty much all the veterans I know around my age are doing work beneath them to pay the bills or have switched out of development completely.
Symantec left our team - which was basically identical before and after acquisition - pretty much alone beyond adding in some (badly needed) release process and i18n requirements.
Almost the entirety of the growth in the size of the imaging executable, which did get hugely bigger, came from a constant drive to add capability to the NTFS support to match the FAT support, most crucially to allow the images to be edited in Ghost Explorer. The initial NTFS support that Ghost had prior to the Symantec releases was really crude, basically the content in the .GHO file wasn't files, but a raw-ish dump of used disk extents that it tried to always put back in the same place to avoid having to fix up attribute runs, whereas the FAT16/FAT32 content was basically a file archive where all the filesystem allocation metadata got recreated on the fly.
Customers wanted and pushed hard to have NTFS images editable, and that made life really hard - the approach that was ultimately taken meant creating a full read/write NTFS implementation, and those aren't small. And the design of that code interacted really badly with the C++ exception implementation in DJGPP (which before that work had begun, I had warned them about), so that eventually exception frame information was taking up ~25% of the on-disk size of the UPX-compressed binary!
The big thing I think with any book is to aim to inspire people (especially younger ones) to create for themselves, and while there's probably a way to do that with an exegesis of the existing source I don't know what it would be. The approach you take in your books is one that I think is inspiring to people to get them trying these things on their own - which is after all where the real hard learning happens.
That's particularly true because so many point design decisions made in commercial codebases like Coherent were conditional on a wider context - both technological and business - that just doesn't exist any more, and that I think gets in the way of helping younger people reaching a deeper understanding and appreciation of the design decisions made.
Recontextualising that code, and re-examining those decisions in the light of the modern day is just more interesting to me, and as well provides the chance to lead people through the process of solving the design problems in a way that's narratively easier to follow. Bootstrapping forces an order on what you do that gives a particular structure to the presentation, while at the same time we have aspirations about what future elements we want to achieve that shape and give context to all kinds of decisions as we work out how to get there.
After showing how to build something today, we can do a compare-and-contrast against the existing Coherent code that's more meaningful, especially because we can then measure the outcomes. Anyone can do (and lots of people do) ill-informed hand-wavy postulations about what-ifs, but the process of actually building things that do work forces your hand in ways that armchair enthusiasts miss.
Hey Nils, love your work - back around '90 when I joined Mark Williams I was a big Scheme fan, so your whole t3x.org site and "Scheme 9 from Empty Space" particularly puts a smile on my dial.
I do think the fact that Coherent really hewed 100% to the original V6 approach based on the basic design constraints of bootstrapped minimalism was part of the charm. Doing the 4.0 release process I really pushed myself hard to get as much POSIX compatibility in as humanly possible, but the balancing act was the size-coding part - making /bin/sh still be 16-bit while having $(()) and shell functions and able to process those insane MB-sized GNU autoconf scripts that embedded >64kb here documents, etc. Another big part of why the size-coding style mattered was keeping the system comprehensible since with only a few developers everyone was spread so thin, but that's again part of the fascination.
I've always wanted to give Coherent a bit of a conceptual reboot for modern x86 (particularly multicore, which means a complete new kernel) but staying to the 16/32-bit size and with a style similar to your books walking through the process of making it and how all the design choices need to interlock to make that possible is something I wish I had the free time for.
Definitely true. And as well, the level of innovation that brought, particularly in places like Australia/NZ or the (then) East Germany/Ukraine, the outrageous cost of components made for a whole generation of quite exceptional "frugal" engineers. UNIX itself was - despite being in the US - born out of a similar level of necessity.
One thing about Coherent (and why I periodically fiddle about with implementing replacement, modernized kernel designs for it, albeit not in C) is that exactly like V6 UNIX it is an exercise in that kind of design, but being for the x86 and partially modernised to an early-90's level it's considerably more accessible than V6 to anyone who wants to look at the source code today to learn about that.
The classic "Nautilus" manual was really interesting on two sides; one was the quality of the writing, which was all down to Fred Butzen - he just had a real knack for clarity and working with him on documenting the work I did during the 4.0 process was really enjoyable. Basically the process was that developers would write up drafts of everything to give to him, and he'd then edit them for clarity before passing them back for an iteration to ensure that none of the accuracy was lost. He just had an excellent sense of when there was going to be a better explanation than the classic dry manpage style.
The other notable thing about that manual was just the economics of getting a perfect-bound book of that sheer size printed and included in the USD99 price tag.
Yes, and also notably Dave Conroy of MicroEMACS fame. I do believe that as with QNX and MKS Systems and Watcom, Coherent was mostly built by former graduates of Waterloo university - their computing department is responsible for a great many notable products.
Yes, prior to 4.x processes were strictly limited to 64kb code and 64kb data - essentially the equivalent to what MS-DOS programmers would call the small memory model only. There are a bunch of reasons for this, all of which stem from the fact every part of Mark Williams C and the assembler/linker toolchain hewed so closely to the exact model set by V6 UNIX - including the same .o and .a formats, hence the same limitations all down the line.
Since so much my previous work before joining work on Coherent was on embedded toolchains and DOS development, the fact that so little use anywhere was made of x86 segmentation was surprising to me, but I knew all the V6 internals well enough to adapt back.
Even in 4.0 where the COFF and the iBCS2 syscalls all worked, actually most of the userland still stayed as 16-bit small model since the aim was to keep most of the installation process the same, which was that essentially the first install floppy was a reasonably full live system. Obviously things grew a ton and during the evolution of 4.0 we slowly had to move the bulk of the 32-bit userland components off to packages in later floppies.
It's not based on V6, although it was basically created as a clone of it (and on the PDP-11, no less - the x86 versions of Coherent until 4.x were basically just minimal ports that kept the PDP-11's classic 16-bit flavour). It was a very very good clone, enough that a lawsuit was threatened and Dennis Ritchie himself got pulled into checking that it wasn't derived from any UNIX source - see dmr's comment at https://groups.google.com/g/alt.folklore.computers/c/_ZaYeY4...
You'd likely find the experience pretty miserable, not so much because of the C compiler - Steve did a great job with that - but actually because of the filesystem. Coherent was around for a long time, remember; it used a very classic V6 UNIX filesystem, complete with odd endianness of 32-bit fields in the inode and most significantly, the classic 14-character file name length.
That right there will be an even bigger showstopper for the build scripts of most things than anything else. During the early 90's we could run GCC and Comeau C++ and all that kind of thing on Coherent fine because those codebases were still built for wide portability even though things like BSD FFS had opened that up - there were still plenty of machines running System III and System V code that most people took care not to create gratuitous collisions if truncation happened. Even shortly post the advent of Linux, that had changed dramatically and basically turned on its head and the effort involved in backporting to any flavour of classic UNIX (not just Coherent) quickly became almost impossible.
Yeah, just to confirm that for v7.1, we got a full source license of it for Symantec Ghost because Microsoft had withdrawn permission for us to distribute anything at all to do with MS-DOS, and we had a lot of customers still using DOS boot disks. Prior to that I believe we had a distribution license for PC-DOS but around that point IBM no longer wanted to maintain it and we had small things we needed to do, so we got the source license and just distributed builds we made as part of the Ghost Boot Wizard disk builder.
What was more interesting from a software archaeology point of view was when I found Symantec had acquired the assets of the then-defunct Quarterdeck; so the source for QEMM and Desqview eventually turned up in the corporate Perforce server and that was certainly fun to read, having made all kinds of things for QEMM back in the day, like a replacement overlay manager for Turbo Pascal (and Turbo C++) that would run overlay code segments directly out of the EMS page frame - this meant pretty much any program we built with the Borland compilers could be turned into a TSR with only the data segment resident.
The one thing that I never did find, though, was the source for the Whitewater Group's Actor. When it was announced that Symantec had acquired those assets from Borland I tried as hard as I could to see if the Actor source code was included but no-one I was ever able to find knew about that. Actor 3.0 was my first introduction to the Smalltalk way of building software, and I still sometimes fire it up in a VM to play enough with its VM format.
Not sure about the specific text the parent had in mind but "Principles of Interactive Computer Graphics" by Newman and Sproull (1979, but I bought it in '81 in the student edition and it influenced me a lot in my youth) made a bit of the connection with Renault in the chapter on Bezier Curves - many of its other examples of advanced graphics (at the time) were on flight simulators and such built by Evans and Sutherland, where the "Evans" was one of Alan Kay's mentors at the University of Utah.
I love this article, and have recommended it many times in discussion with young people learning programming in other forums (although it's rare for them to come back to me to say they'd read it, so...). There are some small differences in the social dynamics when it comes to mentoring in the workplace, but the vast majority of this is still incredibly on point. Thanks so much for writing it.
Anyone from HN who wants to can drop by on the way to a more traditional tourist spot like Mangawhai or further north (although over summer I'm often away at the weekends racing stockcars). The phone number for the library redirects to our house if it's not open.
The cheese shop is fine! I know Keith and Marita who have owned it for many years, they are good people - Ad Clarijs who founded it still occasionally runs cheesemaking classes, his place is not far from mine. Also worth a visit is the amazing Cafe Eutopia next door - https://g.page/eutopiacafe?share - which is a marvel.
It just boggles my mind how US firms don't realise how poor a value proposition - especially to people of an age to have a family or strong community ties, like the roots I have here - they offer to developers in AU/NZ when demanding that people relocate to work for them, and I'm not really optimistic that COVID-19 will really bring any significant change to their corporate cultures. I am hopeful that perhaps more firms in AU may cotton on to the fact that there's a substantial pool of exceptional veteran talent in NZ who are really underutilized, which really represents an opportunity for them if they were to make a serious attempt to hire here and let people work remotely.
I currently live in Kaiwaka not far from Mangawhai and have since Symantec bought out Ghost in '98, as that helped me afford a first house. It's a wonderful place to live. I did a lot remotely back then and once Symantec closed in NZ a decade ago I've been 100% remote from home ever since.
While I don't have an ocean view, and only about 2ha of land, it's a wonderful place to be. However, there are pretty significant drawbacks in getting gigs as there just aren't many local firms doing anything at all intellectually challenging and US firms still just apparently aren't interested in talent in this country who don't want to suffer being forced to relocate to awful places to live.
I don't see that changing, either; the fact is that people who really place proper value on a quality lifestyle tend to be older, and few firms really place much value on older talent either. Younger people are cheaper and more willing to suffer abuses such as moving to unpleasant, expensive metro areas in countries with alien values.
> I think I later upgraded to a 100MB IDE disk, but was still lucky.
Well, it really wasn't luck as much as keeping with the electrical specs. Do that, you'd never see a problem, and the later IDE "cable select" schemes did really help to mitigate a lot of the damage from improperly terminated cables.
> Anyway, thanks for contributing to a great OS!
Well, other than living in infamy due to introducing that bug, I didn't start there until the push to turn 4.0 from a tech demo into a real product. So really all the credit for 3.2 and earlier which set the foundation for Coherent belongs to the other guys, many of whom were long gone by the time I got there like Dave Conroy (who wrote the MicroEMACS I loved to use) and Randall Howard (who went on to found MKS). There were some great people from the earlier days still there though, like Norm Bartek, Hal Snyder and La Monte Yarroll who where there when I joined and of course Steve Ness who was the sole man behind the MWC C Compiler (much as Fred was responsible for that remarkable manual).
Yeah, cross-subsidised products like that were an issue. But it's worth noting that part of the fine line we had to walk was that lots of the people who bought Coherent had hardware on which something like Solaris would have been totally unusable in practice, at least unless they spent US$5K on brand-new machines with enough grunt.
For context, while we we working on 4.x I recall Norm Bartek having a crack at installing one of the early Windows NT 3.1 beta releases on one of the beefiest machines we had in the office. After grinding over half the day copy stuff to the install partition, then when it went to actually boot it just bluescreened because the machine had "only" 12Mb of RAM (which was the minimum spec for release NT 3.1, but the betas were built in debug mode and so needed more).
There was a big gap between what the "workstation" folks thought PC hardware was like and the reality of what most people could actually afford, hence why the minimum spec target for Windows 95 - a couple years later! - was still only down around 4Mb, even though that still tended to page hard.
So, for us, although it sometimes took fairly heroic efforts in size-coding, by keeping everything we possibly could still as 16-bit, our install floppy was actually a fully live system, without any tricks like compressed filesystems expanded to RAMdisk or anything. You booted off the install floppy to the 32-bit kernel, with the regular init and login and /bin/sh and everything - the installer was a shell script that just helped walk through fdisk and mkfs and then did an untar of the remaining stuff on later floppies (with things like the C compiler and libs) but right from the install floppy you could quit (or Alt-F2 to an alternate console TTY) to a root shell and have a usable system that happily ran in 1Mb of RAM.
So, there was still a niche for Coherent (and similarly, QNX), albeit that it wasn't a large enough one to bridge it through the growing pains of the changes at the time, especially given the forces that would shortly wipe out AT&T and DEC and almost all the other UNIX vendors despite their size.
It's interesting, given what I then went on to, to compare that niche to some of the customer response to Ghost. Even well after 2005 we had plenty of paying customers there who absolutely clung to being able to build and boot to DOS via floppies, even though we did have to use the RAMdisk trick once the UPX-packed Ghost binary exceeded about 2Mb by itself (a lot of which was due to a mistake I told the other devs not to make around the time Symantec acquired it, which was to rely on C++ exception handling; alas after several years, something like about 40% of the size of the binary was just GCC's exception metadata).
> What killed it is they, for some reason, spent time and treasure port XWindows to it instead of working on a TCP/IP stack
Just to be clear, there's a whole heaping helping of really big work required to get there either way, since Coherent was still essentially a 16-bit OS that started out as a V6 clone on the PDP-11 and hadn't really kept up during the 1980's with any of what was happening in the wider UNIX world beyond, oh, the Seventh Edition.
The key starting point for Coherent 4.x was essentially a single piece of work (almost all of it provided in a single .c file) that let a 3.x kernel boot into 386 protected mode and provided a system call handler that more-or-less emulated the call sequence specified in the Intel iBCS specification and mapped the iBCS system calls to the original 16-bit Coherent 3.x ones, and a very basic loader for statically linked SVR3 COFF files.
[ See https://en.wikipedia.org/wiki/Intel_Binary_Compatibility_Sta... which essentially formalized a subset of what the SCO port of System V did. Note, that omitted a number of system calls; I'm fairly certain that the iBCS manual didn't include any of the socket-related system calls anyway, so the public spec for what people took to mean "will run SCO binaries" left a lot to be desired. ]
Everything else around that was still, essentially, 16-bit code that was written around some variation of Sixth Edition or Seventh Edition, not System III/ or System V (so to be iBCS compliant the bible was the System V Interface Definition, aka SVID), let alone any of the things that were being added to POSIX.1 or POSIX.2
So, the work required for releasing the V4.x series of Coherent included a massive number of upgrades of almost everything, along with all kinds of awkward technical compromises to a) not break any existing things from the 16-bit userland, which stayed 100% supported, b) navigate between SVID and POSIX because while we on the one hand wanted to be able to run SCO binaries (hence, do things per the SVID) while we had all kinds of enthusiasts really wanting to use the ability to 32-bit code to run various shar archives of GNU project code unmodified.
So, I'd just note about that 16-bit userland from Coherent 3.x was that virtually everything on the (nominally 32-bit) version 4.x install floppies - all the standard utilities, pretty much everything that could be - still was built as 16-bit a.out binaries to keep the install size down. Pretty much everything we did in terms of C libraries and such had to be, and had to stay, buildable both 32-bit and 16-bit from the same source since both subsystems were needed. At some point we could conceivably have (at a technical level) cut over to all-32-bit, but just from the point of view of media duplication, doing that and still staying at a $99 price point wouldn't fly, and also the resulting increased memory footprint wouldn't have made existing customers happen since the kernel didn't have any support for demand paging, at a time when lots of people had just 1-2Mb (or 4Mb for a relatively beefy system) of RAM.
[ So for all the work required to let /bin/sh able to run successfully run multi-megabyte autotools scripts that tried relying on POSIX.1 shell features? Yup, everything involved in POSIX.2 support was all done with code that stayed 16-bit. ]
Without getting too much into the weeds of all the other various challenges involved getting a system like this out, for TCP/IP specifically the question was what kind of TCP/IP stack to pursue and most critically of all the question of where were the NIC drivers going to come from. There was not going to be any plausible way we could work with third-party SCO driver binaries, which were the most common type of UNIX driver that NIC vendors did provide at the time; SVR4.2, however, had STREAMS and the DDI/DDK so there was a plausible, reasonably portable driver model which seemed like a realistic future, but just licensing Lachmann and trying to bolt that into the kernel wasn't realistic at the $99 price point (and at the time, there weren't much of any third-party NIC drivers either).
To make 4.x happen we had to come from a very very long way behind, and getting to where we needed to go while staying within the constraints of a small staff and that $99 price point just made for an incredibly tight set of constraints. None of the resulting tradeoffs were fun to make. We did our best.
The use of trust structures for tax efficiency is one thing, but they are so opaque they largely make a mockery of disclosure both because the size of the trust's asset base is opaque and often the trustee structures are pure paper fictions. It's still good to have this though, well done you.