1. There are multiple levels at which things are hashed, signed and checked when it comes to Python packages. Eg. each file in the Wheel beside the RECORD file is hashed using SHA256. This hash is never checked :( You can also sign your packages (RECORD.jws anyone? Is that still supported?), but nobody checks that either.
2. There are hashes in the HTML served by PyPI. These are updated at the whim of both the index and the publisher. Even though they are checked by pip during install, they are worthless.
3. There are many ways to install packages that work around (2). Custom index server doesn't have to provide hashes, and pip will happily install that. You can install from sources, from a package you've downloaded somewhere, form VCS, you can build it during install, all without even prompting the user to confirm the very scary choices.
NB. I have no idea how do you make the leap from "adding files to release" to "not modifying the release". To me, adding file to release is sure as hell modifying it. Here's a very simple malicious example:
I release package "innocent" with an empty "scripts" section. Then, in the subsequent modification to this release, I add the "scripts" section with a script named "notebook". Now, whenever my user wants to run Jupyter notebook, they will call my "notebook" program, not the one from Jupyter package.
There's no reason a public index should allow any "staging" time for releases. It's the whole point of the release: once it's done, it's sealed. Oh, you messed up? -- Either make a new release, or release patches for the old one. Do not use production release server as your testing ground. It's not meant to do that, it makes it unnecessarily complicated. Test whatever you release on your own, outside of production grounds.
It's kind of hilarious how everything that has to do with Python is so obviously wrong, with the obviously right way of doing things being right there, on the service, having been around since before Python even existed... and yet, Python will fail spectacularly every time.
If you ever used Maven, NPM or... I can't think about any other tool that doesn't automatically check checksums and signatures. Any Linux package manager ever used... Python's Wheel format has provisions for checksums and signatures! But they aren't checked.
Instead Python gets absurdly ineffective workarounds that will probably inconvenience a few developers and will do zilch for users.
Basically, the problem is that the Web browser doesn't give enough control to developers to be used as an application platform. Simple things like key bindings are impossible for example (because there are some key bindings that the browser will not let you modify).
Another problem is integration with other desktop applications and system utilities. The browser intentionally removes and restricts this functionality because the browser's model of the users is that users are incompetent bumbling idiots who must be protected from bad choices at the cost of being able to perform useful actions if necessary. Emacs has some safeguards of that sort, but they are intentionally made easy to disable (eg. loading directory-local code).
Emacs itself serves as a communication interface for other programs and uses operating system interfaces to the same effect. It's trivial to run a program from Emacs and feed its output to another program, while displaying the result in Emacs. In the browser, you'd have to arrange for a custom protocol, a server that listens to requests from that protocol, and write wrappers around every utility you want to run so that they can wrap their outputs into messages of that protocol... essentially re-implementing IPC system abilities in browser (but a lot less efficiently).
> The JS code won't be able to modify the browser itself, but neither can ELisp modify the actual Emacs binary code.
You misunderstood the problem. Remember when Firefox' GUI was made of XUL and that was indeed accessible from JavaScript (before they abandoned add-ons in favor of Google's extensions). Then you had add-ons like Firebug or KeySnail that could indeed take control of the browser's "chrome" (i.e. the visual elements that don't belong to the HTML page, s.a. tabs, menus, address bar, buttons around it).
If you don't remember... life was a lot better back then. I've built a tool for interactively highlighting selected DOM elements in the page from a dedicated toolbar for example (this was before the developer tools). You could completely remap all browser keys. You could call external editor to edit the text in text areas or text inputs. You could call programs on the local filesystem (there used to be an FTP add-on that did that).
I didn't mean to say it was illegal, I meant to say it was an advantage in competition against the open-source counterparts because those usually cannot afford to make discounts of any kind.
More concretely, Microsoft used to claim that the price of their software is compensated for by the operational costs (which is false, but they managed to convince a lot of people).
In the same way, due to having deep pockets, Microsoft can, temporarily, operate at a loss only to out-survive the competition only to crank up the price to the ceiling once the competition is gone.
I'm sure this is a serious problem, but I'm not sure it's the main problem. The other one, that might be an even bigger problem is the way Microsoft sells its products. While, technically, most of the stuff they sell appears as B2C products, they are sold as B2B, and increasingly more so since the emphasis shifted towards the Web versions.
What this means for the users is that their interests are overpowered by the interests of the organization supplying them with the software. An organization can be more easily bribed both "legally" by offering a (temporary) discount and straight-up illegally by paying the procurement officer. Microsoft is known to do both. This is not a battle one can win on technological merit alone, unfortunately...
To comment on my concrete (and quite miserable) situation, here's how it went (and does, even worse with the advent of AI, to this day):
* A report comes from the field (to the support team) that something is broken (or highly desired by the customer).
* The support team assigns the ticket to the lead engineer.
* The lead engineer writes the code and after a quick battle with CI pushes it all the way to the customer in need.
* Since it's the lead engineer, he has the authority to merge PRs w/o consulting anyone, which is what he does. You just pull the changes and marvel at them, or at the sunrise, whichever you like best. The "urgency" is used as a justification to skip the due process.
Now, when AI came into play, two horrible things were added into the mix:
* AI audit and code review. The audit is a 50/50 chance between finding a real problem vs misunderstanding of the purpose of the code or the context in which it is executed. The AI will also come up with a solution that is either completely wrong, unnecessary or touching too many things for a human to track. Code review acts in a similar way, except, technically it fills the role of a human reviewing the code, so it gives a sense of false security to anyone making changes.
* AI generating code that is hard for humans to follow. First and most obvious problem is the volume. In minutes AI generates changes that will take months of effort to review. But this is only the beginning of the problem. When reading code written by humans, another human builds a model of the author's approach, ability, intention and permissions. When it comes to AI, these... let's call them "dimensions" are all over the place. The generated code you read may seem very plausible for a while until you stumble into a crucial functionality that completely invalidates everything you've learned about the change so far. The change may make unnecessary segues into the territory it was never meant to touch.
So, even when there are code reviews, they now miss a lot more than they used to when only humans were writing code.
Also, and this is a pathologically bad, but a very common practice: the reviewer's name is missing from the record. You run git-blame and only see the author, not the reviewer. When things go down, the author takes the heat and the reviewer is nowhere to be found. Subsequently, reviewers don't feel like they need to care as much about the outcomes.
Oh, and one more thing. To be effective at reviewing anything you'd want to try things... as in to test them. Perhaps come up with some idea about what things must happen and what things absolutely must not. But where do you find those? Usually... in a design document, s.a. a PRD, but in the case where that design (allegedly) was solely in the head of the person writing the code, how do you even know if the behavior you conjecture the code has (because there are no tests) is the desired one?
Well, I had to be more specific instead of trying to be dramatic...
The thing is, nothing in the suggested procedure addresses the problem of how you are supposed to distinguish the decisions that matter from those that don't. Whoever came up with this idea must've thought that it's trivial to the point it doesn't need solving, but it's the opposite, unfortunately.
Also, FYO, linters aren't responsible for code formatting. It's the formatter's role. Linters advise on style. I.e. the problem of bracket placement is not solved by using linters, an example of problem that is solved by linters is the maximum number of properties a class is allowed to have.
But the meetings about formatting or using / disabling linter rules still happen. Having tools to do that didn't make the problem go away. What did happen, however, is something more like natural selection. The increased volume of newcomers which the field saw up until maybe five or so years ago were mostly steered in a particular "winning" direction. So, today, naturally, you have most people agree on a set of rules to follow because the majority came from the same background. You need a bunch of old-cadgers to work together on a project for there to be a feud over styling rules.
This is what usually happens if you let incompetent people make design decision: they stay with the first and worst version forever.
Right now I'm in the process of leaving a company where this happened over and over again. The company's product and design aspects were always neglected in this specific way: whenever a problem was discovered its solution was assigned to whoever would write the code to solve it. That's it. The QA would be notified a week or two before the release and after the solution was "finished". The QA would then not be allowed to even comment on the overall design, only the superficial bug reports were allowed to go through.
Needless to say the product's code-base is a dumpster. The morale is low because it's a daunting task to deal with this dumpster on a daily basis. Now, even if you wanted to, you wouldn't have a budget to fix the previous design mistakes because of the layers of more mistakes that were added on top of them due to the low morale and lack of budget / lack of procedure for making better design decisions. The only thing that keeps the product afloat is its uniqueness in a rather niche field and the backing of a large company that acquired it, but doesn't really depend on it and has no time for a thorough audit.
Parent doesn't have experience of working with codebases with slightly high than average code quality requirements.
In products s.a. storage, avionics, medical appliances etc. it's very typical to have a requirement for each commit to compile and to apply tests retroactively. I.e. once a test is added against an existing feature, it is run against every commit since the feature creation (this is also why git-bisect exists).
However, it's true that a lot of companies would probably do better with just rsync instead of Git. Their Git history is in such a bad state that it's basically useless. It just doesn't make sense to use such a complicated tool as Git to deal with the average workflow.
In terms of UI for Git, I've never seen anything getting remotely close to Magit. It's both in-depth and a simplified stateful interface.
Here's what I mean by "stateful": one of the huge problems with command-line tools is that their output becomes mostly inaccessible upon subsequent invocations. Suppose you ran git-log, and now you want to apply the knowledge gained from reading its output to your next action (say, you want to cherry-pick a commit). But you are lucky if the hash of the commit in question is still somewhere in the terminal. Otherwise... you aren't writing that from memory...
Stateful UI to command-line tools allows multiple interactions in parallel and preservation of information obtained in previous interactions.
Git needs a lot of state. You need to remember facts s.a. what branch you are on, what branch did you switch from, what files have been modified, what commits haven't been pushed etc. The requirement to remember all of this is overwhelming and makes the effective use of the tool difficult. This is where various tools come to help (eg. Shell plug-ins that provide auto-completion or modify the prompt to incorporate some of that state info).
I believe that people complaining about difficulties of using Git are often handicapped in their use of surrounding tools that would otherwise mitigate the problems above: they don't know how to add Shell plug-ins that display the current branch, they don't know how to have multiple tabs/panes open in a single terminal session, how to copy text between those tabs/panes, they don't know how to invoke / search auto-complete options.
Genuinely, if you don't have tools to help you use Git, something that people who are used to navigating the world of terminal UI, using Git becomes very unpleasant.
It might be uncalled for, but let me use a philosophical reference: you are trying to derive ought from is. Many tried to prove it to be possible, but most face such propositions with skepticism.
Let me give you some examples of things that were absolutely detrimental to the business of a large company that vanished in a matter of years if not months:
* Adobe Flash. Flash players was for a few years the most installed program on Earth. It vanished without a trace, dropped like a hot potato by everyone who swore they will use it for ever and ever.
* ASP Classic, especially in combination with VBScript. It was the hottest thing twenty (or am I getting too old?) years ago. Everyone was selling books and courses on ASP Classic, probably half the Internet was written in it... and it's gone.
* I expect x86 ISA to eventually die and be completely replaced by ARM or maybe something else. Some big players are already jumping off the bandwagon, and it looks like things are only going to get worse.
There were, of course, more, but I think these three examples should suffice. If tomorrow Google comes up with a better format, or, for whatever reason, tanks its business, or decides to switch to vertical farming in Arizona... say goodby to Protobuf. They owe you zilch. They never promised you anything. You were allowed to use their tech because it was convenient to them, but once it's not, you will be holding the bag.
This is the difference between a standard and a spec published by the designer. The designer is free to make any changes they see fit, up to and including completely destroying their work. They can pull the run from under you making the users pay for the use, they can make modifications to their tools that will require from you to buy things you wouldn't normally want to buy. The possibilities are endless.
It's like in that meme: it's not enough to say it, you need to declare it.
Or, in other words, what makes a standard a standard is that you declare that it is a standard. This declaration carries with it an obligation to respect the standard for ever and ever (like C89), regardless of whether in retrospect you realize you've made mistakes and want to fix them, or maybe wanting to add more stuff etc.
Having a standard is restrictive and uncomfortable for the designers, that's why many opt not to have a standard. Eg. Rust language doesn't have a standard (even though you may, of course, find documents detailing how it works), same for Python, Java and many other popular languages.
I don't know where you get the confidence... When it comes to Protobuf, we are now at version 3 of the format. It's been around for a while, but I'm old enough to have implemented v2 parser myself... v2 is partially supported in v3, even though the support isn't documented. But, emphasis on partially. Some things are no longer there.
So... I'd say that your faith is unfounded. And, in general, there's no reason to believe that a commercial entity will commit to supporting any particular technology if that doesn't generate them a profit. Standard is better.
> Why does it matter for some Python implementation if the Google C++ implementation has a lazy or eager parser?
I wrote about it in my repository, but I'll try to summarize it here: Protobuf is full of bad ideas. One such bad idea is that message fields are allowed to repeat and that the last field wins. So, if you were to write a SAX parser, you'd have a dilemma with how to handle this bizarre idea: do you accept that a callback for some property might be triggered multiple times or do you read the whole message ahead of time and then call callbacks exactly once for each property? If we accept that the parser must be lazy, we "solve" this problem by allowing the parser to read ahead (it needs to do this anyways), but this is a wasteful way to parse (uses more memory than necessary).
> The important part of protobuf is the spec of the wire format. That is what makes the standard an interop format.
I'm not sure what are you trying to say here. All messaging formats are made up of... wire format, that's what they are for. Maybe I'm not seeing it?
> Personally I also prefer code generation over dynamic parsers and generators.
I think you are trying to say that you prefer source code generation over generating supporting definitions at runtime? I wasn't talking about dynamic parser generation (eg. Lark). In my case, the parser was hand-written (using Bison + Flex) and compiled ahead of time, but the Python definitions supporting the Protobuf IML were generated at runtime.
If my guess is true, I'd like to hear your arguments in favor of generating Python source code that translates Protobuf IML into Python. To me it looks like a waste of space on disk... I really can't think of any reason to want that.
I don't know what your metric for reliability is... but I would say that the messaging / serialization format can hardly be blamed for reliability problems s.a. service uptime or service responsiveness etc. Such problems usually arise at a level where details like the choice of messaging format are not important.
Messaging format may affect application performance though. It would affect metrics s.a. throughput or bandwidth.
In a way that is more difficult to measure, a messaging format can affect the number of bugs created by developer using it and indirectly the delivery times. But this can be mitigated by tooling (i.e. Protobuf isn't self-documenting, so if you don't have the schema files, you can't interpret the messages, but you can write a tool that you can feed the schema definitions and then use the tool to interpret the messages).
> It's one of the few things Google _hasn't_ deprecated
You might be unaware of it, but there were Protobuf v1, v2, and now we are at v3. Even though it's not documented, v3 supports most of v2, but not all of it (I think v1 was never used outside of Google itself). Google never properly released Protobuf, so, they can't really deprecate it. Even their formal grammar is full of errors.
Hey. I wrote another Python implementation of Protobuf. (protopy https://gitlab.com/doodles-archive/protopy it was a while ago and haven't touched it since). I'm not saying it's better than whatever this is or that it's any good, I just post it as a proof of sorts that I'm familiar with the problem.
So, without further ado: Protobuf isn't a standard. You can't have a non-standard implementation of something that doesn't have a standard to begin with. In reality, you have Google's implementation for C++ and then everything else. Everything else was, for the most part, not written by Google. And it doesn't always align 100% with the C++ Google's stuff.
Furthermore, C++ implementation has a lot of idiosyncrasies specific to that language that can't be translated one-to-one into other languages, or, in some cases, shouldn't be, even if they could (eg. C++ implementation is all about source code generation because generating runtime entities s.a. classes in C++ is very difficult, while in languages like Python, generating classes at runtime is easy.)
Furthermore, C++ implementation has a specific way of parsing the binary payload (lazy: only the top definitions are parsed, the inner structure of messages is parsed on-demand). But, is this how every parser should behave? What if you want a SAX-like parser?
----
In the hindsight, I just think that Protobuf is not a good format for writing reliable software that aims for decades of usage. We, as in the whole programming world, don't have good formats in general, and whenever we come to the point of having to use some, we either go with an existing popular but crooked or roll our own, probably also crooked. The standard you alluded to would've been great (perhaps a refinement of ASN with more attention to parser implementation, more concrete versions etc.?) But we aren't there yet, and there isn't even a work group to try and address the issue.
I think you are putting the cart before the horse: the code was written intentionally, i.e. someone first had a thought, and then fleshed it out in the code (and maybe had forgotten all about it the minute later).
When LLM generates code, it also has a "thought process" of sorts. But it's very different from the one humans use. An LLM generating code may pretend to have a thought process similar to humans (when asked to elaborate on the reasons the code is this way or the other), but, fundamentally, it's going to be a lie, because the real reasons this LLM created the code in a particular way is very different, and is very hard to grasp for a human, even if they are familiar with the problem (there were some attempts at explaining LLMs "thinking" that would end up looking like heatmaps and other weird things that are useful for debugging them).
Imagine that instead of a stack trace, the program you are debugging spits out only the memory dump with values in registers etc. (the gibberish-looking part of the coredump file). You'd be in a similar situation receiving the "true" explanation from an LLM generating your code.
My very recent story with libvirt and secureboot resulted in blanket disabling of secureboot as part of the preparation for creation of VMs.
The reason: the VM refuses to boot when provided with an ISO (via virtual CDROM) with a meaningless error (permission denied: go figure out what permission and why was it denied and by whom).
Secureboot is meaningless / useless for most people running VMs, be it on own or rented hardware. It takes some pain and extra work to get it to work sometimes, and a huge amount of work to get it to work always. I doubt anyone was dedicated enough to get it to work always. So, I believe you are right. This is extremely unlikely to be a problem for anyone running Linux VMs, and the more VMs they need to run, the less likely it is a problem.
2. There are hashes in the HTML served by PyPI. These are updated at the whim of both the index and the publisher. Even though they are checked by pip during install, they are worthless.
3. There are many ways to install packages that work around (2). Custom index server doesn't have to provide hashes, and pip will happily install that. You can install from sources, from a package you've downloaded somewhere, form VCS, you can build it during install, all without even prompting the user to confirm the very scary choices.
NB. I have no idea how do you make the leap from "adding files to release" to "not modifying the release". To me, adding file to release is sure as hell modifying it. Here's a very simple malicious example:
I release package "innocent" with an empty "scripts" section. Then, in the subsequent modification to this release, I add the "scripts" section with a script named "notebook". Now, whenever my user wants to run Jupyter notebook, they will call my "notebook" program, not the one from Jupyter package.