ahgamut

My early contributions to https://github.com/jart/cosmopolitan were focused towards getting a single-file Python executable. I wanted my Python scripts to run on both Windows and Linux, and now they do. To try out Python, you can:

    wget https://cosmo.zip/pub/cosmos/bin/python -qO python.com
    chmod +x python.com
    ./python.com

Adding pure-Python libraries just means downloading the wheel and adding files to the binary using the zip command:

    ./python.com -m pip download Click
    mkdir -p Lib && cd Lib
    unzip ../click*.whl
    cd ..
    zip -qr ./python.com Lib/
    ./python.com # can now import click

Cosmopolitan Libc provides some nice APIs to load arguments at startup, like cosmo_args() [1], if you'd like to run the Python binary as a specific program. For example, you could set the startup arguments to `-m datasette`.

[1]: https://github.com/jart/cosmopolitan/commit/4e9566cd3328626d...

ahgamut

2 年前議論

most likely before v2 as well. The original post is from July 2021, which is a bit after Cosmopolitan Libc's v1 IIRC.

ahgamut

2 年前議論

My 4-core computer's probably building Cosmopolitan Libc in the background + the Python binary is 2.7 unoptimized. This blog post is originally from 2021.

ahgamut

2 年前議論

I've tried nuitka before, and a recent question that occurred to me was: does nuitka have an option to output just C files? Something like:

  python -m nuitka example.py --no-compile

Might be interesting to see if the above is possible. We could get things like a completely-statically-compiled Python stdlib within the APE.

ahgamut

2 年前議論

Hey, this is my post from 2021, when I was testing Python2.7 and Python3.6 with Cosmopolitan Libc on an old 4-thread Haswell. It's now a lot easier to build Python (and gcc, gnu coreutils, curl etc.), and the binaries are faster, multi-threaded, and quite convenient to use. There are lots of interesting directions to explore when it comes to building software with Cosmopolitan Libc.

ahgamut

2 年前議論

> new methods of communication with the compiler have been established.

From what I understand, this appears to a be separate binary from GCC/Clang that does static analysis and outputs C99.

Can this be a GCC plugin? I know we can write plugins that are activated when a specific macro is provided, and the GCC plugin event list allows intercepting the AST at every function declaration/definition. Unless you're rewriting the AST substantially, I feel this could be a compiler plugin. I'd like to know a bit more about what kinds of AST transformations/checks are run as part of Cake.

ahgamut

3 年前議論

> Instead, many have deeply underestimated LLMs, saying that after all they were nothing more than somewhat advanced Markov chains, capable, at most, of regurgitating extremely limited variations of what they had seen in the training set. Then this notion of the parrot, in the face of evidence, was almost universally retracted.

I'd like to see this evidence, and by that I don't mean someone just writing a blog post or tweeting "hey I asked an LLM to do this, and wow". Is there a numerical measurement, like training loss or perplexity, that quantifies "outside the training set"? Otherwise, I find it difficult to take statements like the above seriously.

LLMs can do some interesting things with text, no doubt. But these models are trained on terabytes of data. Can you really guarantee "there is no part of my query that is in the training set, not even reworded"? Perhaps we can grep through the training set every time one of these claims are made.

ahgamut

3 年前議論

What packages do you use in your app? If you're not using too many C extensions, it may be possible to build your app with Cosmopolitan.

ahgamut

3 年前議論

Try it out! We got Lua/LuaJIT, Python, PHP, and Rust building (the latter two are not fully automatic yet), so Ruby might be possible even now.

ahgamut

3 年前議論

https://github.com/ahgamut/rust-ape-example

My above repo contains example with the Rust standard library that build as fat executables with Cosmopolitan Libc.

I also got ripgrep to build https://github.com/ahgamut/ripgrep/tree/cosmopolitan, but it wasn't part of the cosmocc binaries because some crates that are part of the build require `#!feature(rustc_private)]` to be added to their source code. Same goes for bat and fd.

To summarize, Rust CLI tools can be built with Cosmopolitan Libc, but to build them like we can build gcc/emacs/git right now, we will need some support and Rust expertise.

ahgamut

3 年前議論

Yes, so pip works because the Python APE has OpenSSL built-in.

pip install requires modifying the APE, so I end up installing pure-Python libraries as follows:

    mkdir Lib
    ./python -m pip download httpx
    unzip ./*.whl -d ./Lib
    mv python python.com # in case the next step doesn't work
    zip -r ./python.com Lib
    mv python.com python

Installing CPython extensions like this is an unsolved problem, but I think there might be some interesting workarounds possible.

ahgamut

3 年前議論

The "/zip/" folder refers to the ZIP store inside the actually portable executable -- you can run `unzip -vl ./python` to see what's inside.

Also, to add new pure-python packages, you just need to unzip them into a local folder "./Lib", and add that folder to the APE.

ahgamut

3 年前議論

Thank you Gregory for writing this post. There have been a bunch of announcements about "setup.py has been deprecated", but few have clearly outlined how to move away from setup.py, and more importantly, fewer have outlined what a struggle it is to move away from setup.py.

I was sad to see setuptools officially deprecated, because it looks like another way in which Python packaging is being red-taped away for a non-expert. If someone like the OP (who has 10+ years programming Python) had to do so much for what appears to be a zstd CFFI/Rust wrapper, where does that leave the rest of us?

Here's a python package of mine that uses setup.py: https://github.com/ahgamut/cliquematch/blob/master/setup.py which I have not upgraded to the new tool(s) yet. I think I will need to upgrade it soon. If anyone has suggestions for a tool that will _fully replace_ setup.py, I would like to see tutorials with the following examples:

1. How would I build a package that has pure-Python files and data files? With setuptools I would use maybe MANIFEST.in or package_dir.

2. How would I build a package that has a CPython extension accessed via cffi? (this post points to the answer)

3. How would I build a package that has a CPython extension _without_ cffi, that just wraps some small C code I wrote with CPython's API? What about an extension that uses PyBind11? What about an extension that uses Rust?

4. How would I build a package that requires a "system" package like libblas-dev? Can something like numpy be built optimally without ever writing setup.py? What would a config for that look like? Last I remember numpy used their own patch of distutils to build, I wonder what it is now.

ahgamut

3 年前議論

One of the comments in the OP:

   I believe the problem is that bridge in the US (not true in some other countries) lacks a "critical mass" of 20 and 30 somethings.

I like to play bridge, but all the people I have asked in my age group (the 20 and 30 somethings) say bridge is a game their (grand)parents (used to) play. I am not sure how bridge can be introduced as a fun hobby without the face-to-face aspect, because compared to other online games it does seem a bit monotone.

ahgamut

3 年前議論

"Python 4, but not really", because we want to squeeze out more multithreading performance and be cool again. Some questions from reading the OP:

- How much does performance improve due to this No-GIL thing? Is it greater than 2x? For what workloads? - Do I have to compile two versions of every extension (gil/nogil)? I would prefer building extensions does not get any more complicated. - Can I automatically update my code to handle nogil? (a tool like lib2to3 or six)

ahgamut

3 年前議論

I had tried to build a bunch of different codebases (git, bash, gcc, curl etc.) before trying busybox, and I'd never seen code like that before. Plus that error came alongside a compiler error in my patch that took a long time to debug, so it was a memorable surprise :)

Would you consider a code pattern like that to be common? What other codebases apart from busybox have it? If there are many examples, I might spend some time trying to update my patch to handle patterns of that kind.

ahgamut

3 年前議論

New constants do make compilation a lot easier, but my personal opinion is that the overhead to convert to/from the old constants during runtime is too much.

Look at the list of constants here: https://github.com/jart/cosmopolitan/blob/master/libc/sysv/c...

Every time I used any of these constants, I'd have to load a whole bunch of them into my binary as a large lookup table, and go through that table every time I needed a check in my program. It might not be that slow, but I believe it would definitely be noticeable.

My goal was to make porting easier without changing a lot of source code in either the libc or in the software I was trying to port, and still produce binaries that are close or better in performance. Under those constraints, this gcc patch seemed like the best way to simplify the process.

If I run into enough codebases where SIGHUP is used as an array index initializer, I will probably attempt your suggestion just to measure the tradeoffs. Or you could try it out and let me know if a separate set of constants is better.

ahgamut

3 年前議論

> I highly doubt many would be willing to rewrite the world to make some "clever hack" work.

I specifically wanted to avoid rewriting the world in order to port established codebases like git/curl to Cosmopolitan Libc.

My goal with this gcc patch was to answer the question: "what is the minimum amount of source code I would have to manually change in order to port software to Cosmopolitan Libc?". As we find out, sometimes we don't have to change anything to compile code for an Actually Portable Executable -- just do the usual ./configure && make, and a small `objcopy` command at the end to create the APE.

ahgamut

3 年前議論

Yeah I found out I got banned when attempting to comment about this blog post. I rarely comment on online forums anyway, because I find I don't know enough to comment about most topics.

IIRC over the last year, I've only commented once on lobste.rs, when it came to something about Python's import system, and only upvoted my own blog posts about Cosmopolitan Libc and related comments + maybe one or two other posts. I disagree with the ban, but I can see why my account would get banned as a spammer (one of the accounts banned alongside mine had a double account I think).

lobste.rs is a wonderful community, and I'm happy to see that someone there atleast mentioned this blog post. Hopefully I can get un-banned at some point.

Luckily I can still comment on HN though!

ahgamut

3 年前議論

Right now I use the gcc, git, vim, and ninja APE binaries on my local system for daily work, and so far I have seen no visible regressions.

I agree with you though -- an extensive testing setup will reveal more things for improvement, be it in my patch or in the libc itself. At the moment I only have direct access to Debian 11/Fedora 35/FreeBSD 13, but I expect we will soon find a nice way to run tests across the different operating systems automatically. One of my ideas was to setup a separate drive or folder with all the executables, accessible from different operating systems/VMs, and then run the tests from each. What do you think?

投稿

Patching GCC to Build Actually Portable Executables

Debugging C with Cosmopolitan Libc

Rust is actually portable

コメント