My thoughts on writing a Minecraft server from scratch in Bash(sdomi.pl)
sdomi.pl
My thoughts on writing a Minecraft server from scratch in Bash
https://sdomi.pl/weblog/15-witchcraft-minecraft-server-in-bash/
28 comments
Direct link to the goods, pdd.sh:
https://github.com/bkw777/pdd.sh/blob/main/pdd.sh
Now that is some serious bash-fu!
Edit:
> Similarly, you don't need xxd to convert back & forth between raw binary and text encoding.
How do you convert binary to text without calling out to `bc'?
xxd and bc examples are plentiful, but not "pure bash". For example, see https://gist.github.com/Gosha/5360a3109f77bc0defe7
Now that is some serious bash-fu!
Edit:
> Similarly, you don't need xxd to convert back & forth between raw binary and text encoding.
How do you convert binary to text without calling out to `bc'?
xxd and bc examples are plentiful, but not "pure bash". For example, see https://gist.github.com/Gosha/5360a3109f77bc0defe7
printf % codes and brace expansions, and printf -v to write directly to variables without needing a subshell or a temp file.
See file_to_fhex() for an example of reading binary into hex pairs.
That example is simpler to read than tpdd_read() because reading from a file is simpler than reading from a stream wrt detecting eof.
read a byte, use printf to convert that byte to a hex pair. The funny looking syntax with a single ' in front of the variable is important.
See tpdd_write() for writing hex pairs back out to binary.
brace-expand "aa bb cc" to "\xaa\xbb\xcc", feed that to printf to output binary.
All through this script I'm using arrays instead of simple variables to hold the hex pairs. That's not necessary, I'm just doing that because this app needs to do a lot of byte/position counting on the data, both for parsing the output from the drive and for sending commands to the drive. It's all packets of fixed length fields.
See file_to_fhex() for an example of reading binary into hex pairs.
That example is simpler to read than tpdd_read() because reading from a file is simpler than reading from a stream wrt detecting eof.
read a byte, use printf to convert that byte to a hex pair. The funny looking syntax with a single ' in front of the variable is important.
See tpdd_write() for writing hex pairs back out to binary.
brace-expand "aa bb cc" to "\xaa\xbb\xcc", feed that to printf to output binary.
All through this script I'm using arrays instead of simple variables to hold the hex pairs. That's not necessary, I'm just doing that because this app needs to do a lot of byte/position counting on the data, both for parsing the output from the drive and for sending commands to the drive. It's all packets of fixed length fields.
That is some amazing bash coding right there.
Meanwhile I at least 50% want to apologize for it.
I know full well that it is crammed full of inscrutabilities, relies on side effects, implicit behaviours, an ocean of globals & other state, and isn't even fully consistent with internal conventions.
This would be doing a bad job at work.
Some of that is just working with what you have, like the globals are how you avoid needing subshells so that's justified.
But there's other things like a lot of "business logic" is implimented in the form of arcane conditional brace expansions and particular return values of some commands etc, that could have been done more scrutably with ordinary readable explicit logic, just somewhat more verbosely.
I'm the first to say it aint readable. It's really a great example of "just because you CAN do something..."
But I wanted no dependencies (that aren't likely already there), cross platform, doesn't break in 6 months simply because 6 months have passed, yet interpreted vs a c program. Awk would tick the same boxes. I have both awk and ksh scripts I wrote in the 90's on xenix that still work the same today on linux & mac & bsd, meanwhile a python script breaks in a year.
I know full well that it is crammed full of inscrutabilities, relies on side effects, implicit behaviours, an ocean of globals & other state, and isn't even fully consistent with internal conventions.
This would be doing a bad job at work.
Some of that is just working with what you have, like the globals are how you avoid needing subshells so that's justified.
But there's other things like a lot of "business logic" is implimented in the form of arcane conditional brace expansions and particular return values of some commands etc, that could have been done more scrutably with ordinary readable explicit logic, just somewhat more verbosely.
I'm the first to say it aint readable. It's really a great example of "just because you CAN do something..."
But I wanted no dependencies (that aren't likely already there), cross platform, doesn't break in 6 months simply because 6 months have passed, yet interpreted vs a c program. Awk would tick the same boxes. I have both awk and ksh scripts I wrote in the 90's on xenix that still work the same today on linux & mac & bsd, meanwhile a python script breaks in a year.
> The code is currently lost, due to my extensive use of tmpfs as a project directory, and a system crash.
As if the project wasn't enough of a challenge on it's own, let's throw in some russian roulette!
As if the project wasn't enough of a challenge on it's own, let's throw in some russian roulette!
Windows 98 has me so scarred that, decades later, I always assume my computer is going to crash at any memento, and still periodically hit Control + S as an idle twitch.
This must be someone younger than me.
This must be someone younger than me.
I had a similar occurrence with Windows XP.
Two crashes a couple months apart led me to switching to Linux. I then read up on rsync and using hard links to make incremental snapshot backups. Coupled with doing this over ssh and now I had multiple automated backups.
As my laptop got older, it kept running fine b/c of Linux but I wanted a new laptop. My girlfriend at the time was using said laptop and drinking and said "Oh, do you want me to not drink over your laptop?"
I replied "that laptop has every bit of important data I need backed up in two different locations. Please spill whatever you want on it as that would give me a valid reason to go buy a new one."
Two crashes a couple months apart led me to switching to Linux. I then read up on rsync and using hard links to make incremental snapshot backups. Coupled with doing this over ssh and now I had multiple automated backups.
As my laptop got older, it kept running fine b/c of Linux but I wanted a new laptop. My girlfriend at the time was using said laptop and drinking and said "Oh, do you want me to not drink over your laptop?"
I replied "that laptop has every bit of important data I need backed up in two different locations. Please spill whatever you want on it as that would give me a valid reason to go buy a new one."
Okay you back up your goods. Now how you ever tried to restore <evil grin> ?
Anime avatar, uwu - und completely insane networking projects. Glad to know hacker culture is alive and well xD
> uwu
Debatable part of hacker culture. I'd say it's more a part of postmodern culture.
Debatable part of hacker culture. I'd say it's more a part of postmodern culture.
anime avatars and uwu are as much a part of hacker culture as burritos are
There’s definitely a strong correlation among certain segments of hacker culture.
There’s a lot of furries too.
Might be confirmation bias, we see the people that stick out, but to say there isn’t a large cohort that strongly identifies with anime is just wrong.
There’s a lot of furries too.
Might be confirmation bias, we see the people that stick out, but to say there isn’t a large cohort that strongly identifies with anime is just wrong.
>There’s a lot of furries too.
Wrong part of the internet.
Wrong part of the internet.
Nope. Internet infrastructure almost runs on furries nowadays.
Between O'Reilly's programming books and FOSS mascots, this outcome should be entirely unsurprising.
Furries run them tubes
One of the oldest MUDs was furries based. Go figure.
owo.
Old hacker culture, no, current hacker culture, yes. Anime themed ricing of Linux distros is extremely common, there are probably more anime profile pictures than not in some communities I hang out in
I guess I don't consider "ricing" your Linux distro as hacker culture. The people I knew from college CS courses who did that almost entirely spent more time tweaking themes than hacking(based on their success in the CS program). I see far more anime avatars associated with anti-social Twitter trolls rather than actual hackers.
uwu is rather new, but anime, not so much. SDF pubnix started around anime, but has had a strong hacker culture in it since then (1987).
Burritos definitely seem to be a critical part of hacker culture to me.
Speaking of, a burrito sounds great right now, so thanks for the reminder to go grab some dinner :)
Speaking of, a burrito sounds great right now, so thanks for the reminder to go grab some dinner :)
Burritos must be a regional thing then. At all my CCC interactions burritos never stood out to me.
anime has been a part of hacker culture since a while now
out of touch with new gen
nope.
fair enough
FWIW the Linux users group at my university (in North Carolina) is predominantly anime people
The Fixedsys-esque font in conjunction with the Anime aesthetic seems to tilt more towards vaporwave IMO.
I was expecting this to just be "the netcode was written in Bash. For the actual game state management, we just use existing X." But no, it looks like they've covered ALL of that.
Wow.
Wow.
I guess inline awk is fair game.
There’s a fine line between classy masochism and utter batshit insane lunacy, sir.
It's such a nebulous thing, eh? I'd say `Awk` (like curl, sed, grep, etc.) is part of the Bash toolkit. But then why isn't Python?
I guess the idea is that Bash is about piping together these small programs that typically take stdio and produce stdout. I think these have a name.. like GNU-style programs or something.
I guess the idea is that Bash is about piping together these small programs that typically take stdio and produce stdout. I think these have a name.. like GNU-style programs or something.
and sed
I think the first Q in the FAQ warms my heart:
> Q: Why?
> A: Because I could. And it was fun!
> Q: Why?
> A: Because I could. And it was fun!
my full question was actually "why would you do that to yourself?"
I’ve found bash to be a pretty fulfilling language to work in.
I fed The Linux Documentation Project and Pure Bash Bible into my Anki decks which was really a turning point in writing bash for me.
It’s pretty amazing how much can be accomplished in a relatively small amount of bash.
I fed The Linux Documentation Project and Pure Bash Bible into my Anki decks which was really a turning point in writing bash for me.
It’s pretty amazing how much can be accomplished in a relatively small amount of bash.
I haven't worked extensively in Bash alone but I did work extensively in Make for a while and I can say with confidence: the specialized old tooling languages are extremely good at what they do, Make goes out of its way to make building things extremely slick. These language tools were designed to tackle every problem in their realm because, for a lot of people, there wasn't an alternative - their structure shows clear and careful design intent.
Can you give some examples? I've personally found bash to be among the most difficult to work with. If I need to do something in bash I'd rather keep it as a lightweight wrapper around a ruby script or something, though I know that isn't always an option
For a pretty good example of a readable/fun program written in bash, the KISS package manager stands out: https://github.com/kisslinux/kiss/blob/master/kiss
Neofetch is also written in bash: https://github.com/dylanaraps/neofetch/blob/master/neofetch
For my own work, I once wrote a functional file system based irc client in ~50 lines of bash: https://github.com/retrohacker/irc-sh/blob/master/main.sh
To learn bash:
* https://tldp.org/LDP/abs/html/
* https://github.com/dylanaraps/pure-bash-bible
Neofetch is also written in bash: https://github.com/dylanaraps/neofetch/blob/master/neofetch
For my own work, I once wrote a functional file system based irc client in ~50 lines of bash: https://github.com/retrohacker/irc-sh/blob/master/main.sh
To learn bash:
* https://tldp.org/LDP/abs/html/
* https://github.com/dylanaraps/pure-bash-bible
Bash is about pipelines which is a rather functional concept. If you can exploit these you can write succinct, readable and effective bash IMHO. If you're just stringing together a lot of conditionals it gets ugly fast.
There isn't such a thing as readable bash, as a significant amount of its syntax is symbolic (and cryptic, by the way), and it's full of warts and unintuitive quirks.
Writing very simple scripts is certainly simple, but anything even a bit functionally more advanced (arrays, but even strings cycling or splitting is quirky) is a pain. Heck, even something as simple as process substitution is horribly unsafe (I wonder how many devs know how/why).
It's practically impossible to write correct Bash without shellcheck.
Writing very simple scripts is certainly simple, but anything even a bit functionally more advanced (arrays, but even strings cycling or splitting is quirky) is a pain. Heck, even something as simple as process substitution is horribly unsafe (I wonder how many devs know how/why).
It's practically impossible to write correct Bash without shellcheck.
I disagree that bash needs to be cryptic, but I've also written perl code, so maybe any resemblance of a sane perspective was beaten out of me.
> It's practically impossible to write correct Bash without shellcheck.
I agree with this, since I've never seen a bash script, in the wild, that passed the important bits of shellcheck, without already being passed through spellcheck. I challenge anyone that thinks they've written good bash, with any significant level of complexity, to run that script through shellcheck. Each time I've done this, when starting at a new company, the response has usually been "Will not fix. That's why you don't use spaces in filenames/paths".
> It's practically impossible to write correct Bash without shellcheck.
I agree with this, since I've never seen a bash script, in the wild, that passed the important bits of shellcheck, without already being passed through spellcheck. I challenge anyone that thinks they've written good bash, with any significant level of complexity, to run that script through shellcheck. Each time I've done this, when starting at a new company, the response has usually been "Will not fix. That's why you don't use spaces in filenames/paths".
> I disagree that bash needs to be cryptic
But it's not a developer's choice - Bash's syntax is cryptic itself. Here's a random sampling of common patterns:
Arrays:
But it's not a developer's choice - Bash's syntax is cryptic itself. Here's a random sampling of common patterns:
Arrays:
- `${#myvar[@]}`: unreadable
- `mapfile myvar < <(grep pattern file)`: "mapfile" is not exactly a clear term (there's the synonym
"readarray", but "mapfile" is the reference; worse, you may find both used inconsistently); the
command as I wrote it also has two problems
- `${myvar##.*}`: unreadable
- `echo $(IFS=,; echo "${myvar[*]}")`: unreadable; most also don't know the difference between `[@]`
and `[\*]`; this doesn't also work for two-char join operations
- `[[ $myvar =~ $(echo '\bpattern\b') ]]`: unreadable, and most don't know why command substitution
is needed
- `3>&2 2>&1 1>&3`: most don't know what this means
[deleted]
If you are manipulating files and processes, bash is a very natural and succint language to do so. Programs are much more elegant than a generic programming language.
I found that nobody actually works in "pure bash". There's always sed, awk, perl, something to do fractional math, etc.
Do you have a link to the deck? I want something like that.
the best praise I've received for something I worked on was:
This is an absolutely batshit crazy idea. I love it.
I notice that the author is going into awk for floating point processing. The 1993 language standard for the Korn shell brings floating point into the language as a native data type. This advanced version of the Korn language does not appear to be under active maintenance, but it will likely be much, much faster than handling floating point in awk. Unfortunately, none of the BSD-licensed clones of Korn support the 1993 standards (including floating point).
I also see that the author is having some trouble reading individual bytes. Perhaps these shell tricks might be useful.
http://www.etalabs.net/sh_tricks.html
I also see that the author is having some trouble reading individual bytes. Perhaps these shell tricks might be useful.
http://www.etalabs.net/sh_tricks.html
At one point I've had a NBT parser implementation implemented almost fully, but I decided it was not worth the hassle to finish it. The code is currently lost, due to my extensive use of tmpfs as a project directory, and a system crash.
Also, well done! This looks like it was a lot of fun.
Where there's a will, there's a way. I once needed to do network calculations where I only had some very limited tools, so I used awk to convert 32-bit ints to a 32-character ascii bit string of "1" and "0" and the converse function, then wrote and, or, and not functions that worked on the ascii strings.
https://gist.github.com/jaysoffian/e41ca479d70e60efe59fded93...
https://gist.github.com/jaysoffian/e41ca479d70e60efe59fded93...
gloriously insane. So how do we represent /proc in such a way as to make interacting with it through minecraft useful?
"psDoom" was great fun now i see "tunnel through the DB's page tables searching for a more direct route to the application engine"
"psDoom" was great fun now i see "tunnel through the DB's page tables searching for a more direct route to the application engine"
> So how do we represent /proc in such a way as to make interacting with it through minecraft useful?
Not exactly what you wanted, but Hajime can already query info from /proc to make it accessible from within Minecraft.
https://github.com/Slackadays/Hajime
Not exactly what you wanted, but Hajime can already query info from /proc to make it accessible from within Minecraft.
https://github.com/Slackadays/Hajime
The one thing that people who understand bash have in the forefront of their minds is error handling.
You can be a wizard in bash if nothing ever goes wrong. Functions calling functions calling functions. It’s basically programming.
Until something goes wrong. Then there’s nothing you can do.
Anyone who has ever set-dash-ee’d and then seen a function execute without set-dash-ee when used in an if context knows what I mean.
You can be a wizard in bash if nothing ever goes wrong. Functions calling functions calling functions. It’s basically programming.
Until something goes wrong. Then there’s nothing you can do.
Anyone who has ever set-dash-ee’d and then seen a function execute without set-dash-ee when used in an if context knows what I mean.
That's why the cool kids today use set -euo pipefail.
Or set -x if you're debugging.
Or set -x if you're debugging.
You may have missed my point.
Set -e is not global. It only applies in some cases. In other contexts (inside a function when called as if <function>) errors will not be caught and once you realise that you realise there’s no hope of writing error proof logic in bash.
Set -e is not global. It only applies in some cases. In other contexts (inside a function when called as if <function>) errors will not be caught and once you realise that you realise there’s no hope of writing error proof logic in bash.
Ah, I'm sorry. I didn't realize the problem with functions and indeed misunderstood.
I mean, while it's amazing what you can do with bash, I'd be wary to use it for "production" stuff. So far, I mostly used it for "toolbox" type if stuff that is only supposed to be used by the devs. For that purpose, it worked well though.
I mean, while it's amazing what you can do with bash, I'd be wary to use it for "production" stuff. So far, I mostly used it for "toolbox" type if stuff that is only supposed to be used by the devs. For that purpose, it worked well though.
That ... doesn't solve the problem GP is noting.
This reminds me of the xmpp chat bot written in sed: https://github.com/horazont/xmpp-echo-bot
While I admire the skill involved, I keep wondering what drives people into starting such projects :D
While I admire the skill involved, I keep wondering what drives people into starting such projects :D
I saw a Z-Machine interpreter (Zork, Infocom games, IF archive...) written in damn PostScript.
Projects like this make me warm inside. Lovely work.
Reading stuff like this makes me think about how much more... "Fun" game programming is than the stuff I do in web dev.
These problems (yes, self-constrained) look like so much fun to solve, in a way that "I did xyz thing in pure CSS" is just less so to me.
Maybe It's that I miss rapid cycle of Completely Lost -> Earth-shattering Realization of How To Make This Work that the first ~1-2 years of my programming journey was full of
These problems (yes, self-constrained) look like so much fun to solve, in a way that "I did xyz thing in pure CSS" is just less so to me.
Maybe It's that I miss rapid cycle of Completely Lost -> Earth-shattering Realization of How To Make This Work that the first ~1-2 years of my programming journey was full of
The web has some pretty cool stuff to tinker with these days. WebGL in particular would give you that sense of learning you’re craving, is practical to know and IMHO has a much more pleasant feedback loop than C++.
> As in my opinion the wiki page doesn't explain it well enough to quickly comprehend, here's another drawing:
Contributions welcome :) Our wiki is open and anyone can edit. If you can ELI5 something to make it easier to understand, go for it.
Contributions welcome :) Our wiki is open and anyone can edit. If you can ELI5 something to make it easier to understand, go for it.
Such a fan. 'The code is currently lost, due to my extensive use of tmpfs as a project directory, and a system crash.'
And I was sad to see some things farmed out to awk. I mean what's next? bc? ed? ;-)
And I was sad to see some things farmed out to awk. I mean what's next? bc? ed? ;-)
I am in awe. Also, I think you are a huge masochist! It's like wood working with cutlery. But it's actually really informative about some advanced bash knowledge. Kudos!
I'm surprised this person didn't switch to using wcalc, instead of awk. For comparison:
fao_@blob:~$ time $(echo '' | awk '{print (2*-1)}' >/dev/null)
real 0m0.006s user 0m0.006s sys 0m0.000s
fao_@blob:~$ time $(wcalc '2*-1' >/dev/null)
real 0m0.005s user 0m0.001s sys 0m0.005s
I'd imagine you could use something like xargs to insert the number so that wcalc can interpret the bits for you
fao_@blob:~$ time $(echo '' | awk '{print (2*-1)}' >/dev/null)
real 0m0.006s user 0m0.006s sys 0m0.000s
fao_@blob:~$ time $(wcalc '2*-1' >/dev/null)
real 0m0.005s user 0m0.001s sys 0m0.005s
I'd imagine you could use something like xargs to insert the number so that wcalc can interpret the bits for you
The results you post don't seem to differ much, do you mean the difference is bigger as the inputs get larger?
Spawning an extra subshell inside 'time' seems strange, as does using echo instead of </dev/null.
Also: people wiriting things in bash usually try not to rely on utilities that are not present by default, and wcalc is not.
Also: people wiriting things in bash usually try not to rely on utilities that are not present by default, and wcalc is not.
I think it's supposed to run on BusyBox.
Besides the nostalgia, the website font is so easy on my eyes I'm considering to give up space and start to use it as well
I searched for "font" in this thread and certainly did not expect this comment. I found it so hard to read that I added custom CSS rules to replace it with 'sans-serif' for the body and 'monospace' for the code.
This is a good site to check out:
https://int10h.org/oldschool-pc-fonts/
https://int10h.org/oldschool-pc-fonts/
What's next, a database in pure bash? It's really impressive what some people can do with a couple of shell scripts.
From a quick search didn't find anyone going all out, but did find this that could be used as a starting point: https://matt.might.net/articles/sql-in-the-shell/
Not as cool as this :) but I recently released an app for running Minecraft servers from your phone. CraftBox: https://play.google.com/store/apps/details?id=tech.ula.craft...
Example for reading and writing binary data (i.e. including 0's) in pure Bash, check the "enc_hex" function:
https://github.com/faragon/lzb/blob/master/lzb
https://github.com/faragon/lzb/blob/master/lzb
I just love this, this is why i love programming and writing a forking minecraft server in C was one of the most fun things i did in computing, i still have the source for that experiment, good times!
Appreciate the QR code! I scanned it. You should too.
I wonder if there's C compiler written in shell that's good enough to bootstrap something like gcc?
> The Protocol Is Not Really Good, Actually
Title of my programming memoir right there
Title of my programming memoir right there
this is so cool.
This is a driver/client for a vintage portable disk drive that has an rs232 interface. Any disk drive obviously has to handle arbitrary data including binary including 0x00.
It's entirely in native bash with no external tools and not even any subshells. It does have to call stty and mkfifo once at startup, but that's just startup like to open the serial port, not part of the work.
https://github.com/bkw777/pdd.sh
The gimmick for reading nulls is not exactly efficient. Basically you read one character at a time and look at the errorlevel from read to detect the difference between "got nothing" and "got 0x00"
It's fine for this case because the drive is so slow and small that even the entire disk is only 200k max. Making bash read one byte at a time for 200k is still nothing today but only because hardware is insane today.
But it's possible. And the beginning of the post does say "as a thought experiment".
Similarly, you don't need xxd to convert back & forth between raw binary and text encoding.
My own similar thought experiment included the idea that, if I'm not going to use any external process I can possibly avoid and only use internal bash features, on the flip side of that, I will squeez bash for all it's worth, allow every bashism possible.