It depends how much the code uses manually spawned goroutines, and how complex the lifecycle of these goroutines is… in big codebases such as kubernetes, docker, etc, it has been a problem. There have been research papers and blogs about this, but most Go developers are not aware of this issue it seems.
I am not an expert in incident reaction, but I thought the safe way was to image the affected machine, turn it off, take a clean machine, boot a clean OS image with the affected image mounted read only in a VM, and do the investigation like that ?
Assume that the malware has replaced system commands, possibly used a kernel vulnerability to lie to you to hide its presence, so do not do anything in the infected system directly ?
Nice article, thank you.
Did you also consider using bpftrace while debugging?
I do not have much experience with it, but I think you can see the kernel call stack with it and I know you can also see the return value (in eax).
That would be less effort than qemu + gdb + disabling kernel aslr, etc.
Author here, thanks for the feedback on legibility, I have now just learned about the CSS `tab-size` property to control how much space tabs get rendered with. I have reduced it, should be better now.
Just one point about testable assembly: I like the Golang guidelines which require to write a Go version before implementing the same in assembly. That way there is a baseline to compare with in terms of correctness and performance. Also the Go project has recently introduced mutation testing and test coverage on assembly code which is very interesting.
Finally about detecting at runtime the presence of simd: I am confused why a third-party dependency or even the std is needed. At least on x64, it’s just one instruction (“cpuid”) and then checking if a bit is set, right?
I agree with the premise, just use a specific version of your dependencies, that’s generally fine.
However: You absolutely do need a lock file to store a cryptographic hash of each dependency to ensure that what is fetched has not been tampered with. And users are definitely not typing a hash when adding a new dependency to package.json or Cargo.toml.
I wrote a toy Kotlin compiler, for fun.
Then one day a Jetbrains employee opens an issue which only says: “Why? Just why?”.
Maybe it’s the language barrier… but I did not find that particularly polite.
On the other hand I open sourced my blog and received lots of small contributions to fix typos or such which were nice.
I have implemented polling against a cluster of mixed mariadb/mysql databases which do not offer listen/notify. It was a pain in the neck to get right.
- The batch size needs to be adaptative for performance, latency, and recovering smoothly after downtime.
- The polling timeouts, frequency etc the same.
- You need to avoid hysteresis.
- You want to be super careful about not disturbing the main application by placing heavy load on the database or accidentally locking tables/rows
- You likely want multiple distributed workers in case of a network partition to keep handling events
It’s hard to get right especially when the databases at the time did not support SKIP LOCKED.
In retrospect I wish I had listened to the WAL. Much easier.
The x64 assembly would probably work natively on the Mac, no need for docker, provided the 2 syscall numbers (write and exit) are adjusted. Which llms can likely do.
If it’s an ARM Mac, under Rosetta. Otherwise directly.
About not having perf on macOS: you can get quite far with dtrace for profiling. That’s what the original flame graph script in Perl mentions using and what the flame graph Rust reimplementation also uses. It does not have some metrics like cache misses or micro instructions retired but still it can be very useful.
I have worked for financial services companies that used FreeBSD both in EC2 and on the metal in data centers (self managed).
The two features we used all the time were zfs and jails.
Each service ran in its own jail for isolation. One (not even beefy) server could run all the services which was insanely cost efficient.
A cloud migration was undertaken at some point to have a hybrid setup, using a mix of Linux (k8s) and FreeBSD, and costs skyrocketed. It’s a trade off because in the data center we had to buy and replace our own disks, react to fires taking place, being only in one country etc.
AWS gives you multi region, and tons of good stuff, and that has a price.
ZFS was not leveraged that much but it saved our beacon once when a table in the production database was accidentally dropped and we could instantly rollback to the previous zfs snapshot (there was a tiny bit of data loss as a result but this did not matter too much for this application - uptime was more important). ZFS was also used for backups I believe.
A few times I used dtrace in production to troubleshoot.
When we introduced Linux to our fleet of FreeBSD servers, every team picked a different distro organically so it was a bit of a zoo. With FreeBSD on the server you only have the one variant.
I still use and like both, but I must say I really like that FreeBSD is a kernel+OS integrated together.
I can tell you some financial services I have worked for do use FreeBSD on EC2 as well as on the metal in data centers to do millions of transactions a month.
I like the OS, thanks for your work.
Good article, I also have a C program (a compiler) that I would like to compile to webassembly to offer a playground web page, so that is good information. Thank you!
About the file system stuff: modern browsers ship with SQLite which is available to JavaScript (is it available to webassembly? No idea) so I would probably use that instead. Ideally you could use the sqlite API directly in C and emscripten would bridge the calls to the browser SQLite db. Something to investigate.