mostly strace and it's macos equivalent; Later we moved to ptrace and ebpf. LD_PRELOAD unfortunately doesn't work for statically linked libc.
There are also kernel probes but didn't like that it required root permissions...
Love it!
We did something similar using strace/dtruss back in 2018 with https://buildinfer.loopperfect.com/ and were generating graphs (using eg. graphviz and perfetto.dev) and BUCK files on the back of that
Whilst we regrettably never came around to package it as a propper product, we found it immensly valuable in our consulting work, to pinpoint issues and aid the conversion to BUCK/Bazel.
We used graphviz, https://perfetto.dev/ and couple other tools to visualise things
Recently we cicled back to this too but with a broader usecase in mind.
There are some inherent technical challanges with this approach & domain:
- syscall logs can get huge - especially when saved to disk. Our strace logs would get over 100GB for some projects (llvm was around ~50GB)
- some projects also use https and inter process communications and that needs ot be properly handled too. (We even had a customer that was retriving code from a firebird database via perl as part of the compilation step!)
- It's runtime analysis - you might need to repeat the analysis for each configuration.
> Each binary gate takes about 13 milliseconds single-core time to evaluate, which improves [DM15] by a factor 53, and the mux gate takes about 26 CPU-ms
Addition of two bits can be implemented using 5 binary gates (fulladder)
Hence to add 2 32 bit numbers ~416ms => 2 additions per second
Great work!
This allows to perform arbitrary computation on untrusted devices.
However last time I checked computation in this scheme is ridiculously slow: on modern machines, cutting edge implementation of FHE manage to get around 100 integer operations per second.
Never the less there have been some brave startups trying to commercialise this technology:
- Buck leverages remote caches currently better than Bazel [3]
- Bazel is very easy to install
- Bazels "toolchains" make it very easy to onboard newcomers (to any project and language) but also ensure the build will run as expected.
- Bazel is less opinionated and more extensible than Buck.
In fact Bazel is so powerful that you can have Buildfiles that download a package manager and use it to resolve more dependencies.
This is great to get things off the ground, but makes things less composable because the package manager won't see the whole dependency graph. As a result you might get version conflicts somewhere down the line.
To summarize: I think having a very opinionated build-system is easier to reason and scales usually better.
Communities with very opinionated packaging and build-systems are proving this by having orders of magnitude more packages that eg. the highly fragmented C++ community where configuration is prefered over convention.
> Have you considered offering support in Bazel for your package manager?
Yes we did. As soon as this feature [1] is implemented we will have a 1:1 mapping for C++ Buck Projects and Bazel. Then after a small (automated) refactoring of our Buckaroo packages, you should be able to build any package from the Buckaroo ecosystem with either Buck or Bazel.
Btw. The cppslack community is attempting to create a feature matrix of various build-systems here [2]
Overall Buck and Bazel are quite similar as they are both converging to Starlark DSL.
However there are still some differences:
Buck is much more opinionated than Bazel.
Buck models slightly better C++ projects[1] and currently it's remote cache is more efficient[2].
Bazel is more extensible and offers also remote execution.
Bazel has a bigger community and it's roadmap is public.
There has been more than 350 C++ [4] libraries been ported to Buck for the Buckaroo Package Manager[3].
There are also technical details that manifest in some odd ways but are not significant.
There is a nice paper by Simon Peyton Jones (creator of haskell) and others that goes into the design details [5]
We think Buck is great.
It's Deterministic hermetic builds and it's composable and declarative high-level build description language made packaging very easy.
We built even built a package manager for Buck:
https://github.com/LoopPerfect/buckaroo
Currently we marketing it for C++ but it can be used for any language that is supported by Buck.
As one of the creators of a distributed package manager for C++ and friends [1] we made a funny discovery:
Many C libraries that a big chunk of the ecosystem depends on, have not been updated for many years. Some of those can only be downloaded from sourceforge or ftp server.
Even worse, some libraries are copy and pasted from project to project and have no actual home.
We uploaded them to github and started maintaining them.
If you know any abandoned C/C++ projects or C/C++ projects you need a hand in maintaining, we are happy to help.
This is great!
This will make bootstrap more attractive and competitive as a framework.
JQuery was one of the most important frameworks in Javascript history. It has enabled us to built real webapps.
However since then differences between browsers shrunk significantly and we learned how to build maintainable and scalable apps in a more declarative fashion (hello React, Angular and friends)
buck implements sophisticated (disk & network) caching, optimization and scheduling strategies to make things fast.
Furthermore it will use services like watchman (if available) to precompute what things need building if you change some files for fast incremental builds.
Lastly it will strip and sort symbols in your binary to make sure the hash of your binary is always the same for the same set of inputs.
All this complexity is handled for you and all you need to do to run your executable is
But all those dependency packages will be resolved against a conan server not against the origin eg. github.
Furthermore I'd need to put every version of the package into the conan cache.
This is impractical if a complex dependency graph as it would require the user to solve a SAT problem just to determine which packages and versions to put into the conan cache to avoid pulling from the conan server.
May projects use some document generators or tools to collect metrics. However some build-systems that were really interesting were OpenCV and OpenSSL.
OpenSSL buildsystem is over 6k lines of perl scripts which communicate over pipes.
OpenCV uses cmake, python and prolog for it's build. Until this day, I have no clue why there is prolog involved.