Use weird tests to capture tacit knowledge(jmduke.com)
jmduke.com
Use weird tests to capture tacit knowledge
https://jmduke.com/posts/essays/weird-tests-tacit-knowledge/
5 comments
More info on pytest's test parametrization: https://docs.pytest.org/en/7.1.x/how-to/parametrize.html
> The perfect solution to this problem is creating a script that automatically adds a new app to all the relevant places and stuffing it into a Justfile, but that's a pretty big piece of work that requires thought and error handling and a whole slew of other stuff. Instead, it's comparatively easy to just capture these constraints in a test:
Just write the goddamn script. You can put comments in that too. Not saying it shouldn't have tests but the solution to bunch of steps that need to be done always, and in same, easy to automate way is not "make sure the checklist is always there so other people waste the optimum amount of time on it".
Just write the goddamn script. You can put comments in that too. Not saying it shouldn't have tests but the solution to bunch of steps that need to be done always, and in same, easy to automate way is not "make sure the checklist is always there so other people waste the optimum amount of time on it".
But then no one adds an app for a few months, during which the requirements change but the script isn't updated. Then it fails and someone who wasn't there when it was written is off in the weeds trying to sherlock the reason this thing exists in the first place and how to update it, making their minor change seem much larger & riskier to reviewers. Or worse it doesn't fail, but the incomplete run introduces a defect no one is looking for.
You simply can't make judgements about what should be done to a codebase from a distance in the abstract like this. Sometimes you should write the script, sometimes you shouldn't. You need specific knowledge of the circumstances to exercise good judgement about it which we certainly don't have from this comment section.
You simply can't make judgements about what should be done to a codebase from a distance in the abstract like this. Sometimes you should write the script, sometimes you shouldn't. You need specific knowledge of the circumstances to exercise good judgement about it which we certainly don't have from this comment section.
Another thing that came to mind: do-nothing scripting[0].
[0] https://news.ycombinator.com/item?id=29083367
[0] https://news.ycombinator.com/item?id=29083367
> pytest.ini, so tests are run;
Seems weird to me. I don't have to add anything to this file when I add an app to my Django projects. How did you end up there?
> pyproject.toml, so files are linted;
This too. Same comment as above.
> modules.txt, so metrics are exported.
I don't know what this is. Sounds like something specific to your code base?
Seems weird to me. I don't have to add anything to this file when I add an app to my Django projects. How did you end up there?
> pyproject.toml, so files are linted;
This too. Same comment as above.
> modules.txt, so metrics are exported.
I don't know what this is. Sounds like something specific to your code base?
If you're able to automate a script for it, or make tests for it, it's not tacit knowledge.
Also, as someone else has said, why not just automate the prcoess? Why spend the time to write a test when that time could be spent on the automation and saving everyone's time in the future?
Also, as someone else has said, why not just automate the prcoess? Why spend the time to write a test when that time could be spent on the automation and saving everyone's time in the future?
There was a code quality tool (Snyk) which had a blog post [citation needed] about focusing fixes on only recently changed files. eg: You start adopting Snyk, it starts complaining about a bunch of stuff ("ERR: Too much whitespace in if statement..." or whatever), and you _would_ like to conform to that, but it's a massive undertaking to modify your whole codebase to support it. To be clear: much of the results from Snyk are actual functional issues, I'm just throwing out an overly simplistic example.
Anyway, their blog post mentioned something like "fix the tip of the wave, not the ocean", and that metaphor has stayed with me ever since.
Internally in your codebase, you may have `TheDatabase.GetUsers(...)`, and want to migrate to something like `PerformantShardingV2.GetSomeUsers(...)`. It's a perfect example of where something like this technique would work.
You would/could/should use code to `grep -l TheDatabase.GetUsers` and get a list of filenames _known_ to use the old-and-busted way of doing things. You would/could/should have a test that says "ERR: $new_filename uses TheDatabase, but is not pre-approved, use PerformantShardingV2 instead!".
Additionally, it'd be great if there was a way to say something like: "ERR: Your PR modified files that use `TheDatabase` but you didn't migrate to `PerformantShardingV2`, please upgrade!".
Long way of saying: the technique is incredibly useful for gradual (not big-bang) changes/rewrites, which is mostly what you should be doing. Standards, compliance, and best-practices do tend to change and skew over time, and having a technique like this is very valuable.
Why you wouldn't have `just make-a-new-enum-json` as you're suggesting? It isn't very discoverable (unless you have a VERY good business vocabulary and constant training) and if you don't know you'd need to use it then you wouldn't have used it in the first place. Additionally, you'd still want to have this double-checker to prevent errors or standards/best-practices violations from getting in to your codebase.
This _is_ automation that saves everyone's time in the future, and is complementary to some of the automation that you're talking about.
Anyway, their blog post mentioned something like "fix the tip of the wave, not the ocean", and that metaphor has stayed with me ever since.
Internally in your codebase, you may have `TheDatabase.GetUsers(...)`, and want to migrate to something like `PerformantShardingV2.GetSomeUsers(...)`. It's a perfect example of where something like this technique would work.
You would/could/should use code to `grep -l TheDatabase.GetUsers` and get a list of filenames _known_ to use the old-and-busted way of doing things. You would/could/should have a test that says "ERR: $new_filename uses TheDatabase, but is not pre-approved, use PerformantShardingV2 instead!".
Additionally, it'd be great if there was a way to say something like: "ERR: Your PR modified files that use `TheDatabase` but you didn't migrate to `PerformantShardingV2`, please upgrade!".
Long way of saying: the technique is incredibly useful for gradual (not big-bang) changes/rewrites, which is mostly what you should be doing. Standards, compliance, and best-practices do tend to change and skew over time, and having a technique like this is very valuable.
Why you wouldn't have `just make-a-new-enum-json` as you're suggesting? It isn't very discoverable (unless you have a VERY good business vocabulary and constant training) and if you don't know you'd need to use it then you wouldn't have used it in the first place. Additionally, you'd still want to have this double-checker to prevent errors or standards/best-practices violations from getting in to your codebase.
This _is_ automation that saves everyone's time in the future, and is complementary to some of the automation that you're talking about.
Practically all knowledge was once tacit knowledge. I bet protohumans found fire from a lightning strike long before they figured out how to make it themselves.
Even the idea of a city, say London, is just a non-tacit version of the idea of the area where London is.
Even the idea of a city, say London, is just a non-tacit version of the idea of the area where London is.
I've done this before, it's a good solution. It's helpful to include relevant error messages to precisely explain what a future developer needs to do.