I had a similar issue. The blind spot was unit tests.
I think the issue is just that it's incredibly hard to sell an abstract idea and incredibly hard to convince people to abandon ingrained habit.
I created a testing framework where you wrote half a test in YAML and the framework filled in the rest based on program output.
It made writing tests quick, easy and even kinda fun.
Moreover if you added a bit of explanation prose to the YAML and used a slightly nicer example scenario it would generate you guaranteed up-to-date readable markdown how to docs. For free.
But, these things are culturally chorey and there's a shame culture built around them.
CUE lets you embed functions too it looks like it's almost a programming language itself.
The closer a configuration language gets to a programming language the less of a reason I see for it to exist.
>I would argue that it's a slippery slope. Consider v1 where an enum is statically defined as Employee or Manager. Then in v2 we add VP and CEO. Then in v3 actually the list of permitted titles needs to be fetched from a database populated by HR. Is it still correct to put this in configuration validation?
No, coupling to a database would be bad design IMO, but grabbing those enums from other config files in the same folder that are parsed earlier I have done a lot.
I've also used libraries that provide lists of timezones and country codes as enums and plugged them in to the parser so you couldnt invent your own country code.
And Ive written validators that reference other bits of the config (e.g. the list of permitted titles is in another part of the config).
All of these things I would argue are good and useful and not worth sacrificing in exchange for preventing possible misuse (like coupling parsers to a DB).
I actually wrote this parser in the first place because I wanted to create a good metalanguage for tersely defining strongly typed executable specifications in YAML (i.e. Gherkin done right). Tons of stuff I wanted to strictly validate wouldnt have been possible with config-based schema validation and with YAML's weak, implicit typing it was a fucking mess.
The difference is because full validation/parsing is a task that can rarely be always fully accomplished with JUST a non-turing complete schema. Every time I use JSON schema I have to add additional validation on top written in turing complete code.
This happened to me literally just an hour ago when I wanted to put a DSL in a field in a config file. json-schema (the "config" schema) doesn't let me write code to validate this and reject it. It's a string or it's not. With StrictYAML schemas written in code it's pretty straightforward to create a parser/validator that rejects invalid DSL with a meaningful error.
You might argue that "these rules bolted on top aren't part of the schema" or "this is validation that you can do after the json schema validates" but there is benefit to combining them - namely, code coherence and validation error consistency.
(there are also down sides - namely that json schema can be used in multiple languages. strictness comes at the expense of reusability).
In practice almost every schema I build I want to have stricter validation rules that are not enforceable with something like json-schema alone.
These are both instances of the law of least power. There are plenty of languages which are too powerful for the task at hand and plenty which are not powerful enough and people hack around and even rage against both. There are other "goldilocks" languages that are just right for the task at hand.
>you sort of can't make standalone ELisp apps (or at least nobody does). Otherwise you'd just have an `org2html` and `org2pdf` application at the command line.
Because I don't like emacs and I wanted to lean on command line apps for getting data into and out of my orgzly org-mode electronic brain, I've been doing this by combining orgmode with jinja2 (I do this with a cli app I wrote called 'orji').
I think it's a very powerful combination. I use it to generate latex/html/whatever and short bash scripts which are then run. E.g.
* Generate my CV using a latex CV template I found on overleaf that I converted into a jinja2 template.
* Send the contents of a TODO note tagged 'sendemail' as an email with a tiny templated bash script.
* Create jira ticket with the details of a note.
* Generate reveal.js presentations.
All from my phone, using orgzly and one big button which runs a single termux script that seeks out TODO notes with labels (e.g. cv/sendemail) that match the scripts in my library (cv.sh / sendemail.sh).
jinja2 certainly has the capability to end up creating a huge old mess if you abuse it, especially to generate code, but I find that using it to template tiny 5-6 line bash scripts or a latex or an HTML file hits a sweet spot of usability and flexibility.
It's not purely about skill, code quality also improves as a function of discipline, a willingness to take risks and outside pressure.
fwiw I dont think Ive ever seen clean code that didnt make use of something at the very least resembling both exhaustive CI/CD and TDD. There are some practices which are basically essential even if some of the mavens of "best" practices mistakenly label a few that arent necessary as necessary.
The best way to conceptualize hexagonal is as a kind of crutch to accomodate the inability of unit tests to effectively fake stuff like the db and their tendency to tightly couple to everything.
It's not intrinsically good design but it does improve unit testability (which sometimes has value and sometimes has zero value).
>And stuff like mocks etc take far longer to write than a 5 line fix
If you test from the outside in and build up a library of functional and realistic fakes then over time then this gets quicker and easier.
Ideally I think people shouldnt use mock objects at all, ever - only fakes of varying realism at the edge of the project, populated with realistic data.
One reason for doing TDD is that it compels you to match a realistic test with a realistic scenario. I tend to find people lose that when they do test after, and they instead lock the test in to the current behavior of the code instead. This is not just tedious work, it's also harmful.
Literate programming shouldnt necessarily apply to application code for large systems but I almost always want more literate tests that clearly explain not just what should happen under all kinds of scenarios but why it does that. Even better if I had hyperlinked explanations for domain specific terms.
If I always had this I think I'd become fully productive on large systems 2x as quickly.
I dont think this need be achieved by hiring a unicorn coder who is also a great writer but by making tests more accessible and editable by good technical writers so that both parties can collaborate around behavioral tests that double as documentation.
They're attacking the problem from two directions - bottom up and top down.
I view the type system as a way to whittle down the state space to the bare minimum while the tests validate that that the behavior of that whittled down execution space matches what is required.
I think investment in a type system to replace tests very quickly reaches a point of diminishing returns and vice versa. A really good type system won't ever beat ok tests + an ok type system and vice versa.
They are useful if you program in a somewhat object oriented way. They are deeply unhelpful if they're all either "method(self) -> dict" or "method(input: pd.DataFrame)"
I think the issue is just that it's incredibly hard to sell an abstract idea and incredibly hard to convince people to abandon ingrained habit.
I created a testing framework where you wrote half a test in YAML and the framework filled in the rest based on program output.
It made writing tests quick, easy and even kinda fun.
Moreover if you added a bit of explanation prose to the YAML and used a slightly nicer example scenario it would generate you guaranteed up-to-date readable markdown how to docs. For free.
But, these things are culturally chorey and there's a shame culture built around them.