We might be moving towards putting more intent - in text - into the code itself, for agents. We should've been doing that for other humans in the past, but it wasn't always encouraged - "code should be self-explanatory, and comments are a sign of poor naming and abstraction", and all that.
These days I run a prompt of this shape to ensure that my code contains as much tacit knowledge as I have -- "I want to make sure all the implicit invariants and structures and concepts described in these plans, are present in the code comments, namings, comments around major types and modules. So even if the original plan file is not present, we should still be able to understand it (Peter Naur - Programming is Theory Building)"
However - I still need the type system as another preserver of invariants and intents. Coding agents still make subtle mistakes and miss very obvious things as the code gets more intricate and conceptually large.
A type error where an expected case is not handled - that is a feedback that can ground it back and force it to think through whether windspeed matters to is_dangerous or not. And the choice of is_dangerous in the article is a bit misleading, because anyone can at first read make the connection from windspeed to is_dangerous. It is a pretty bold variable name and attention grabbing situation.
However codebases are made of far more mundane but widespread and braided sets of tiny invariants that wouldn't catch either a human's or a coding agent's attention without needing a more firmer hand, and so I still do love my type systems despite my best laid comments.
But couple that with a deterministic system - the type checker with exhaustive destructing (and pattern matching, while we are it it) - that would become a fool-proof combination.
I hadn't considered that, you are right. Though I would still prefer TS because the language is all about types; the entire ecosystem is well-typed and the type system is quite powerful (I do enjoy an occasional Omit and Pick). But it is a personal choice; as long as LLMs can generate well-typed Python, then for people who like the language it makes sense. However from the article, I got the sense that they went completely dynamic.
I'm curious about the choice of Python, rather than TypeScript.
I find Ruby a very beautiful language, and Rails is an excellent web framework, but I need typed functions, record types and sum types.
They help not just with correctness, but also as living documentation that lets me understand AI generated code. TypeScript provides discriminated union, but not exhaustive pattern-matching, and its syntax is a bit verbose, but since I'm no longer writing most of the code myself, I can live with it.
However I can't imagine using Python or any other dynamic language going forward. There is likely good reason for you to choose it, and I'm curious to know what that is.
There is no other stack in the last 15 years or so that I look back with the same fond affection as building business applications in Clipper. One important aspect was that software was delivered physically, and I could watch my users using them. If I made a report that helped the operator, or made a workflow easier, their happiness was immediate and nurturing. And there was an immediacy to programming in xBase - if we're building a database backed business application, which was their strong suit, there was very little standing in between us and the problem space.
"There was no exploit. No vulnerability disclosure. No CVE for me to write. The attacker filled out my signup form 942 times, made 942 workspaces, sent 942 batches of about a hundred invitations each, and stopped. They used my tool exactly as designed. The design was just bad enough that the tool was good for phishing."
In my early programming days, working with Clipper, I used to look at Delphi from a distance with awe and a bit of jealousy. There also used to be PowerBuilder and Paradox, as competition to the xBase platforms.
I'd love to hear more about how you're using Delphi and what it excels at, compared to current web and native software stacks.
Everytime someone mentions Clipper (or dBase or FoxPro, or even FoxBase, but Clipper the most), I feel a sene of productive nostalgia, and a constructive anger at the state of technology today. xBase was a beautiful thing - I haven't had as much fun at building software that I've had from the first plink86 till CA-Clipper 5.3's blinker and exospace. Even prolific use of Opus 4.6 doesn't bring the sense of quality and satisfaction that those systems produced.
I'm building a new database tool for the web, a frankenstein of Lotus 1-2-3, dBase, MS-Access, and Claude Code. It is where that anger goes these days.
The technique described in the article, seems to use this key-value pair to store pointers to the additional metadata (in this case a distinct index) embedded in the file. Note that we can embed arbitrary binary data in the Parquet file between each data page. This is perfectly valid since all Parquet readers rely on the exact offsets to the data pages specified in the footer.
This means that DataFusion does not need to specify how the metadata is interpreted. It is already well specified as part of the Parquet file format itself. DataFusion is an independent project -- it is a query execution engine for OLAP / columnar data, which can take in SQL statements, build query plan, optimize them, and execute. It is an embeddable runtime with numerous ways to extend it by the host program. Parquet is a file format supported by DataFusion because it is one of the most popular ways of storing data in a columnar way in object storages like S3.
Note that the readers of Parquet need to be aware of any metadata to exploit it. But if not, nothing changes - as long as we're embedding only supplementary information like indices or bloom filters, a reader can still continue working with the columnar data in Parquet as it used to; it is just that it won't be able to take advantage of the additional metadata.
I think this post is a response to some new file format initiatives, based on the criticism that the Parquet file format is showing its age.
One of the arguments is that there is no standardized way to extend Parquet with new kinds of metadata (like statistical summaries, HyperLogLog etc.)
This post was written by the DataFusion folks, who have shown a clever way to do this without breaking backward compatibility with existing readers.
They have inserted arbitrary data between footer and data pages, which other readers will ignore. But query engines like DataFusion can exploit it. They embed a new index to the .parquet file, and use that to improve query performance.
In this specific instance, they add an index with all the distinct values of a column. Then they extend the DataFusion query engine to exploit that so that queries like `WHERE nation = 'Singapore'` can use that index to figure out whether the value exists in that .parquet file without having to scan the data pages (which is already optimized because there is a min-max filter to avoid scanning the entire dataset).
Also in general this is a really good deep dive into columnar data storage.
Well put. It should always be "Created by <person>" rather than "Processed by LLM". We can already see it with Claude Code - its commit messages contain a "Generated by Claude Code" line, and it guarantees a pandemic of diffused responsibility in software engineering. But I think there is no point in railing against it - market forces, corporate incentives, and tragedy of the commons all together make it an inevitability.
I'm sorry I don't follow. The fact that you use an LLM to classify a transaction does not mean there is no audit trail for the fact. There should also be a manual verifier who's ultimately responsible for the entries, so that we do not abdicate responsibility to black boxes.
Accounting, specifically book-keeping, really plays to the strengths of LLMs - pattern matching within a bounded context.
The primary task in book-keeping is to classify transactions (from expense vouchers, bank transactions, sales and purchase invoices and so on) and slot them into the Chart of Accounts of the business.
LLMs can already do this well without any domain/business specific context. For example - a fuel entry is so obvious that they can match it into a similar sounding account in the CoA.
And for others where human discretion is required, we can add a line of instruction in the prompt, and that classification is permanently encoded. A large chunk of these kind of entries are repetitive in nature, and so each such custom instruction is a long-term automation.
You might have not been speaking about simple book-keeping. If so, I'm curious to learn.
I can think of two instances, where the LLM embracing best practices for human thought leads to better results.
Claude Code breaks down large implementations to simpler TODOs, and produces far better code than single-shot prompts. There is something about problem decomposition that works well no matter whether it is in mathematics, LLMs, or software engineers.
The decomposition also shows a split between planning and execution. Doing them separately somehow provides the LLM more cognitive space to think.
Another example is CHASE-SQL. This is one of the top approaches in Text-to-SQL benchmark in bird-bench. They take a human textual data requirement, and instead of directly asking the LLM to generate a SQL query, they run it through multiple passes: generating portions of the requirement as pseudo-SQL fragments using independent LLM calls, combining them, then using a separate ranking agent to find the best one. Additional agents like a fixer to fix invalid SQL are also used.
What could've been done with a single direct LLM query is instead broken down into multiple stages. What was implicit (find the best query) is made explicit. And from how well it performs, it is clear that articulating fuzzy thoughts and requirements into explicit smaller clearer steps works as well for LLMs as it does for humans.
With negative/positive, the invariant would be sum(amount) = 0; with your approach, it would be sum(debit-credit)=0. Both are valid, it is just two ways of expressing the same thing.
I think it is useful to think about double-entry book-keeping in two layers. One is the base primitive of the journal - where each transaction has a set of debits and credits to different accounts, which all total to 0.
Then above that there is the chart of accounts, and how real-world transactions are modelled. For an engineer, to build the base primitive, we only need a simple schema for accounts and transactions. You can use either amount (+/-ve), or debit/credit for each line item.
Then if you're building the application layer which creates entries, like your top-up example, then you also need to know how to _structure_ those entries. If you have a transfer between two customer accounts, then you debit the one who's receiving the money (because assets are marked on the debit side) and credit the other (because liabilities are on the credit side). If you receive payment, then cash is debited (due to assets), and the income account is credited (because income balances are on the credit side).
However, all of this has nothing to do with how we structure the fundamental primitive of the journalling system. It is just a list of accounts, and then a list of transactions, where each transaction has a set of accounts that get either debited/credit, with the sum of the entire transaction coming to 0. That's it -- that constraint is all there is to double-entry book-keeping from a schema point.
> Instead of using negative numbers, Accounts have normal balance: normal credit balance literally means that they are normal when its associated entries with type credit have a total amount that outweighs its associated entries with type debit. The reverse is true for normal debit balance.
But that is an interpretation made by the viewer. A customer typically is an asset account, whose balances are in the debit column. But if we somehow owe them money because let's say they paid us an advance, then their balance should be in the credit column. The accounting system need not bother with what the "right" place for each account is.
It is quite practical to have only a simple amount column rather than separate debit/credit columns in a database for journal entries. As long as we follow a consistent pattern in mapping user input (debit = positive, credit = negative) into the underlying tables, and the same when rendering accounting statements back, it would remain consistent and correct.
That part of the article felt quite wrong to me as well. I've built accounting systems that worked well for a decade, where internally the values were a single amount column in the journal table. If it was a debit, it'd be positive, if a credit, it'd be negative.
In fact, we could call these values yin and yang, for all it mattered.
Also, I'm not able to really follow what he means by "money = assets in the future".
Money is money, but if you wanted to track the intermediate state until the customer gets receipt, you would use an In Transit account (Good In Transit / Service In Transit etc.)
Yet, it doesn't change the fundamental definition of the value in the accounting system. I think the author confuses an engineering concept (sagas, or thunks, or delayed but introspectable/cancellable actions in general) with accounting.
Thanks, that makes sense. I avoid using LLMs for math because it is only a text token prediction system (but a magical one at that), and can't do true numeric computation. But making it write code to compute works well.
I'm curious to hear more about this. I've seen very little hallucination with mainstream LLMs where the conversation revolves around concepts that were well-represented in the training data. Most educational topics thus have been quite solid. Even asking for novel analogies between distant and unrelated topics seem to work well.
https://twitter.com/jasim_ab
[email protected]