One of their line items complains about being unable to bind 65k PostgreSQL placeholders (the linked post calls them "parameters") in a single query. This is a cursed idea to begin with, so I can't fully blame PostgreSQL.
From the linked GitHub issue comments, it looks like they adopted the sensible approach of refactoring their ORM so that it splits the big query into several smaller queries. Anecdotally, I've found 3,000 to 5,000 rows per write query to be a good ratio.
Someone else suggested first loading the data into a temp table and then joining against that, which would have further improved performance, especially if they wrote it as a COPY … FROM. But the idea was scrapped (also sensibly) for requiring too many app code changes.
Overall, this was quite an illuminating tome of cursed knowledge, all good warnings to have. Nicely done!
During the period for which we have evidence, Neanderthals seem to have had smaller populations than Homo sapiens, and they reproduced more slowly (longer gestation plus longer interval between pregnancies). So it may be that they went extinct for those reasons.
He was critical of blockchain, because he considered a permanent register of transactions to be the antithesis of privacy. How much store you want to set by that is up to you, but my instinct says that he was sincere.
This made me think of Dirk Gently's Holistic Detective Agency, Douglas Adams's 1987 novel, where there's an AI company that takes military and political decisions and comes up with post hoc rationalizations for them.
There's an excellent SF novella, Walter Jon Williams' The Green Leopard Plague, where Transnistrian corruption happens to be a major plot device. The other characters derisively call the Transnistrian government "Trashcanistanis." This story was my first encounter with Transnistria, and it seemed weird enough to be fictional, but later I found out it was real.
Tiny correction: I heard it as "all languages are really just dialects of Perl 6," which makes a bit more sense in context. Either way it's a cute Wall witticism, of which there were many that night. Word has it that there will be a video in a couple of weeks, at which point you can judge for yourself!
From the linked GitHub issue comments, it looks like they adopted the sensible approach of refactoring their ORM so that it splits the big query into several smaller queries. Anecdotally, I've found 3,000 to 5,000 rows per write query to be a good ratio.
Someone else suggested first loading the data into a temp table and then joining against that, which would have further improved performance, especially if they wrote it as a COPY … FROM. But the idea was scrapped (also sensibly) for requiring too many app code changes.
Overall, this was quite an illuminating tome of cursed knowledge, all good warnings to have. Nicely done!