German-style strings is a way to store array of strings for columnar dbs. The idea is to have an array of metadata. Metadata has a fixed size (16 bytes) The metadata includes the string length and either a pair of pointer + string prefix or the full string for short strings. For some operations the string prefix is enough in many cases avoiding the indirection.
> Your coworkers and QA will thank you for learning LINQ and ditching the imperative methods that plague your Python brain.
This is a very unfortunate joke: Python has list (and generator) comprehension expression for a long time (2.3?) which are similar to LINQ. At some point in the history many languages stole useful expressions from other paradigms.
> As luck would have it, the Arrow spec was also finally making progress with adding the long anticipated German Style string types to the specification. Which, spoiler alert, is the type we implemented.
Congratulations to the team, Pydantic is an amazing library.
If you find JSON serialization/deserialization a bottleneck, another interesting library (with much less features) for Python is msgspec: https://github.com/jcrist/msgspec
I don't know what were the issues Yelp was facing, I have upgraded several times my past Kafka clusters and really never experienced any issues. Normally the upgrade instructions are documented (e.g. https://kafka.apache.org/31/documentation.html#upgrade) and the regular rolling upgrade comes with no downtimes.
Besides this, operating Kafka never required much effort a part when we needed to re-balance partitions across brokers. Earlier versions of Kafka required to handle it with some external tools to avoid network congestions, but I think this is part of the past now.
On the other hand, Kafka still needs to be used carefully, especially you need to plan topics/partitions/replications/retention but that really depends by the application needs.
One thing that is underestimated is keep the tools version in sync between your app dev dependencies and pre-commit. This also includes plugins for specific tools (for instance flake8). A solution would be to define the hooks in pre-commit to run the tools inside your venv.
About typings: I agree the eco-system is not mature enough, especially for some frameworks such as Django, but the effort is still valuable and in many cases the static analysis provided by mypy is more useful than not using it at all. So I would suggest to try do your best to make it work.
We are currently using Kafka Manager (https://github.com/yahoo/kafka-manager). It seems Kafka HQ have the same features (it is not clear from the doc if you can actually manage the partition arrangement or only view it) plus the possibility to view the actual messages that are streaming in Kafka that might be very useful.
Much of the effort for the next release of Druid (0.9) has been put into docs. Even with that you still need to handle a complex cluster made of several services: zookeeper, druid overlord, druid middlemanager, druid broker, etc... still it sounds more complicate than it actually is. What was the alternative solution adopted by your company?
Many people here are comparing this feature of Postgres with MongoDB or other document-oriented NoSQL dbs. I think the comparison is just wrong and unfair.
Even if I'm amazed by the performance of Postgres for this particular task (considering also that is a relatively new feature), I don't think performance is the reason why people are using NoSQL dbs. The problem that NoSQL dbs are helping with is scaling. I don't see this as a priority for an RDBMS such as Postgres. Take for instance MongoDB (just because it was named by many of the other comments here, but I guess the same apply to Couch or others): it's relatively simple to deploy a cluster with automatic sharding, replica, failover, etc.. because these are all builtin features.
A framework is not only a good way to start. It's also a good way to uniform code, make things easier for new comers (larger projects usually have lots of new people that eventually write code), solve multiple common issues you'd fight by your own (e.g. browsers compatibility), community support and documentation. If you write your own you need to document well all things and things very careful how to arrange css classes to make code reusable and flexible.
I had some "memory leak" issues a few months ago with the api hosts of the service I'm working on. As the code changes made during the days before didn't seem to have affected anything that could be related to this leak I realized it was something about the requests we were receiving but I wasn't able to replicate the same issues in my dev host nor in staging.
Eventually I dumped the memory of a running process using Pyrasite and after hacking a little bit with meliae (https://launchpad.net/meliae) I was able to find the issue and solve it.
Very useful.
By the way is worth noticing that you won't probably be able to attach the interactive console to a running web server as the output is usually handled by the supervisor process, at least I wasn't able to do that in my first tentative and the memory dump was good enough for me.
If you cut and paste probably the misaligned indentation is the least problem. Cut and paste should be avoided if it's not done during a refactoring (so actual moving of code for better design).
Beginners usually think Python strong indentation is a weakness of the language. I actually find C++ freedom being more error prone:
if (a < b);
a = b
It's a not very frequent bug, but when it happens it takes you hours to spot. ;)