FWIW, this is addressed in the article and the paper it covers: the slowdown involved reduced "proactive policing" -- "systematic and aggressive enforcement of low-level violations" -- but the observed drop is in major crimes like burglary and assault. The authors also do some work to control for the overall crime reporting rate.
Worth mentioning that, while the `git bisect` algorithm is pretty easy to run by hand if you have a linear history, `bisect` also handles very complex branching and merging histories as well. Very clever and worth using!
The issue being pointed out here isn't that the computation is out of sync with the outside world... it's that it's our of sync with _itself_! It will return answers that are not just stale, but inaccurate for any time in history.
This might still be fine, depending on your needs, but IMO a legitimate distinction.
Based on that description of Pijul, it does seem to be a semilattice: it has a merge operation that's associative and commutative, and while it doesn't explicitly state that it's idempotent it's hard to imagine merging two states and getting anything else back than the same state.
Of course, there really is something special about 0, 1, and Inf -- they're the kind of numbers that tend to arise from theory, not just from tuning. Tuning is hugely important for real systems, so an allergy to 'arbitrary' numbers is not useful; on the other hand, this rule is often a nice heuristic to keep in mind when you're picking which numbers to expose in your config files.
It's worth noting that green threads are not 'enough' -- in any language with concurrency, you still need to be explicit about what work can run concurrently and what needs to be sequential. In Go, for example, the default is sequential operation; if you want to run a bunch of code concurrently, you need to do extra work to launch a bunch of new goroutines / wait for everyone to finish. In Haskell, which also (typically) has green threads, there's a popular `async` package that supplies a promise-like construct -- it's quite useful when you want to be explicit about concurrency and ordering.
It's an odd definition of personal privilege you have, where the privileged person suffers two years of harassment and abuse, and the person without privilege immediately returns to a lucrative career.
It's also an odd definition of professional victim, where publicly being a 'victim' results in you getting booted out of the profession.
I don't want to comment on the event itself here -- but I think that the aftermath, and the continued reaction to it, say a lot of sad things about the tech community.
- Any monoid operation is trivially parallelizable: take the dataset, split it into chunks, combine all the elements in each chunk together, then combine those results together as a final step.
- If I'm updating a row in my database with a monoid operation, I can always 'pre-aggregate' a batch of values together on the client side before taking that result and combining it with the stored value.
- If I store some statistics for every day of the year, I can calculate the monthly statistics very cheaply -- as long as those stats are a monoid.
The monoid abstraction seems weird at first, because it's so dang general, but it ends up hitting a bit of a sweet spot: the rules are just strong enough to be useful for a bunch of things, but simple enough that be applied to all kinds of different situations. You can think of it kind of like a hub-and-spokes thing -- this interface connects all kinds of different data types to all kinds of different cool situations, so you get a lot of functionality with a lot less typing and thinking.
- Some arbitrary type of thing. (Lists, sets, strings, numbers, whatever.) Let's call it 'A'.
- A value of type A. Let's call it the 'empty' value.
- A function that takes two A values as arguments, and returns a single one. Let's call it 'combine' function.
Here are some examples of monoids:
- String concatenation: 'combine' is string concatenation, and 'empty' is the empty string.
- Integer plus: 'combine' is the usual + operation, and 'empty' is the value 0.
- Set union: 'combine' is the union of two sets, and 'empty' is the empty set.
You can come up with monoids for all sorts of things: maps, lists, functions, bloom filters, futures / promises, ...
There are, however, a couple rules:
- It doesn't matter which order you combine things in: `("hello" + "world") + "!"` has the same result as `"hello " + ("world" + "!")`; and `1 + (2 + 3)` has the same result as `(1 + 2) + 3`.
- You can add the 'empty' value to either end without changing anything. `"" + "hello"` is the same as `"hello" + ""` and plain old `"hello"`. Likewise, `3 + 0 == 0 + 3 == 3`.
That's it! To get an intuition, you might want to pick a couple more types from the list above and pick a good 'empty' / 'combine' function that satisfies the rules.
Of course, why you'd want to do this is another question. (And one I'd be happy to answer, if you're curious.)
-- little-endian sequence of bytes to arbitrary integral type
integerWithBytes :: (Bits a, Integral a) => [Word8] -> a
integerWithBytes = foldr (\byte acc -> (acc `shiftL` 8) + fromIntegral byte) 0
At least, I'm pretty sure that's what the article was trying to do...
I don't think this is some amazing showcase of Haskell; after all, the article includes a (working?) implementation in C++. But it does serve as a nice counterexample to the idea -- expressed elsewhere in the thread -- that being generic is necessarily a hugely painful or complicated thing.
The main similarity is that 'methods' are defined 'outside' the data type they belong to. But there are a huge number of differences: Haskell's typeclasses are used for many things that don't look like methods at all, while Go's behave more like a classic OOP interface you'd find in Java.
So, perhaps superficially similar-looking, but they work quite differently in practice.
> Demanding this mind of self censorship is a sign that a society is fragile rather than robust. A robust society can take the jokes.
Sure, but you don't magically make society robust by telling racist jokes. The healthy society comes first; the fact that these jokes no longer sting is just a pleasant side effect.
In Canada, as I understand it, it's illegal to refer to yourself as an engineer / a position as an engineering position unless you have Real Engineering Credentials. So a lot of the touchiness is just the usual sensitivity you find around a legal issue.
Yes, Kafka certainly is good at those things, and I suspect Kafka will only get better at them. But it's actually quite a simple bit of infrastructure at heart[0] and this means it's useful for a large variety of other things, many of which we're only figuring out now.
Re: optimistic locking... there are a couple proposals for this floating around, and I'm not sure which / if one will get in. It certainly seems consistent with the general mission, though.
I enjoyed and totally endorse the content of your blog post, but I did find the title / intro a bit misleading. Exactly-once is pretty much always what people want: it's the easiest model for most folks to reason about, especially when they're new to the whole distributed-systems thing. (If exactly-once delivery was easy, you certainly wouldn't find people going out of their way to build systems that dropped or duplicated messages.) Of course, you might not need exactly-once delivery to get what you want -- which is good, because like you say, that's a hard / impossible problem to solve in general.
It feels to me like people in the stream processing universe have taken that message a bit too much to heart -- things like the 'lambda architecture' treat stream processing as "too hard to get right" and relegate it to approximate, disposable calculations only. My post was partly an effort to push back on this; as a community, I don't think we should give up without a fight.
The Samza ecosystem seems to be a magnet for this sort of thing these days -- the community's really committed to taking full advantage of Kafka, and not just papering over it so it looks just like any other queuing system. Really excited to see where all this will lead.
One additional note: this is a survey article, not based on a single study. This seems to be the case of NPR staff using a questionable reference, not just rewriting a press release.