To summarize something from a little over a year after I joined there: Cloudflare was building out a way to ship logs from its edge to a central point for customer analytics and serving logs to enterprise customers. As I understood it, the primary engineer who built all of that out, Albert Strasheim, benchmarked the most likely serialization options available and found Cap'n Proto to be appreciably faster than protobuf. It had a great C++ implementation (which we could use from nginx, IIRC with some lua involved) and while the Go implementation, which we used on the consuming side, had its warts, folks were able to fix the key parts that needed attention.
Anyway. Cloudflare's always been pretty cost efficient machine wise, so it was a natural choice given the performance needs we had. In my time in the data team there, Cap'n Proto was always pretty easy to work with, and sharing proto definitions from a central schema repo worked pretty well, too. Thanks for your work, Kenton!
I also think his thesis isn't to celebrate poverty but to consider that development (or modernity, globalization, pick a term) comes with its own problems.
(Yes to the earlier post - had a similar experience with an early metrics system for Twilio in 2012-2013)
The 1st issue is that ZMQ implementations generally have an IO thread handling all socket IO. REQ/REP patterns tend to work out just fine, because the client is going to wait for a reply from the server. But things like PUSH/PULL become...harder to follow. You can push things in the client, but how do you know when they've flushed out of the IO thread and made it to the server? Those sorts of things matter when shutting down a process, running integration tests, etc. In contrast, with an HTTP request, you're basically always doing REQ/REP, and so you know when the data has been pushed.
The 2nd issue is that ZMQ implementations tend to be harder to observe / operate than a more common path like HTTP requests. (HTTP requests can go through a load balancer, have standard response codes, use headers for authorization, etc.) For these reasons, I've tended to avoid ZMQ ever after -- it seems like often, you're best off either using REST or GRPC if by HTTP, or raw TCP if it's purely a data push kind of operation (e.g. forwarding structured logs, with framing, to a remote TLS endpoint).
You don't have to agree with her politics, but I think Peggy Noonan had the right answer on this one: it's a show and there's no good to be found in taking part.
Just say no and hit send.
> In February 2018 Nicholas Thompson and Fred Vogelstein of Wired wrote a deeply reported piece that mentioned the 2016 meeting. It was called so that the company could “make a show of apologizing for its sins.” A Facebook employee who helped plan it said part of its goal—they are clever at Facebook and knew their mark!—was to get the conservatives fighting with each other. “They made sure to have libertarians who wouldn’t want to regulate the platform and partisans who would.” Another goal was to leave attendees “bored to death” by a technical presentation after Mr. Zuckerberg spoke.
I've not spent enough time on Pearl, but Migual Hernan and Jamie Robins have a free textbook on causal inference at https://www.hsph.harvard.edu/miguel-hernan/causal-inference-... ; parts of it were pretty useful for a causal inference assignment I did earlier this year.
It'd be hard to understand without feeling it for yourself, but in actual instrument conditions (IMC), it's easy for your sense of balance to mislead you into thinking you're level, when in fact you're turning.
There's some physiology to explain this, but one key point is that once you're in a coordinated turn, even a descending one, your weight is still going straight down the vertical axis of the seat. So, you'll think you're doing fine, and possibly even pull the turn tighter (the graveyard spiral), and fixate on things other than the attitude indicator and your plummeting altitude.
Michel Foucault also wrote about this essay. His essay starts thus:
> Today when a periodical asks its readers a question, it does so in order to collect opinions on some subject about which everyone has an opinion already; there is not much likelihood of learning anything new. In the eighteenth century, editors preferred to question the public on problems that did not yet have solutions. I don't know whether or not that practice was more effective; it was unquestionably more entertaining.
> In any event, in line with this custom, in November 1784 a German periodical, Berlinische Monatschrift published a response to the question: Was ist Aufklärung? And the respondent was Kant.
* Hour restrictions for passenger transport are significantly more stringent than for freight.
* Southern Air's fleet consists 737s and 777s -- not small planes.
* These plans have to fly over and into major metropolitan areas.
* Many parts of flying can be automated, but in the end, and for aircraft this size, you probably want a human operator in the plane for the sheer reason that they are better equipped to handle unplanned emergencies.
It's a tough call whether flying freight or regional airlines pays worse -- but in either case, you're anywhere from a quarter to half of what a software engineer of similar years experience in the bay area would make. The hours are in a lot of ways worse -- mostly because you're away from home a lot of the time, and you have really limited control of your schedule. (As someone who's worked as an on-call engineer for ten years, and who also flies, I'd take the on-call responsibilities over flying freight any day of the week.)
I'd agree that the article lacks for details. But, a relatively quick reading through things like airliners.net will give you an idea of the kind of hours and pay these guys typically work -- it's not an easy job. If we care about their safety -- and our own, given the number of freight aircraft flying over us every night -- we shouldn't write them off as just asking for money they don't deserve.
If you haven't read it, you may find Hannah Arendt's Origins of Totalitarianism a worthwhile read. She offers a lot of analysis on the rise of a classless society, paired with a deep sort of aimlessness in state leadership.
Correct. CTEs are pretty much a no-go, and you also can't do things like join between distributed and non-distributed tables.
Your queries tend to break down into ones where you're hitting a small number of shards (such as when we serve data for the analytics page), or else ones where you tend to aggregate up into temporary, non-distributed tables for further analysis (such as you'd do for infrequent business reporting).
All told, it's been one of the easier tools for us to operationalize, largely thanks to the fact that it's "just PostgreSQL." The one thing I wish we had was better documentation of what kind of consistency guarantees to expect, although for an append-only store like our current use case, that's less of a concern. And to be fair, Ozgun and the other guys at Citus have always been really happy to answer any questions we have.
With this code now going open source, it should be pretty easy to look into these sorts of internals.
The best part of using Helix was that the people doing data entry, and the people who worked closely with them, really could and did update the forms themselves.
I like paper for a lot of things, but in software, I've always hoped we can arrive at systems where users get more of a hand in shaping the tools they use. We're not there yet.
Anyway. Cloudflare's always been pretty cost efficient machine wise, so it was a natural choice given the performance needs we had. In my time in the data team there, Cap'n Proto was always pretty easy to work with, and sharing proto definitions from a central schema repo worked pretty well, too. Thanks for your work, Kenton!