While this is clearly xenophobic and just plain awful, it seems like the most sensible workaround for this is for universities to offer very small (4 students or less per session) in-person classes specifically for their new international students. This could work essentially like office hours with faculty leading small group discussions to help acclimate international students to their new lives abroad. I can imagine anyone moving to the US in the year 2020 (the peak of our crazy) could use some emotional support.
I'm not quite certain I understand that last part. To the best of my understanding, Kafka's design is such that consumer ingestion is completely decoupled from production throughput. All messages from the producer are first copied to disk and then zero-copied from disk to network to the consumers at the byte array level. If a consumer falls behind, it has its own independent offset stored on the broker that keeps track of where in the byte array (or log) it left off. This, by design, allows Kafka to handle different profiles of consumers and even have consumers drop off entirely and later join to catch up. But perhaps I'm missing something about what you're saying.
That works for some use cases, but for others (Elasticsearch, Zookeeper, Kafka, etc) the service inside the container needs to bind to an interface associated with an IP that's also addressable by the host. Even in host networking mode, eth0 inside a DFM-powered container will bound something like 192.168.x.y but that 192.168.x.0 subnet is completely inaccessible from the host.
us-east-1 is where they typically deploy new features/hardware first (with the exception of efs which went to us-west first for some reason). it's also by far the largest region, with the most tenants and the heaviest traffic, so it's approaching the limit on what's physically possible to do in a public data center.
that's completely untrue. there are many ways to do fault-tolerance in AWS. it's expensive, but it's possible. netflix even goes as far as simulating the failure of entire aws regions in their simian army testing suite:
So one of the challenges in a containerized environment is that services start up on random ports. If I'm running a Postgres container with the default Docker networking mode, for example, the internal port of 5432 may be bound to the host port of 12345. This allows me to spin up multiple instances of Postgres on the same machine for greater service density.
However, in a distributed environment, services can spin up on different machines. The instance of Postgres my application needs could be on server1:12345 or server2:23456. But in a distributed system, you need a cross-cutting service that's available to all servers so that if my app is running on server1, it can find the right Postgres instance running on server2.
I'm not an expert on etcd, but my understanding is that the most common use case is to run etcd on each host machine. When services start up, their supervisor registers the service's hostname, port, etc with etcd's key-value store. This registration is then propagated to other etcd nodes in a consistent manner, using a consensus algorithm called Raft:
Consensus actually turns out to be one of the harder problems in a distributed system design. If I have a network partition that prevents etcd instances from seeing each other, you don't want one instance reporting incorrect or stale data. Otherwise, my application could be writing to the wrong service, causing data loss.
Etcd does consensus extremely well, and in a way that scales to support hundreds of nodes. It's one of the two distributed systems I'm aware of that have (mostly) passed Jepsen testing:
There are also alternatives like Consul and Zookeeper, but in the case of Zookeeper you have to do a lot of the heavy lifting yourself to support service discovery. There are also some well-documented caveats:
This seems quite cool. For very large clusters, I'd also consider looking into Optaplanner, which exposes a variety of probabilistic metaheuristics for balancing. There's a "cloud balancing" example in the documentation which is fairly close in terms of the use case: https://docs.jboss.org/drools/release/6.0.0.Beta1/optaplanne...
Spantree Technology Group is looking for a new member to join our existing US-based team members in Grand Rapids or Chicago. We'd also be interested in a remote team member (preferably based in the Midwest or East Coast US) depending on expertise.
---
Hello, we’re Spantree, a rapidly growing boutique software engineering consultancy headquartered in West Loop of Chicago. We’re looking for a brilliant and personable engineer to join our team. Our clients range from small and scrappy startups to large banks and insurance companies. While we have a pretty broad technical focus, a key element to most of the stuff we do is building tools to help people make better decisions.
Our technical role on projects varies a bit depending on the customer, so we’re hiring organically for generalists that can be flexible up and down the stack. Though if you have deep expertise in some of the technologies listed above (esp Elasticsearch or Docker/Mesos), we can teach you the rest on an as-needed basis.
We offer competitive compensation, relatively flexible work schedules, health/vision/dental, 401K matching and the usual tech company perks (a copious amount of catered lunches in our Chicago offices, etc).
Whether or not this is the same person or an imposter? Who knows. But it wouldn't be surprising.
If true, it's entirely possible that Airbnb is more concerned about setting a precedent for any and all auxiliary damages that go along with a bad stay. What happens when someone gets kicked out of their apartment because a guest accidentally tipped off management that they were an Airbnb renter?
Ge0ffrey, any plans to release the source code on this optimization? It'd be great to have a writeup on how to actually run the tsp tests for this. It's sort of buried in the test suite at the moment.
Optaplanner is an incredibly useful, often underrated framework. Keep up the good work!
This is a great way to handle offline search, but as the article mentions, indexing is still a pretty heavy operation to perform on the client-side. I'm curious to see if it'd make sense to invert the index server-side, and pass the pre-inverted index along with the documents via REST when an app loads. Then, you could still perform offline queries, but the client would only be responsible tokenizing/analyzing the search terms, comparing them against the index and doing the BM25 scoring piece.
Looks pretty awesome, esp the clean DSL for your page model, but it seems like most of the documentation might be missing. How sophisticated is the crawler portion? Does it support Nutch-style generators that crawl more frequently updated pages more frequently? Or is it more designed for focused, one-off crawls a la Scrapy?
