frant-hartm · HackerTrans

frant-hartm

no profile record

Submissions

Processing 10M queries / second on a single node using Hazelcast Jet and gRPC

jet-start.sh

2 pointsby frant-hartm6 năm trước0 comments

comments

năm ngoáidiscuss

I wonder what people use/need apart from the usual stuff in Maven nowadays.

In the last 5 years, I didn't need anything more than:

- resource plugin - compiler plugin - jar plugin (jars, test jars, javadoc jars) - surefire/failsafe - shade plugin for repackaging to avoid classpath hell - assemble plugin - license plugin

Most of the issues I had were with the shade/assemble/license plugins.

I consider myself a Maven power-user and like the tool compared to others (Gradle is too ant-like, resulting in non-standard builds, sbt is just a torture tool).

With time, I concluded that the simpler it is, the better.

2 năm trướcdiscuss

Just a note: I am not claiming it is working correctly, only saying there is a clear and documented way how the client knows when to commit, and that it works as expected in a simple scenario.

2 năm trướcdiscuss

> how does Kafka know when you've processed the messages it gave you?

By calling `poll()` again. It doesn't commit the records returned from poll until auto commit interval expires AND you call poll again.

At least this is what the javadoc says quite clearly: https://kafka.apache.org/39/javadoc/org/apache/kafka/clients...

Note: Using automatic offset commits can also give you "at-least-once" delivery, but the requirement is that you must consume all data returned from each call to poll(Duration) before any subsequent calls, or before closing the consumer.

E.g. the following commits every 10s - on each call to `poll`, it doesn't automagically commit every 5 s.

        Properties props = new Properties();
        props.setProperty("bootstrap.servers", "localhost:9092");
        props.setProperty("group.id", "test");
        props.setProperty("enable.auto.commit", "true");
        props.setProperty("auto.commit.interval.ms", "5000");
        props.setProperty("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
        props.setProperty("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
        KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
        consumer.subscribe(Arrays.asList("my-topic"));
        while (true) {
            ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
            for (ConsumerRecord<String, String> record : records) {
                System.out.printf("offset = %d, key = %s, value = %s%n", record.offset(), record.key(), record.value());
            }
            Thread.sleep(10_000);
        }

2 năm trướcdiscuss

Depending on where you are going in London, flying to Stansted / Gatwick might be faster than to Heathrow or City.

2 năm trướcdiscuss

You probably don't have many roundabouts where you live, right?

2 năm trướcdiscuss

That's why I hate sales people. Give me all the info on your product, ideally, make it easy to compare to others and I will decide myself.

The idea that one can ask me a few questions and give good advice when buying a phone, a car, a house etc.. is just bizarre.

Maybe it is not like that in the general population, but it certainly is within technically-minded people.

2 năm trướcdiscuss

And on top of that this _end-to-end exactly-once guarantees_ makes the result look like the message was processed in the pipeline only once, while in reality a message will be processed 2x (or more times) in the pipeline in case of a temporary failure but only one version will be committed to the external system.

2 năm trướcdiscuss

> Engineers look down on advertising and advertising people

I don't think this is entirely true. We certainly look down on marketing fluff and playing on emotions, but I certainly appreciate well-made marketing material full of facts and specs.

2 năm trướcdiscuss

Why would you do it for every tab switch? You do it once to fit your preferred width and all websites adjust. You would probably have a shortcut for that in your window manager.

There was a great time at some point in the past where everyone went all in on responsive design and most websites behaved well, what we have nowadays is a regression in usability (at least for power users).

2 năm trướcdiscuss

While I agree with the research, let me be the judge of what works (for me..), span the text across the full width, so I can resize the window as I see fit.

It's a shame that one needs to use scrips to modify websites used daily (e.g. I do that with Github to span the full width).

2 năm trướcdiscuss

I remember that Neo4j Enterprise used to be available under AGPL. They pulled it and now it's available only under a commercial license.

AGPL is not a problem for server-side software if you don't need to modify it. Your application (talking to the server) doesn't become infected by AGPL.

2 năm trướcdiscuss

This is called mounting/un-mounting and is much cheaper than a context switch.

2 năm trướcdiscuss

Ok. This JVM-level switching is called mounting/un-mounting of the virtual thread and is supposed to be several orders of magnitude cheaper compared to normal context switch. You should be fine with millions of virtual threads.

2 năm trướcdiscuss

What do you mean by context switching?

My understanding is that virtual threads mostly eliminate context switching - for N CPUs JVM creates N platform threads and they run virtual threads as needed. There is no real context switching apart from GC and other JVM internal threads.

A platform thread picking another virtual thread to run after its current virtual thread is blocked on IO is not a context switch, that is an expensive OS-level operation.

2 năm trướcdiscuss

Complain to the author of your direct dependencies (or better send a PR if open source). There is absolutely no reason why lombok should get pulled in as transitive dependency apart from ignorance on library author side.

2 năm trướcdiscuss

There are multiple attack vectors via the supply chain that this new settings prevents.

Sure, if the compromised library is one of your core libraries that runs during build, test and runtime, then this does not help. But there are

  test libraries only
  compile time only libraries (hello lombok?)
  transitive dependencies that may not be used during runtime (or run only in rare code paths)

Sometimes compromising the build environment is more valuable than the app's runtime environment - e.g. it may allow the attacker to compromise all apps.

Explicitly enabling a particular annotation processor I want to run is small price to pay for the increased security.

2 năm trướcdiscuss

> legal department works for you, not the other way around.

Not sure what planet you live on. Unless you are one of the execs, Legal (and other compliance departments like HR) work pretty much against you. They exist to protect the company and the exec team.

2 năm trướcdiscuss

You can, but one has to be careful with the transfers. If not you could incur ridiculous costs.

And some architectures are still prohibited - any replicated database across cloud vendors - imagine Cassandra hosted across AWS and Azure to protect against one cloud vendor outage.

2 năm trướcdiscuss

Same here. I am so used to

  /**
   *
   */

that I find this visually off-putting. Almost as much the string interpolation in string templates.

  /***

would be much better.

Also /// will be read as a comment with a line starting with a slash until the tooling is updated.

3 năm trướcdiscuss

> still don't have a legal requirement to blur people in the background

What countries require to "blur people in the background"?