Hey, I'm Mark Nadal and I was accepted into Mozilla's "Fix the Internet" program - it has been fantastic so far & huge respect to everyone at Mozilla for putting this on.
Note: I doubt anyone will see this comment as HackerNews has shadow banned my account because my Open Source project competes with some of YC's investments (it undermines & destroys the need/market for some of their crypto-coin scams).
I interviewed with YC a few years ago and their entire process was terribly unprofessional - I've gone through 3 accelerator programs now, and interviewed at many more. YC itself seems pretty awesome, but beware of the politics - if a single person (HN mod in my case) feels like you are a threat to their status, that person has enough marketing power to hurt you. Find the GOOD people in YC and work with them, don't leave it to chance or you'll fall out of good graces.
In contrast, look what Mozilla is doing! They're helping move the internet (the world, the community) forward, and are a non-profit foundation. I highly recommend applying to their program. They are GOOD people, doing good work. Please please do everything you can to help them make this a success - the internet depends on it.
For anybody that is interested in an Open Source example of the "mesh" that is E2EE that the article mentions (tho not scalable, without switching to SFU & doing what they say), you can check out this repo ( https://github.com/meething/webrtc-gun ) that our community built in a couple of days.
https://mozilla.org/builders is an 8 week paid hackathon Mozilla is sponsoring, and teams are forming there, thinking about building a better Open Source E2EE multi-party Zoom alternative.
I think that is a good hill to die on, tho I would rather prioritize UX (browser not freezing) and server responsiveness. Ideally we'd have no CPU chunking & good UX, but if we have to choose one, which would you sacrifice?
Meanwhile I can barely get Chrome/NodeJS to parse 20MB in less than 100ms :(.
How useful (or useless) would Simdjson as a Native Addon to V8 be? I assume transferring the object into JS land would kill all the speed gains?
I wrote my own JSON parser just last week, to see if I could improve the NodeJS situation. Discovered some really interesting factoids:
(A) JSON parse is CPU-blocking, so if you get a large object, your server cannot handle any other web request until it finishes parsing, this sucks.
(B) At first I fixed this by using setImmediate/shim, but discovered to annoying issues:
(1) Scheduling too many setImmediates will cause the event loop to block at the "check" cycle, you actually have to load balance across turns in the event loop like so (https://twitter.com/marknadal/status/1242476619752591360)
(2) Doing the above will cause your code to be way slow, so a trick instead, is to actually skip setImmediate and invoke your code 3333 (some divider of NodeJS's ~11K stack depth limit) times or for 1ms before doing a real setImmediate.
(D) I'm seeing this pure JS parser be ~2.5X slower than native for big complex JSON objects (20MB).
(E) Interestingly enough, I'm seeing 10X~20X faster than native, for parsing JSON records that have large values (ex, embedded image, etc.).
(F) Why? This happened when I switched my parser to skip per-byte checks when encountering `"` to next indexOf. So it would seem V8's built in JSON parser is still checking every character for a token which slows it down?
(G) I hate switch statements, but woah, I got a minor but noticeable speed boost going from if/else token checks to a switch statement.
Happy to answer any other Qs!
But compared to OP's 2.5GB/s parsing?! Ha, mine is a joke.
I get some hate my Open Source project & actively abuse downvotes/flagging to censor me.
But for every 1 hater there are 100s of hackers that have found, starred, used, or told me (in our chat channel) they were thankful they discovered GUN via HackerNews.
& HN guidelines encourage on-topic submissions & comments (https://news.ycombinator.com/newsguidelines.html) "Anything that good hackers would find interesting ... anything that gratifies one's intellectual curiosity" even you have to admit that GUN discussions have sparked a lot of intellectual & algorithm chats.
Finally, you state I'm not spamming then say I am spamming. Spam is indiscriminate posting of something, yet your very own comment says "you always post GUN in discussions about CRDTs & decentralization." That is indeed on-topic according to HN policy.
Hey, my Open Source database is the #1 CRDT rated system on GitHub, and is used in-production by 10M+ people a month, at non-profits like the Internet Archive, and others.
If you ever have time, it'd be great to snag whatever code or data you had, to create a replicable test case for us to make a fix against. Sorry about the wasted time you experienced, I hate that.