In my opinion it's the opposite. This type of associations is welcome, and they are fine to promote free software and help people, but they are exactly like neighborhood associations: they're mostly local, relying on volunteers with limited time who become a new dependency for people who were not using these services. That's fine for limited use case but it doesn't scale at all, and causes a huge duplication of efforts (organization, software creation -- several of them reinvented stuff that already exists, advertising etc). And associations rarely if ever merge, because most often association creators have a very clear view of what they're seeking (often idealist) and are rarely willing to compromise it and accept to adopt another association's mostly similar but not exactly identical goals (it often works very similarly to political parties). GAFAMs would never exist under this model.
The problem is that such services which proudly run on low budget, volunteers and recycled hardware, cannot be relied on by companies without risking to enter legal trouble in case of major incident, so it means that a higher-grade service is still needed, with a dedicate funding, and we're facing fragmentation. We must not reproduce the scheme of cloudwatt either. Too much money injected into a wet dream that was only used to spend lots of money in consultants coming here just to confirm their presence and get their check.
What is needed instead is to sponsor the development of such activities by a few (2-3) well-established competing companies, so as to avoid the regular risk of monoculture that diverges from what users expect, and help them reach the point where their offerings can compete with GAFAM's for both end users and enterprise. The contract should be clear that services must rely on open formats, make it possible for leaving users to retrieve all their data, that software developed under such funding must be opensource, though technology acquisition is fine, and that these offerings must become self-sustaining at one point (i.e. a mix of free+paid services). The EU funders should have enough shares of these activities so that their permission is required for business acquisition and that they can restrict it to EU-based companies, so that such companies can still grow and seek public funding.
What we need is a few durable big players, not 10000 incompatible associations each with their own software suite, that no enterprise can trust over the long term and that cannot resist a trivial DDoS by lack of a robust infrastructure, and who are not organized enough to run full-stack security audits to make sure that user data are properly protected. These ones are only fine for friends and family but that's not what we're missing the most (the proof is that they already exist).
No, it needs first to encourage local investment. Companies who seek investors or who get sold do not do it by pleasure, but as a last resort before dying. And in the EU you don't get any offer to save a company that has a limited commercial activity. Many companies die every day by lack of money. It turns our that US etc are willing to take much more risk and invest sufficient money to transform such fragile companies into durable ones. And in the case of software it's great, because software is sold all over the world, and the income serves to hire more local people, so in the end it's a way to really develop EU sales to the rest of the world. It would clearly be better if the investors were EU-based, but at least it's better than nothing that some investors are willing to risk their money on such companies.
Blaming the audience makes sense because after all, they're the ones not getting the message right and not asking the presenter to explain it better. But it remains the presenter's failure to catch their attention better and try to deliver a clear message.
Every time I had a presentation, I tried to analyze the failures (including listening to me when it was recorded, a really painful experience). Certain mistakes such as like having slides on a white background that makes attendees look at the screen and read instead of watching the presenter and listening to him can be devastating. Just because attendees are naturally attracted by light. It's not the audience's fault, it's the presenter's fault (and to some extents the tools in use). A good exercise is to stop slides from time to time during the presentation (i.e. switch to a black one), you'll be amazed how much you suddenly catch the attention, you feel like you're at a theater. It even manages to catch attention of those who were looking at their smartphones because the light in the room suddenly changes.
Also another difficulty which is specific to English native speakers is that many of them initially underestimate the difficulties of the audience to catch certain expressions (with some people it's very hard to distinguish "can" from "can't" for example, which complicates the understanding), or idiomatic ones, or references to local culture, because such things are part of their daily vocabulary. Of course, after a few public talk, when they get questions at the end proving there were misunderstandings, they realize that speaking slower, articulating a bit more and avoiding such references does help with non-native listeners. Conversely, when you present in a language that is not yours, you stick to very simple vocabulary using longer sentences to assemble words that try to form a non-ambiguous meaning. It can probably sound boring for native speakers but the message probably reaches the audience better.
In any case, it definitely always is the presenter's failure when a message is poorly delivered and their responsibility to try to improve this, however difficult this is. It's just important never to give up.
Thanks for the pointer, it looks particularly interesting. I'm not good with the terminology and it always takes me a while to figure which properties we're talking about starting from a name. But the reported times in the article look pretty good and certainly interesting to consider. One difficulty I'm facing with ebtrees and strings is that I need the position of the first difference, that strcmp() doesn't return to you, and that if you reimplement yourself in multi-byte matches at once (I did it already), it will quickly upset valgrind and sanitizers for reading out of bounds. That makes such functions annoying to adopt in various projects as it requires more customizations. So I kept the hand-crafted one-byte-at-a-time comparison, but figured that other approaches based on just a comparison (strcmp, like used in rbtrees) could actually be a win for this reason.
The problem is not the performance of the low-level crypto code IMHO, but how it interfaces with the rest, which is where you're crossing a myriad of locks (and atomic ops for newer versions) that cost a lot as soon as you're interested in using more than one CPU core :-/
Just to be clear, we don't care at all about performance of 1.0. The tests resulting in the pretty telling graphs were done in 1.3 only, as that's what users care about.
Absolutely. Sometimes when using OpenSSL in performance tests, you notice that performances vary significantly just by switching to a different memory allocator, which is totally scary.
I hadn't seen the conversation above, thanks for the pointer. It's surrealistic. I don't see how having to support multiple file formats requires to invest so many allocations. In the worst case you open the file (1 malloc and occasionally a few realloc) and you try to parse it into a struct using a few different decoders. I hope they're not allocating one byte at a time when reading a file...
Not to mention the catastrophic security that comes with these systems. On a local ubuntu, I've had exactly 4 different versions of the sudo binary. One in the host OS and 3 in different snaps (some were the same but there were a total of 4 different). If they had a reason to be different, it's likely for bug fixes, but not all of them were updated, meaning that even after my main OS was updated, there were still 3 bogus binaries exposed to users and waiting for an exploit to happen. I find that this is the most shocking aspect of these systems (and I'm really not happy with the disrespect of my storage, like you mention).
But it's the same for other long sessions such as slow downloads and git clones. Sites concerned by the number of source ports are not those dealing with just favicon.ico and bullet.png, but mainly those dealing with long transfers.
Also there's a cascade effect on large sites, where as long as your servers respond fast, everything's OK. Suddenly a database experiences a hiccup, everything saturates, and once you enter the situation where the LB has all of its ports in use, it can take a while to recover because of connect() getting much slower (I already observed delays up to 50ms!). At this point there's no hope to recover in a sane time, because excess connections are not even served by the servers, they're in the accept queue in the system, so they keep a port busy, slowing down connect() which means more even connections are needed for other incoming requests. If the LB is not properly sized and tuned, you'd rather just kill it to get rid of all the connections at once, wait a second or two for the RST storm to calm down and start again.
H2 can avoid that, at the expense of other issues I mentioned in another response above (i.e. don't multiplex too much to the servers, 5-10 streams max, to avoid the risk of inter-client HoL). But H2 also comes with higher xfer costs than H1 for large objects due to framing.
Yep. Actually H1/H2/H3 do have the same problem (remember the good old days when everyone was trying to pipeline over H1?), except that H1 generally comes with multiple connections and H3 currently goes over QUIC and it's QUIC that addresses HoL by letting streams progress independently.
If you transfer large objects, H2 on the backend will increase transfer costs (due to framing). If you deal with many moderate or small objects however, H2 can improve the CPU usage for both the LB and the backend server because they will have less expensive parsing and will be able to merge multiple messages in a single packet over the wire, thus reducing the number of syscalls. Normally it's just a matter of enabling H2 on both and you can run some tests. Be careful not to mix too many clients over a backend connection if you don't want slow client to limit the other ones' xfer speed or even cause head of line blocking, though! By typically supporting ~10 streams per backend connection does improve things quite a bit over H1 for regular sites.
It's amazing how people having visibly never dealt with high loads can instantly become vehement against those reporting a real issue.
The case where ports are quickly exhausted is with long connections, typically WebSocket. And with properly tuned servers, reaching the 64k ports limit per server comes very quickly. I've seen several times the case where admins had to add multiple IP addresses to their servers just to hack around the limit, declaring each of them in the LB as if they were distinct servers. Also, even if Linux is now smart enough to try to pick a random port that's valid for your tuple, once your ports are exhausted, the connect() system call can cost quite a lot because it performs multiple tries until finding one that works. That's precisely what IP_BIND_ADDRESS_NO_PORT improves, by letting the port being chosen at the last moment.
H2 allows to work around all this more elegantly by simply multiplexing multiple client streams into a single connection. And that's very welcome with WebSocket since usually each stream has little traffic. The network also sees much less packets since you can merge many small messages into a single packet. So there are cases where it's better.
Another often overlooked point is that cancelling a download over H1 means breaking the connection. Over H2 you keep the connection opened since you simply send an RST_STREAM frame for that stream in the connection. The difference is important on the frontend when clients abort downloads multiple times per browsing session (you save the TLS setup again), but it can also make a difference on the backend, because quite often an aborted transfer on the front will also abort an H1 connection on the back, and then that's much less fun for your backend servers.
I suspect it might feel indecent to tell others you suffer when you're both free and rich, and it's difficult for them to figure what's wrong with you.
Instead, people in such position should probably go out and join associations which distribute food to those who need it. At least they'll see that they're doing something good to improve others' condition and would probably feel better.
What you're describing is for TCP. On TCP you can perform a write(64kB) and see the stack send it into 1460 segments. On UDP if you write(64kB) you'll get a single 64kB packet composed of 45 fragments. Needless to say, it suffices that any of them is lost in a buffer somewhere for the whole packet never being received and all of them having to be retransmitted by the application layer.
GSO on UDP allows the application to send a large chunk of data, indicating the MTU to be applied, and lets the kernel pass it down the stack as-is, until the lowest layer that can split it (network stack, driver or hardware). In this case they will make packets, not fragments. On the wire there will really be independent datagrams with different IP IDs. In this case, if any of them is lost, the other ones are still received and the application can focus on retransmitting only the missing one(s). In terms of route lookups, it's as efficient as fragmentation (since there's a single lookup) but it will ensure that what is sent over the wire is usable all along the chain, at a much lower cost than it would be to make the application send all of them individually.
> > There's still the problem of sending to multiple destinations: OK sendmmsg() can send multiple datagrams, but for a given socket.
> Hmm? sendmsg takes the destination address in the `struct msghdr` structure, and sendmmsg takes an array of those structures.
But that's still pointless on a connected socket. And if you're not using connected sockets, you're performing destination lookups for each and every datagram you're trying to send. It also means you're running with small buffers by default (the 212kB default buffers per socket are shared with all your destinations, no longer per destination). Thus normally you want to use connected socket when dealing with UDP in environments having performance requirements.
Nowadays the vast majority of CVEs have nothing to do with security, they're just Curriculum Vitae Enhancers, i.e. a student finding that "with my discovery, if A, B, C and D were granted, I could possibly gain some privileges", despite A/B/C/D being mutually exclusive. That's every days job for any security people to sort out that garbage. So what the kernel does is not worse at all.
There's still the problem of sending to multiple destinations: OK sendmmsg() can send multiple datagrams, but for a given socket. When you have small windows (thank you cubic), you'll just send a few datagrams this way and don't save much.
The default UDP buffers of 212kB are indeed a big problem for every client at the moment. You can optimize your server as you want, all your clients will experience losses if they pause for half a millisecond to redraw a tab or update an image, just because the UDP buffers can only store so few packets. That's among the things that must urgently change if we want UDP so start to work well on end-user devices.
Something that nobody seems to be talking about here is the congestion control algorithm, which is the problem here. Cubic doesn't like losses. At all. In the kernel, pacing is implemented to minimise losses, allowing Cubic to work acceptably for TCP, but if the network is slightly lossy, the perfs are terrible anyway. QUIC strongly recommends to implement pacing but it's less easy to implement accurately in userland when you have to cross a whole chain than at the queue level in the kernel.
Most QUIC implementations use different variations around the protocol to make it behave significantly better, such as preserving the last metrics when facing a loss so that in case it was only a reorder, they can be restored, etc. The article should have compared different server-side implementations, with different settings. We're used to see a ratio of 1:20 in some transatlantic tests.
And testing a BBR-enabled QUIC implementation shows tremendous gains compared to TCP with Cubic. Ratios of 1:10 are not uncommon with moderate latency (100ms) and losses (1-3%).
At least what QUIC is enlightening is that if TCP has worked so poorly for a very long time (remember that the reason for QUIC was that it was impossible to fix TCP everywhere), it's in large part due to congestion control algorithms, and that since they were implemented in kernel by people carefully reading an academic paper that never considers reality but only in-lab measurements, such algorithms behave pretty poorly in front of the real internet where jitter, reordering, losses, duplicates etc are normal. QUIC allowed many developers to put their fingers in the algos, adjust some thresholds and mechanisms and we're seeing stuff improve fast (it could have improved faster if OpenSSL didn't decide to play against QUIC a few years ago by cowardly refusing to implement the API everyone needed, and imposing to rely on locally-built SSL libs to use QUIC). I'm pretty sure that within 2-3 years, we'll see some of the QUIC improvements ported to TCP, just because QUIC is a great playground to experiment with these algos that for 4 decades had been the reserved territory of just a few people who denied the net as it is and worked for the net how they dreamed it.
The problem is that such services which proudly run on low budget, volunteers and recycled hardware, cannot be relied on by companies without risking to enter legal trouble in case of major incident, so it means that a higher-grade service is still needed, with a dedicate funding, and we're facing fragmentation. We must not reproduce the scheme of cloudwatt either. Too much money injected into a wet dream that was only used to spend lots of money in consultants coming here just to confirm their presence and get their check.
What is needed instead is to sponsor the development of such activities by a few (2-3) well-established competing companies, so as to avoid the regular risk of monoculture that diverges from what users expect, and help them reach the point where their offerings can compete with GAFAM's for both end users and enterprise. The contract should be clear that services must rely on open formats, make it possible for leaving users to retrieve all their data, that software developed under such funding must be opensource, though technology acquisition is fine, and that these offerings must become self-sustaining at one point (i.e. a mix of free+paid services). The EU funders should have enough shares of these activities so that their permission is required for business acquisition and that they can restrict it to EU-based companies, so that such companies can still grow and seek public funding.
What we need is a few durable big players, not 10000 incompatible associations each with their own software suite, that no enterprise can trust over the long term and that cannot resist a trivial DDoS by lack of a robust infrastructure, and who are not organized enough to run full-stack security audits to make sure that user data are properly protected. These ones are only fine for friends and family but that's not what we're missing the most (the proof is that they already exist).