Too bitter. I referenced a little of that "adventure" here, in 2021... gfiber was attempting to restart with refreshing their now obsolete hardware... https://blog.cerowrt.org/post/trouble_in_paradise/
They laid us all off. They had huge plans - millions of users! Then they intersected reality in KC where all people wanted was 5Mbit service and free TV... There were many, many people working to perfect the settop box for example. We got fq_codel running on the wifi, we never got anywhere on the shaper, the plan was to move 1+m units of that (horrible integrated chip the comcerto C2000 - it didn´t have coherent cache in some cases), I think they barely cracked 100k before pulling the plug on it all....
and still that box was better than what most fiber folk have delivered to date.
At least some good science was done about how ISPs really work... and published.
The users that call less post-libreqos are the gamers & zoomers.
I am sorry you are are so down on the lack of motivations care for customer service that many ISPs have. I am thrilled by how much libreqos's customer base cares.
Pie's principal advantage is that it is slightly easier to implement in hardware, it's disadvantages are that it does tail drop, rather than head drop, and struggles to be stable at a target of 16ms, where codel can go down to us and targets 5ms by default. I haven't really revisited pie since the above paper was published.
COBALT in cake is a codel derivative. It is slightly tighter in some respects (hitting slow start sooner), and looser in others (it never drops the last packet in a queue, which fq_codel does.fq_codel scales to hundreds of instances and 10s of thousands of queues, still aiming for 5ms across that target, where it would be easier to essentially DOS that many instances of cake with tons of flows.
Being one of the authors of fq_codel, cake and to a small extent, LibreQos, I remain boggled after 15 years of trying, to get these simple points across..
OpenWrt depreciated pfifo_fast in favor of fq_codel in 2012, and have not looked back. It (and BQL) is ever present on all their Ethernet hardware and most of their wifi, no configuration required. It's just there.
That said many OpenWrt chips have offloads that bypass that, and while speedier and low power, tend to be over buffered.
We have seen small ISPs get LibreQos running in under an hour, which includes installing ubuntu. Configuring it right and getting it fully integrated with the customer management system takes longer.
We're pretty sure most of those ISPs see reduced opex from support calls.
Capex until the appearance of fq_codel (Preseem, Bequant) or cake (LibreQos, Paraqum) middleboxes was essentially infinite. Now it's pennies per subscriber and many just a get a suitable box off of ebay.
I agree btw, that how to monitor and scale is a learned thing. For example many naive operators look at "drops" as reported by CAKE as a bad thing, when it is actually needed for good congestion control.
speedtest.net added support for tracking latency under load a few years ago. they show ping during up/dl now. That's the number to show your colleague.
However they tend to use something like the 75th percentile and throw out real data. The waveform bufferbloat test does 95% and supplies whisker charts. cloudflare also.
No web test tests up + down at the same time, which is the worst scenario. crusader and flent.org's rrul test do.
Rathan than argue with your colleague, why not just slap an OpenWrt box as a transparent bridge inline and configure CAKE SQM?
It really is astounding to me how so many still do not understand that tcp is not a function call, or behaviors like slow start and congestion avoidance.
Recently a new rate limiter for TCP went by that was so terribly, terribly broken, and I cannot help but imagine that most of the containers of the world suffer from Bufferbloat in general.
Thank you. I enjoyed the sense of play and this story more than anything I have read about the internet in a long time.
I am def burned out, and need to come up with something frivolous. I am reminded of Richard Feynman´s story of spending 10 years depressed after the war, and him finding joy of the physics in a spinning disk one day at lunch, so he could disregard what he had done before.
That said, I loved the zero page on the 6502...