Congrats to the team at Stainless, it's a great team to be joining over at Anthropic.
We were an ealy adopter of their Node SDK generator at Mux (and latterly their Typescript and other generators), and the product worked great, and I'm sad to see it be shut down.
At the same time, it's easy to understand why this is a complciated product/market to be in at the moment - it's very tempting and easy to vibe code SDKs from a OpenAPI spec files right now. I would think a lot of teams will just go in that direction (for better or worse), using the same toolchain that the product developers are using today for the product, for effectively no extra cost.
In the interests of transparent disclosure on such a positive blog post, It might be worth calling out that all the links on the page are all linked to the Bunny Affiliate Program. [1]
I know it's not public domain per-say, but for me, the thing that's most exciting is that in 2025, the last remaining patents on the h.264 (AVC) video codec will expire [1].
Now if only HEVC wasn't such a hot patent / licensing mess.
Disclaimer: We're an early adopter of Stainless at Mux.
I've spent more of my time than I'd like to admit managing both OpenAPi spec files [1] and fighting with openapi-generator [2] than any sane person should have to. While it's great having the freedom to change the templates an thus generated SDKs you get with using that sort of approach, it's also super time consuming, and when you have a lot of SDKs (we have 6 generated SDKs), in my experience it needs someone devoted to managing the process, staying up with template changes etc.
Excited to see more SDK languages come to Stainless!
Super interesting, it really depends on what you mean by "choppy".
Are you seeing the video re-buffer (a loading spinner re-appearing), or just the video looking stuttery?
My video engineer gut tells me that this is a frame rate conversion issue - do you know what frame rate your graphics card is outputting to your display, and do you happen to know the frame rate of the content that you're seeing issues with (or more generically, what type of content is it - film? TV) ?
Great question! The real answer is it varies, but for H.264, most just encode on software right now, because GPUs are expensive (especially in the cloud), and the failure rates are really high (if you try to build your own). ffmpeg and lib264 is really fast on modern hardware with decent X86 extensions.
It's also worth noting that YouTube also now builds its own transcoding chips [1], and AWS just launched a dedicated transcoding instances based on Xilinx chips:
Hey, I work in the Product team at Mux, and worked on the LL-HLS spec and our implementation, I own our real-time video strategy too.
We do offer LL-HLS in an open beta today [1], which in the best case will get you around 4-5 seconds of latency on a good player implementation, but this does vary with latency to our service's origin and edge. We have some tuning to do here, but best case, the LL-HLS protocol will get to 2.5-3 seconds.
We're obviously interested in using WebRTC for use cases that require more real-time interactions, but I don't have anything I can publicly share right now. For sub-second streaming using WebRTC, there are a lot of options out there at the moment though, including Millicast [2] and Red5Pro [3] to name a couple.
Two big questions comes up when I talk to customers about WebRTC at scale:
The first is how much reliability and perceptual quality people are willing to sacrifice to get to that magic 1 second latency number. WebRTC implementations today are optimised for latency over quality, and have a limited amount of customisability - my personal hope is that the client side of the WebRTC will become more unable for PQ and reliability, allowing target latencies of ~1s rather than <= 200ms.
The second is cost. HLS, LL-HLS etc. can still be served on commodity CDN infrastructure, which can't currently serve WebRTC traffic, making it an order of magnitude cheaper than WebRTC.
We actually built How Video Works as a side project at Mux [1] (inspired by How DNS Works [2]) - there's a note about it at the top of the page. We have contributions from our own team as well as others in the industry.
Our main motivation is to try to educate on the complexities and intracies of streaming video. Despite streaming video representing 80+% of the internet, it's all underpinned by a fairly small community of engineers, which we're eager to help grow through tools like this, and the Demuxed community [3].
Edit: I should also mention that Leandro was kind enough to adapt a this content from his amazing Digital Video Introduction [4]
DRM'd video content uses the same video codecs and containers, but introduces segment encryption during the packaging phase. In most cases, this encryption is within the audio and video samples rather than on the entire segment. Most content is encrypted using MPEG Common Encryption (CENC) - though there are a couple of variants.
Decryption keys are then exchanged using one of the common proprietary DRM protocols, usually Widevine (Google), Playready (Microsoft), or Fairplay (Apple). The CDM (Content Decryption Module) in the browser is then passed the decryption key, so the browser. Can decrypt the content for playback.
While HLS isn't the cleanest protocol (Yay for extensions of the M3U playlist format...), it's actually really good at what it's designed to do - provide reliable video streaming while using HTTP/S over variable networks.
Ultimately, HLS isn't designed for downloading and storing videos, it's designed for streaming.
Do you have a marketing site or landing page for Padka? It sounds interesting, but just linking to a .pkg file on S3 really isn't the way to educate HN users about your offering!
So many permissions, but meanwhile autoplaying video and audio is still hidden behind a horrible heuristic model in Chrome that fundamentally prioritises internet giants like YouTube, Netflix and alike.
I work in the video streaming industry, and we continue to support TLS 1.1 extensively for a wide range of "smart" TVs and set-top boxes, very frustrating, there was even a long period where large CDNs were trying to shut down 1.1 and realised they'd lose a lot of business in the streaming space if they did...
I don't realistically see this as being a killer feature for most users, but it's super interesting none the less.
Sony are saying the bitrate will be "Up to 80Mbps", which is around 3x that of most UHD streaming service profiles. However there will be significant diminishing returns on the perceptual quality at this bitrate - 20Mbps to 40Mbps for example will only increase perceptual quality by a couple of percentage points. Much of this "Up to 80Mbps" will be wasted perceptually speaking.
It's also worth keeping in mind that the encoding on BluRays is often lazy because there's lot of space available, where as a huge amount of work goes into encoding content for streaming delivery in the lowest bandwidth possible for a given target quality, using technologies like per-title and per-scene encoding. [1] [2] [3]
This isn't accurate at all. HLML 5 video, MSE etc. have a world of improvements over the mess of FLV / Flash / RTMP streaming.
Can you give some examples of what's wrong with HTML5 video streaming and these implementation issues? There's a huge community of video engineers working to make streaming video better, and we'd love to hear feedback.
There's a few approaches depending on your target latency. If you need to get below 2 seconds, yes, you'll likely want to use a WebRTC based technology to deliver the video. There's a couple of open source approaches out there, including Pion [1], which is a go implementation of a WebRTC stack. You could also build something on top of Jitsi [2]. Commercially there's also a few solutions, including Milicast [3], Red5Pro [4] and others.
The biggest problem with WebRTC based stacks is that the cross device compatibility is still generally poor, and the cost of operation is generally very high, as you can't use commodity CDNs for delivery.
If you're comfortable around the 2-5 seconds latency mark, there's more traditional HTTP based technologies available. MPEG-DASH has a Low Latency mode which uses chunk transferred HTTP fragments, and Apple has recently introduced a Low Latency HLS mode, which works in much the same way [5] [6]. You can build LL-HLS and DASH-LL solutions on top of open source toolchains like Streamline [7], and use commodity CDNs for delivery to reduce cost.
While some of the larger UGC platforms do use VP9 for live streaming, it is only for a limited subset of high concurrent viewer streams, and yes, in many cases those aren't running the encodes on commodity hardware.
As for AV1, there really aren't any live implementations ready right now, a few have been demo'd, but I'm not aware of any deployments today.
Sure, but this was my point, x264 is very fast on commodity hardware, the overwhelming amount of video delivered is still h.264, especially on live video, and in particular for low-viewership, live user generated content.
If you want/need to transcode to VP9 or AV1 at scale with good quality, yes, you'll absolutely need GPU or ASIC accelerated encoders, of which there are a couple for VP9, and none for AV1 today.
h.264 will get you to 95-99% market penetration on the device landscape.
We were an ealy adopter of their Node SDK generator at Mux (and latterly their Typescript and other generators), and the product worked great, and I'm sad to see it be shut down.
At the same time, it's easy to understand why this is a complciated product/market to be in at the moment - it's very tempting and easy to vibe code SDKs from a OpenAPI spec files right now. I would think a lot of teams will just go in that direction (for better or worse), using the same toolchain that the product developers are using today for the product, for effectively no extra cost.