How Modern Video Players Work(blog.streamroot.io)
blog.streamroot.io
How Modern Video Players Work
http://blog.streamroot.io/how-modern-video-players-work/
3 comments
Oh. Well that's disappointing. I was hoping this would be about a modern video/audio decode, synchronize, and display pipeline... not that knowing how video streamed on the web isn't interesting, but it's really about how modern video streaming works.
Not that its particularly casually browsable but there are some details in the programmer reference manuals for the Intel integrated graphics encode/decode hardware. If you head to volume 8 the MFx introduction section has some sample flows before it starts getting into command packet details: https://01.org/sites/default/files/documentation/intel-gfx-p...
It talks about modern streaming indeed, but if you take a closer look, most of the video consumption today is done by streaming: on Desktop you still have a small portions of download then watch, but on mobile, Set Top Boxes like AppleTV and Roku, connected TVs, etc. You watch everything from the web.
Besides, the decoding/sync/rendering part didn't really change in the last 10 years, the innovations mostly arrived from the Web streaming use cases, with adaptive bitrate formats, that partly solved the multi-devices streaming problem we started to have with the rise of Mobile and OTT (over-the-top) video streaming.
Besides, the decoding/sync/rendering part didn't really change in the last 10 years, the innovations mostly arrived from the Web streaming use cases, with adaptive bitrate formats, that partly solved the multi-devices streaming problem we started to have with the rise of Mobile and OTT (over-the-top) video streaming.
All this fancy software and yet watching videos on the web is just as painful as it was 15 years ago if you don't have a fast enough connection. The download won't keep up with the play speed and they don't accommodate that nearly as well as they could. In the early 2000's with dial-up, you could just press pause then come back a few minutes later to watch what had been buffered. Nowadays that trick often doesn't work - pause also pauses the download. Not only that but the play/pause button sometimes doesn't work at all! If the playback is stopped because it's waiting to download more, the pause button usually does nothing - so you have to rewind to a previously buffered part (if you're lucky enough and it kept the buffer) then let it play, then press pause.
This!
Related gripe: WTH is there no full-track buffering mode for any web players (looking at You, Tube!) and apps like Pandora or Spotify, for low- or sparse- bandwidth travel?!
I don't care if you limit it geographically or only make it work with bandwidth throttling or whatever...
...just make it work.
Then your apps would work and I would not be cursing and throwing my phone out the window every road trip.
This is costing me a fortune in phones.
Related gripe: WTH is there no full-track buffering mode for any web players (looking at You, Tube!) and apps like Pandora or Spotify, for low- or sparse- bandwidth travel?!
I don't care if you limit it geographically or only make it work with bandwidth throttling or whatever...
...just make it work.
Then your apps would work and I would not be cursing and throwing my phone out the window every road trip.
This is costing me a fortune in phones.
The other side of the equation involves at least:
1) Video provider pays for bytes sent to the user that are not used. If the cache policy is "cache the whole thing", you would find yourself paying (a guestimate) 2x for bandwidth. This is because most people do not watch even nearly the entirety of every video that they start loading.
2) Using server bandwidth for bytes that are not used. This increases contention for bandwidth at the origin. Perhaps less of an issue if your site is small and you're using a giant CDN. This does become an issue if you are youtube or you are serving the bytes yourself.
3) Using client bandwidth for bytes that are not used. This slows down any other download that might be happening on the client computer, even when the video is paused. This could be, a different video in a different tab that the user is trying to watch, other assets on the page, other pages, or even other programs on the computer.
Of course adding an option for the user to cache the whole thing wouldn't be so bad, as opposed to having this behavior by default for everyone.
1) Video provider pays for bytes sent to the user that are not used. If the cache policy is "cache the whole thing", you would find yourself paying (a guestimate) 2x for bandwidth. This is because most people do not watch even nearly the entirety of every video that they start loading.
2) Using server bandwidth for bytes that are not used. This increases contention for bandwidth at the origin. Perhaps less of an issue if your site is small and you're using a giant CDN. This does become an issue if you are youtube or you are serving the bytes yourself.
3) Using client bandwidth for bytes that are not used. This slows down any other download that might be happening on the client computer, even when the video is paused. This could be, a different video in a different tab that the user is trying to watch, other assets on the page, other pages, or even other programs on the computer.
Of course adding an option for the user to cache the whole thing wouldn't be so bad, as opposed to having this behavior by default for everyone.
I live in China which has crap international bandwidth. Youtube via VPN is the only way ... my solution is to use VideoDownloadHelper plugin for Firefox @ https://addons.mozilla.org/en-US/firefox/addon/video-downloa...
OTOH, China has an awesome "who cares about gwailo IP" progressive IP policy which allows you to stream lots of copyrighted stuff instantly via multiple domestic providers. Hell, new TVs have this sort of thing built in! It's awesome for Hollywood crap, just not the content (eg. EU / art-house films) or type (eg. documentaries) that I am typically looking for. Then again we have torrents...
Somewhere, somehow, the internet finds a way. When all is said and done, it's only people who are scared, stupid or spendthrift that pay... or those who want the authentic cinema experience, collect media, or are commercial operations that need to worry about legal challenges.
A few years ago I actually used to live in Hollywood and London working for a big DRM-based video solutions provider that did work for literally most of the world's major mobile device manufacturers. Boring, but a great insight in to how corrupt the industry is. (Very)
OTOH, China has an awesome "who cares about gwailo IP" progressive IP policy which allows you to stream lots of copyrighted stuff instantly via multiple domestic providers. Hell, new TVs have this sort of thing built in! It's awesome for Hollywood crap, just not the content (eg. EU / art-house films) or type (eg. documentaries) that I am typically looking for. Then again we have torrents...
Somewhere, somehow, the internet finds a way. When all is said and done, it's only people who are scared, stupid or spendthrift that pay... or those who want the authentic cinema experience, collect media, or are commercial operations that need to worry about legal challenges.
A few years ago I actually used to live in Hollywood and London working for a big DRM-based video solutions provider that did work for literally most of the world's major mobile device manufacturers. Boring, but a great insight in to how corrupt the industry is. (Very)
Yes, I was thinking specifically of client-side caching. I would like my mobile apps to aggressively cache forward in the play list so that I always have
a) the complete current song, and ideally b) the complete next song
Except in game over insufficient bandwidth full stop scenarios, natch.
I would imagine the most user-friendly version of this would be to start playback using as soon as there is a conventional "sufficient buffer," but to be much more aggressive about sucking every possible byte to maintain the above caching...
Personally I would solve the 'very long delay before play' problem by defaulting to the 80% case of playing one of N songs from the last used playlist/station, _as cached locally from last play_. (Maybe not simply the most recently played, maybe the highest ranked, least skipped, whatever...)
That wouldn't help with people who are flipping stations, but my road trip case would be sudden bliss... no more constant drop-outs.
Disclosure: part of my problem is no doubt sticking with Sprint because I don't want to lose unlimited (sic) data. :P
a) the complete current song, and ideally b) the complete next song
Except in game over insufficient bandwidth full stop scenarios, natch.
I would imagine the most user-friendly version of this would be to start playback using as soon as there is a conventional "sufficient buffer," but to be much more aggressive about sucking every possible byte to maintain the above caching...
Personally I would solve the 'very long delay before play' problem by defaulting to the 80% case of playing one of N songs from the last used playlist/station, _as cached locally from last play_. (Maybe not simply the most recently played, maybe the highest ranked, least skipped, whatever...)
That wouldn't help with people who are flipping stations, but my road trip case would be sudden bliss... no more constant drop-outs.
Disclosure: part of my problem is no doubt sticking with Sprint because I don't want to lose unlimited (sic) data. :P
Very true, bandwidth is very expensive today. Some comments to your post:
- Actually, and quite surprisingly, the "cache everything" mode only adds around 25% of used bandwidth in practice. But even 25% is already a big difference if you pay 10 or 50K of CDN bills per month.
- Indeed useless bytes download can create overall congestion brings harm to the network in general, and should be avoided. It's especially true for websites that don't have the money to pay big CDNs and stream with dedicated server with limited capacity.
- Actually, and quite surprisingly, the "cache everything" mode only adds around 25% of used bandwidth in practice. But even 25% is already a big difference if you pay 10 or 50K of CDN bills per month.
- Indeed useless bytes download can create overall congestion brings harm to the network in general, and should be avoided. It's especially true for websites that don't have the money to pay big CDNs and stream with dedicated server with limited capacity.
most people probably don't have the patience anymore to just pause and wait 30 minutes until the download is finished.
And most of the new modern video players support adaptive bitrate streaming, that adapts the quality to your network conditions. Of course if you want to stream 4K video on 2G, you won't be able to do it today, but in that case it's probably easier to download the mp4 file directly
And most of the new modern video players support adaptive bitrate streaming, that adapts the quality to your network conditions. Of course if you want to stream 4K video on 2G, you won't be able to do it today, but in that case it's probably easier to download the mp4 file directly
Spotify has preloading offline mode in the mobile player. Maybe not exactly what you are looking for.
Yes! Pause&wait was my default behavior on slow connections (i.e. smartphone in the middle of nowhere) and removing it is a huge step back. I mostly can't use Youtube on my phone anymore and forced to look for videos somewhere else, where this "old-fashioned" method still works. Is there any technical reasons that won't let the video download while paused?
Apps are built for the average, everyday user. I'm sure their user testing has shown that the majority of users don't do the "pause and wait" ritual, and a large amount of pauses are never resumed.
So, they choose to save the bandwidth.
So, they choose to save the bandwidth.
Actually as the article mentions, most broadcasters today are using adaptive bitrate formats like HLS and DASH, that are solving this issue by serving you a bitrate that is low enough for you to handle.
Today the major broacasters often have 6 or 7 different qualities, going from 300kbps up to 4000Mbps for HD.
Today the major broacasters often have 6 or 7 different qualities, going from 300kbps up to 4000Mbps for HD.
A very good article for those who would like to follow the lead of NYT, Youtube or others who recently switched to all HTML5